Per-process block I/O top

bcc biotop: which pid drives block I/O by bytes/latency; modern: bpftrace one-liners

biotop-bpfcc -C 5 1
tool
troubleshooting32 commands
tags
troubleshootingdiskperformanceobservability

More troubleshooting commands

  • Conntrack table full check
    conntrack -L 2>/dev/null | wc -l; conntrack -S

    Count tracked flows vs limit; -S shows insert_failed/drop when table full

  • Conntrack live events
    conntrack -E -p tcp --dport 443 -o timestamp

    Stream NEW/DESTROY events live to catch NAT/timeout drops on a port

  • Ephemeral port exhaustion
    ss -tan state time-wait | wc -l; sysctl net.ipv4.ip_local_port_range

    Count TIME-WAIT sockets vs port range; note: tune tw_reuse not tw_recycle

  • Socket summary stats
    ss -s; ss -tan state syn-sent | wc -l

    One-shot socket totals by state; SYN-SENT pileup hints at SYN drops/firewall

  • Path MTU discovery
    tracepath -n <host>; ping -M do -s 1472 -c2 <host>

    Find PMTU drop hop; ping -M do forbids frag to pinpoint blackhole MTU

all 32 troubleshooting commands