Check disk I/O saturation

%util near 100 with growing await means the device is saturated; then find the process behind it.

iostat -x 1 | awk '$1!~/^$/{print $1, "util="$NF, "await="$(NF-2)}'

tool
troubleshooting32 commands
tags
troubleshootingdiskperformancemonitoring

More troubleshooting commands

  • biotop-bpfcc -C 5 1

    bcc biotop shows which process makes the most block I/O by bytes and latency; for custom questions, use bpftrace.

  • conntrack -L 2>/dev/null | wc -l; conntrack -S

    Count tracked connections and compare with nf_conntrack_max; -S shows insert_failed and drop when the table is full.

  • conntrack -E -p tcp --dport 443 -o timestamp

    Stream NEW and DESTROY events for a port to catch NAT or timeout drops.

  • ss -tan state time-wait | wc -l; sysctl net.ipv4.ip_local_port_range

    Count TIME-WAIT sockets and compare with the local port range; tune tw_reuse, not tw_recycle.

  • ss -s; ss -tan state syn-sent | wc -l

    Show socket totals by state; many SYN-SENT sockets point to dropped SYNs or a firewall.

all 32 troubleshooting commands