Compare the run queue with cores

If the vmstat r column (runnable tasks) is above nproc, the CPU is saturated, not just showing a high load average.

vmstat 1 5 | awk 'NR>2{print "runq="$1" blocked="$2}'; nproc

tool
troubleshooting32 commands
tags
troubleshootingperformancecpudebug

More troubleshooting commands

  • ps -o nlwp= -p <pid>; ps -eLf | wc -l

    Count threads for one process and for the whole system; runaway threads can use up kernel.pid_max.

  • while sleep 5; do ps -o rss= -p <pid> | awk '{print strftime("%T"),$1/1024"MB"}'; done

    Sample RSS every 5 seconds to tell a real leak (steady growth) from normal ups and downs.

  • cat /proc/<pid>/smaps_rollup

    Get Rss, Pss and Swap totals in one read without parsing the full smaps; cheap even for huge processes.

  • pmap -x <pid> | sort -k3 -rn | head

    Sort mappings by RSS to spot a leaking heap, a large mmap or a bloated shared library.

  • slabtop -o -s c | head -20

    Sort slab caches by size; large dentry or inode caches often explain kernel memory pressure.

all 32 troubleshooting commands