troubleshooting
32 commands
32 shown
Per-thread CPU breakdown
Per-thread CPU%, find the hot TID inside a busy process for jstack/perf
pidstat -t -p <pid> 1
Hot thread live view
Live per-thread view; convert hot TID to hex (printf %x) to match jstack nid
top -H -p <pid>
Per-core CPU saturation
Per-core %usr/%sys/%iowait — spot a single saturated core vs balanced load
mpstat -P ALL 1
Run-queue vs cores
vmstat r column (runnable) above nproc means CPU saturation, not just high load
vmstat 1 5 | awk 'NR>2{print "runq="$1" blocked="$2}'; nproc
Process thread count
Thread count per pid and system-wide; runaway threads exhaust kernel.pid_max
ps -o nlwp= -p <pid>; ps -eLf | wc -l
RSS leak over time
Sample RSS every 5s to confirm a real leak (monotonic growth) vs steady churn
while sleep 5; do ps -o rss= -p <pid> | awk '{print strftime("%T"),$1/1024"MB"}'; done
Fast memory rollup
One-shot Rss/Pss/Swap totals without parsing full smaps; cheap on huge maps
cat /proc/<pid>/smaps_rollup
Largest memory mappings
Top RSS mappings by region; spot a leaking heap, mmap or shared lib bloat
pmap -x <pid> | sort -k3 -rn | head
Kernel slab top consumers
Sort slab caches by size; dentry/inode bloat explains kernel memory pressure
slabtop -o -s c | head -20
Reclaimable slab memory
Split slab into reclaimable vs unreclaimable; high SUnreclaim = real kernel leak
grep -E 'Slab|SReclaimable|SUnreclaim' /proc/meminfo
OOM killer in dmesg
Find OOM victims with human timestamps; -T avoids unreadable kernel ticks
dmesg -T | grep -iE 'killed process|out of memory' | tail
OOM score of process
Higher oom_score = likelier victim; set oom_score_adj -1000 to immunize a pid
cat /proc/<pid>/oom_score /proc/<pid>/oom_score_adj
cgroup v2 OOM events
oom_kill/max counters per cgroup; pod hitting memory.max before node-wide OOM
cat /sys/fs/cgroup/<name>/memory.events | grep -E 'oom|max'
Deleted files holding disk
df full but du clean? Unlinked files held open by a pid; restart it to reclaim
lsof +L1 | sort -k7 -rn | head
Disk I/O latency saturation
%util near 100 with rising await = saturated device; correlate with the noisy pid
iostat -x 1 | awk '$1!~/^$/{print $1, "util="$NF, "await="$(NF-2)}'
Per-process block I/O top
bcc biotop: which pid drives block I/O by bytes/latency; modern: bpftrace one-liners
biotop-bpfcc -C 5 1
Conntrack table full check
Count tracked flows vs limit; -S shows insert_failed/drop when table full
conntrack -L 2>/dev/null | wc -l; conntrack -S
Conntrack live events
Stream NEW/DESTROY events live to catch NAT/timeout drops on a port
conntrack -E -p tcp --dport 443 -o timestamp
Ephemeral port exhaustion
Count TIME-WAIT sockets vs port range; note: tune tw_reuse not tw_recycle
ss -tan state time-wait | wc -l; sysctl net.ipv4.ip_local_port_range
Socket summary stats
One-shot socket totals by state; SYN-SENT pileup hints at SYN drops/firewall
ss -s; ss -tan state syn-sent | wc -l
Path MTU discovery
Find PMTU drop hop; ping -M do forbids frag to pinpoint blackhole MTU
tracepath -n <host>; ping -M do -s 1472 -c2 <host>
Confirm egress route and ARP
Show actual src/iface for a dest; FAILED neigh entries = L2/ARP problem
ip route get <ip>; ip neigh show | grep -E 'FAILED|INCOMPLETE'
TCP port reachability past firewall
TCP-SYN traceroute survives ICMP filtering; nc -zv confirms open port
traceroute -T -p 443 <host>; nc -zv <host> 443
Bypass DNS with curl --resolve
Pin host to a specific IP to test backend while keeping SNI/Host header
curl -sv --resolve <host>:443:<ip> https://<host>/healthz
In-cluster DNS smoke test
Throwaway pod to resolve in-cluster names; verifies CoreDNS + kube-dns svc
kubectl run dnsutils --image=tutum/dnsutils -it --rm --restart=Never -- nslookup kubernetes.default
Query CoreDNS directly
Hit CoreDNS clusterIP straight; note: ndots:5 makes short names spam queries
dig +short @$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}') <name>.<ns>.svc.cluster.local
Inspect CoreDNS config and logs
Review Corefile (forward/cache) and tail logs for SERVFAIL/upstream errors
kubectl -n kube-system get cm coredns -o yaml; kubectl -n kube-system logs deploy/coredns --tail=50
TLS cipher and protocol enum
Grade supported TLS versions/ciphers; flags weak/deprecated suites per port
nmap --script ssl-enum-ciphers -p 443 <host>
Verify full TLS chain and SNI
Show full chain with correct SNI; Verify line catches missing intermediates
openssl s_client -connect <host>:443 -servername <host> -showcerts 2>/dev/null | grep -E 'Verify|s:|i:'
Batch cert expiry scan
Loop many hosts to print notAfter dates; pipe to sort for soonest expiry
for h in $(cat hosts.txt); do echo -n "$h "; echo | openssl s_client -connect $h:443 -servername $h 2>/dev/null | openssl x509 -noout -enddate; done
Decode JWT claims
Decode payload to inspect exp/iss/aud; note: fixes base64url, skips sig check
cut -d. -f2 <<<"$JWT" | tr '_-' '/+' | base64 -d 2>/dev/null | jq .
List gRPC services via reflection
Enumerate services/methods over reflection; -plaintext skips TLS for debug
grpcurl -plaintext <host>:<port> list; grpcurl -plaintext <host>:<port> describe <svc>
no commands match