troubleshooting commands

32 commands, each with a one-line explanation. Click a command to copy it, or its title for details.

Commands 32 shown

  • Per-thread CPU breakdown
    pidstat -t -p <pid> 1

    Per-thread CPU%, find the hot TID inside a busy process for jstack/perf

    performance · cpu · debug

  • Hot thread live view
    top -H -p <pid>

    Live per-thread view; convert hot TID to hex (printf %x) to match jstack nid

    performance · cpu · debug

  • Per-core CPU saturation
    mpstat -P ALL 1

    Per-core %usr/%sys/%iowait — spot a single saturated core vs balanced load

    performance · cpu · monitoring

  • Run-queue vs cores
    vmstat 1 5 | awk 'NR>2{print "runq="$1" blocked="$2}'; nproc

    vmstat r column (runnable) above nproc means CPU saturation, not just high load

    performance · cpu · debug

  • Process thread count
    ps -o nlwp= -p <pid>; ps -eLf | wc -l

    Thread count per pid and system-wide; runaway threads exhaust kernel.pid_max

    performance · debug · kernel

  • RSS leak over time
    while sleep 5; do ps -o rss= -p <pid> | awk '{print strftime("%T"),$1/1024"MB"}'; done

    Sample RSS every 5s to confirm a real leak (monotonic growth) vs steady churn

    memory · debug · monitoring

  • Fast memory rollup
    cat /proc/<pid>/smaps_rollup

    One-shot Rss/Pss/Swap totals without parsing full smaps; cheap on huge maps

    memory · debug · performance

  • Largest memory mappings
    pmap -x <pid> | sort -k3 -rn | head

    Top RSS mappings by region; spot a leaking heap, mmap or shared lib bloat

    memory · debug

  • Kernel slab top consumers
    slabtop -o -s c | head -20

    Sort slab caches by size; dentry/inode bloat explains kernel memory pressure

    memory · kernel · performance

  • Reclaimable slab memory
    grep -E 'Slab|SReclaimable|SUnreclaim' /proc/meminfo

    Split slab into reclaimable vs unreclaimable; high SUnreclaim = real kernel leak

    memory · kernel · debug

  • OOM killer in dmesg
    dmesg -T | grep -iE 'killed process|out of memory' | tail

    Find OOM victims with human timestamps; -T avoids unreadable kernel ticks

    memory · oom · kernel

  • OOM score of process
    cat /proc/<pid>/oom_score /proc/<pid>/oom_score_adj

    Higher oom_score = likelier victim; set oom_score_adj -1000 to immunize a pid

    memory · oom · kernel

  • cgroup v2 OOM events
    cat /sys/fs/cgroup/<name>/memory.events | grep -E 'oom|max'

    oom_kill/max counters per cgroup; pod hitting memory.max before node-wide OOM

    memory · oom · k8s

  • Deleted files holding disk
    lsof +L1 | sort -k7 -rn | head

    df full but du clean? Unlinked files held open by a pid; restart it to reclaim

    disk · debug · performance

  • Disk I/O latency saturation
    iostat -x 1 | awk '$1!~/^$/{print $1, "util="$NF, "await="$(NF-2)}'

    %util near 100 with rising await = saturated device; correlate with the noisy pid

    disk · performance · monitoring

  • Per-process block I/O top
    biotop-bpfcc -C 5 1

    bcc biotop: which pid drives block I/O by bytes/latency; modern: bpftrace one-liners

    disk · performance · observability

  • Conntrack table full check
    conntrack -L 2>/dev/null | wc -l; conntrack -S

    Count tracked flows vs limit; -S shows insert_failed/drop when table full

    network · kernel · performance

  • Conntrack live events
    conntrack -E -p tcp --dport 443 -o timestamp

    Stream NEW/DESTROY events live to catch NAT/timeout drops on a port

    network · debug · kernel

  • Ephemeral port exhaustion
    ss -tan state time-wait | wc -l; sysctl net.ipv4.ip_local_port_range

    Count TIME-WAIT sockets vs port range; note: tune tw_reuse not tw_recycle

    network · performance · kernel

  • Socket summary stats
    ss -s; ss -tan state syn-sent | wc -l

    One-shot socket totals by state; SYN-SENT pileup hints at SYN drops/firewall

    network · performance · debug

  • Path MTU discovery
    tracepath -n <host>; ping -M do -s 1472 -c2 <host>

    Find PMTU drop hop; ping -M do forbids frag to pinpoint blackhole MTU

    network · dns · debug

  • Confirm egress route and ARP
    ip route get <ip>; ip neigh show | grep -E 'FAILED|INCOMPLETE'

    Show actual src/iface for a dest; FAILED neigh entries = L2/ARP problem

    network · debug

  • TCP port reachability past firewall
    traceroute -T -p 443 <host>; nc -zv <host> 443

    TCP-SYN traceroute survives ICMP filtering; nc -zv confirms open port

    network · security · debug

  • Bypass DNS with curl --resolve
    curl -sv --resolve <host>:443:<ip> https://<host>/healthz

    Pin host to a specific IP to test backend while keeping SNI/Host header

    network · dns · debug

  • In-cluster DNS smoke test
    kubectl run dnsutils --image=tutum/dnsutils -it --rm --restart=Never -- nslookup kubernetes.default

    Throwaway pod to resolve in-cluster names; verifies CoreDNS + kube-dns svc

    k8s · dns · debug

  • Query CoreDNS directly
    dig +short @$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}') <name>.<ns>.svc.cluster.local

    Hit CoreDNS clusterIP straight; note: ndots:5 makes short names spam queries

    k8s · dns · debug

  • Inspect CoreDNS config and logs
    kubectl -n kube-system get cm coredns -o yaml; kubectl -n kube-system logs deploy/coredns --tail=50

    Review Corefile (forward/cache) and tail logs for SERVFAIL/upstream errors

    k8s · dns · observability

  • TLS cipher and protocol enum
    nmap --script ssl-enum-ciphers -p 443 <host>

    Grade supported TLS versions/ciphers; flags weak/deprecated suites per port

    tls · security · network

  • Verify full TLS chain and SNI
    openssl s_client -connect <host>:443 -servername <host> -showcerts 2>/dev/null | grep -E 'Verify|s:|i:'

    Show full chain with correct SNI; Verify line catches missing intermediates

    tls · security · debug

  • Batch cert expiry scan
    for h in $(cat hosts.txt); do echo -n "$h "; echo | openssl s_client -connect $h:443 -servername $h 2>/dev/null | openssl x509 -noout -enddate; done

    Loop many hosts to print notAfter dates; pipe to sort for soonest expiry

    tls · security · monitoring

  • Decode JWT claims
    cut -d. -f2 <<<"$JWT" | tr '_-' '/+' | base64 -d 2>/dev/null | jq .

    Decode payload to inspect exp/iss/aud; note: fixes base64url, skips sig check

    security · secrets · api

  • List gRPC services via reflection
    grpcurl -plaintext <host>:<port> list; grpcurl -plaintext <host>:<port> describe <svc>

    Enumerate services/methods over reflection; -plaintext skips TLS for debug

    network · api · debug