cgroup v2 OOM events

oom_kill/max counters per cgroup; pod hitting memory.max before node-wide OOM

cat /sys/fs/cgroup/<name>/memory.events | grep -E 'oom|max'
tool
troubleshooting32 commands
tags
troubleshootingmemoryoomk8s

More troubleshooting commands

  • Deleted files holding disk
    lsof +L1 | sort -k7 -rn | head

    df full but du clean? Unlinked files held open by a pid; restart it to reclaim

  • Disk I/O latency saturation
    iostat -x 1 | awk '$1!~/^$/{print $1, "util="$NF, "await="$(NF-2)}'

    %util near 100 with rising await = saturated device; correlate with the noisy pid

  • Per-process block I/O top
    biotop-bpfcc -C 5 1

    bcc biotop: which pid drives block I/O by bytes/latency; modern: bpftrace one-liners

  • Conntrack table full check
    conntrack -L 2>/dev/null | wc -l; conntrack -S

    Count tracked flows vs limit; -S shows insert_failed/drop when table full

  • Conntrack live events
    conntrack -E -p tcp --dport 443 -o timestamp

    Stream NEW/DESTROY events live to catch NAT/timeout drops on a port

all 32 troubleshooting commands