Reclaimable slab memory

Split slab into reclaimable vs unreclaimable; high SUnreclaim = real kernel leak

grep -E 'Slab|SReclaimable|SUnreclaim' /proc/meminfo
tool
troubleshooting32 commands
tags
troubleshootingmemorykerneldebug

More troubleshooting commands

  • OOM killer in dmesg
    dmesg -T | grep -iE 'killed process|out of memory' | tail

    Find OOM victims with human timestamps; -T avoids unreadable kernel ticks

  • OOM score of process
    cat /proc/<pid>/oom_score /proc/<pid>/oom_score_adj

    Higher oom_score = likelier victim; set oom_score_adj -1000 to immunize a pid

  • cgroup v2 OOM events
    cat /sys/fs/cgroup/<name>/memory.events | grep -E 'oom|max'

    oom_kill/max counters per cgroup; pod hitting memory.max before node-wide OOM

  • Deleted files holding disk
    lsof +L1 | sort -k7 -rn | head

    df full but du clean? Unlinked files held open by a pid; restart it to reclaim

  • Disk I/O latency saturation
    iostat -x 1 | awk '$1!~/^$/{print $1, "util="$NF, "await="$(NF-2)}'

    %util near 100 with rising await = saturated device; correlate with the noisy pid

all 32 troubleshooting commands