Find top kernel slab users

Sort slab caches by size; large dentry or inode caches often explain kernel memory pressure.

slabtop -o -s c | head -20

tool
troubleshooting32 commands
tags
troubleshootingmemorykernelperformance

More troubleshooting commands

  • grep -E 'Slab|SReclaimable|SUnreclaim' /proc/meminfo

    Split slab into reclaimable and unreclaimable parts; a high SUnreclaim points to a real kernel leak.

  • dmesg -T | grep -iE 'killed process|out of memory' | tail

    See who the OOM killer picked, with readable timestamps; -T turns kernel time into normal dates.

  • cat /proc/<pid>/oom_score /proc/<pid>/oom_score_adj

    A higher oom_score means the process is more likely to be killed; oom_score_adj -1000 protects it from the OOM killer.

  • cat /sys/fs/cgroup/<name>/memory.events | grep -E 'oom|max'

    Read oom_kill and max counters for a cgroup to see a pod hitting memory.max before the whole node runs out of memory.

  • lsof +L1 | sort -k7 -rn | head

    df says the disk is full but du disagrees? A process still holds deleted files open; restart it to free the space.

all 32 troubleshooting commands