Find OOM kills in dmesg

See who the OOM killer picked, with readable timestamps; -T turns kernel time into normal dates.

dmesg -T | grep -iE 'killed process|out of memory' | tail

tool
troubleshooting32 commands
tags
troubleshootingmemoryoomkernel

More troubleshooting commands

  • cat /proc/<pid>/oom_score /proc/<pid>/oom_score_adj

    A higher oom_score means the process is more likely to be killed; oom_score_adj -1000 protects it from the OOM killer.

  • cat /sys/fs/cgroup/<name>/memory.events | grep -E 'oom|max'

    Read oom_kill and max counters for a cgroup to see a pod hitting memory.max before the whole node runs out of memory.

  • lsof +L1 | sort -k7 -rn | head

    df says the disk is full but du disagrees? A process still holds deleted files open; restart it to free the space.

  • iostat -x 1 | awk '$1!~/^$/{print $1, "util="$NF, "await="$(NF-2)}'

    %util near 100 with growing await means the device is saturated; then find the process behind it.

  • biotop-bpfcc -C 5 1

    bcc biotop shows which process makes the most block I/O by bytes and latency; for custom questions, use bpftrace.

all 32 troubleshooting commands