Check a process OOM score
A higher oom_score means the process is more likely to be killed; oom_score_adj -1000 protects it from the OOM killer.
cat /proc/<pid>/oom_score /proc/<pid>/oom_score_adj
- tool
- troubleshooting32 commands
More troubleshooting commands
-
cat /sys/fs/cgroup/<name>/memory.events | grep -E 'oom|max'Read oom_kill and max counters for a cgroup to see a pod hitting memory.max before the whole node runs out of memory.
-
lsof +L1 | sort -k7 -rn | headdf says the disk is full but du disagrees? A process still holds deleted files open; restart it to free the space.
-
iostat -x 1 | awk '$1!~/^$/{print $1, "util="$NF, "await="$(NF-2)}'%util near 100 with growing await means the device is saturated; then find the process behind it.
-
biotop-bpfcc -C 5 1bcc biotop shows which process makes the most block I/O by bytes and latency; for custom questions, use bpftrace.
-
conntrack -L 2>/dev/null | wc -l; conntrack -SCount tracked connections and compare with nf_conntrack_max; -S shows insert_failed and drop when the table is full.