Node NotReady triage

Read node Conditions then tail kubelet journal to find why node is NotReady

kubectl describe node <node> | grep -A12 Conditions; ssh <node> 'journalctl -u kubelet -n 100 --no-pager'

tool
k8s-debug27 commands
tags
k8s-debugtroubleshootingk8skernel

More k8s-debug commands

  • kubectl get events -n <ns> --field-selector reason=Failed -o wide | grep -i 'pull\|manifest\|unauthorized'

    Surface registry pull failures: bad tag, missing imagePullSecret, or rate limit

  • kubectl get po -A | grep Evicted | awk '{print $2" -n "$1}' | xargs -L1 kubectl delete po

    Bulk-delete all Evicted pods left behind by node-pressure eviction

  • kubectl patch po <pod> -n <ns> -p '{"metadata":{"finalizers":null}}' --type=merge; kubectl delete po <pod> -n <ns> --grace-period=0 --force

    Clear finalizers to unstick a pod hung in Terminating; note: confirm node is gone first

  • kubectl get --raw /api/v1/nodes/<node>/proxy/stats/summary | jq '.pods[] | {name:.podRef.name, mem:.memory.workingSetBytes}'

    Pull live per-pod memory from kubelet Summary API, bypassing metrics-server

  • kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -S

    Check per-CPU conntrack drops/insert_failed; nf_conntrack table exhaustion drops packets

all 27 k8s-debug commands