Node NotReady triage k8s-debug
Read node Conditions then tail kubelet journal to find why node is NotReady
kubectl describe node <node> | grep -A12 Conditions; ssh <node> 'journalctl -u kubelet -n 100 --no-pager'
more k8s-debug
all 27 commands →
ImagePullBackOff root cause
kubectl get events -n <ns> --field-selector reason=Failed -o wide | grep -i 'pull\|manifest\|unauthorized'
Cleanup evicted pods
kubectl get po -A | grep Evicted | awk '{print $2" -n "$1}' | xargs -L1 kubectl delete po
Force-remove stuck Terminating pod
kubectl patch po <pod> -n <ns> -p '{"metadata":{"finalizers":null}}' --type=merge; kubectl delete po <pod> -n <ns> --grace-period=0 --force
Raw kubelet stats summary
kubectl get --raw /api/v1/nodes/<node>/proxy/stats/summary | jq '.pods[] | {name:.podRef.name, mem:.memory.workingSetBytes}'
Conntrack stats on node
kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -S