Confirm egress route and ARP

Show actual src/iface for a dest; FAILED neigh entries = L2/ARP problem

ip route get <ip>; ip neigh show | grep -E 'FAILED|INCOMPLETE'
tool
troubleshooting32 commands
tags
troubleshootingnetworkdebug

More troubleshooting commands

  • TCP port reachability past firewall
    traceroute -T -p 443 <host>; nc -zv <host> 443

    TCP-SYN traceroute survives ICMP filtering; nc -zv confirms open port

  • Bypass DNS with curl --resolve
    curl -sv --resolve <host>:443:<ip> https://<host>/healthz

    Pin host to a specific IP to test backend while keeping SNI/Host header

  • In-cluster DNS smoke test
    kubectl run dnsutils --image=tutum/dnsutils -it --rm --restart=Never -- nslookup kubernetes.default

    Throwaway pod to resolve in-cluster names; verifies CoreDNS + kube-dns svc

  • Query CoreDNS directly
    dig +short @$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}') <name>.<ns>.svc.cluster.local

    Hit CoreDNS clusterIP straight; note: ndots:5 makes short names spam queries

  • Inspect CoreDNS config and logs
    kubectl -n kube-system get cm coredns -o yaml; kubectl -n kube-system logs deploy/coredns --tail=50

    Review Corefile (forward/cache) and tail logs for SERVFAIL/upstream errors

all 32 troubleshooting commands