Take a snapshot, then defrag

Back up etcd, then free space: defrag returns space after compaction, alarm disarm clears the NOSPACE alarm. Run it off-peak

ETCDCTL_API=3 etcdctl snapshot save /backup/snap.db && etcdctl defrag --cluster && etcdctl alarm disarm

tool
etcd3 commands
tags
etcddisktroubleshooting

More etcd commands

  • curl -sk https://localhost:6443/healthz && kubectl -n kube-system get lease kube-scheduler kube-controller-manager

    Check apiserver health and see which node holds the leader lease for the scheduler and controller-manager

  • ETCDCTL_API=3 etcdctl --endpoints=https://<ip>:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --cluster -w table

    Show the leader, DB size and raft index for each member. Bloat and split-brain are easy to spot

all 3 etcd commands