how to update the etcd member list

Question

my cluster have 3 master node, now I have shutdown 1 master node, then I check the member from etcd database:

[root@fat001 bin]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl member list
56298c42af788da7, started, azshara-k8s02, https://172.19.104.230:2380, https://172.19.104.230:2379
5ab2d0e431f00a20, started, azshara-k8s01, https://172.19.104.231:2380, https://172.19.104.231:2379
84c70bf96ccff30f, started, azshara-k8s03, https://172.19.150.82:2380, https://172.19.150.82:2379

still show 3 nodes started. why the etcd did not refresh the node status? what should I do to update the etcd status to the latest? is it possible to refresh the status manually? the kubernetes version is 1.15.x.

Answer 1

If you delete a node that was in a cluster, you should manually delete it from the etcd also i.e. by doing 'etcdctl member remove 84c70bf96ccff30f '.

Make sure that etcd container is no longer running on the failed node, and that the node does not contain any data anymore:

rm -rf /etc/kubernetes/manifests/etcd.yaml /var/lib/etcd/

crictl rm "$CONTAINER_ID"

The commands above will remove the static-pod for etcd and data-directory /var/lib/etcd on the node.Of course, you can also use the kubeadm reset command as an alternative. However, it also will remove all Kubernetes-related resources and certificates from this node.