delete the unreachable node from server-side and only re-add it when it is available and explicitly desired

[boppanasreevishnu]

When some agent node is unreachable, I am deleting the node via kubectl delete node and then I am deleting the secret of the unreachable node this method is working fine but I am facing issue when I want to add the node back after it is available, Once the node becomes available, I am cleaning the node by running rke2-killall.sh and deleting the rancher folders in /var/lib and /etc then trying to add back node but it is throwing error "response to 1xx.xxx.xxx.xx:45606: unable to verify password for node slave1: hash does not match" because the " cat /etc/rancher/node/password " is being changed when the node is launched again and this newly changed password is not matching with the slave1 hash which was already present in server node how to fix this issue ?

Timeline

1. kubectl delete node
2. Delete the secret (though RKE2 often manages this, it's good to confirm).
3. Node becomes available.
4. You clean the node (rke2-killall.sh, rm -rf /var/lib/rancher /etc/rancher). This generates a new /etc/rancher/node/password.
5. You try to add the node back.

Problem Summary: The RKE2 server still has the old hash for slave1's /etc/rancher/node/password stored internally. When the node tries to connect with its new /etc/rancher/node/password, the server sees a mismatch and rejects it with "hash does not match."

[brandond]

1. What version of K3s are you using?
2. Is this about K3s or RKE2? If this is an RKE2 issue then this is the wrong repo.
3. If the node and node password secret are deleted, it shouldn't matter what's in the node password file on the node. The node password isn't stored anywhere else other than the password file and secret.
4. Why are you deleting /etc/rancher when killing the node? This should not be necessary.