How to support (rolling) upgrade #82
Description
Is there a recommended way to upgrade k8s-HDFS? Would it require down time or can a rolling upgrade be performed. Not sure if the followings (from Apache Hadoop) can be used or can we follow k8s updating StatefulSets (e.g. .spec.updateStrategy.type = RollingUpdate)? A pointer would be great.
Upgrading Non-Federated Clusters
Suppose there are two namenodes NN1 and NN2, where NN1 and NN2 are respectively in active and standby states. The following are the steps for upgrading an HA cluster:
Prepare Rolling Upgrade
Run “hdfs dfsadmin -rollingUpgrade prepare” to create a fsimage for rollback.
Run “hdfs dfsadmin -rollingUpgrade query” to check the status of the rollback image. Wait and re-run the command until the “Proceed with rolling upgrade” message is shown.
Upgrade Active and Standby NNs
Shutdown and upgrade NN2.
Start NN2 as standby with the “-rollingUpgrade started” option.
Failover from NN1 to NN2 so that NN2 becomes active and NN1 becomes standby.
Shutdown and upgrade NN1.
Start NN1 as standby with the “-rollingUpgrade started” option.
Upgrade DNs
Choose a small subset of datanodes (e.g. all datanodes under a particular rack).
Run “hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT> upgrade” to shutdown one of the chosen datanodes.
Run “hdfs dfsadmin -getDatanodeInfo <DATANODE_HOST:IPC_PORT>” to check and wait for the datanode to shutdown.
Upgrade and restart the datanode.
Perform the above steps for all the chosen datanodes in the subset in parallel.
Repeat the above steps until all datanodes in the cluster are upgraded.
Finalize Rolling Upgrade
Run “hdfs dfsadmin -rollingUpgrade finalize” to finalize the rolling upgrade.