Using tf-controller for cluster management

[tomhuang12]

I have been thinking about cluster management using tf-controller, and a few people have also asked about this so I want to document my idea and invite discussions around this topic.

Similar to the CAPI model, tf-controller can be used to manage clusters infrastructure. A management cluster can be bootstrapped by a local cluster running tf-controller. Then, the resources can be migrated to the new management cluster.

• If the terraform state is stored external to the cluster, the migration process is very straightforward, simply syncing the same repository that the local cluster Flux is synced on. The terraform resource will reconcile on the same state and results in no drift.
• If the state is stored in the cluster, the state secret will need to be migrated to the new cluster for the tf-runner to reconcile on so it doesn't try to re-create the cluster.

During a node upgrade process in the management cluster, I would recommend breaking the upgrade process into various steps.

1. Create new nodes with terraform/tf-controller. The tf-runner would run in the existing nodes.
2. Once new nodes are ready, cordon and drain existing nodes so that all pods are running on the new nodes.
3. Delete/destroy the existing nodes by deleting the Terraform resource. This tf-runner would run in the new nodes.
   This way you don't have tf-runner potentially be evicted during upgrade.

Open to ideas and feedback!

[chanwit]

We can also pause the reconciliation to make sure that the TF object will not kick off a new runner pod.