Consistent EKS addon management for vpc-cni and kube-proxy

[MinhNguyen-at]

/kind feature
Thanks to #5520 CAPA is on its way to provide some resemblance of the ability to migrate to self-managed versions of eks addons.

Currently, vpc-cni and kube-proxy are explicitly reconciled separately regardless of what is listed as an addon. vpc-cni reconciliation has the label "aws.cluster.x-k8s.io/prevent-deletion" to allow short-circuiting the ReconcileCNI() while ReconcileKubeProxy() does not.

It would be nice to have a consistent story for addons and migrating to self-managed versions of addons.

Proposed solution(s):
a) add annotation/label check for deletion prevention to short circuit ReconcileKubeProxy, similar to ReconcileCNI.
b) remove ReconcileKubeProxy and ReconcileCNI and have them as managed addons.

B is cleaner schema wise but would make it more difficult for existing users to upgrade CAPA versions.

• Cluster-api-provider-aws version: v2.8.2 (big thanks to 🐛 fix(addons): avoid infinite reconciliation loop #5458 for fixing infinite reconciliation loop with addons)
• Kubernetes version: 1.30
• OS debian

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.