Ensure we don't have unexpected double rollouts in ClusterClass e2e tests

[sbueringer]

Context:

• An unexpected double rollout in the ClusterClass context e.g. means that when we apply changes to Cluster.spec.topology (e.g. setting etcdImageTag and upgrading the version) those changes are rolled out with a single rollout.
• A double rollout would be:
  • The topology controller propagates the etcdImageTag to KCP
  • Then waits until KCP is stable again
  • The topology controller then bumps the version in KCP

We found an unexpected double rollout when triaging: #7833 (essentially what is described under context).

In this issue we should:

• analyze in which test cases double rollouts could occur
• find a way to validate that they didn't occur
• also consider how this works with other providers where double rollouts might be expected
  • It can depend on the cluster-template. So maybe the validation must be opt-in / opt-out

At this point the issue requires a bit more research and an idea how to implement the validation before it is actionable.

/area topology
/area testing
/kind feature

[fabriziopandini]

/triage accepted
I think this will be a great addition to our tests!

[fabriziopandini]

Note, ideally, we should detect double rollout both for CP upgrade and MD upgrades

The first idea that comes to my mind is a go routine watching for machines to be created while the upgrade happens; we do expect the same number of machines to be created, not more.
The tricky part is to decide if/how to account for remediations that might happen during upgrades, which could impact the number of machines created; hopefully the work I'm doing for #7855 might help here.

[sbueringer]

I wonder if remediations is another unexpected thing that we don't want

[sbueringer]

/assign

(mostly to have it somewhere in my queue, could become more relevant soon. If someone else wants to work on it, just let me know!)