Unexpected behaviour using kubectl scale with a topology managed cluster

[killianmuldoon]

When using kubectl scale machinedeployment or kubectl scale kcp with a topology managed cluster the command succeeds but does not have the expected result. Depending on how quickly the topology controller reacts to the change the resource either doesn't appear to scale at all or else it scales up and then scales down quickly (in line with replicas specified in the cluster topology).

There's two options to deal with this IMO:

1. Intercept the scale request and send it to the Cluster Topology instead.
2. Block a scale request when the scalable resource is topology managed and replicas is set.

Blocking the request is the most feasible IMO as intercepting the request would be control plane specific i.e. we could implement it on KCP but it would have to be done in all other control plane providers. It would also mean adding a client to all scalable resources, and it subverts some expectations about how the API should work. When blocking we can return a useful error message explaining how to make a resource independently scalable i.e. unsetting replicas.

I haven't done a POC, but I think it would be feasible to both block changing the replicas field and block access to the scale subresource. This would have an impact on autoscaling, with the autoscaler returning a useful error when trying to improperly change a managed cluster's scalable resource rather than having the scale operation be apparently successful, but resulting in no spec change.

This is also, presumably, an issue when a MachineSet is managed by a MachineDeployment, but scaling an MS individually is probably a less established pattern.

/area topology
/cc @jackfrancis @sbueringer

[fabriziopandini]

/triage accepted

Ideally, the cluster should become the scalable resource in this case, but I'm not sure we can achieve this given that there could be many scalable things in one cluster.

+1 to explore ideas about how to make this possible/less surprising for the users

[killianmuldoon]

Ideally, the cluster should become the scalable resource in this case, but I'm not sure we can achieve this given that there could be many scalable things in one cluster

Agreed - especially with multiple MachineDeployments this could become really unergonomic - i.e. kubectl scale cluster md $MD_NAME --replicas=3

[sbueringer]

Looking at how scale is implemented on the Kubernets side this means we need spec/status fields for replicas in the Cluster and I think we can't have multiple of them:

cluster-api/config/crd/bases/cluster.x-k8s.io_machinedeployments.yaml

Lines 508 to 512 in 3b3d969

	scale:
	labelSelectorPath: .status.selector
	specReplicasPath: .spec.replicas
	statusReplicasPath: .status.replicas
	status: {}

So I think making this happen somehow requires a very creative solution.

Intercept the scale request and send it to the Cluster Topology instead.

I can't imagine a good solution for this

This would have an impact on autoscaling, with the autoscaler returning a useful error when trying to improperly change a managed cluster's scalable resource rather than having the scale operation be apparently successful, but resulting in no spec change.

We would have to make sure we block this request only if the topology controller actually cares about the replicas field. Otherwise we break the autoscaler.

Additionally, I have no idea if the autoscaler is using update/patch or the scale subresource.

[sbueringer]

Agreed - especially with multiple MachineDeployments this could become really unergonomic - i.e. kubectl scale cluster md $MD_NAME --replicas=3

This CLI syntax 100% requires a KEP for kubectl as it's not supported today.

[killianmuldoon]

Additionally, I have no idea if the autoscaler is using update/patch or the scale subresource.

I think if we implement either of the solutions it would have to work for both the scale subresource and for updating the replicas field.

This CLI syntax 100% requires a KEP for kubectl as it's not supported today.

I'm definitely not up for proposing it 😆 - just expressing the level of granularity this would need because of the multiple types of scalable resource and the multiple actual resource in the Cluster Topology.

[sbueringer]

Just wanted to make clear that we're talking about changing a feature / trying to workaround a feature which is almost entirely implemented upstream (kubectl / kube-apiserver) :)

[killianmuldoon]

Just had a look - it looks like the Cluster Autoscaler uses https://pkg.go.dev/k8s.io/client-go/scale which works on the scale subresource. But we can't be sure this is the case for other autoscaler implementations of course.

[jackfrancis]

Thanks for facilitating this discussion @killianmuldoon

@elmiko a brief tour through the capi provider of cluster-autoscaler suggests that it doesn't work with ClusterClass at all

Maybe this option will solve for a lot of the issues we're observing:

Intercept the scale request and send it to the Cluster Topology instead.

But it would probably be better to have autoscaler be ClusterClass-aware, and initiate scale events against the appropriate Cluster sub-resource instead of the MachineDeployments (and when MachinePools support lands for autoscaler, same idea).

Does that sound right?

[fabriziopandini]

I was thinking at this and looking at options to make it work for the 80% of the use cases

Is it acceptable to say that kubectl scale should scale the first MachineDeployment?

Why this approach?
I assume scaling the control plane is not a frequent operator, so I don't see value in providing a shortcut on this operation (it could also be seen as a sort of protection to the control plane)

So, with the CP out of the table, kubectl scale should target MachineDeployments, and assuming having only 1 MD is the most common use case, it should target the first

Does this makes sense?

[fabriziopandini]

Wrt to the auto scaler, AFAIK we are not setting replicas in the cluster topology, so everything should work as usual

[killianmuldoon]

The first MachineDeployment is probably the 80 percent case but IMO it's not a great solution to this problem. It's not extensible to cover any other use case which makes it confusing and frustrating for users. It introduces inconsistency into the management of Clusters, and in the API which could cause further complications down the line.

I think blocking the request is probably a better solution, even if it means losing kubectl scale.