Scalability tests for beta releases

[alejandrox1]

Current state of affairs:
We have the following jobs to gauge the quality of the current release

• https://testgrid.k8s.io/sig-release-master-informing#gce-master-scale-performance
• https://testgrid.k8s.io/sig-release-master-informing#gce-master-scale-correctness

These run against the latest on the master branch of k/k.
These jobs provide critical signal during the release cycle.
However, after code freeze, when we reopen the master branch for the next release, we may occasionally cherry pick multiple commits from master to the release-x.y branch.
During this period, between code thaw and the official release-x.y, we occasionally see failures in our master-informing scalability jobs and are unsure if the changes that brought on the failure are have been cherry picked into the release-x.y branch.

The thing I want to bring upfor discussion in this issue is the possibility of creating scalability jobs for the beta release (the version of the Kubernetes code from code thaw until the official release).
An additional caveat is that besides testing a certain portion of the lubernetes source code (the contents of the release-X.Y branch from code thaw to release) we may also have to set up the tests to run with the equivalent version of https://github.com/kubernetes/perf-tests (to make sure changes to this repo dont obscure signal from k/k).
In short, what do you all think?

Additional resources:

• Figure out a mechanism to avoid breaking tests in k/k repo perf-tests#797 Figure out a mechanism to avoid breaking tests in k/k repo

/cc @kubernetes/sig-scalability-feature-requests
/cc @kubernetes/release-team @kubernetes/release-engineering
/sig release
/sig scalability
/priority important-longterm
/milestone v1.18

[justaugustus]

xref 1.15 Retro AIs: #806

[alejandrox1]

/cc @wojtek-t @mm4tt
would love to hear your thoughts on this proposal

[wojtek-t]

While in general I support it, currently we don't have resources to run this.
We're working on speeding up our tests (among others, we're trying to merge existing two tests into a single one: kubernetes/perf-tests#1008). We believe, we will be able to speed up our 5k-node tests to take hopefully less than 8 hours.
Once this is done, we should be able to leave the 1-day frequency, but run the job for both master and k8s-beta release.

But that still requires non-trivial amount of work to do on our side to speed up our tests.

[alejandrox1]

Thank you @wojtek-t for your comment. The work you are doing to speed up the tests would be of great help for us.

One thing that has come up a couple times, please correct me if Im wrong, is that Google is running all scalability jobs, correct?
If possible, we would like to (at some point) move these jobs into CNCF infrastructure.

In order to do this we would need some idea of what we would need to run these tests. For example billing information that could be shared with us and wg-k8s. This way we can ask and possibly start planning how we can make the move.
What do you think?

[wojtek-t]

One thing that has come up a couple times, please correct me if Im wrong, is that Google is running all scalability jobs, correct?
If possible, we would like to (at some point) move these jobs into CNCF infrastructure.

I think the release-blocking ones has already been moved. But TBH I don't know how to confirm it.
They are running in kubernetes-scale project - do you know how to check if this was already transferred to CNCF?

[mm4tt]

Sorry for not jumping in earlier, I was OOO.
FTR, we run 100 gce node and 500 kubemark node tests continuously on all active release branches - https://k8s-testgrid.appspot.com/sig-scalability-gce. Also recently we've started running the same tests as presubmits in those branches. They are not as sensitive as 5k node tests, but treating them (gce 100 node tests) as beta release blocking tests might be a good intermediate solution (if you don't do that already).

[alejandrox1]

Sorry for the super delayed response on this: between release team and checking on the current state of infra I lost track of time but couple details...

I think the release-blocking ones has already been moved. But TBH I don't know how to confirm it.
They are running in kubernetes-scale project - do you know how to check if this was already transferred to CNCF?

Currently, runs in CI borrow (at least for GCP-based jobs) projects/credentials from Boskos.
I went around asking wg k8s infra and it seems that all credentials (projects) are currently owned by Google.

bentheelder
Prow itself is on Google infra.
Most of the infra used from Prow is in google.com GCP projects.
A little bit is not. Including:

• AWS accounts via CNCF (formerly Google, not for a year or so now)
• GCB projects for some release automation, which are Google funded CNCF owned

Jobs execute arbitrary code though, so some of them could be using some other infra
Like I think we might actually still have some jobs running EKS tests :face_with_rolling_eyes:
And some of the windows testing from Azure involves images built out of band by some Azure owned process we know nothing about 🙃

alejandrox1
The pool of GCP creds that boskos uses, are all of those google-owned accounts?

bentheelder
Yes.
Also fwiw boskos only hands out project names, the same credential owns all of them :witnessprotectionparrot:

So I guess the work involved in moving scalability tests onto CNCF resources would involve some tweaking on Boskos, in which case it would land us (SIG scalability and release) to work with wg-k8s-infra and figure this out. wdyt @justaugustus @mm4tt @wojtek-t ? Should we proceed with this and try and figure out a way to use CNCF resources for the 5k scalability job?

---

FTR, we run 100 gce node and 500 kubemark node tests continuously on all active release branches - https://k8s-testgrid.appspot.com/sig-scalability-gce. Also recently we've started running the same tests as presubmits in those branches. They are not as sensitive as 5k node tests, but treating them (gce 100 node tests) as beta release blocking tests might be a good intermediate solution (if you don't do that already).

Thank you Matt for mentioning these. We, release team, do consider these jobs release blocking as well.