Measure SLOs across whole suite, not single tests

[oxddr]

Currently we measure all SLOs per-test. We think about measuring it across whole testing suite (density + load).

Measuring SLO across performance suite will increase number of windows (as defined in SLO description). Single bad request will have a smaller chances to sink the whole tests. Currently we see test flakiness cause by a single request (e.g. kubernetes/kubernetes#82377). Also this would put us closer to intention behind the two-level SLO, which is defined per cluster-day.

Implementation wise, this would involve merging density and load test into a single test and moving some measurements to the very end of it.

/area slo

[wojtek-t]

Implementation wise, this would involve merging density and load test into a single test and moving some measurements to the very end of it.

Alternatively (I'm actually leaning towards saying that it would be better, but I'm happy to hear arguments against) we could:

• add a concept of measurement that is run before the first test case and then after the last
  [conceptually BeforeSuite and AfterSuite from ginkgo]
• start and gather all SLOs there

[wojtek-t]

I think that we can even hardcode the set of things that we start there - it would also visibly simplify the configs...

[wojtek-t]

@mm4tt - FYI

[oxddr]

I like the new per-suite measurements you propose. This is indeed a better, long-term solution. Mine was suppose to be rather a hacky prototype to validate the concept.

However I'd rather avoid hard-coding any set of measurements, this would decrease the flexibility clusterloader2 gives us. But I'm open for debate.

[mm4tt]

Could it become part of the TestSuite - https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/test_suite_design.md ?

[wojtek-t]

However I'd rather avoid hard-coding any set of measurements, this would decrease the flexibility clusterloader2 gives us. But I'm open for debate.

I'm not going to push hard for this. My main goal is just to simplify the configs.
So the next iteration of that proposal is:

1. opaque the SLO-based measurement (api-call-latencies, network-programming, ...) into single "SLO" measurement [we can't yet add pod-startup-time, but we hopefully will be in the future]
   [this is similar to what we did for TestMetrics]

2. Make that a default measurement for Before/After suite.

[wojtek-t]

Could it become part of the TestSuite - https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/test_suite_design.md ?

yes - it perfectly fits there

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[wojtek-t]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale