Need a way to make run results more granular

[edeandrea]

The current setup allows us to be super flexible in what we want to run (i.e. only quarkus, only spring, certain versions, which comparisons, etc). However, when we run a job most of the time we want all permutations/combinations of runtimes & tests because of the way we store the results (both in horreum & in https://github.com/quarkusio/benchmarks).

The challenge is that a "full" run takes 4+ hours and blocks the entire lab environment while running. It would be nice if we could split it into multiple "sub-jobs" that were part of the same run, and then "combine" the results in horreum and https://github.com/quarkusio/benchmarks once all the sub-runs are done.

Note

The term "sub-run" could be purely conceptual where we "glue" multiple runs together or it could be an actual thing that exists. I'm trying to separate theory from implementation.

This way each sub-run could be scheduled independently in the lab and other jobs besides ours could compete for compute time (we want to be good citizens after all!).

Ideally we would want to continue to publish results incrementally. Currently we treat each run as a standalone thing with a unique date/time, and then that whole run becomes the "latest". We would need to re-think how we do that. There will probably be some work within https://github.com/quarkusio/benchmarks as well once we figure out a plan.

[edeandrea]

@zakkak @franz1981 @holly-cummins I've been thinking about this a bit and wanted to get your thoughts.

After the addition of the leyden runs we now have quite a large matrix of "runtimes":

• quarkus3-jvm
• quarkus3-leyden
• quarkus3-virtual
• quarkus3-virtual-leyden
• quarkus3-native
• spring3-jvm
• spring3-leyden
• spring3-virtual
• spring3-virtual-leyden
• spring3-jvm-aot
• spring3-native
• spring4-jvm
• spring4-leyden
• spring4-virtual
• spring4-virtual-leyden
• spring4-jvm-aot
• spring4-native

17 total variants - 5 Quarkus, 6 Spring 3, 6 Spring 4

Running them all together in the lab takes 8.5 hours, an average of ~30 minutes per runtime (the native variants take a bit longer than jvm, but let's stay with average here).

I'm not necessarily advocating that we remove the ability to do any of these. I want to re-define what we consider a "full" run (or what we mean when we say "everything").

When looking at all of this I came up with the following questions:

1. Do we care about the variants that measure the impacts of Leyden with virtual threads? Or is just measuring the overall impact of Leyden "good enough"?
2. Do we care about Spring 3 anymore?
3. Do we care about spring3-jvm-aot and/or spring4-jvm-aot?

If the answer is No to all 3 of these, we can shrink down the number of variants from 17 -> 8, which would cut the run time down to around 4 hours.

If we care about spring 3, maybe all we care about is spring3-jvm as a baseline? That would bring it to 9. Still more manageable.

[holly-cummins]

I agree running everything every time isn't sustainable. I suggest we do the following:

• Add the ability to have 'distributed in time' comparisons (which I think was sort of what this issue originally covered)
• Have an 'a-list' and a 'b-list' set of jobs (and maybe a few other lists), so that we don't need to run everything every time; maybe we'd only run everything on the release boundaries, or something. This would need a bit of work on the graphics side, to maybe not generate some of the 'groups' plots if there's not a critical mass.

Do we care about the variants that measure the impacts of Leyden with virtual threads? Or is just measuring the overall impact of Leyden "good enough"?

In the long term, it seems measuring both variants of Leyden is combinatorics overkill. But in the short term, I'm really intrigued about how Leyden seems to 'cancel' the throughput impact of Leyden in some cases, and think we (or @Sanne) should investigate that. It might be just noise, but I was surprised.

Do we care about Spring 3 anymore?

This is another one where I think long term, definitely not, but for now, maybe? Spring 3 and 4 have different tradeoffs, which is kind of interesting, and I assume there's a lot of usage of Spring 3 still.

Do we care about spring3-jvm-aot and/or spring4-jvm-aot?

I wonder how widely used it is? It feels like this one can dropped now, but if it's widely used, maybe we'd want to wait for Leyden adoption to be broader?

[edeandrea]

I wonder how widely used it is? It feels like this one can dropped now, but if it's widely used, maybe we'd want to wait for Leyden adoption to be broader?

Whenever I do my Quarkus v Spring talk and i'm in an interactive setting I always ask the audience if anyone is doing this. Its very rare that I get a single hand.

[edeandrea]

which I think was sort of what this issue originally covered

It is. I was just trying to think of a way to get some "quick wins" now vs trying to figure out how to refactor everything, especially not knowing what tools/etc are going to be used on the new hardware at IBM vs the current hardware at Red Hat.

[edeandrea]

@holly-cummins @franz1981 I was also thinking of another way to save some time - currently for each test (load test, measure ttfr, etc) we build the app before running the tests.

Maybe at the very beginning we can go through all the various runtimes and build each one, that way if we are doing multiple tests we don't have to rebuild the app each time. This will help mostly for native variants, as they can take several minutes to build.

[holly-cummins]

That seems really sensible.

[zakkak]

FWIW +1 to all things proposed so far. They all seem reasonable to me.

[edeandrea]

I'm going to break some of these suggestions/thoughts out into separate sub-issues so that we can tackle them in smaller chunks.