Currently the benchmarks report results that can vary constantly with GitHub Actions availability, suffer from environment variations (or any external factor related to the environment in which they are being executed).
Personally, I like to work with ratio-based benchmarks, where there is a baseline and a comparison for each benchmark run, and the comparison is made based on the ratios, not in the execution itself.
In MySQL2 this is a bit more complicated due to the MySQL Server (more external factors), but I believe we can explore something viable and even compare/document the results between other drivers.
Also, workflows that require secrets don't run properly on PRs from forks (even if it's secrets.GITHUB_TOKEN).
Furthermore, the benchmark report only negative cases. It's also interesting to have positive feedback.
Related
Currently the benchmarks report results that can vary constantly with GitHub Actions availability, suffer from environment variations (or any external factor related to the environment in which they are being executed).
Personally, I like to work with ratio-based benchmarks, where there is a baseline and a comparison for each benchmark run, and the comparison is made based on the ratios, not in the execution itself.
In MySQL2 this is a bit more complicated due to the MySQL Server (more external factors), but I believe we can explore something viable and even compare/document the results between other drivers.
Also, workflows that require secrets don't run properly on PRs from forks (even if it's
secrets.GITHUB_TOKEN).Furthermore, the benchmark report only negative cases. It's also interesting to have positive feedback.
Related