How to measure flakiness?

[guettli]

I use the CI at Github and Gitlab, and up to now our CI does not have a memory. The CI does not know if a test failed for the first time, or if it fails randomly once a week.

Some years ago we used Jenkins. I don't miss it, but it provided a GUI where you could see the history of one particular test.

Dear pytest community, do you have recommendation, so that everybody in the team can easily see the fail/pass history of each test?

[nicoddemus]

Dear pytest community, do you have recommendation, so that everybody in the team can easily see the fail/pass history of each test?

Not exactly what you asked for, but at work we have a process in place to avoid having flaky tests get in the way of the day to day work.

When working on a PR and an unrelated test fails in CI, we run the CI again. If the test passes, it means it is flaky, so we:

1. Create a new issue that tells which test failed, its traceback, and any other relevant information such as platform.
2. Mark the test with @pytest.mark.xfail(run=False, reason="issue #XXX") (or @pytest.mark.flaky if using flaky). If we notice it only happens in one platform, we only xfail on that platform.
3. At each sprint, we make sure to take 1 or 2 flaky tests to investigate, to fix the cause for them to be flaky and remove the mark.

With this approach we make sure flaky tests don't become a problem (frustration, eroding the trust in the test suite), and also don't let the number of flaky tests grow out of control.

[guettli]

This workflow looks pragmatic. Thank you for the hint.

Nevertheless numbers (per single test) would be nice.

Maybe someone else has a hint?