Skip to content

TestRunner: add --skip-screenshots option to accelerate local UI test#24150

Merged
sgiehl merged 2 commits into5.x-devfrom
feature/ui-skip-screenshot-comparison
Mar 2, 2026
Merged

TestRunner: add --skip-screenshots option to accelerate local UI test#24150
sgiehl merged 2 commits into5.x-devfrom
feature/ui-skip-screenshot-comparison

Conversation

@tzi
Copy link
Copy Markdown
Contributor

@tzi tzi commented Feb 28, 2026

Description

This PR introduces a new UI test runner flag: --skip-screenshots.

When enabled, UI tests still execute normally (fixtures, navigation, interactions, API calls, waits, JS/runtime checks), but screenshot diff assertions are skipped.

Why this is valuable

  • Screenshot comparison is often unreliable locally because rendering depends on OS-level differences (for example, font rendering).
  • Makes local debugging faster when visual diffs are not the focus, or when investigating logic/regression issues.
  • Unblocks development in environments missing image tooling.

How it looks

Screenshot 2026-02-28 at 22 08 21

Checklist

  • [✔] I have understood, reviewed, and tested all AI outputs before use
  • [✔] All AI instructions respect security, IP, and privacy rules

Review

@tzi tzi added this to the 5.9.0 milestone Feb 28, 2026
@tzi tzi added not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. labels Mar 2, 2026
@sgiehl
Copy link
Copy Markdown
Member

sgiehl commented Mar 2, 2026

@tzi Nice idea. But actually the screenshot in the PR and how it really looks does not macht. There is no info message that screenshots are skipped when that happens, the tests just silently turn green.

@tzi
Copy link
Copy Markdown
Contributor Author

tzi commented Mar 2, 2026

@tzi Nice idea. But actually the screenshot in the PR and how it really looks does not macht. There is no info message that screenshots are skipped when that happens, the tests just silently turn green.

Thanks for the feedback.
It's interesting, because I use it a lot since I created it. I will check if I forgot something in this PR.

@tzi
Copy link
Copy Markdown
Contributor Author

tzi commented Mar 2, 2026

@sgiehl Yes, I've forgot to add the console.log() is this PR. It's updated now

@sgiehl sgiehl merged commit 517f6b7 into 5.x-dev Mar 2, 2026
30 checks passed
@sgiehl sgiehl deleted the feature/ui-skip-screenshot-comparison branch March 2, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.

Development

Successfully merging this pull request may close these issues.

2 participants