Fix flaky playground e2e tests#11159
Draft
timotheeguerin wants to merge 3 commits into
Draft
Conversation
Contributor
|
No changes needing a change description found. |
The Playwright webServer used the Vite dev server (`pnpm watch`). The dev server transforms modules on demand and re-optimizes dependencies on the first page load; when it discovers a new dependency mid-load it issues a full page reload that aborts the in-flight `page.goto` (net::ERR_ABORTED). This is fast (not a timeout) and only hits whichever test runs first, so the first test(s) flaked while later ones passed. Serve the pre-built app with `vite preview` instead (the `test:e2e` turbo task already depends on `build`), so there is no on-demand transform, dependency re-optimization, or HMR reload. Set `cwd` to the package root so preview resolves `vite.config.ts` (outDir dist/web). Keep a single CI retry as a safety net.
timotheeguerin
force-pushed
the
fix-playground-e2e-flaky
branch
from
172c83c to
a079033
Compare
|
You can try these changes here
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
@typespec/playground-websitee2e tests are flaky in CI — the first one or two tests (compiled http sample,report compilation errors) intermittently fail atpage.goto(ui.e2e.ts:24), while the later tests pass.This is not a timeout —
page.gotocompletes in ~1–2.5s locally. The root cause is the PlaywrightwebServerusing the Vite dev server (pnpm watch). The dev server transforms modules on demand and re-optimizes dependencies on the first page load; when it discovers a new dependency mid-load it issues a full page reload that aborts the in-flight navigation (net::ERR_ABORTED). That fails fast and only affects whichever test runs first — subsequent tests hit an already-optimized server and pass.Change
vite previewinstead of the dev server. Thetest:e2eturbo task alreadydependsOn: ["build"], sodist/webis always available. There is no on-demand transform, dependency re-optimization, or HMR reload, so the first navigation is deterministic (full suite: ~6s vs ~2m before).cwdto the package root sovite previewresolvesvite.config.ts(build.outDir = dist/web).Validation
CI=1 pnpm test:e2e— 4/4 tests pass consistently in ~6s (previously ~2.1m with intermittent first-test failures).pnpm lintpasses.Note
The dev-server dependency re-optimization reload is timing-dependent, so it does not reproduce on every cold local run; CI's slower/different timing is what surfaces it. Serving the static build removes the mechanism entirely regardless of timing.