Tests rely on missing infrastructure

[jugglinmike]

Tests for a few proposals rely on files in a top-level directory named /gen/:

$ git grep -l '/gen/'
idle-detection/interceptor.https.html
shape-detection/resources/shapedetection-helpers.js
webxr/resources/webxr_util.js

Further references have recently been proposed for tests of the "sms" proposal.

This directory does not exist in WPT's source tree, and to the best of my knowledge, it is not generated by any of WPT's infrastructure.

@Hexcles mentioned that this is generated code in Chromium's infrastructure. In that case, I don't think this can be resolved by checking in the files (doing so would presumably interfere with Chromium's tests, and the files are unlikely to be useful for other implementations).

@Hexcles @LukeZielinski Can we formalize a contract which would allow others to use these tests? Do you think it could be stable enough to support implementation by other browsers?

[jdm]

I brought up the idle-detection issue in #15753.

[Hexcles]

These are all Mojo stuff. These tests won't work even on Chrome unless 1. generated Mojo JS files are available (this is non-trivial and I'll explain further); 2. a few test flags are turned on (see web-platform-tests/results-collection#582).

Mojo JS stubs are generated from source code at compile time so technically we need the Mojo files generated from the specific revision of Chrome we're testing against, which means we need to publish & download a bundle of files instead of simply checking them into WPT.

More discussions can be found in web-platform-tests/results-collection#81 from last year and cc @foolip .

[jugglinmike]

That's helpful information for getting results from Chromium, but it doesn't seem like it will help others run these tests.

In the issue that Josh created for idle-detection, there's been some discussion about documenting a testing API. That's just the kind of contract I had in mind. It seems essential for the tests to be shared. @jdm would you agree?

I can't find instruction like that for idle-detection, shape-detection, webxr, or sms. @samuelgoto @reillyeon would you mind referencing those docs from the tests?

[Hexcles]

There's https://github.com/immersive-web/webxr-test-api/blob/master/explainer.md for WebXR.
(From https://chromium-review.googlesource.com/c/chromium/src/+/1070778/15#message-f47cb5752847daa0e0c550d4a033d9fc21103c74)

[reillyeon]

Are you looking for documentation on how to run these tests or documentation of the API that the tests expect to be provided by a browser implementation?

[jugglinmike]

The latter--something along the lines of the document @Hexcles shared for WebXR

[thejohnjansen]

I feel like there is a larger concern here: Google (or anyone working in Chromium) adding tests that rely on internal infra and therefore cannot be run in other browsers. It may be even larger than that: adding tests that are not designed to be run in other browsers. I think this needs to be addressed at that level: given the 2-way sync between Chromium and WPT, I worry this may become a larger problem over time. Should these specific tests even be in the WPT repo?

If not, how do we prevent them from syncing in the future?
if they should be, how do we get them to run in other browsers?

[reillyeon]

@thejohnjansen the details matter a lot here. Every browser running WPT comes with a bit of implementation-specific infrastructure for automation. A well-written test abstracts over these differences in order to test a feature on all implementations. The reason why this issue was filed was because those tests weren't properly abstracted from the implementation-specific automation they require.

To get these tests to run in other browsers (aside from the browser implementing the API being tested) the engine needs to add its own version of necessary automation hooks.

[jugglinmike]

Thanks for bringing this up again, @thejohnjensen.

given the 2-way sync between Chromium and WPT, I worry this may become a larger problem over time.

You're already correct. Today, there are more references than when this issue was reported:

$ git grep -l '/gen/'
contacts/resources/helpers.js
idle-detection/interceptor.https.html
shape-detection/resources/shapedetection-helpers.js
sms/sms_provider.js
web-nfc/resources/nfc-helpers.js
webxr/resources/webxr_util.js

It's been four months, so I think some action is necessary from WPT. gh-18509 is a patch that causes the project linter to flag these references. Removing the existing tests from WPT directly would likely regress coverage in Chromium, so the patch makes exceptions for those. Probably they should be relocated in the Chromium source tree (and hopefully reintroduced when documentation is available). The new linting rule will at least help us avoid further regression.

[Hexcles]

Following TPAC discussion today, I think we have a consensus here:

1. It's fine to have test APIs backed by browser-specific implementations (be it test APIs behind prefs or MojoJS interfaces).
2. We should not check automatically generated MojoJS stubs (i.e. the /gen/ stuff) into WPT (these are build artifacts that are specific to Chromium revisions). In order to make MojoJS work with public Chrome, we would need to figure out a way to archive and release MojoJS stubs per Chrome version. (I'm looking into this.)
3. It's fine to check the implementations of test APIs that rely on MojoJS into WPT. However, currently these impls would fetch some /gen/ URLs that only exist in Chromium infra, which is unfortunate. We should first at least make the fetches conditional and asynchronous so that they don't cause network errors in the upstream. Once 2 is resolved, we will work out a way to make this more transparent.

Thanks for pushing for the right thing and taking some actions, @jugglinmike and sorry that I haven't done more -- in case it's not clear, I'd like to tackle this. Here are the concrete and parallel steps I'm taking:

• Make sure the lint rule is working in Chromium (for some reason, it's not currently). Once it is, new references of /gen/ would need to be whitelisted in lint.whitelist, of which I'm an owner in Chromium and I'll be vigilant reviewing the changes.
• Convert existing fetches of MojoJS to non-blocking & conditional; the tests should fail in some more obvious way other than a network error.
• Explore releasing MojoJS stubs and make it available to upstream WPT.