Proposed for browser testing spec

[jgraham]

For browser testing (i.e. web-platform-tests) there are some required APIs that aren't suitable to expose to web content, but don't fit into existing web developer testing paradigms like WebDriver. For example there's a relatively common request to have a way for tests to wait for a garbage collection to happen.

To address such use cases I started a Browser Testing spec document. Assuming there's vendor buy-in, I think it would make sense to make this a WHATWG Living Standard. This is intended to complement feature-specific testing APIs e.g. the one in WebUSB, and provide a home for cross-cutting functionality that isn't tied to a specific feature. In the long run I hope we can move in the direction of having a small number (maybe 1) of top level interfaces containing test APIs, with additional specs extending that top level interface as required.

As an alternative to creating a new spec here, we could consider adding these features to an existing spec, probably HTML. I think the main argument against that is that it seems likely the people specifying testing APIs will be different to the people working on HTML, and the overhead of working on the HTML spec is considerable. Also, given the features are not exposed to general web content, it seems helpful to be able to write requirements around expose that apply to the entire testing spec and nothing else, without worrying about reader confusion.

[foolip]

Thanks for filing this, @jgraham!

With my WHATWG SG hat off, I'm very happy to see this proposal, test-only APIs is something we've discussed many times, and I think it makes a great deal of sense for some things that don't really work as WebDriver APIs.

With SG hat on, I think WHATWG would be a good venue for this, but the WHATWG doesn't really have an incubation track, and I wonder if it could be too early to commit to a workstream for this. In particular, https://whatwg.org/working-mode#additions might be an issue in early days while things are a little bit experimental and we're not certain the overall approach will multi-implementor support. Perhaps @annevk or @domenic can share some experiences of how large-ish new things tends to come into existence in Living Standards, perhaps this isn't a big issue.

If WHATWG is the final destination, then the work to get it set up is to define a new workstream, with some well defined scope, and getting some legal review of that scope.

A possible starting point would be the WICG. We haven't yet had anything graduate from the WICG to WHATWG, but if that intention is made clear up-front I think we could make that work. Incubate until implementer interest is clear, then do the work to define a workstream.

[othermaciej]

It's a little weird to specify something that must not be present in default configuration but only a non-default mode, but I can see how it's useful for tests.

We have not previously executed the process of adding a Living Standard since adopting the new governance model. So there may be bugs. I think we'd be willing to change it as we go.

For starters, I think a new Living Standard should meet the same bar of implementer interest as we'd expect in a new feature in an existing spec. In some ways the Steering Group stands in for implementers but likely we should get feedback from people who would be close to this work. It may be correct to do some incubation first.

[annevk]

Thus far we have started with features that were already implemented, but did not have an adequate standard. New features would then be added once the existing feature set was a bit more cemented.

So I guess it depends a bit. If there's a very small minimal viable standard here that can get multi-implementer support a draft that meets https://whatwg.org/working-mode#additions would be adequate I think. If it needs to be a bit bigger WICG might be reasonable or we formalize https://idea.whatwg.org/ somehow.

[othermaciej]

If the contents of the spec as-is can meet the Working Mode additions guidelines, then I agree, it is reasonable to create a Workstream directly. The policy for that is defined here: https://whatwg.org/workstream-policy#new-workstreams

It doesn’t really say where to put a Wrokstream Proposal, but it seems to me putting it into the top comment of an issue would be fine.

[othermaciej]

Right now the spec has one interface with one method. Not sure if that is the whole expected scope, or if more is planned.

[foolip]

That's a good question, what should go in this spec and what should other specs define themselves? partial interface TestUtils allows other specs to define their bits, and that will probably be the most common in the end, but how much stuff do we have that doesn't fit into any spec that we'd add to this spec directly?

[domenic]

I'm also excited by this proposal!

On the question of incubate-first versus straight-to-Living-Standard, I think that's largely a question for @jgraham. Most new specs benefit from an incubation period, where they can iterate without getting multi-implementer interest on every pull request; they can write specs and explainers ahead of tests; etc. But this might be different. It seems like the intention is to start small, with the gc() function (plus maybe the exposure technology; see below), and expand over time in a methodical way, getting implementer interest for each addition. As long as @jgraham would find that working mode OK, then I think the WHATWG community and the Steering Group should be supportive.

My only other concern is whether this really needs a new standalone spec, as the OP mentions. In particular, I'm curious how many instances of "cross-cutting functionality that isn't tied to a specific feature" @jgraham or others envision in the long run. We certainly do have Living Standards that end up small and largely static, e.g. Console or Compat. But it'd be a bit extreme if we had such a standard with just a single method (gc()). So any speculations on future additions to testUtils would be great to have.

Finally, a bit of technical background on the exposure technology: see whatwg/webidl#896. If we do the refactoring described in whatwg/webidl#883 to create a generic "exposure attribute" concept in Web IDL, then it would make sense for this Browser Testing API standard to contain the definition for the [TestOnly] extended attribute, based on top of that Web IDL concept. So that would be at least two concrete things that the Browser Testing API Living Standard could define :).

[jgraham]

So in terms of the venue, my preference would be to deal with one standards body rather than exploring how to bounce things between two groups. So if possible I'd prefer to figure out how to make progress in WHATWG directly rather than first going to WCIG. Obviously if that turns out to be impossible I can take a different approach, but based on what's been said so far I think the requirements match my expectations of how this work will proceed. In particular:

• I think it makes sense to start with a small scope i.e. just an interface and the gc function. I'm confident there's implementer interest in that and it has enough value to be useful without anything else.
• There are no specific plans for additional APIs at this time, but there is a plan to audit existing Gecko/WebKit/Blink tests for internal test APIs that are common between the implementations. That may throw up some additional candidates (for an example, the ability to pierce shadow-dom encapsulation came up in discussions as one possibility, but I don't yet have any evidence that's widely needed, or possible to implement, so I don't want to over fixate on the specifics).
• The scope of this spec would remain "stuff that doesn't obviously belong to a specific platform feature". For example I would expect WebUSB testing APIs to remain in Web USB. But if we wanted to define a [TestOnly] extended attribute, I agree this would be the right spec to do that.
• Any further additions would go through the normal process for getting implementer acceptance.

In general I think limited incubation i.e. just getting buy-in on the general concept of test-only DOM APIs and the one specific API works in this case because there isn't a lot of "architecture" here; decisions taken now don't strongly constrain the future evolution of the spec. That's because each feature is basically orthogonal to existing features except in trivial aspects like the name of the base interface.

In terms of actual implementations, I'm hoping to implement this in Gecko soon, but it's subject to internal priorities. Getting another implementation would obviously be great; so far all the signals are positive, but nothing concrete.

If this approach is acceptable, what's the next step? Making a formal workstream proposal? @othermaciej suggested creating that as an issue; is that in this repo or elsewhere?

[othermaciej]

In this repo, and you can even recycle this issue if you want.

[annevk]

@jgraham as discussed at #143 the next steps here would be settling on "scope" and the wider Workstream proposal, which can be either inline or a linked resource.

[hsivonen]

There are no specific plans for additional APIs at this time

While I'm not committing to implementing the Gecko side of such an API myself, I'd like to have the ability to write WPTs that generate mouse clicks at CSS-pixel coordinates relative to the top-left corner of the content area and WPTs that generate key events (particularly tab key and shift-tab). Specifically, I care much more about the ability to synthesize clicks at a location than about the ability to synthesize specific mouse movements.

[jgraham]

I believe the mouse click and key event use cases are mostly covered by testdriver and we should continue to pursue that approach rather than add test-only APIs for things we can reasonably implement in a way that works in release-configuration browsers.

Based ont the feedback, I'll create a scope/workstream proposal for this work (but not immediately due to short term priorities).