Proposal: Restructure test files to enable per-test API results

[lolaodelola]

As part of work happening in Accessibility Compat Data, we are creating a dataset to highlight the availability of web features specifically for accessibility interoperability. After discussing with the WPT Accessibility sub-group, it was suggested that we may be able to use the current WPT tests for the browser portion of the support data; however, the current wpt.fyi API doesn't provide enough granularity for the results we need.

The wpt.fyi API returns results at the file level. When a file contains multiple tests, there's no way to query results for a specific test within that file. I'd like to propose that we split multi-test files into individual files based on the HTML element being tested. For example, in the accname tests, the current structure is: accname/name/comp_label.html
I'm proposing: accname/name/comp_label/button.html, accname/name/comp_label/input.html, accname/name/comp_label/a.html, etc.

This would enable precise per-test result querying and make it easier to track specific test outcomes across browser versions.

I'd love to know if there are there concerns with this approach, or if there are alternative ways to expose individual test results through the API? @cookiecrook mentioned that there may have been previous discussions about exposing specific tests to the API if I remember correctly?

[hitdepani]

I WOULD LIKE TO WORK ON THIS

[gsnedders]

I suspect the right approach is to just download the raw_results_url of what the runs API returns and process the WPT Report JSON file, rather than making one request per browser per test file, which given even 3 browsers and the 155 accessibility-labelled tests is 465 requests.

The dashboard certainly does fetch subtest results without downloading the full JSON data; https://github.com/web-platform-tests/wpt.fyi/blob/9c624d869f2e5a04a8895d2a1117c2c4a51e2903/webapp/components/test-file-results.js#L271-L282 seems to be its (kinda hideous) logic. Adding an actual API here is probably sensible; I've filed web-platform-tests/wpt.fyi#4606 for that.

(And as an aside: for comp_label.html splitting by HTML element this seems like the probably-wrong approach, especially looking at the current test with its h2 elements — and the motivation here is very much about querying the data, nothing else.)

[lolaodelola]

Thanks @gsnedders, being able to get the subtest data is the most important thing for us at the moment. Right now I'm running a search filtered by a label e.g. "a11y" (I'm going to add our own metadata soon) and then getting the relevant tests that way.

In wpt-metadata, there are some examples of subtests being specified, but the response includes all the tests, not only the specified subtests.

Something I investigated today was adding a series of metadata labels e.g. "acd-button" in the relevant folders and specify the tests and subtests for each thing, if I were able to do that then for instances like comp_lablel.html I can specify which subtests I care about for the associated label.

In an ideal scenario, we'd introduce a new metadata key web-feature to match the web features defined for BCD and Baseline & define the tests and subtests under that key, query the API and get the relevant results.

[gsnedders]

Can you actually step back and explain what you're trying to do, without getting into detail of potential solutions?

If I understand correctly, there's three things you're trying to do:

1. Map subtests to Web Features.
2. Get the results from the wpt.fyi API for those subtests.
3. Have wpt.fyi be able to show only those subtests.

---

To try and address those three:

1. There is already have a means to map tests to Web Features (via WEB_FEATURES.yml files) — though I realise we've ended up with entirely too many metadata files — but there indeed isn't any way to do this at a subtest level.
2. Most of the current tooling (e.g., for Interop) works via the full JSON report files, and does its own filtering based on the metadata; if you're working with a large number of tests and subtests, this is almost certainly the right approach, as otherwise you'll be making large numbers of requests to the server to fetch all the relevant data.
3. wpt.fyi's query syntax does sorta have support for this, see for example, path:/dom/historical.html subtest:"Historical DOM features must be removed: DOMError", though clearly this just matches substrings.

(1) kinda suggests we do need to loop back around to the initial suggesting of splitting the tests; this could also be done as variants rather than actually splitting them into separate files, but what makes sense there is likely a question for those who know those tests better.

[lolaodelola]

If I understand correctly, there's three things you're trying to do:

1. Map subtests to Web Features.
2. Get the results from the wpt.fyi API for those subtests.
3. Have wpt.fyi be able to show only those subtests.

Yes to 1 & 2. Unless there's some kind of constraint that means the API and dashboard need the same capabilities, for this project we don't need wpt.fyi to have to show the subtests.

Most of the current tooling (e.g., for Interop) works via the full JSON report files, and does its own filtering based on the metadata; if you're working with a large number of tests and subtests, this is almost certainly the right approach, as otherwise you'll be making large numbers of requests to the server to fetch all the relevant data.

This makes sense but when I look at an example of the JSON report files, I can't see where metadata is included to do the filtering ourselves without having to make many calls, do you have an example report that has the metadata included in it?

(1) kinda suggests we do need to loop back around to the initial suggesting of splitting the tests; this could also be done as variants rather than actually splitting them into separate files, but what makes sense there is likely a question for those who know those tests better.

Variants look interesting, I can ask in Matrix for folks with more experience there.