Skip to content

Integrate Test262 into WPT #229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Integrate Test262 into WPT #229

Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0b4d345
Integrate Test262 into WPT
jcscottiii Nov 4, 2025
b2bd357
updates
jcscottiii Nov 4, 2025
1d618f8
minor edits
jcscottiii Nov 4, 2025
4ac9407
remove fluff
jcscottiii Nov 4, 2025
0e196b0
address feedback
jcscottiii Nov 12, 2025
b19bb88
edits
jcscottiii Nov 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions rfcs/test262_integration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# RFC 229: Integrating Test262

## Summary

This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT. Due to its large size (>50,000 files), tests will be fetched on-demand rather than vendored into the repository. Execution will be opt-in via a new `--test262` flag. This provides a unified way to run this conformance suite while protecting the WPT repository from bloat and performance degradation.
Copy link
Contributor

@jgraham jgraham Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly prefer just vendoring the test262 files directly.

Although that number is large it's about a 50% increase on the current number of files in the repo, so it's not unmanagable.

Doing the vendoring directly means that there's no novel requirements for wpt consumers who want to run these tests (e.g. in vendor repos), apart from selecting the new test type.

I'd also propose a path like third_party/test262 so that it's at least clear that this is not code that lives in wpt. I do think we're likely to run into cases where people are surprised when they make changes to this directory and they aren't upstreamed; at least the third_party convention provides a clue that this might not work.

Copy link
Member

@foolip foolip Nov 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like a shallow clone of Test262 is only 16M in the .git/objects/ directory, so I think that's about how much WPT would grow, which seems OK.


## Motivation

* **Browser-based Testing:** Run Test262 in real browsers (stable and experimental), not just JS shells. This provides a more complete conformance signal for how ECMAScript behaves in the full web platform environment.
* **Unified Infrastructure:** Use the existing WPT infrastructure (`wpt`, CI) and `wpt.fyi` for test execution and results analysis.
* **Operational Efficiency:** Remove the need for vendors to maintain separate infrastructure for running Test262 and correlating its results with WPT.

## Detailed Design

### 1. Source Management

The WPT repository will contain a reference to the official `tc39/test262` repository (e.g., via a `.gitmodules` entry). This reference specifies the expected local path for the Test262 source and a pinned commit SHA.
Copy link
Contributor

@jgraham jgraham Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should try and be specific and not use "e.g." here.

I don't think we should use git submodules, or .gitmodules, but just use our own format e.g. a file like third_party/vendor.toml with a format like:

[test262]
source = <git url>
commit = <sha1>

Copy link
Contributor Author

@jcscottiii jcscottiii Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and changed the language to specify that we would be copying the files into the directory instead. Similar to the wasm/core directory.


* The pinned commit SHA ties each WPT commit to a precise version of Test262, ensuring reproducible test runs.
* An automated process (e.g., a weekly GitHub Action) will create pull requests to update this pinned SHA, providing a regular cadence for ingesting upstream updates.

See Section 3 for details on how the Test262 source is acquired.

### 2. Test Integration

WPT's manifest generation and serving logic will be extended to support the on-demand nature of Test262 tests.

* **Manifest Generation:** The `wpt manifest` command will recognize the Test262 source directory and traverse its `test` directory to discover tests.
* **Metadata Parsing:** A new, Test262-specific parser will read the YAML frontmatter from each `.js` file to extract metadata (e.g., ES features, negative test expectations, execution flags).
Copy link
Contributor

@jgraham jgraham Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear: where will this end up? In MANIFEST.json? Or is the proposal to have a new special manifest format just for test262? Ideally I'd like to avoid that, but the RFC isn't providing enough information to understand the tradeoffs.

Copy link
Contributor Author

@jcscottiii jcscottiii Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same MANIFEST.json.

I did try it out with this PR: web-platform-tests/wpt#55997

The uncompressed manifest.json file is 36M in size now.

* **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use the parsed metadata to construct the correct execution environment and generate distinct URLs for different test modes.

### 3. Acquiring Test262 Source Code

The Test262 source code is acquired on-demand. `wpt` can handle this automatically for users in a Git environment, while providing a manual path for CI systems.
Copy link
Contributor

@jgraham jgraham Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As already mentioned I don't think this is the correct approach.

I think we should have a CI job that fetches the latest commit of test262, copies that over to the destination path and then opens a PR with the update. The CI should run some kind of smoketests to ensure that the update hasn't broken test262 support. Possibly a bot should be trusted to auto-merge these PRs (although that means we have to trust the test262 repository).

Copy link
Contributor Author

@jcscottiii jcscottiii Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to this suggested approach.


When the `--test262` flag is used, `wpt` first checks if the Test262 source directory exists. If missing, `wpt` determines if the main WPT repository is a Git checkout.
- **If it is a Git repository,** `wpt` will attempt to run `git submodule update --init --recursive`.
- **If it is not a Git repository,** `wpt` will fail with an error, instructing the user to acquire the source manually.

This allows CI systems and vendors with non-Git checkouts to use custom tooling, while providing a seamless experience for the common user.

### 4. Execution Control

Running Test262 is **opt-in**.

* **Default Behavior:** By default, `wpt run` will not look for the Test262 directory and will not include its tests.
* **Opt-in Flag:** A new `--test262` flag will be added to `wpt run`.
* **Flag Behavior:** When present, `wpt` ensures the Test262 source code is available (fetching it if possible, per Section 3) and includes the Test262 tests **in addition to** the standard WPT tests.
* **CI Integration:** CI systems can enable Test262 runs by adding this flag, after ensuring the source has been acquired.

## Alternatives Considered

### Using `--test-types=test262`
Copy link
Contributor Author

@jcscottiii jcscottiii Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should move to this instead.

Copy link
Member

@foolip foolip Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do use a separate test type internally (not testharness.js) then I agree, this would be the natural way.

Copy link
Contributor

@jgraham jgraham Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the internal implementation details are the important thing. From the point of view of a consumer test262 tests are just another kind of test we can run, just like crashtests are basically a special kind of reftest, but we expose them as a separate test type because that makes more sense to users.


This was rejected because Test262 is fundamentally different from internal WPT test types (e.g., `testharness`, `reftest`). It is an external suite with its own metadata format and harness. A dedicated `--test262` flag provides a clearer separation of concerns and a cleaner implementation.

## Implementation Considerations & Risks

* **Vendor Integration:** The responsibility for acquiring the Test262 source is left to the user/CI system. This allows vendors with non-Git source control to implement their own sync mechanisms.
* **CI Submodule Behavior:** Some CI systems may check out submodules by default, causing an unintentional, resource-intensive checkout of the large Test262 repository. The impact and configuration options for common CI systems require investigation.
* **CI Performance:** A full Test262 run is lengthy. It is anticipated that full runs will be scheduled on a nightly or weekly basis, not on every pull request.
* **Initial Source Checkout:** The first-time fetch of the Test262 repository will cause a one-time delay.
* **Source Code Visibility on wpt.fyi:** Linking to test source on `wpt.fyi` is not straightforward. A solution would require `wpt.fyi` to have special-case logic for Test262, including access to a manifest mapping test names to file paths and knowledge of the pinned submodule SHA for a given run.

## Proof of Concept

A prototype implementation exists in the following commit:
[https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504](https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504)

*Caveat: The YAML parsing logic in this prototype will be moved to a more Test262-specific location.*