Skip to content

Commit 2f42553

Browse files
authored
ci: use a container and HEADLESS=false for running benchmarks (#37725)
## **Description** Previously, benchmarks would crash if you ran them too many times. The solution here is to use a container and run them `HEADLESS=false`, just like the E2E tests run. Also undo the `pingTimeout` change. The container did not have the AWS CLI installed, so we had to install it. TODO is pre-install this in the container. ## **Changelog** CHANGELOG entry: null ## **Related issues** Fixes: #36935 <!--## **Manual testing steps** ## **Screenshots/Recordings** ## **Pre-merge author checklist** ## **Pre-merge reviewer checklist**--> <!-- CURSOR_SUMMARY --> --- > [!NOTE] > Runs benchmarks in a container with HEADLESS=false and removes the `pingTimeout` option from Snaps execution service (and its tests). > > - **CI / Benchmarks (`.github/workflows/run-benchmarks.yml`)** > - Run in container `ghcr.io/metamask/metamask-extension-e2e-image:v24.11.0` with credentials. > - Set `HEADLESS=false`; configure Xvfb for display. > - Install AWS CLI inside container. > - Increase `powerUserHome` loads: `--browserLoads 10 --pageLoads 10`. > - **Snaps Execution Service** > - Remove `pingTimeout` from `OffscreenExecutionService` and `IframeExecutionService` initialization in `app/scripts/controller-init/snaps/execution-service-init.ts`. > - Update test `execution-service-init.test.ts` to drop `pingTimeout` expectation. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit f3d9eeb. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent 6f70c0f commit 2f42553

File tree

3 files changed

+14
-5
lines changed

3 files changed

+14
-5
lines changed

.github/workflows/run-benchmarks.yml

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ env:
1313
COMMANDS: |
1414
{
1515
standardHome: 'yarn tsx test/e2e/benchmarks/benchmark.ts --out test-artifacts/benchmarks/benchmark-{0}-{1}-standardHome.json --retries 2',
16-
powerUserHome: 'yarn tsx test/e2e/benchmarks/benchmark.ts --persona powerUser --browserLoads 4 --pageLoads 4 --out test-artifacts/benchmarks/benchmark-{0}-{1}-powerUserHome.json --retries 2',
16+
powerUserHome: 'yarn tsx test/e2e/benchmarks/benchmark.ts --persona powerUser --browserLoads 10 --pageLoads 10 --out test-artifacts/benchmarks/benchmark-{0}-{1}-powerUserHome.json --retries 2',
1717
userActions: 'yarn tsx test/e2e/benchmarks/user-actions-benchmark.ts --out test-artifacts/benchmarks/benchmark-{0}-{1}-userActions.json --retries 2',
1818
}
1919
@@ -22,6 +22,11 @@ jobs:
2222
if: ${{ github.event_name != 'merge_group' }} # Skip this job for the Merge Queue
2323
runs-on: ubuntu-22.04
2424
timeout-minutes: 30
25+
container:
26+
image: ghcr.io/metamask/metamask-extension-e2e-image:v24.11.0
27+
credentials:
28+
username: ${{ github.actor }}
29+
password: ${{ secrets.GITHUB_TOKEN }}
2530
strategy:
2631
fail-fast: false
2732
matrix:
@@ -31,7 +36,7 @@ jobs:
3136
name: ${{ matrix.browser }}-${{ matrix.buildType }}-${{ matrix.pageType }}
3237
env:
3338
SELENIUM_BROWSER: ${{ matrix.browser }}
34-
HEADLESS: true
39+
HEADLESS: false
3540
ARTIFACT_NAME: build-test-${{ matrix.buildType }}
3641
OUTPUT_NAME: benchmark-${{ matrix.browser }}-${{ matrix.buildType }}-${{ matrix.pageType }}.json
3742
INFURA_PROJECT_ID: ${{ secrets.INFURA_PROJECT_ID }}
@@ -63,6 +68,13 @@ jobs:
6368
github-token: ${{ secrets.GITHUB_TOKEN }} # This is required when downloading artifacts from a different repository or from a different workflow run.
6469
run-id: ${{ inputs.builds-from-run }} # Download from whatever run the identify-builds job said.
6570

71+
- name: Configure Xvfb
72+
run: Xvfb -ac :99 -screen 0 1280x1024x16 &
73+
74+
# TODO: Pre-install this in the container image
75+
- name: Install AWS CLI (needed in the container)
76+
run: sudo apt-get install python3-pip && sudo pip install awscli --break-system-packages
77+
6678
- name: Run the benchmark
6779
# Choose a benchmark command from env.COMMANDS
6880
# Then replace the {0} placeholder with the browser, and the {1} placeholder with the buildType

app/scripts/controller-init/snaps/execution-service-init.test.ts

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,6 @@ describe('ExecutionServiceInit', () => {
6969
messenger: expect.any(Object),
7070
offscreenPromise: expect.any(Promise),
7171
setupSnapProvider: expect.any(Function),
72-
pingTimeout: process.env.IN_TEST ? 60000 : 5000, // see https://github.com/MetaMask/metamask-extension/issues/36935
7372
});
7473
});
7574
});

app/scripts/controller-init/snaps/execution-service-init.ts

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,6 @@ export const ExecutionServiceInit: ControllerInitFunction<
5656
messenger: controllerMessenger,
5757
setupSnapProvider,
5858
offscreenPromise,
59-
pingTimeout: process.env.IN_TEST ? 60000 : 5000, // see https://github.com/MetaMask/metamask-extension/issues/36935
6059
}),
6160
};
6261
}
@@ -71,7 +70,6 @@ export const ExecutionServiceInit: ControllerInitFunction<
7170
messenger: controllerMessenger,
7271
iframeUrl: new URL(iframeUrl),
7372
setupSnapProvider,
74-
pingTimeout: process.env.IN_TEST ? 60000 : 5000, // see https://github.com/MetaMask/metamask-extension/issues/36935
7573
}),
7674
};
7775
};

0 commit comments

Comments
 (0)