[Scout] Test config manifests #245135

dolaru · 2025-12-03T17:11:27Z

Because Scout is based on Playwright, detailed information about tests is only available while Playwright is running. This means that any time we need test-level information from a test config file we have to start a Playwright process which can be quite slow and resource-intensive if you want to do this for a larger number of test configs.

These changes are trying to address that by introducing a system where this runtime-only information is stored in manifest files, in close proximity of test config files, automatically kept up to date by CI.

Usage

Locally, run the new kbn-scout CLI command (via node scripts/scout update-test-config-manifests) to regenerate manifests.

Running without --includingUpToDate will only regenerate manifests that appear outdated (component directory SHA mismatch)
Use --includingUpToDate to force regenerate all manifests

In Buildkite, node scripts/scout update-test-config-manifests runs as a quick check and any changes will be auto-commited.

Notable changes

Add Scout reporting package with a registry of test configs and components
Add a Playwright reporter that writes manifest files that capture test information for static analysis
Add a Scout CLI command update-test-config-manifests to create/update manifests
Add a versatile Playwright CLI wrapper for Scout tooling to call Playwright programmatically
Add Buildkite quick check to run manifest updates and commit any changes as [Scout] Automated config manifest updates
Add new path constants/globs/regex for discovering Scout configs
Changed how test IDs are calculated for Scout: combine test file path and test title when hashing
Remove project name and test file path from Scout test titles

Because Scout is based on Playwright, detailed information about tests is only available while Playwright is running. This means that any time we need test-level information from a test config file we have to start a Playwright process which can be quite slow and resource-intensive if you want to do this for a larger number of tests configurations. These changes are trying to address that by introducing a system where this runtime-only information is stored in manifest files, stored in the proximity of test config files, automatically kept up to date by CI. ### Usage Locally, run the new kbn-scout CLI command (via `node scripts/scout update-test-config-manifests`) to regenerate manifests. - Running without `--includingUpToDate` will only regenerate manifests that appear outdated (component directory SHA mismatch) - Use `--includingUpToDate` to force regenerate all manifests In Buildkite, `node scripts/scout update-test-config-manifests` runs as a quick check and any changes will be auto-commited. ### Notable changes - Add Scout reporting package with a registry of test configs and components - Add a Playwright reporter that writes manifest files that capture test information for static analysis - Add a Scout CLI command `update-test-config-manifests` to create/update manifests - Add a versatile Playwright CLI wrapper for Scout tooling to call Playwright programmatically - Add Buildkite quick check to run manifest updates and commit any changes as `[Scout] Automated config manifest updates` - Add new path constants/globs/regex for discovering Scout configs - Changed how test IDs are calculated for Scout: combine test file path and test title when hashing - Remove project name and test file path from Scout test titles

...ges/private/kbn-scout-reporting/src/reporting/playwright/failed_test/failed_test_reporter.ts

src/platform/packages/private/kbn-scout-reporting/src/helpers/test_id_generator.ts

src/platform/packages/private/kbn-scout-reporting/src/registry/test_config.ts

src/platform/packages/private/kbn-scout-reporting/src/registry/testable_component.ts

.../packages/private/kbn-scout-reporting/src/reporting/playwright/events/playwright_reporter.ts

src/platform/packages/private/kbn-scout-reporting/src/registry/test_config.ts

- add tests for `test_config` module - add missing `lastModified` and `sha1` props to empty manifest object - readability improvements - rename, component -> module

elasticmachine · 2025-12-08T16:24:47Z

💚 Build Succeeded

Buildkite Build
Commit: afcc301

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/scout-info`	15	19	+4
`@kbn/scout-reporting`	111	112	+1
total			+5

Unknown metric groups

API count

id	before	after	diff
`@kbn/scout-info`	15	19	+4
`@kbn/scout-reporting`	130	131	+1
total			+5

ESLint disabled line counts

id	before	after	diff
`@kbn/scout-reporting`	3	4	+1

Total ESLint disabled count

id	before	after	diff
`@kbn/scout-reporting`	3	4	+1

History

💔 Build #368950 failed 0640d9c
💔 Build #368935 failed c9db057
💛 Build #368665 was flaky f796004
💔 Build #368631 failed 6bd6697

cc @dolaru

dmlemeshko

LGTM

tylersmalley · 2025-12-09T18:08:44Z

src/platform/packages/shared/kbn-scout/test/scout/.meta/ui/standard.json

+  "lastModified": "2025-12-03T19:04:20.842Z",
+  "sha1": "cf987cc228a08cc2083d5d705322a88092e4ba5a",


Do we need the lastModified and sha1 properties here? Want to make sure it's something we need as it will likely be a cause of merge conflicts.

dolaru self-assigned this Dec 3, 2025

dolaru requested review from a team as code owners December 3, 2025 17:11

dolaru added release_note:skip Skip the PR/issue when compiling release notes backport:all-open Backport to all branches that could still receive a release labels Dec 3, 2025

elastic-vault-github-plugin-prod bot requested review from a team as code owners December 3, 2025 17:36

elastic-vault-github-plugin-prod bot requested review from a team as code owners December 3, 2025 18:10

dolaru force-pushed the scout-mfs branch from aa22fff to 99cc6b0 Compare December 3, 2025 18:42

kibanamachine and others added 4 commits December 3, 2025 19:04

[Scout] Automated config manifest updates

f796004

Merge branch 'main' into scout-mfs

f5a51fc

fix: test props; title and test end results

c9db057

[Scout] Automated config manifest updates

0640d9c