Skip to content

Conversation

@olapawlus
Copy link
Member

@olapawlus olapawlus commented Oct 31, 2025

Resolves #234490
The link to Flaky Test Runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709

Summary

This PR fixes a flaky test in the maps geo file upload suite.
The test was asserting an exact document count (Found ~281 documents), but due to MVT tile rendering, the number of documents may vary slightly depending on the zoom level.

Details

In some runs, the count fluctuated between ~280 and ~281, causing the test to fail intermittently.

Reasoning

The failing test screenshot showed a slightly different zoom level compared to the local FTR environment.
To stabilize the test, I explicitly set the zoom level at the beginning of the test to ensure consistent MVT data loading.

@olapawlus olapawlus self-assigned this Oct 31, 2025
@olapawlus olapawlus added Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas t// loe:small Small Level of Effort technical debt Improvement of the software architecture and operational architecture failed-test A test failure on a tracked branch, potentially flaky-test release_note:skip Skip the PR/issue when compiling release notes impact:critical This issue should be addressed immediately due to a critical level of impact on the product. backport:version Backport to applied version labels v9.2.0 v9.3.0 labels Oct 31, 2025
@olapawlus olapawlus marked this pull request as ready for review October 31, 2025 10:10
@olapawlus olapawlus requested a review from a team as a code owner October 31, 2025 10:10
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-presentation (Team:Presentation)

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#9693

[✅] x-pack/platform/test/functional/apps/maps/group4/config.ts: 150/150 tests passed.

see run history

const tooltipText = await maps.getLayerTocTooltipMsg(indexName);
expect(tooltipText).to.be(`${indexName}\nFound ~281 documents. This count is approximate.`);
expect(tooltipText).to.contain(`${indexName}`);
expect(tooltipText).to.match(/Found ~\d+ documents/);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should understand why the number is changing before committing this change. If the browser size is the same and the file is the same, then it should be uploaded the same each time.

Copy link
Member Author

@olapawlus olapawlus Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that the GeoJSON is loaded via MVT tiles, which fetch only the visible tiles.
Can this cause the tooltip count to vary (~280 vs ~281) depending on zoom or viewport size?
If yes, then setting zoom explicitly after openNewMap() could make the test more stable.
However, is it okay to check for an exact value here if we use the word “approximate”, since the number can differ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that the zoom level differs slightly between runs —When I load this document layer locally, the initial 1.42 zoom becomes 1.51.
However, in the screenshot from the failing test, the zoom is 1.38, which might explain the slight difference in the tooltip count (~280 vs ~281).

Screen.Recording.2025-10-31.at.15.40.15.mov

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for determining the root cause. Lets try having the test set the view to ensure the test is always looking at the same thing between runs. Then we should not need variable validation.

Try setting zoom to '1' and see how stable this makes the tests.

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#9709

[✅] x-pack/platform/test/functional/apps/maps/group4/config.ts: 150/150 tests passed.

see run history

@olapawlus olapawlus enabled auto-merge (squash) November 3, 2025 12:19
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

cc @olapawlus

Copy link
Contributor

@nreese nreese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Thanks for digging into this one and fixing by stabilizing the flakyness.
code review only

@olapawlus olapawlus merged commit 7943c3e into elastic:main Nov 3, 2025
13 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 9.2

https://github.com/elastic/kibana/actions/runs/19039516251

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 3, 2025
…proximate document count (elastic#241426)

Resolves elastic#234490
The link to Flaky Test Runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709

### Summary
This PR fixes a flaky test in the maps geo file upload suite.
The test was asserting an exact document count (Found ~281 documents),
but due to MVT tile rendering, the number of documents may vary slightly
depending on the zoom level.

### Details
In some runs, the count fluctuated between ~280 and ~281, causing the
test to fail intermittently.

### Reasoning
The failing test screenshot showed a slightly different zoom level
compared to the local FTR environment.
To stabilize the test, I explicitly set the zoom level at the beginning
of the test to ensure consistent MVT data loading.

(cherry picked from commit 7943c3e)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Nov 3, 2025
… on approximate document count (#241426) (#241649)

# Backport

This will backport the following commits from `main` to `9.2`:
- [[Maps] Fix flaky test in maps geo file – remove exact assertion on
approximate document count
(#241426)](#241426)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Ola
Pawlus","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-11-03T15:17:31Z","message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Presentation","loe:small","technical
debt","failed-test","release_note:skip","impact:critical","backport:version","v9.2.0","v9.3.0"],"title":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document
count","number":241426,"url":"https://github.com/elastic/kibana/pull/241426","mergeCommit":{"message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},"sourceBranch":"main","suggestedTargetBranches":["9.2"],"targetPullRequestStates":[{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/241426","number":241426,"mergeCommit":{"message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}}]}]
BACKPORT-->

Co-authored-by: Ola Pawlus <[email protected]>
@olapawlus
Copy link
Member Author

💚 All backports created successfully

Status Branch Result
9.1
8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

olapawlus added a commit to olapawlus/kibana that referenced this pull request Nov 4, 2025
…proximate document count (elastic#241426)

Resolves elastic#234490
The link to Flaky Test Runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709

### Summary
This PR fixes a flaky test in the maps geo file upload suite.
The test was asserting an exact document count (Found ~281 documents),
but due to MVT tile rendering, the number of documents may vary slightly
depending on the zoom level.

### Details
In some runs, the count fluctuated between ~280 and ~281, causing the
test to fail intermittently.

### Reasoning
The failing test screenshot showed a slightly different zoom level
compared to the local FTR environment.
To stabilize the test, I explicitly set the zoom level at the beginning
of the test to ensure consistent MVT data loading.

(cherry picked from commit 7943c3e)
olapawlus added a commit that referenced this pull request Nov 4, 2025
… on approximate document count (#241426) (#241780)

# Backport

This will backport the following commits from `main` to `9.1`:
- [[Maps] Fix flaky test in maps geo file – remove exact assertion on
approximate document count
(#241426)](#241426)

<!--- Backport version: 10.1.0 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Ola
Pawlus","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-11-03T15:17:31Z","message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Presentation","loe:small","technical
debt","failed-test","release_note:skip","impact:critical","backport:version","v9.2.0","v9.3.0","v8.19.7","v9.1.7","v9.2.1"],"title":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document
count","number":241426,"url":"https://github.com/elastic/kibana/pull/241426","mergeCommit":{"message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.1"],"targetPullRequestStates":[{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/241649","number":241649,"state":"MERGED","mergeCommit":{"sha":"96243bb98eb73b007cfc01fff78c3b8ecf48250e","message":"[9.2]
[Maps] Fix flaky test in maps geo file – remove exact assertion on
approximate document count (#241426) (#241649)\n\n# Backport\n\nThis
will backport the following commits from `main` to `9.2`:\n- [[Maps] Fix
flaky test in maps geo file – remove exact assertion on\napproximate
document
count\n(#241426)](https://github.com/elastic/kibana/pull/241426)\n\n\n\n###
Questions ?\nPlease refer to the [Backport
tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by:
Ola Pawlus
<[email protected]>"}},{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/241426","number":241426,"mergeCommit":{"message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},{"branch":"8.19","label":"v8.19.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
olapawlus added a commit that referenced this pull request Nov 4, 2025
…n on approximate document count (#241426) (#241781)

# Backport

This will backport the following commits from `main` to `8.19`:
- [[Maps] Fix flaky test in maps geo file – remove exact assertion on
approximate document count
(#241426)](#241426)

<!--- Backport version: 10.1.0 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Ola
Pawlus","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-11-03T15:17:31Z","message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Presentation","loe:small","technical
debt","failed-test","release_note:skip","impact:critical","backport:version","v9.2.0","v9.3.0","v8.19.7","v9.1.7","v9.2.1"],"title":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document
count","number":241426,"url":"https://github.com/elastic/kibana/pull/241426","mergeCommit":{"message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.1"],"targetPullRequestStates":[{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/241649","number":241649,"state":"MERGED","mergeCommit":{"sha":"96243bb98eb73b007cfc01fff78c3b8ecf48250e","message":"[9.2]
[Maps] Fix flaky test in maps geo file – remove exact assertion on
approximate document count (#241426) (#241649)\n\n# Backport\n\nThis
will backport the following commits from `main` to `9.2`:\n- [[Maps] Fix
flaky test in maps geo file – remove exact assertion on\napproximate
document
count\n(#241426)](https://github.com/elastic/kibana/pull/241426)\n\n\n\n###
Questions ?\nPlease refer to the [Backport
tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by:
Ola Pawlus
<[email protected]>"}},{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/241426","number":241426,"mergeCommit":{"message":"[Maps]
Fix flaky test in maps geo file – remove exact assertion on approximate
document count (#241426)\n\nResolves #234490\nThe link to Flaky Test
Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n###
Summary\nThis PR fixes a flaky test in the maps geo file upload
suite.\nThe test was asserting an exact document count (Found ~281
documents),\nbut due to MVT tile rendering, the number of documents may
vary slightly\ndepending on the zoom level.\n\n### Details\nIn some
runs, the count fluctuated between ~280 and ~281, causing the\ntest to
fail intermittently.\n\n### Reasoning\nThe failing test screenshot
showed a slightly different zoom level\ncompared to the local FTR
environment.\nTo stabilize the test, I explicitly set the zoom level at
the beginning\nof the test to ensure consistent MVT data
loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},{"branch":"8.19","label":"v8.19.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
@olapawlus olapawlus changed the title [Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count [Maps] Fix flaky test in maps geo file Nov 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:version Backport to applied version labels failed-test A test failure on a tracked branch, potentially flaky-test impact:critical This issue should be addressed immediately due to a critical level of impact on the product. loe:small Small Level of Effort release_note:skip Skip the PR/issue when compiling release notes Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas t// technical debt Improvement of the software architecture and operational architecture v8.19.7 v9.1.7 v9.2.0 v9.2.1 v9.3.0

Projects

None yet

4 participants