-
Notifications
You must be signed in to change notification settings - Fork 8.5k
[Maps] Fix flaky test in maps geo file #241426
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Pinging @elastic/kibana-presentation (Team:Presentation) |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#9693[✅] x-pack/platform/test/functional/apps/maps/group4/config.ts: 150/150 tests passed. |
| const tooltipText = await maps.getLayerTocTooltipMsg(indexName); | ||
| expect(tooltipText).to.be(`${indexName}\nFound ~281 documents. This count is approximate.`); | ||
| expect(tooltipText).to.contain(`${indexName}`); | ||
| expect(tooltipText).to.match(/Found ~\d+ documents/); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should understand why the number is changing before committing this change. If the browser size is the same and the file is the same, then it should be uploaded the same each time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand that the GeoJSON is loaded via MVT tiles, which fetch only the visible tiles.
Can this cause the tooltip count to vary (~280 vs ~281) depending on zoom or viewport size?
If yes, then setting zoom explicitly after openNewMap() could make the test more stable.
However, is it okay to check for an exact value here if we use the word “approximate”, since the number can differ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I noticed that the zoom level differs slightly between runs —When I load this document layer locally, the initial 1.42 zoom becomes 1.51.
However, in the screenshot from the failing test, the zoom is 1.38, which might explain the slight difference in the tooltip count (~280 vs ~281).
Screen.Recording.2025-10-31.at.15.40.15.mov
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for determining the root cause. Lets try having the test set the view to ensure the test is always looking at the same thing between runs. Then we should not need variable validation.
Try setting zoom to '1' and see how stable this makes the tests.
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#9709[✅] x-pack/platform/test/functional/apps/maps/group4/config.ts: 150/150 tests passed. |
💚 Build Succeeded
Metrics [docs]
History
cc @olapawlus |
nreese
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM - Thanks for digging into this one and fixing by stabilizing the flakyness.
code review only
|
Starting backport for target branches: 9.2 |
…proximate document count (elastic#241426) Resolves elastic#234490 The link to Flaky Test Runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709 ### Summary This PR fixes a flaky test in the maps geo file upload suite. The test was asserting an exact document count (Found ~281 documents), but due to MVT tile rendering, the number of documents may vary slightly depending on the zoom level. ### Details In some runs, the count fluctuated between ~280 and ~281, causing the test to fail intermittently. ### Reasoning The failing test screenshot showed a slightly different zoom level compared to the local FTR environment. To stabilize the test, I explicitly set the zoom level at the beginning of the test to ensure consistent MVT data loading. (cherry picked from commit 7943c3e)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
… on approximate document count (#241426) (#241649) # Backport This will backport the following commits from `main` to `9.2`: - [[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)](#241426) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Ola Pawlus","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-11-03T15:17:31Z","message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Presentation","loe:small","technical debt","failed-test","release_note:skip","impact:critical","backport:version","v9.2.0","v9.3.0"],"title":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count","number":241426,"url":"https://github.com/elastic/kibana/pull/241426","mergeCommit":{"message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},"sourceBranch":"main","suggestedTargetBranches":["9.2"],"targetPullRequestStates":[{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/241426","number":241426,"mergeCommit":{"message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}}]}] BACKPORT--> Co-authored-by: Ola Pawlus <[email protected]>
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…proximate document count (elastic#241426) Resolves elastic#234490 The link to Flaky Test Runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709 ### Summary This PR fixes a flaky test in the maps geo file upload suite. The test was asserting an exact document count (Found ~281 documents), but due to MVT tile rendering, the number of documents may vary slightly depending on the zoom level. ### Details In some runs, the count fluctuated between ~280 and ~281, causing the test to fail intermittently. ### Reasoning The failing test screenshot showed a slightly different zoom level compared to the local FTR environment. To stabilize the test, I explicitly set the zoom level at the beginning of the test to ensure consistent MVT data loading. (cherry picked from commit 7943c3e)
… on approximate document count (#241426) (#241780) # Backport This will backport the following commits from `main` to `9.1`: - [[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)](#241426) <!--- Backport version: 10.1.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Ola Pawlus","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-11-03T15:17:31Z","message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Presentation","loe:small","technical debt","failed-test","release_note:skip","impact:critical","backport:version","v9.2.0","v9.3.0","v8.19.7","v9.1.7","v9.2.1"],"title":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count","number":241426,"url":"https://github.com/elastic/kibana/pull/241426","mergeCommit":{"message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.1"],"targetPullRequestStates":[{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/241649","number":241649,"state":"MERGED","mergeCommit":{"sha":"96243bb98eb73b007cfc01fff78c3b8ecf48250e","message":"[9.2] [Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426) (#241649)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.2`:\n- [[Maps] Fix flaky test in maps geo file – remove exact assertion on\napproximate document count\n(#241426)](https://github.com/elastic/kibana/pull/241426)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by: Ola Pawlus <[email protected]>"}},{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/241426","number":241426,"mergeCommit":{"message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},{"branch":"8.19","label":"v8.19.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
…n on approximate document count (#241426) (#241781) # Backport This will backport the following commits from `main` to `8.19`: - [[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)](#241426) <!--- Backport version: 10.1.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Ola Pawlus","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-11-03T15:17:31Z","message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Presentation","loe:small","technical debt","failed-test","release_note:skip","impact:critical","backport:version","v9.2.0","v9.3.0","v8.19.7","v9.1.7","v9.2.1"],"title":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count","number":241426,"url":"https://github.com/elastic/kibana/pull/241426","mergeCommit":{"message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.1"],"targetPullRequestStates":[{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/241649","number":241649,"state":"MERGED","mergeCommit":{"sha":"96243bb98eb73b007cfc01fff78c3b8ecf48250e","message":"[9.2] [Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426) (#241649)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.2`:\n- [[Maps] Fix flaky test in maps geo file – remove exact assertion on\napproximate document count\n(#241426)](https://github.com/elastic/kibana/pull/241426)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by: Ola Pawlus <[email protected]>"}},{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/241426","number":241426,"mergeCommit":{"message":"[Maps] Fix flaky test in maps geo file – remove exact assertion on approximate document count (#241426)\n\nResolves #234490\nThe link to Flaky Test Runner:\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709\n\n### Summary\nThis PR fixes a flaky test in the maps geo file upload suite.\nThe test was asserting an exact document count (Found ~281 documents),\nbut due to MVT tile rendering, the number of documents may vary slightly\ndepending on the zoom level.\n\n### Details\nIn some runs, the count fluctuated between ~280 and ~281, causing the\ntest to fail intermittently.\n\n### Reasoning\nThe failing test screenshot showed a slightly different zoom level\ncompared to the local FTR environment.\nTo stabilize the test, I explicitly set the zoom level at the beginning\nof the test to ensure consistent MVT data loading.","sha":"7943c3e3f3e145a94c985d5aca1a08ee51360610"}},{"branch":"8.19","label":"v8.19.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
Resolves #234490
The link to Flaky Test Runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9709
Summary
This PR fixes a flaky test in the maps geo file upload suite.
The test was asserting an exact document count (Found ~281 documents), but due to MVT tile rendering, the number of documents may vary slightly depending on the zoom level.
Details
In some runs, the count fluctuated between ~280 and ~281, causing the test to fail intermittently.
Reasoning
The failing test screenshot showed a slightly different zoom level compared to the local FTR environment.
To stabilize the test, I explicitly set the zoom level at the beginning of the test to ensure consistent MVT data loading.