bk(smart exhaustive tests): tune the steps to support GH comments and changesets #18383

v1v · 2025-10-30T15:25:29Z

Release notes

[rn:skip]

What does this PR do?

Validate the smart exhaustive tests pipeline works as expected

Why is it important/What is the impact to the user?

Avoid duplicated runs for changes in files and also a valid github comment.
Tune the configuration to run as requested

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files (and/or docker env variables)
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Important to highlight that the behaviour of the BK PR bot when running for push and comment events differ, hence the need to have two different implementations when triggering the exhaustive pipeline:

In addition, https://github.com/elastic/logstash/pull/18383/files#diff-37e558fe06dbd6c97017aa38d26835c3d4516285ee456dcda11c862715392d71R3 was required, for the same reason, if a GH comment is triggered and the last commit includes changes in the qa/acceptance folder, then it would have been triggered twice. That's an edge case I discovered and fixed with the above-mentioned line.

Screenshots

Logs

github-actions · 2025-10-30T15:25:39Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
/run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

mergify · 2025-10-30T15:26:09Z

This pull request does not have a backport label. Could you fix it @v1v? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

v1v · 2025-10-30T16:53:17Z

/run exhaustive tests

v1v · 2025-10-30T17:10:23Z

/run exhaustive tests

v1v · 2025-10-30T17:24:02Z

/run exhaustive tests

v1v · 2025-10-30T17:26:08Z

/run exhaustive tests

v1v · 2025-10-30T17:30:38Z

/run exhaustive tests

v1v · 2025-10-30T17:31:06Z

Changes caused https://buildkite.com/elastic/logstash-smart-exhaustive-tests-pipeline/builds/10

While github comment caused https://buildkite.com/elastic/logstash-smart-exhaustive-tests-pipeline/builds/11

ci/acceptance_tests.sh

.buildkite/smart_exhaustive_tests_pipeline.yml

v1v · 2025-10-30T17:35:01Z

/run exhaustive tests

Warning: Checkout failed! checking out commit "ecdc1d984499a63372fcb8992c24864c80ab1e5b": exit status 128 (Attempt 3/3)

v1v · 2025-10-30T17:39:27Z

Test

v1v · 2025-10-30T17:40:47Z

/run exhaustive tests

v1v · 2025-10-30T17:46:39Z

/run exhaustive tests

v1v · 2025-10-30T17:50:13Z

/run exhaustive tests

v1v · 2025-10-30T17:57:36Z

/run exhaustive tests

v1v · 2025-10-30T18:02:36Z

.buildkite/smart_exhaustive_tests_pipeline.yml

-      commit: "${BUILDKITE_COMMIT}"
-      branch: "${BUILDKITE_BRANCH}"
+      commit: "HEAD"
+      branch: "pull/${BUILDKITE_PULL_REQUEST}/merge"


reviewer: BK PR bot ehaves awkwardly when supporting GitHub comments, and the previous approach did not work. Please refer to https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/2815

2025-10-30 17:36:23 UTC | fatal: reference is not a tree: ecdc1d984499a63372fcb8992c24864c80ab1e5b -- | -- | 2025-10-30 17:36:23 UTC | ⚠️ Warning: Checkout failed! checking out commit "ecdc1d984499a63372fcb8992c24864c80ab1e5b": exit status 128 (Attempt 3/3) | 2025-10-30 17:36:23 UTC | 🚨 Error: checking out commit "ecdc1d984499a63372fcb8992c24864c80ab1e5b": exit status 128

That is odd. Seems like you found a decent workaround though.

v1v · 2025-10-30T18:03:13Z

.buildkite/smart_exhaustive_tests_pipeline.yml

 steps:
  - label: "Trigger logstash-exhaustive-tests-pipeline for PRs with qa/acceptance/ changes"
-    if: build.pull_request.id != null
+    if: build.pull_request.id != null && build.env("GITHUB_PR_TRIGGER_COMMENT") != "/run exhaustive tests"


in case of two events at the same time, a changeset and also a comment

v1v · 2025-10-30T18:03:18Z

.buildkite/smart_exhaustive_tests_pipeline.yml

-                    - ELASTIC_SLACK_NOTIFICATIONS_ENABLE=false
+                    BUILDKITE_PULL_REQUEST: "${BUILDKITE_PULL_REQUEST}"
+                    BUILDKITE_PULL_REQUEST_BASE_BRANCH: "${BUILDKITE_PULL_REQUEST_BASE_BRANCH}"
+                    ELASTIC_SLACK_NOTIFICATIONS_ENABLED: false


v1v · 2025-10-30T18:03:43Z

.buildkite/smart_exhaustive_tests_pipeline.yml

+                    BUILDKITE_PULL_REQUEST: "${BUILDKITE_PULL_REQUEST}"
+                    BUILDKITE_PULL_REQUEST_BASE_BRANCH: "${BUILDKITE_PULL_REQUEST_BASE_BRANCH}"


reviewer: it requires this format, although, I tried to use the same as we have in other places, like beats and didn't work.

A map of env var name with key and its value as the value makes sense

logstash/.buildkite/windows_jdk_matrix_pipeline.yml

Lines 4 to 5 in be64ebf

DEFAULT_MATRIX_OS: "windows-2022"

DEFAULT_MATRIX_JDK: "adoptiumjdk_21"

over a list of string. Good with this.

v1v · 2025-10-30T18:03:52Z

.buildkite/smart_exhaustive_tests_pipeline.yml

-        - GITHUB_PR_LABELS=${GITHUB_PR_LABELS}
-        - ELASTIC_SLACK_NOTIFICATIONS_ENABLE=false
+        BUILDKITE_PULL_REQUEST: "${BUILDKITE_PULL_REQUEST}"
+        ELASTIC_SLACK_NOTIFICATIONS_ENABLED: false


v1v · 2025-10-30T18:05:36Z

GH check failed because I cancelled a run related to the changeset by running a GitHub comment immediately after, https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/2820 is running, but the GH check is not updated for some reason, maybe a race condition with the BK PR bot - where cancelled is happening after a new run instead of being sequential :/

elasticmachine · 2025-10-30T18:35:34Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 0057a8e

Failed CI Steps

cc @v1v

donoghuc

Looks like you've got this working

fr4nc1sc0-r4m0n

LGTM

… changesets (elastic#18383)

v1v added 2 commits October 30, 2025 16:24

chore: test

e414c51

chore: test

5d5ccab

v1v added 4 commits October 30, 2025 17:57

fix

748564e

fix

4ad6fed

revert

93ddef7

fix

f81e169

finer grained

8847b3f

fix

8ad4433

v1v commented Oct 30, 2025

View reviewed changes

ci/acceptance_tests.sh Outdated Show resolved Hide resolved

v1v commented Oct 30, 2025

View reviewed changes

.buildkite/smart_exhaustive_tests_pipeline.yml Outdated Show resolved Hide resolved

v1v and others added 2 commits October 30, 2025 18:33

revert

de9741f

Apply suggestion from @v1v

ecdc1d9

fix: issues when running on GH Comments

9df5ac8

Warning: Checkout failed! checking out commit "ecdc1d984499a63372fcb8992c24864c80ab1e5b": exit status 128 (Attempt 3/3)

v1v changed the title ~~chore: test~~ bk(smart exhaustive tests): configure the steps to run as requested Oct 30, 2025

fix

2240d82

fix

01b4d7a

revert

ffd0899

fix

0057a8e

v1v commented Oct 30, 2025

View reviewed changes

v1v changed the title ~~bk(smart exhaustive tests): configure the steps to run as requested~~ bk(smart exhaustive tests): tune the steps to support GH comments and changesets Oct 30, 2025

v1v self-assigned this Oct 30, 2025

v1v added the backport-skip Skip automated backport with mergify label Oct 30, 2025

v1v marked this pull request as ready for review October 30, 2025 18:06

v1v requested a review from a team October 30, 2025 18:07

donoghuc approved these changes Oct 30, 2025

View reviewed changes

fr4nc1sc0-r4m0n approved these changes Oct 31, 2025

View reviewed changes

v1v merged commit 02aa5e1 into elastic:main Oct 31, 2025
13 checks passed

This was referenced Oct 31, 2025

chore: remove test file for exhaustive tests and normalise github commands #18384

Merged

feat(github-commands): support logstash specifics elastic/oblt-actions#393

Closed

feat(github-commands): support logstash specifics elastic/oblt-actions#394

Merged

donoghuc pushed a commit to donoghuc/logstash that referenced this pull request Oct 31, 2025

bk(smart exhaustive tests): tune the steps to support GH comments and…

426fce4

… changesets (elastic#18383)

donoghuc mentioned this pull request Oct 31, 2025

Backport smart exhaustive tests #18386

Open

		BUILDKITE_PULL_REQUEST: "${BUILDKITE_PULL_REQUEST}"
		BUILDKITE_PULL_REQUEST_BASE_BRANCH: "${BUILDKITE_PULL_REQUEST_BASE_BRANCH}"

	DEFAULT_MATRIX_OS: "windows-2022"
	DEFAULT_MATRIX_JDK: "adoptiumjdk_21"

bk(smart exhaustive tests): tune the steps to support GH comments and changesets #18383

bk(smart exhaustive tests): tune the steps to support GH comments and changesets #18383

Conversation

v1v commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

github-actions bot commented Oct 30, 2025 • edited by v1v Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 GitHub comments

Uh oh!

mergify bot commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

Uh oh!

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v commented Oct 30, 2025

Uh oh!

v1v Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

v1v commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Oct 30, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Uh oh!

donoghuc left a comment

Choose a reason for hiding this comment

v1v commented Oct 30, 2025 •

edited

Loading

github-actions bot commented Oct 30, 2025 •

edited by v1v

Loading

v1v commented Oct 30, 2025 •

edited

Loading

v1v commented Oct 30, 2025 •

edited

Loading