Use logstash-versions.yml for observabilitySRE stack versions #18375

donoghuc · 2025-10-27T22:37:59Z

Previously the version of logstash checked in to the logstash repo was used to determine stack version for smoke and acceptance testing of the observabilitySRE artifact. Given the ES/Filebeat containers are subject to a unified release the delay between a logstash version bump and those artifacts being available was causing failures when the smoke/acceptance tests were run. In plugin testing the way we deal with this is to ensure the logstash-versions.yml file is updated when stack components are ready. This commit introduces the use of that pattern to the observabilitySRE artifact testing. The important step will be to set the STACK_VERSION to correspond to the desired stack version. For example the 9.2 logstash branch should look up 9.current.

closes #18358

github-actions · 2025-10-27T22:38:10Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-10-27T22:38:37Z

This pull request does not have a backport label. Could you fix it @donoghuc? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

donoghuc · 2025-10-27T22:53:39Z

exhaustive tests: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/2798

mashhurs

LGTM!

mashhurs · 2025-10-28T03:42:57Z

ci/observabilitySREsmoke_tests.sh

-export FILEBEAT_IMAGE_VERSION="${FILEBEAT_IMAGE_VERSION:-$QUALIFIED_VERSION}"
+# Look up corresponding LOSTASH_STREAM to test against. MAKE SURE TO UDPATE this per branch for example the 9.2 branch in logstash would
+# correspond to the '9.current' key in the logstash-versions.yml file
+LOGSTASH_STREAM=main


With this initial step no concern and for the long term it would be great if we can dynamically resolve artifact version (eg: if branch 9.1, consider values of 9.prev or 9.current if matches 9.1)

I think ultimately this is going to be a "moving target". IE just recently we had to update 9.1 -> 9.previous 9.2 -> 9.current so computing from branch name wont be possible without an external input. The other thought I had was to centralize this to look up programatically. Perhaps a new section in logstash-versions.yml:

logstash-streams: "8.19": "8.previous" "9.1": "9.previous" "9.2": "9.current" "main": "9.next"

computing from branch name wont be possible without an external input

Agree! In the past I had a python script in elastic_integration plugin to check if stream value/artifact-version contains the branch we are running from, if it works for you. Then we moved to matrix because for the elastic_integration case it can be 1:N mapping.

Thinking about this more, i actually think we want to track the stream information in the logstash repo (this would only require a single update when a new branch is created I added a commit 1985c4b showing an option for that. The main reason is that we should be able to run the scripts for executing the tests from any branch, there is not really a clean way to determine what the intended target stream is for the current branch. For example, if i raise a PR from a branch called patch-n there is not really any information to reliable determine what target branch/stream of logstash that should be targeted at.

donoghuc · 2025-10-30T00:12:49Z

versions.yml

 logstash-core: 9.3.0
 logstash-core-plugin-api: 2.1.16
+# Logstash stream corresponding to keys in https://github.com/logstash-plugins/.ci/blob/1.x/logstash-versions.yml
+logstash-stream: main


If this is approved/agreen upon I will update https://github.com/elastic/logstash/blob/main/.github/workflows/bump-logstash.yml to manage this along with logstash version.

mashhurs

LGTM!

Previously the version of logstash checked in to the logstash repo was used to determine stack version for smoke and acceptance testing of the observabilitySRE artifact. Given the ES/Filebeat containers are subject to a unified release the delay between a logstash version bump and those artifacts being available was causing failures when the smoke/acceptance tests were run. In plugin testing the way we deal with this is to ensure the logstash-versions.yml file is updated when stack components are ready. This commit introduces the use of that pattern to the observabilitySRE artifact testing. The important step will be to set the STACK_VERSION to correspond to the desired stack version. For example the 9.2 logstash branch should look up `9.current`.

donoghuc · 2025-11-04T00:08:45Z

@jsvd You good with this approach/naming convention?

jsvd

Minor nitpick: please fix the commends that still mention "stream" instead of release track.

Otherwise LGTM

elasticmachine · 2025-11-04T18:53:24Z

💚 Build Succeeded

Buildkite Build
Commit: 9ec398c

History

💚 Build #3807 succeeded 3001241
💛 Build #3780 was flaky 1985c4b
💚 Build #3768 succeeded 1baf415

* Use logstash-versions.yml for observabilitySRE stack versions Previously the version of logstash checked in to the logstash repo was used to determine stack version for smoke and acceptance testing of the observabilitySRE artifact. Given the ES/Filebeat containers are subject to a unified release the delay between a logstash version bump and those artifacts being available was causing failures when the smoke/acceptance tests were run. In plugin testing the way we deal with this is to ensure the logstash-versions.yml file is updated when stack components are ready. This commit introduces the use of that pattern to the observabilitySRE artifact testing. The important step will be to set the STACK_VERSION to correspond to the desired stack version. For example the 9.2 logstash branch should look up `9.current`. * track stream in versions.yml * Finalize name/location for versions.yml mapping * better comment (cherry picked from commit d9fda37)

* Use logstash-versions.yml for observabilitySRE stack versions Previously the version of logstash checked in to the logstash repo was used to determine stack version for smoke and acceptance testing of the observabilitySRE artifact. Given the ES/Filebeat containers are subject to a unified release the delay between a logstash version bump and those artifacts being available was causing failures when the smoke/acceptance tests were run. In plugin testing the way we deal with this is to ensure the logstash-versions.yml file is updated when stack components are ready. This commit introduces the use of that pattern to the observabilitySRE artifact testing. The important step will be to set the STACK_VERSION to correspond to the desired stack version. For example the 9.2 logstash branch should look up `9.current`. * track stream in versions.yml * Finalize name/location for versions.yml mapping * better comment (cherry picked from commit d9fda37) # Conflicts: # ci/observabilitySREacceptance_tests.sh # ci/observabilitySREsmoke_tests.sh

donoghuc · 2025-11-04T19:48:40Z

@Mergifyio backport 9.2

mergify · 2025-11-04T19:48:51Z

backport 9.2

✅ Backports have been created

#18402 [9.2] (backport #18375) Use logstash-versions.yml for observabilitySRE stack versions has been created for branch 9.2

* Use logstash-versions.yml for observabilitySRE stack versions Previously the version of logstash checked in to the logstash repo was used to determine stack version for smoke and acceptance testing of the observabilitySRE artifact. Given the ES/Filebeat containers are subject to a unified release the delay between a logstash version bump and those artifacts being available was causing failures when the smoke/acceptance tests were run. In plugin testing the way we deal with this is to ensure the logstash-versions.yml file is updated when stack components are ready. This commit introduces the use of that pattern to the observabilitySRE artifact testing. The important step will be to set the STACK_VERSION to correspond to the desired stack version. For example the 9.2 logstash branch should look up `9.current`. * track stream in versions.yml * Finalize name/location for versions.yml mapping * better comment (cherry picked from commit d9fda37)

…E stack versions (#18401) * Use logstash-versions.yml for observabilitySRE stack versions (#18375) * Use logstash-versions.yml for observabilitySRE stack versions Previously the version of logstash checked in to the logstash repo was used to determine stack version for smoke and acceptance testing of the observabilitySRE artifact. Given the ES/Filebeat containers are subject to a unified release the delay between a logstash version bump and those artifacts being available was causing failures when the smoke/acceptance tests were run. In plugin testing the way we deal with this is to ensure the logstash-versions.yml file is updated when stack components are ready. This commit introduces the use of that pattern to the observabilitySRE artifact testing. The important step will be to set the STACK_VERSION to correspond to the desired stack version. For example the 9.2 logstash branch should look up `9.current`. * track stream in versions.yml * Finalize name/location for versions.yml mapping * better comment (cherry picked from commit d9fda37) # Conflicts: # ci/observabilitySREacceptance_tests.sh # ci/observabilitySREsmoke_tests.sh * Update versions.yml --------- Co-authored-by: Cas Donoghue <[email protected]>

…RE stack versions (#18400) * Use logstash-versions.yml for observabilitySRE stack versions (#18375) * Use logstash-versions.yml for observabilitySRE stack versions Previously the version of logstash checked in to the logstash repo was used to determine stack version for smoke and acceptance testing of the observabilitySRE artifact. Given the ES/Filebeat containers are subject to a unified release the delay between a logstash version bump and those artifacts being available was causing failures when the smoke/acceptance tests were run. In plugin testing the way we deal with this is to ensure the logstash-versions.yml file is updated when stack components are ready. This commit introduces the use of that pattern to the observabilitySRE artifact testing. The important step will be to set the STACK_VERSION to correspond to the desired stack version. For example the 9.2 logstash branch should look up `9.current`. * track stream in versions.yml * Finalize name/location for versions.yml mapping * better comment (cherry picked from commit d9fda37) * Update versions.yml --------- Co-authored-by: Cas Donoghue <[email protected]>

…E stack versions (#18402) * Use logstash-versions.yml for observabilitySRE stack versions (#18375) * Use logstash-versions.yml for observabilitySRE stack versions Previously the version of logstash checked in to the logstash repo was used to determine stack version for smoke and acceptance testing of the observabilitySRE artifact. Given the ES/Filebeat containers are subject to a unified release the delay between a logstash version bump and those artifacts being available was causing failures when the smoke/acceptance tests were run. In plugin testing the way we deal with this is to ensure the logstash-versions.yml file is updated when stack components are ready. This commit introduces the use of that pattern to the observabilitySRE artifact testing. The important step will be to set the STACK_VERSION to correspond to the desired stack version. For example the 9.2 logstash branch should look up `9.current`. * track stream in versions.yml * Finalize name/location for versions.yml mapping * better comment (cherry picked from commit d9fda37) * Update versions.yml --------- Co-authored-by: Cas Donoghue <[email protected]>

donoghuc added backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch backport-9.2 Automated backport to the 9.2 branch labels Oct 27, 2025

donoghuc mentioned this pull request Oct 27, 2025

Use logstash-versions in .ci as source of truth for ES dependent fips tests #18358

Closed

mashhurs approved these changes Oct 28, 2025

View reviewed changes

donoghuc mentioned this pull request Oct 29, 2025

Map logstash branches to streams logstash-plugins/.ci#93

Closed

donoghuc requested a review from mashhurs October 29, 2025 23:39

donoghuc commented Oct 30, 2025

View reviewed changes

mashhurs approved these changes Oct 30, 2025

View reviewed changes

donoghuc added 3 commits October 31, 2025 12:01

track stream in versions.yml

fda69b9

Finalize name/location for versions.yml mapping

3001241

donoghuc force-pushed the use-logstash-versions-as-source-of-truth-fips branch from 1985c4b to 3001241 Compare October 31, 2025 19:15

donoghuc requested review from a team as code owners October 31, 2025 19:15

donoghuc requested a review from jsvd November 4, 2025 00:08

jsvd approved these changes Nov 4, 2025

View reviewed changes

better comment

9ec398c

donoghuc merged commit d9fda37 into elastic:main Nov 4, 2025
11 checks passed

mergify bot mentioned this pull request Nov 4, 2025

[8.19] (backport #18375) Use logstash-versions.yml for observabilitySRE stack versions #18400

Merged

mergify bot mentioned this pull request Nov 4, 2025

[9.1] (backport #18375) Use logstash-versions.yml for observabilitySRE stack versions #18401

Merged

mergify bot mentioned this pull request Nov 4, 2025

[9.2] (backport #18375) Use logstash-versions.yml for observabilitySRE stack versions #18402

Merged

Use logstash-versions.yml for observabilitySRE stack versions #18375

Use logstash-versions.yml for observabilitySRE stack versions #18375

Conversation

donoghuc commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 27, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Oct 27, 2025

Uh oh!

donoghuc commented Oct 27, 2025

Uh oh!

mashhurs left a comment

Choose a reason for hiding this comment

Uh oh!

mashhurs Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

mashhurs Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

mashhurs left a comment

Choose a reason for hiding this comment

Uh oh!

donoghuc commented Nov 4, 2025

Uh oh!

jsvd left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Nov 4, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

donoghuc commented Nov 4, 2025

Uh oh!

mergify bot commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

donoghuc commented Oct 27, 2025 •

edited

Loading

mergify bot commented Nov 4, 2025 •

edited

Loading