Skip to content

feat: Preserve event.original when pipeline errors in integration-experience integrations (part 3) #15805

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 6, 2025
Merged

feat: Preserve event.original when pipeline errors in integration-experience integrations (part 3) #15805

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c1a52e3
feat: Preserve event.original when errors occur in pipelines
taylor-swanson Oct 29, 2025
95795c3
Merge branch 'main' into chore/event-original-part3
taylor-swanson Nov 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/imperva/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.9.0"
changes:
- description: Preserve event.original on pipeline error.
type: enhancement
link: https://github.com/elastic/integrations/pull/15805
- version: "1.8.2"
changes:
- description: Generate processor tags and normalize error handler.
Expand Down
10 changes: 10 additions & 0 deletions packages/imperva/data_stream/securesphere/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,12 @@ processors:
tag: set_pipeline_error_to_event_kind
value: pipeline_error
if: ctx.error?.message != null
- append:
tag: append_preserve_original_event_on_error
field: tags
value: preserve_original_event
allow_duplicates: false
if: ctx.error?.message != null
on_failure:
- append:
field: error.message
Expand All @@ -347,3 +353,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
2 changes: 1 addition & 1 deletion packages/imperva/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.3
name: imperva
title: Imperva
version: "1.8.2"
version: "1.9.0"
description: Collect logs from Imperva devices with Elastic Agent.
categories: ["network", "security"]
type: integration
Expand Down
5 changes: 5 additions & 0 deletions packages/iptables/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.22.0"
changes:
- description: Preserve event.original on pipeline error.
type: enhancement
link: https://github.com/elastic/integrations/pull/15805
- version: "1.21.4"
changes:
- description: Generate processor tags and normalize error handler.
Expand Down
10 changes: 10 additions & 0 deletions packages/iptables/data_stream/log/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,6 +415,12 @@ processors:
field:
- _tmp
ignore_failure: true
- append:
tag: append_preserve_original_event_on_error
field: tags
value: preserve_original_event
allow_duplicates: false
if: ctx.error?.message != null
on_failure:
- remove:
field:
Expand All @@ -430,3 +436,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
2 changes: 1 addition & 1 deletion packages/iptables/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: iptables
title: Iptables
version: "1.21.4"
version: "1.22.0"
description: Collect logs from Iptables with Elastic Agent.
type: integration
icons:
Expand Down
5 changes: 5 additions & 0 deletions packages/juniper_srx/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.26.0"
changes:
- description: Preserve event.original on pipeline error.
type: enhancement
link: https://github.com/elastic/integrations/pull/15805
- version: "1.25.2"
changes:
- description: Generate processor tags and normalize error handler.
Expand Down
4 changes: 4 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/atp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -427,3 +427,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
10 changes: 10 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,12 @@ processors:
- juniper.srx.dstzone
- syslog_pri
ignore_missing: true
- append:
tag: append_preserve_original_event_on_error
field: tags
value: preserve_original_event
allow_duplicates: false
if: ctx.error?.message != null
on_failure:
- append:
field: error.message
Expand All @@ -445,3 +451,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/flow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -427,3 +427,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/idp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,3 +338,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/ids.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,3 +429,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/secintel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,3 +411,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/system.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -941,3 +941,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/utm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,3 +458,7 @@ on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
2 changes: 1 addition & 1 deletion packages/juniper_srx/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.0.3"
name: juniper_srx
title: Juniper SRX
version: "1.25.2"
version: "1.26.0"
description: Collect logs from Juniper SRX devices with Elastic Agent.
categories: ["network", "security", "firewall_security"]
type: integration
Expand Down
5 changes: 5 additions & 0 deletions packages/modsecurity/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.22.0"
changes:
- description: Preserve event.original on pipeline error.
type: enhancement
link: https://github.com/elastic/integrations/pull/15805
- version: "1.21.2"
changes:
- description: Generate processor tags and normalize error handler.
Expand Down
4 changes: 4 additions & 0 deletions packages/modsecurity/data_stream/auditlog/elasticsearch/ingest_pipeline/apache-modsec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,3 +300,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
10 changes: 10 additions & 0 deletions packages/modsecurity/data_stream/auditlog/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,12 @@ processors:
tag: pipeline_7206172d
name: '{{ IngestPipeline "apache-modsec" }}'
if: (ctx.modsec?.audit?.server != null && ctx.modsec.audit.server.toLowerCase().contains('apache')) || (ctx.modsec?.audit?.connector != null && ctx.modsec.audit.connector.toLowerCase().contains('apache'))
- append:
tag: append_preserve_original_event_on_error
field: tags
value: preserve_original_event
allow_duplicates: false
if: ctx.error?.message != null
on_failure:
- set:
field: event.kind
Expand All @@ -52,3 +58,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/modsecurity/data_stream/auditlog/elasticsearch/ingest_pipeline/nginx-modsec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,3 +210,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
2 changes: 1 addition & 1 deletion packages/modsecurity/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.0.3"
name: modsecurity
title: "ModSecurity Audit"
version: "1.21.2"
version: "1.22.0"
description: Collect logs from ModSecurity with Elastic Agent
type: integration
categories:
Expand Down
5 changes: 5 additions & 0 deletions packages/netflow/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.24.0"
changes:
- description: Preserve event.original on pipeline error.
type: enhancement
link: https://github.com/elastic/integrations/pull/15805
- version: "2.23.2"
changes:
- description: Generate processor tags and normalize error handler.
Expand Down
10 changes: 10 additions & 0 deletions packages/netflow/data_stream/log/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,12 @@ processors:
field:
- _tmp_
ignore_missing: true
- append:
tag: append_preserve_original_event_on_error
field: tags
value: preserve_original_event
allow_duplicates: false
if: ctx.error?.message != null
on_failure:
- set:
field: event.kind
Expand All @@ -199,3 +205,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
2 changes: 1 addition & 1 deletion packages/netflow/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.0.3"
name: netflow
title: NetFlow Records
version: "2.23.2"
version: "2.24.0"
description: Collect flow records from NetFlow and IPFIX exporters with Elastic Agent.
type: integration
categories:
Expand Down
5 changes: 5 additions & 0 deletions packages/panw/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "5.4.0"
changes:
- description: Preserve event.original on pipeline error.
type: enhancement
link: https://github.com/elastic/integrations/pull/15805
- version: "5.3.6"
changes:
- description: Prefer set with copy_from and ignore_empty_value.
Expand Down
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/audit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/authentication.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,3 +116,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,3 +158,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/correlated_event.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/decryption.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,3 +386,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
10 changes: 10 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2057,6 +2057,12 @@ processors:
return false;
}
dropEmptyFields(ctx);
- append:
tag: append_preserve_original_event_on_error
field: tags
value: preserve_original_event
allow_duplicates: false
if: ctx.error?.message != null

on_failure:
- set:
Expand All @@ -2075,3 +2081,7 @@ on_failure:
- _conf
- message
ignore_missing: true
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/globalprotect.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,3 +164,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/gtp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,3 +205,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/hipmatch.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,3 +96,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/ip_tag.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/sctp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,3 +170,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/system.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
4 changes: 4 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/threat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -470,3 +470,7 @@ on_failure:
{{#_ingest.on_failure_processor_tag}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{/_ingest.on_failure_processor_tag}}in pipeline '{{{ _ingest.pipeline }}}'
failed with message '{{{ _ingest.on_failure_message }}}'
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
Loading