You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/detections/alert-suppression.asciidoc
+31-40Lines changed: 31 additions & 40 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,14 @@
1
1
[[alert-suppression]]
2
2
== Suppress detection alerts
3
3
4
+
Alert suppression allows you to reduce the number of repeated or duplicate detection alerts created by <<about-rules,detection rules>>. Normally, when a rule meets its criteria repeatedly, it creates multiple alerts, one for each time the rule's criteria are met. When alert suppression is configured, alerts for duplicate events are not created. Instead, the qualifying events are grouped, and only one alert is created for each group.
5
+
6
+
Depending on the rule type, you can configure alert suppression to create alerts each time the rule runs, or once within a specified time window. You can also specify multiple fields to group events by unique combinations of values.
7
+
8
+
The {security-app} displays several indicators in the Alerts table and the alert details flyout when a detection alert is created with alert suppression enabled. You can view the original events associated with suppressed alerts by investigating the alert in Timeline.
9
+
10
+
=== Configure alert suppression
11
+
4
12
.Requirements and notices
5
13
[sidebar]
6
14
--
@@ -9,59 +17,29 @@
9
17
* {ml-cap} rules have <<ml-requirements,additional requirements>> for alert suppression.
10
18
--
11
19
12
-
Alert suppression allows you to reduce the number of repeated or duplicate detection alerts created by these detection rule types:
13
-
14
-
* <<create-custom-rule,Custom query>>
15
-
* <<create-threshold-rule,Threshold>>
16
-
* <<create-indicator-rule,Indicator match>>
17
-
* <<create-eql-rule,Event correlation>>
18
-
* <<create-new-terms-rule,New terms>>
19
-
* <<create-esql-rule,{esql}>>
20
-
* <<create-ml-rule,{ml-cap}>>
21
-
22
-
Normally, when a rule meets its criteria repeatedly, it creates multiple alerts, one for each time the rule's criteria are met. When alert suppression is configured, duplicate qualifying events are grouped, and only one alert is created for each group. Depending on the rule type, you can configure alert suppression to create alerts each time the rule runs, or once within a specified time window. You can also specify multiple fields to group events by unique combinations of values.
23
-
24
-
The {security-app} displays several indicators in the Alerts table and the alert details flyout when a detection alert is created with alert suppression enabled. You can view the original events associated with suppressed alerts by investigating the alert in Timeline.
25
-
26
-
NOTE: Alert suppression is not available for Elastic prebuilt rules. However, if you want to suppress alerts for a prebuilt rule, you can duplicate it, then configure alert suppression on the duplicated rule.
27
-
28
-
=== Configure alert suppression
29
-
30
-
You can configure alert suppression when you create or edit a supported rule type. Refer to documentation for creating <<create-custom-rule,custom query>>, <<create-threshold-rule, threshold>>, <<create-eql-rule,event correlation>>, <<create-new-terms-rule,new terms>>, <<create-esql-rule,{esql}>>, or <<create-ml-rule,{ml}>> rules for detailed instructions.
20
+
You can configure alert suppression when <<rules-ui-create,creating>> or editing a rule.
31
21
32
-
. When configuring the rule type (the *Define rule* step for a new rule, or the *Definition* tab for an existing rule), specify how you want to group events for alert suppression:
22
+
. When configuring the rule (the **Define rule** step for a new rule, or the **Definition** tab for an existing rule), specify how you want to group alerts for alert suppression:
33
23
+
34
24
--
35
-
* **Custom query, indicator match, threshold, event correlation, new terms, {ml}, and {esql} rules:** In *Suppress alerts by*, enter 1-3 field names to group events by the fields' values.
36
-
* **Threshold rule:** In *Group by*, enter up to 3 field names to group events by the fields' values, or leave the setting empty to group all qualifying events together.
25
+
** **All rule types except the threshold rule**: In *Suppress alerts by*, enter 1-3 field names to group events by the fields' values.
26
+
** **Threshold rule only:** In **Group by**, enter up to 3 field names to group events by the fields' values, or leave the setting empty to group all qualifying events together.
37
27
38
28
--
39
29
+
40
-
[NOTE]
41
-
======
42
-
If you specify a field with multiple values, alerts with that field are handled as follows:
43
-
44
-
* **Custom query or threshold rules:** A group of alerts is created for each value. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts will be suppressed separately for each value of `127.0.0.1`, `127.0.0.2`, and `127.0.0.3`.
45
-
* **Indicator match, event correlation (non-sequence queries only), new terms, {esql}, or {ml} rules:** Alerts with the specified field name and identical array values are grouped together. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts with the entire array are grouped and only one alert is created for the group.
46
-
* **Event correlation (sequence queries only) rules:** If the specified field contains an array of values, suppression only happens if the field's values are an exact match and in the same order. For example, if you specify the field `myips` and one sequence alert has [1.1.1.1, 0.0.0.0] and another sequence alert has [1.1.1.1, 192.168.0.1], neither of those alerts will be suppressed, despite sharing an array element.
47
-
======
48
-
49
-
. If available, select how often to create alerts for duplicate events:
50
-
+
51
-
NOTE: Both options are available for custom query, indicator match, event correlation, new terms, {esql}, and {ml} rules. Threshold rules only have the *Per time period* option.
30
+
TIP: Refer to <<suppression-fields-with-multiple-values,Suppression for fields with an array of values>> to learn how fields with multiple values are handled.
52
31
+
53
-
--
54
-
* *Per rule execution*: Create an alert each time the rule runs and meets its criteria.
55
-
* *Per time period*: Create one alert for all qualifying events that occur within a specified time window, beginning from when an event first meets the rule criteria and creates the alert.
32
+
. If available, choose how often to create alerts for duplicate events:
33
+
** *Per rule execution*: Create an alert each time the rule runs and meets its criteria.
34
+
** *Per time period*: Create one alert for all qualifying events that occur within a specified time window, beginning from when an event first meets the rule criteria and creates the alert. This is the only option available when configuring alert suppression for threshold rules.
56
35
+
57
36
For example, if a rule runs every 5 minutes but you don't need alerts that frequently, you can set the suppression time period to a longer time, such as 1 hour. If the rule meets its criteria, it creates an alert at that time, and for the next hour, it'll suppress any subsequent qualifying events.
. Under *If a suppression field is missing*, choose how to handle events with missing suppression fields (events in which one or more of the *Suppress alerts by* fields don't exist):
63
41
+
64
-
NOTE: These options are not available for threshold rules.
42
+
NOTE: These options are available for all rule types except threshold rules.
65
43
66
44
* *Suppress and group alerts for events with missing fields*: Create one alert for each group of events with missing fields. Missing fields get a `null` value, which is used to group and suppress alerts.
67
45
* *Do not suppress alerts for events with missing fields*: Create a separate alert for each matching event. This basically falls back to normal alert creation for events with missing suppression fields.
@@ -76,6 +54,19 @@ NOTE: These options are not available for threshold rules.
76
54
77
55
====
78
56
57
+
[discrete]
58
+
[[suppression-fields-with-multiple-values]]
59
+
=== Suppression for fields with an array of values
60
+
61
+
62
+
When specifying fields to suppress alerts by, you can select fields that have multiple values. When alerts for those fields are generated, they're handled as follows:
63
+
64
+
* **Custom query or threshold rules:** Alerts are grouped by each unique value and an alert is created for each group. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts are grouped separately for each value of `127.0.0.1`, `127.0.0.2`, and `127.0.0.3` and an alert is created for each group.
65
+
66
+
* **Indicator match, event correlation (non-sequence queries only), new terms, {esql}, or {ml} rules:** Alerts with identical array values are grouped together. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts with the entire array are grouped and only one alert is created for the group.
67
+
68
+
* **Event correlation (sequence queries only) rules:** Alerts that are an exact match are grouped. To be an exact match, array values must be identical and in the same order. For example, if you specify the field `myips` and one sequence alert has `[1.1.1.1, 0.0.0.0]` and another sequence alert has `[1.1.1.1, 192.168.0.1]`, neither of those alerts is suppressed, despite sharing an array element.
69
+
79
70
=== Confirm suppressed alerts
80
71
81
72
The {security-app} displays several indicators of whether a detection alert was created with alert suppression enabled, and how many duplicate alerts were suppressed.
* In the `elasticsearch.yml` {ref}/settings.html[configuration] file:
29
+
30
+
** Set the `xpack.security.enabled` setting to `true`. For more information, refer to {ref}/security-settings.html[general security settings in {es}].
31
+
** If the `search.allow_expensive_queries` setting is set to `false`, remove it. If set to its default value of `true` or not included in the file, you don't need to change it. This setting must be `true` for key detection features, such as {kibana-ref}/alerting-getting-started.html#_rules[alerting rules] and rule exceptions, to work.
32
+
32
33
33
34
IMPORTANT: After changing the `xpack.encryptedSavedObjects.encryptionKey` value
34
35
and restarting {kib}, you must restart all detection rules.
This rule monitors for attempts to clear logs using the "journalctl" command on Linux systems. Adversaries may use this technique to cover their tracks by deleting or truncating log files, making it harder for defenders to investigate their activities. The rule looks for the execution of "journalctl" with arguments that indicate log clearing actions, such as "--vacuum-time", "--vacuum-size", or "--vacuum-files".
5
+
6
+
*Rule type*: eql
7
+
8
+
*Rule indices*:
9
+
10
+
* auditbeat-*
11
+
* endgame-*
12
+
* logs-auditd_manager.auditd-*
13
+
* logs-crowdstrike.fdr*
14
+
* logs-endpoint.events.process*
15
+
* logs-sentinel_one_cloud_funnel.*
16
+
17
+
*Severity*: low
18
+
19
+
*Risk score*: 21
20
+
21
+
*Runs every*: 5m
22
+
23
+
*Searches indices from*: now-9m ({ref}/common-options.html#date-math[Date Math format], see also <<rule-schedule, `Additional look-back time`>>)
24
+
25
+
*Maximum alerts per execution*: 100
26
+
27
+
*References*: None
28
+
29
+
*Tags*:
30
+
31
+
* Domain: Endpoint
32
+
* OS: Linux
33
+
* Use Case: Threat Detection
34
+
* Tactic: Defense Evasion
35
+
* Data Source: Elastic Defend
36
+
* Data Source: Elastic Endgame
37
+
* Data Source: Auditd Manager
38
+
* Data Source: Crowdstrike
39
+
* Data Source: SentinelOne
40
+
* Resources: Investigation Guide
41
+
42
+
*Version*: 1
43
+
44
+
*Rule authors*:
45
+
46
+
* Elastic
47
+
48
+
*Rule license*: Elastic License v2
49
+
50
+
51
+
==== Investigation guide
52
+
53
+
54
+
55
+
*Triage and analysis*
56
+
57
+
58
+
> **Disclaimer**:
59
+
> This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
60
+
61
+
62
+
*Investigating Attempt to Clear Logs via Journalctl*
63
+
64
+
65
+
This detection flags attempts to purge systemd journal logs by invoking journalctl with vacuum options, which attackers use to erase evidence and impede investigations. A common pattern is a compromised user escalating to root and immediately running sudo journalctl --vacuum-time=1s or --vacuum-size=1M, sometimes via a script or cron job, to rapidly truncate the journal across all boots and hide prior execution traces.
66
+
67
+
68
+
*Possible investigation steps*
69
+
70
+
71
+
- Enrich with user/UID, effective privileges, parent and command-line, session/TTY, and origin (SSH IP or local), and determine if execution came from a scheduled job (cron/systemd timer) or a script.
72
+
- Quantify destructiveness by extracting the exact vacuum parameter value(s) and immediately checking journal state (journalctl --disk-usage and --list-boots) and /var/log/journal size/mtime to see how much history was removed.
73
+
- Inspect configuration and persistence paths for intentional log suppression, including recent changes in /etc/systemd/journald.conf (Storage=volatile, SystemMaxUse, SystemMaxFileSize, MaxRetentionSec) and any new systemd units or scripts invoking journalctl vacuum.
74
+
- Correlate the vacuum timestamp with preceding activity to identify what might be concealed (privilege escalation, new accounts, sudoers edits, suspicious binaries), using auditd/EDR telemetry and shell history to rebuild the timeline.
75
+
- Verify remote log forwarding and SIEM ingestion for this host, compare gaps around the vacuum time, and recover pre-vacuum events from central storage to assess impact and intent.
76
+
77
+
78
+
*False positive analysis*
79
+
80
+
81
+
- A sysadmin or maintenance script ran journalctl --vacuum-time or --vacuum-size to reclaim space on a host under log disk pressure, which should correlate with low-free-space alerts, approved retention policy, and a scheduled systemd timer or cron job.
82
+
- OS provisioning or image-preparation steps vacuumed the journal with journalctl --vacuum-files to sanitize logs before snapshotting, typically a one-time root action occurring near installation and matching documented build procedures.
83
+
84
+
85
+
*Response and remediation*
86
+
87
+
88
+
- Immediately kill any active journalctl vacuum invocation (e.g., pkill -x journalctl), lock or remove sudo for the initiating user, and network-quarantine the host to prevent further tampering.
89
+
- Remove persistence by disabling systemd units/timers and cron jobs that call "journalctl --vacuum-*", inspecting /etc/systemd/system/* for ExecStart=journalctl vacuum and /etc/crontab, /etc/cron.*, and user crontabs, then deleting the offending scripts.
90
+
- Recover logging by setting Storage=persistent and policy-compliant SystemMaxUse/SystemMaxFileSize/MaxRetentionSec in /etc/systemd/journald.conf, restarting systemd-journald, and backfilling missing events from central log archives.
91
+
- Harden by enabling remote forwarding (ForwardToSyslog=yes and rsyslog/syslog-ng to SIEM), adding auditd rules to alert on "journalctl --vacuum-*", and tightening sudoers to require MFA and record command I/O for journalctl on critical hosts.
92
+
- Preserve evidence by archiving remaining /var/log/journal entries, journald.conf and its mtime, modified unit files under /etc/systemd/system, and shell/auth logs, and capture a disk snapshot before making further changes.
93
+
- Escalate to incident response if root executed "journalctl --vacuum-time/size/files" outside a documented maintenance window, if Storage=volatile was set or retention reduced below policy, or if the same actor performed vacuums on multiple hosts within 24 hours.
94
+
95
+
96
+
==== Setup
97
+
98
+
99
+
100
+
*Setup*
101
+
102
+
103
+
This rule requires data coming in from Elastic Defend.
104
+
105
+
106
+
*Elastic Defend Integration Setup*
107
+
108
+
Elastic Defend is integrated into the Elastic Agent using Fleet. Upon configuration, the integration allows the Elastic Agent to monitor events on your host and send data to the Elastic Security app.
109
+
110
+
111
+
*Prerequisite Requirements:*
112
+
113
+
- Fleet is required for Elastic Defend.
114
+
- To configure Fleet Server refer to the https://www.elastic.co/guide/en/fleet/current/fleet-server.html[documentation].
115
+
116
+
117
+
*The following steps should be executed in order to add the Elastic Defend integration on a Linux System:*
118
+
119
+
- Go to the Kibana home page and click "Add integrations".
120
+
- In the query bar, search for "Elastic Defend" and select the integration to see more details about it.
121
+
- Click "Add Elastic Defend".
122
+
- Configure the integration name and optionally add a description.
123
+
- Select the type of environment you want to protect, either "Traditional Endpoints" or "Cloud Workloads".
124
+
- Select a configuration preset. Each preset comes with different default settings for Elastic Agent, you can further customize these later by configuring the Elastic Defend integration policy. https://www.elastic.co/guide/en/security/current/configure-endpoint-integration-policy.html[Helper guide].
125
+
- We suggest selecting "Complete EDR (Endpoint Detection and Response)" as a configuration setting, that provides "All events; all preventions"
126
+
- Enter a name for the agent policy in "New agent policy name". If other agent policies already exist, you can click the "Existing hosts" tab and select an existing policy instead.
127
+
For more details on Elastic Agent configuration settings, refer to the https://www.elastic.co/guide/en/fleet/8.10/agent-policy.html[helper guide].
128
+
- Click "Save and Continue".
129
+
- To complete the integration, select "Add Elastic Agent to your hosts" and continue to the next section to install the Elastic Agent on your hosts.
130
+
For more details on Elastic Defend refer to the https://www.elastic.co/guide/en/security/current/install-endpoint.html[helper guide].
131
+
132
+
133
+
==== Rule query
134
+
135
+
136
+
[source, js]
137
+
----------------------------------
138
+
process where host.os.type == "linux" and event.type == "start" and
139
+
event.action in ("exec", "exec_event", "start", "ProcessRollup2", "executed", "process_started") and
140
+
process.name == "journalctl" and process.args like ("--vacuum-time=*", "--vacuum-size=*", "--vacuum-files=*")
0 commit comments