-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Add IOC Explorer doc; small edits #33413
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
c09d650
4567efb
3d95609
d9be222
11e67e1
a0d3880
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,20 +1,21 @@ | ||
| --- | ||
| title: Threat Intelligence | ||
| title: Bring Your Own Threat Intelligence | ||
| disable_toc: false | ||
| aliases: | ||
| - /security/cloud_siem/threat_intelligence | ||
| further_reading: | ||
| - link: "security/cloud_siem/detection_rules" | ||
| tag: "Documentation" | ||
| text: "Create custom detection rules" | ||
| - link: /security/cloud_siem/triage_and_investigate/ioc_explorer/ | ||
| tag: documentation | ||
| text: IOC Explorer | ||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| Datadog provides built-in [threat intelligence][1] for Cloud SIEM logs. This article explains how to extend that functionality by enriching logs with your own custom threat intelligence feeds. | ||
|
|
||
| ## Bring your own threat intelligence | ||
|
|
||
| Cloud SIEM supports enriching and searching logs using threat intelligence indicators of compromise (IOCs) stored in Datadog reference tables. [Reference Tables][7] allow you to combine metadata with information already in Datadog. | ||
|
|
||
| ### How bring your own threat intelligence works | ||
|
|
@@ -38,21 +39,21 @@ When Cloud SIEM processes a log, the log's IP and domain attributes are evaluate | |
| - Cloud SIEM evaluates logs in real time and uses both [Datadog-curated threat intelligence][10] and your own reference tables. | ||
| - Reference tables are the mechanism for storing and joining your custom IoCs with logs and detections. | ||
|
|
||
| ### Storing indicators of compromise in reference tables | ||
| ### Store indicators of compromise in reference tables | ||
|
|
||
| Threat intelligence is supported in the CSV format, and requires a table for each Indicator type (for example, IP address or domain) and requires the following columns: | ||
|
|
||
| #### CSV structure for IP address | ||
|
|
||
| | Field | Data | Description | Required | Example | | ||
| |-------------------|-------|-------------------------------------------------------------------------------------------------|----------|----------------------------------| | ||
| | ip_address | text | The primary key for the reference table in the IPv4 dot notation format. | true | 192.0.2.1 | | ||
| | additional_data | json | Additional data to enrich the logs. | false | `{"ref":"hxxp://example.org"}` | | ||
| | category | text | The threat intel [category][8]. This is used by some out-of-the-box detection rules. | true | Malware | | ||
| | intention | text | The threat intel [intent][9]. This is used by some out-of-the-box detection rules. | true | malicious | | ||
| | source | text | The name of the source and the link to its site, such as your team and your team's wiki. | true | `{"name":"internal_security_team", "url":"https://teamwiki.example.org"}` | | ||
| | Field | Data | Description | Required | Example | | ||
| |-----------------|------|------------------------------------------------------------------------------------------|----------|---------------------------------------------------------------------------| | ||
| | ip_address | text | The primary key for the reference table in the IPv4 dot notation format. | true | 192.0.2.1 | | ||
| | additional_data | json | Additional data to enrich the logs. | false | `{"ref":"hxxp://example.org"}` | | ||
| | category | text | The threat intel [category][8]. This is used by some out-of-the-box detection rules. | true | Malware | | ||
| | intention | text | The threat intel [intent][9]. This is used by some out-of-the-box detection rules. | true | malicious | | ||
| | source | text | The name of the source and the link to its site, such as your team and your team's wiki. | true | `{"name":"internal_security_team", "url":"https://teamwiki.example.org"}` | | ||
|
|
||
| <div class="alert alert-info">JSON in a CSV requires double quoting. The following is an example CSV.</div> | ||
| <div class="alert alert-info">JSON in a CSV requires double quoting. The following is an example CSV:</div> | ||
|
|
||
| ``` | ||
| ip_address,additional_data,category,intention,source | ||
|
|
@@ -63,15 +64,15 @@ ip_address,additional_data,category,intention,source | |
|
|
||
| #### CSV structure for domain | ||
|
|
||
| | Field | Data | Description | Required | Example | | ||
| |-------------------|-------|-------------------------------------------------------------------------------------------------|----------|----------------------------------| | ||
| | domain | text | The primary key for the reference table. | true | mal-domain.com | | ||
| | additional_data | json | Additional data to enrich the trace. | false | `{"ref":"hxxp://example.org"}` | | ||
| | category | text | The threat intel [category][8]. This is used by some out-of-the-box detection rules. | true | Phishing | | ||
| | intention | text | The threat intel [intent][9]. This is used by some out-of-the-box detection rules. | true | malicious | | ||
| | source | text | The name of the source and the link to its site, such as your team and your team's wiki. | true | `{"name":"internal_security_team", "url":"https://teamwiki.example.org"}` | | ||
| | Field | Data | Description | Required | Example | | ||
| |-----------------|------|------------------------------------------------------------------------------------------|----------|---------------------------------------------------------------------------| | ||
| | domain | text | The primary key for the reference table. | true | mal-domain.com | | ||
| | additional_data | json | Additional data to enrich the trace. | false | `{"ref":"hxxp://example.org"}` | | ||
| | category | text | The threat intel [category][8]. This is used by some out-of-the-box detection rules. | true | Phishing | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as above; consider code/backtick formatting for all examples in this column |
||
| | intention | text | The threat intel [intent][9]. This is used by some out-of-the-box detection rules. | true | malicious | | ||
| | source | text | The name of the source and the link to its site, such as your team and your team's wiki. | true | `{"name":"internal_security_team", "url":"https://teamwiki.example.org"}` | | ||
|
|
||
| ### Uploading and enabling your own threat intelligence | ||
| ### Upload and enable your own threat intelligence | ||
|
|
||
| Datadog supports creating reference tables either by a manual upload or by periodically retrieving the data from Amazon S3, Azure storage, or Google Cloud storage. | ||
|
|
||
|
|
@@ -110,15 +111,15 @@ In Datadog Event Management, it may appear that data has been fetched from the c | |
| - The update replaces the entire table with the new data. | ||
| In case of a duplicated primary key, the rows with the duplicated key are not written, and an error is shown in the reference table detail page. | ||
|
|
||
| ## Threat intelligence in the user interface | ||
| ## View threat intelligence data in Datadog | ||
|
|
||
| To enable Cloud SIEM threat intelligence data for reference tables: | ||
| 1. Navigate to [Threat Intelligence][3]. | ||
| 1. For the table you want to see Cloud SIEM threat intelligence data, click the dropdown menu in the **Enabled** column and select Cloud SIEM. | ||
| 1. For the table you want to see Cloud SIEM threat intelligence data for, click the dropdown menu in the **Enabled** column and select Cloud SIEM. | ||
|
|
||
| After applying a reference table to Cloud SIEM, all incoming logs are evaluated against the table using a specific Indicator of Compromise (IoC) key, such as an IP address or domain. If a match is found, the log is enriched with relevant Threat Intelligence (TI) attributes from the table, which enhances detection, investigation, and response. | ||
| After applying a reference table to Cloud SIEM, all incoming logs are evaluated against the table using a specific Indicator of Compromise (IoC) key, such as an IP address or domain. If a match is found, the log is enriched with relevant Threat Intelligence (TI) attributes from the table, which enhances detection, investigation, and response. A threat intelligence reference table can be shared across multiple security products. | ||
|
|
||
| A threat intelligence reference table can be shared across multiple security products. | ||
| You can view your threat intelligence data in the [IOC Explorer][11]. | ||
|
|
||
| ## Further reading | ||
|
|
||
|
|
@@ -133,4 +134,5 @@ A threat intelligence reference table can be shared across multiple security pro | |
| [7]: /reference_tables/ | ||
| [8]: /security/threat_intelligence/#threat-intelligence-categories | ||
| [9]: /security/threat_intelligence/#threat-intelligence-intents | ||
| [10]: /security/threat_intelligence#threat-intelligence-sources | ||
| [10]: /security/threat_intelligence#threat-intelligence-sources | ||
| [11]: /security/cloud_siem/triage_and_investigate/ioc_explorer/ | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| --- | ||
| title: IOC Explorer | ||
| further_reading: | ||
| - link: /security/threat_intelligence/ | ||
| tag: documentation | ||
| text: Threat Intelligence | ||
| - link: /security/cloud_siem/ingest_and_enrich/threat_intelligence | ||
| tag: documentation | ||
| text: Bring Your Own Threat Intelligence | ||
| --- | ||
|
|
||
| {{< callout url="" btn_hidden="true" header="false" >}} | ||
| The IOC Explorer is in Preview. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I can't find a guideline around capitalizing Preview/preview in our SG or Tech Content's, and i don't think we've been especially consistent (i just looked back at a few pages i knew were in preview). Lowercase seems preferable and somewhat more prevalent to me (i would also say the same for its counterpart "general availability"), and while I don't think this is a primary source of truth, it's all lowercase on this D4D page: https://datadoghq.atlassian.net/wiki/x/e4rliw. I realize it's entirely possible and even likely that you've found precedent for capitalizing, so just calling out as a suggestion.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for calling this out, Jeff! It's funny, when I started, it was really drilled into me that we always need to capitalize Preview, but now I can't find guidance about it either. Maybe it changed! I know the terminology around beta/preview/limited availability/etc. has changed quite a bit over time. |
||
| {{< /callout >}} | ||
|
|
||
| ## Overview | ||
|
|
||
| Indicators of Compromise (IOC) are evidence that your systems have experienced a security breach. With the [IOC Explorer][1], you can view more details about compromises, and see related signals and logs. | ||
|
|
||
| {{< img src="security/security_monitoring/ioc_explorer.png" alt="The IOC Explorer, showing an IP address that has been flagged as an indicator of compromise" style="width:100%;" >}} | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| To view data in the IOC Explorer, all of the following must be true: | ||
| - Your organization must subscribe to Cloud SIEM. | ||
| - The indicator of compromise must be in a threat feed that was available to Datadog at the time of the log acquisition. | ||
| - For more information on the threat intelligence feeds the IOC Explorer displays content from, see [Threat intelligence sources][2]. | ||
| - A log that has a matching entity in threat intelligence must be acquired. | ||
| - The time frame for the Explorer is fixed to the last 30 days. The log must be from within that time frame. | ||
|
|
||
| ## Use the IOC Explorer | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I do think "Using the IOC Explorer" reads a bit more naturally. Non-critical suggestion. I know we're pretty scattered consistency-wise across the docs for using the participle form or not in these types of headings. |
||
|
|
||
| To access the IOC Explorer in Datadog, go to **Security** > **Cloud SIEM** > **Investigate** > [**IOC Explorer**][1]. | ||
|
|
||
| ### Query and filter indicators of compromise | ||
|
|
||
| You can write custom queries or apply filters to determine which indicators of compromise you can see in the explorer. You can query or filter by: | ||
| - Severity score | ||
| - [Entity type][3] | ||
| - [Threat intelligence source][2] | ||
| - [Threat intelligence category][4] | ||
|
|
||
| Additionally, you can click a column heading in the Explorer to sort by that column's values. | ||
|
|
||
| ### Get more context on an indicator of compromise | ||
|
|
||
| Click an indicator of compromise to open a side panel that contains additional information about it: | ||
| - When the indicator was first and last seen in a threat intelligence feed | ||
| <div class="alert alert-info" style="margin-bottom: 0">This is distinct from the first or last time the indicator was seen in a log.</div> | ||
| - Any categories and ratings assigned to it, and the threat intelligence feeds associated with those ratings | ||
| - A breakdown of the indicator's severity score | ||
| - Signal matches, which you can view in Signals Explorer | ||
| - Related logs, which you can view in Log Explorer | ||
|
|
||
| ## Understand severity scoring | ||
|
|
||
| It's important to have proper context for the severity score for an indicator, so you can properly prioritize investigations. For example, [IP addresses][5] can be volatile and require frequent reassessments as a result. | ||
|
|
||
| In the IOC Explorer side panel, you can see the factors that contribute to the severity score. Severity score starts from a base score based on classification, and increases or decreases based on additional factors: | ||
| - **Classification**: The base score associated with the indicator's category and intent | ||
| - **Corroboration**: Whether the indicator appears on multiple threat intelligent feeds | ||
| - **Persistence**: How long threat intelligence feeds have been reporting on the indicator | ||
| - **Hosting Type**: Used for IP and domain entity types; evaluates whether the hosting infrastructure type is commonly used for attacks | ||
| - **Signal Activity**: Whether the indicator has been observed in Signals | ||
|
|
||
| ## Further reading | ||
|
|
||
| {{< partial name="whats-next/whats-next.html" >}} | ||
|
|
||
| [1]: https://app.datadoghq.com/security/siem/ioc-explorer | ||
| [2]: /security/threat_intelligence/#threat-intelligence-sources | ||
| [3]: /security/threat_intelligence/#entity-types | ||
| [4]: /security/threat_intelligence/#threat-intelligence-categories | ||
| [5]: /security/threat_intelligence/#ip-addresses-dynamic-and-transient | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might consider putting everything in this column in code/backtick formatting, since they're all examples of values for the CSV? Your call, non-blocking.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I appreciate the suggestion! I tried it, and annoyingly, it caused some wrapping in the first column that made it more difficult to read. Maybe that's why we avoided the backticks to begin with...