You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ti_opencti: support filtering of indicators and deduplication mechanism (#15876)
This PR contains changes listed below for the OpenCTI integration:
- Added support for the following filters:
- Pattern Types
- Indicator Types
- Revoked Status
- Valid From
- Valid Until
- Label IDs (UUIDs)
- Minimum Confidence Level
- Author IDs (UUIDs)
- Creator User IDs (UUIDs)
- Created After
- Modified After
- Marking Definition IDs (UUIDs)
- Added tracking of the last modified timestamp in state
to prevent re-fetching already processed indicators.
- Added fingerprint processor to prevent duplicate indicators.
- Added useful fields to events for the creation of detection rules.
- Updated OpenCTI logo.
Copy file name to clipboardExpand all lines: packages/ti_opencti/_dev/build/docs/README.md
+93-1Lines changed: 93 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ Each event in the Indicator data stream collected by the OpenCTI integration is
15
15
16
16
This integration requires Filebeat version 8.16.0, or later.
17
17
18
-
It has been updated for OpenCTI version 5.12.24 and requires that version or later.
18
+
It has been updated for OpenCTI version 6.1.0 and requires that version or later.
19
19
20
20
## Setup
21
21
@@ -25,6 +25,98 @@ When adding the OpenCTI integration, you will need to provide a base URL for the
25
25
26
26
The simplest authentication method to use is an API key (bearer token). You can find a value for the API key on your profile page in the OpenCTI user interface. Advanced integration settings can be used to configure various OAuth2-based authentication arrangements, and to enter SSL settings for mTLS authentication and for other purposes. For information on setting up the OpenCTI side of an authentication strategy, please refer to [OpenCTI's authentication documentation](https://docs.opencti.io/latest/deployment/authentication/).
27
27
28
+
### Filtering
29
+
30
+
The OpenCTI integration supports advanced filtering capabilities to help you control which indicators are ingested. This allows you to focus on specific types of indicators, confidence levels, authors, or time ranges that are most relevant to your security operations.
31
+
32
+
#### Available Filters
33
+
34
+
The following filters can be configured when setting up the integration (Note: The integration automatically filters for entity type 'Indicator' only):
35
+
36
+
-**Pattern Types**: Filter indicators by pattern type (e.g., 'stix'). The values are customizable in OpenCTI, and any custom pattern types defined in your OpenCTI instance are supported (if an observable is associated).
37
+
38
+
-**Indicator Types**: Filter indicators by type. Values are customizable in OpenCTI. Common defaults include: 'malicious-activity', 'attribution', 'benign', 'anomalous-activity', 'compromised', 'unknown'. Custom types defined in your OpenCTI instance are also supported.
39
+
40
+
-**Revoked Status**: Filter by revoked status. Set to 'true' to get only revoked indicators, 'false' for only active (non-revoked) indicators, or leave empty to get all indicators regardless of revoked status.
41
+
42
+
-**Valid From (Start Date)**: Filter indicators with valid_from date after this date. Use ISO 8601 format (e.g., '2024-01-01T00:00:00Z') or relative date expressions (e.g., 'now-30d', 'now-7d').
43
+
44
+
-**Valid Until (End Date)**: Filter indicators with valid_until date before this date. Use ISO 8601 format (e.g., '2024-12-31T23:59:59Z') or relative date expressions (e.g., 'now+30d', 'now+7d').
45
+
46
+
-**Label IDs**: Filter by label IDs. Enter the UUIDs of the labels to filter indicators that have these labels applied. **Important: You must use label IDs (UUIDs), not label names.** You can find label IDs in the OpenCTI interface by navigating to Settings > Taxonomies > Labels, or via the API.
47
+
48
+
-**Minimum Confidence Level**: Filter indicators with confidence level greater than or equal to a specified value (0-100).
49
+
50
+
-**Author IDs**: Filter by author IDs (createdBy relationship). Enter the UUIDs of the authors to filter indicators created by them. **Important: You must use author IDs (UUIDs), not author names.** You can find author IDs in the OpenCTI interface by clicking on an entity and checking its details, or via the API.
51
+
52
+
-**Creator IDs**: Filter by technical creator IDs. Enter the UUIDs of the internal users who created the indicators in OpenCTI.
53
+
54
+
-**Created After**: Filter indicators created after a specific date. Use ISO 8601 format (e.g., '2024-01-01T00:00:00Z') or relative date expressions (e.g., 'now-30d', 'now-7d', 'now-24h').
55
+
56
+
-**Modified After**: Filter indicators modified after a specific date. Use ISO 8601 format (e.g., '2024-01-01T00:00:00Z') or relative date expressions (e.g., 'now-30d', 'now-7d', 'now-24h').
57
+
58
+
-**Marking Definition IDs**: Filter by marking definitions (e.g., TLP levels). Enter the UUIDs of the marking definitions. **Important: You must use marking definition IDs (UUIDs), not names.** Common TLP marking IDs:
Here are some practical examples of filter configurations:
67
+
68
+
1.**High-confidence indicators only**: Set `Minimum Confidence Level` to 75 to ingest only indicators with high confidence.
69
+
70
+
2.**Active threat indicators**: Set `Indicator Types` to ['malicious-activity', 'compromised'] and `Revoked Status` to 'false' to focus on active, non-revoked threats.
71
+
72
+
3.**Currently valid indicators**: Set `Valid From (Start Date)` to 'now-365d' and `Valid Until (End Date)` to 'now+30d' to get indicators that are currently within their validity period.
73
+
74
+
4.**Recent indicators**: Set `Created After` to 'now-7d' to collect only indicators created in the last 7 days.
75
+
76
+
5.**Specific pattern types**: Set `Pattern Types` to ['stix'] to collect only STIX pattern indicators, or include your custom pattern types defined in OpenCTI.
77
+
78
+
6.**Specific campaign tracking**: Use `Label IDs` filter with specific campaign label UUIDs (e.g., ['550e8400-e29b-41d4-a716-446655440000']) to track indicators related to particular threat campaigns.
79
+
80
+
7.**Indicators from specific sources**: Use `Author IDs` with the UUIDs of specific threat intelligence sources (e.g., ['123e4567-e89b-12d3-a456-426655440000']) to filter indicators from trusted sources.
81
+
82
+
8.**Recently modified high-value indicators**: Combine `Modified After` set to 'now-24h', `Minimum Confidence Level` to 80, and `Revoked Status` to 'false' to get recently updated, high-confidence active indicators.
83
+
84
+
9.**TLP-restricted indicators**: Use `Marking Definition IDs` with TLP:CLEAR and TLP:GREEN UUIDs to only ingest indicators that are safe to share broadly: ['marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da', 'marking-definition--bab4a63c-aed9-4cf5-a766-dfca5abac2bb'].
85
+
86
+
All filters work together using AND logic at the top level. Within each multi-value filter (like pattern types or label IDs), OR logic is applied between values.
87
+
88
+
#### High Availability and Deduplication
89
+
90
+
The OpenCTI integration supports running on multiple Elastic Agents for high availability. When multiple agents fetch the same indicators:
91
+
92
+
-**Automatic Deduplication**: The integration uses a fingerprint-based document ID to prevent duplicates. Each indicator gets a consistent ID based on its `standard_id` and `modified` timestamp.
93
+
-**No Manual Configuration Needed**: Deduplication works automatically - just deploy the integration to multiple agents.
94
+
-**Update Handling**: When an indicator is updated in OpenCTI, the new version replaces the old one in Elasticsearch.
95
+
96
+
#### Best Practices for HA Setup
97
+
98
+
1.**Stagger Execution Times**: To avoid all agents hitting OpenCTI simultaneously, consider offsetting their schedules slightly (e.g., Agent 1 at :00, Agent 2 at :02).
99
+
2.**Use the Same Configuration**: Ensure all agents use identical filter settings to fetch the same dataset.
100
+
3.**Monitor Performance**: Check OpenCTI server load when multiple agents are polling.
101
+
102
+
### Finding IDs in OpenCTI
103
+
104
+
Since several filters require UUIDs rather than names, here are ways to find these IDs:
105
+
106
+
1.**Label IDs**:
107
+
- In OpenCTI UI: Navigate to Settings → Taxonomies → Labels. Click on a label to see its ID in the URL or details.
108
+
- Via API: Query the `labels` endpoint to list all labels with their IDs.
109
+
110
+
2.**Author IDs**:
111
+
- In OpenCTI UI: Click on any entity that has an author, then click on the author name to see its details including the ID.
112
+
- Via API: Query the `identities` endpoint to list all identities (organizations, individuals) with their IDs.
113
+
114
+
3.**Creator IDs**:
115
+
- In OpenCTI UI: Navigate to Settings → Security → Users to see user IDs.
116
+
- Via API: Query the `users` endpoint (requires appropriate permissions).
117
+
118
+
For more information about OpenCTI's filtering system, refer to the [OpenCTI filters documentation](https://docs.opencti.io/latest/reference/filters/).
Copy file name to clipboardExpand all lines: packages/ti_opencti/changelog.yml
+14Lines changed: 14 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,18 @@
1
1
# newer versions go on top
2
+
- version: "2.10.0"
3
+
changes:
4
+
- description: Add comprehensive filtering support for indicators including pattern types, confidence levels, labels, dates, authors, creators, and marking definitions.
0 commit comments