Skip to content

Commit c230de7

Browse files
brijesh-elasticbrijesh-elastic
authored
proofpoint_essentials: Initial Release of Proofpoint Essentials
The Proofpoint Essentials integration with Elastic enables the collection of threats for monitoring and analysis. This valuable data can be leveraged within Elastic to analyze potential threat signals, including spam, phishing, business email compromise (BEC), imposter emails, ransomware, and malware. The integration collects threats using the (Essentials Threat API) [1], after which they are rerouted to different data streams based on their types. - clicks_blocked - clicks_permitted - message_blocked - message_delivered Sanitized test case inputs were obtained from live Proofpoint Essentials instance using the Essentials Threat API. [1] https://help.proofpoint.com/Essentials/Additional_Resources/API_Documentation/Essentials_Threat_API
1 parent 6bcdcde commit c230de7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+8845
-0
lines changed

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -378,6 +378,7 @@
378378
/packages/prometheus/data_stream/query @elastic/obs-infraobs-integrations
379379
/packages/prometheus/data_stream/remote_write @elastic/obs-ds-hosted-services
380380
/packages/prometheus_input @elastic/obs-infraobs-integrations
381+
/packages/proofpoint_essentials @elastic/security-service-integrations
381382
/packages/proofpoint_itm @elastic/security-service-integrations
382383
/packages/proofpoint_on_demand @elastic/security-service-integrations
383384
/packages/proofpoint_tap @elastic/security-service-integrations

‎packages/proofpoint_essentials/_dev/build/build.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
dependencies:
2+
ecs:
3+
reference: [email protected]

‎packages/proofpoint_essentials/_dev/build/docs/README.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Proofpoint Essentials Integration for Elastic
2+
3+
## Overview
4+
The Proofpoint Essentials integration with Elastic enables the collection of threats for monitoring and analysis. This valuable data can be leveraged within Elastic to analyze potential threat signals, including spam, phishing, business email compromise (BEC), imposter emails, ransomware, and malware.
5+
6+
This integration utilizes the [Proofpoint Essentials Threat API](https://help.proofpoint.com/Essentials/Additional_Resources/API_Documentation/Essentials_Threat_API) to collect threat events.
7+
8+
### Compatibility
9+
10+
The Proofpoint Essentials integration uses the REST API. It uses the `/v2/siem/all` to collect threat events.
11+
12+
### How it works
13+
14+
The **threat** data stream uses the `/v2/siem/all` endpoint to gather all threats starting from the configured initial interval. Subsequently, it fetches the recent threats available at each specified interval.
15+
16+
The gathered threat data is subsequently routed into individual data streams, each corresponding to a specific threat type.
17+
18+
## What data does this integration collect?
19+
20+
The Proofpoint Essentials integration collects threat events of the following types:
21+
22+
- `clicks_blocked`: events for clicks on malicious URLs blocked by URL Defense.
23+
- `clicks_permitted`: events for clicks on malicious URLs permitted by URL Defense.
24+
- `message_blocked`: events for blocked messages that contain threats recognized by URL Defense or Attachment Defense.
25+
- `message_delivered`: events for delivered messages that contain threats recognized by URL Defense or Attachment Defense.
26+
27+
### Supported use cases
28+
Integrating Proofpoint Essentials with Elastic SIEM enriches your security operations with targeted email threat intelligence. It enables the detection, investigation, and analysis of phishing, malware, and other email-based threats by leveraging detailed data on clicks and message events.
29+
30+
## What do I need to use this integration?
31+
32+
### From Proofpoint Essentials
33+
34+
#### Collecting data from Essentials Threat API
35+
36+
1. Navigate to
37+
- Go to **Account Management > Integrations**, then select the **Integration Keys** tab.
38+
2. Add a New Key
39+
- Click **Add Integration Key** in the upper right-hand corner.
40+
3. Enter Key Details
41+
- Provide a **description** to help identify the purpose of the key.
42+
- In the **Access Type** dropdown, select **SIEM Threat Events**
43+
4. Set Scope
44+
- If you are part of an **organisation**, the **Scope** field will be locked to **My Organisation Only**.
45+
- If you are a **partner**, you can choose between:
46+
- **My Organisation Only**
47+
- **My Organisation and All Child Organisations**
48+
5. Create and Save Credentials
49+
- After clicking **Create**, you’ll receive **API Key** and **API Key Secret**.
50+
6. Activation Time
51+
- The key may take up to **30 minutes** to become active.
52+
53+
For more details, check [Documentation](https://help.proofpoint.com/Essentials/Product_Documentation/Account_Management/Integrations/Integration_Keys).
54+
55+
## How do I deploy this integration?
56+
57+
### Agent-based deployment
58+
59+
Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host.
60+
61+
Elastic Agent is required to stream data from the syslog or log file receiver and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.
62+
63+
### Agentless deployment
64+
65+
Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. Agentless deployments provide a means to ingest data while avoiding the orchestration, management, and maintenance needs associated with standard ingest infrastructure. Using an agentless deployment makes manual agent deployment unnecessary, allowing you to focus on your data instead of the agent that collects it.
66+
67+
For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html)
68+
69+
### Onboard / configure
70+
71+
1. In the top search bar in Kibana, search for **Integrations**.
72+
2. In the search bar, type **Proofpoint Essentials**.
73+
3. Select the **Proofpoint Essentials** integration from the search results.
74+
4. Select **Add Proofpoint Essentials** to add the integration.
75+
5. Enable and configure only the collection methods which you will use.
76+
77+
* To **Collect Proofpoint Essentials logs via API**, you'll need to:
78+
79+
- Configure **URL**, **API Key**, and **API Key Secret**.
80+
- Adjust the integration configuration parameters if required, including the Interval, Collect Customer Data, Collect Own Data, Preserve original event etc. to enable data collection.
81+
82+
6. Select **Save and continue** to save the integration.
83+
84+
### Validation
85+
86+
#### Dashboards populated
87+
88+
1. In the top search bar in Kibana, search for **Dashboards**.
89+
2. In the search bar, type **Proofpoint Essentials**.
90+
3. Select a dashboard for the dataset you are collecting, and verify the dashboard information is populated.
91+
92+
## Troubleshooting
93+
94+
For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems).
95+
96+
## Scaling
97+
98+
For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation.
99+
100+
## Reference
101+
102+
### ECS field reference
103+
104+
#### Clicks Blocked
105+
106+
{{fields "clicks_blocked"}}
107+
108+
#### Clicks Permitted
109+
110+
{{fields "clicks_permitted"}}
111+
112+
#### Messages Blocked
113+
114+
{{fields "message_blocked"}}
115+
116+
#### Messages Delivered
117+
118+
{{fields "message_delivered"}}
119+
120+
### Inputs used
121+
122+
These inputs are used in this integration:
123+
124+
- [cel](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-cel)
125+
126+
### API usage
127+
128+
This integration uses the following APIs:
129+
130+
- [Proofpoint Essentials Threat API](https://help.proofpoint.com/Essentials/Additional_Resources/API_Documentation/Essentials_Threat_API).

‎packages/proofpoint_essentials/changelog.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# newer versions go on top
2+
- version: "0.1.0"
3+
changes:
4+
- description: Initial release.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/16073

‎packages/proofpoint_essentials/data_stream/clicks_blocked/fields/base-fields.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
- name: data_stream.type
2+
external: ecs
3+
- name: data_stream.dataset
4+
external: ecs
5+
- name: data_stream.namespace
6+
external: ecs
7+
- name: event.module
8+
external: ecs
9+
type: constant_keyword
10+
value: proofpoint_essentials
11+
- name: event.dataset
12+
external: ecs
13+
type: constant_keyword
14+
value: proofpoint_essentials.clicks_blocked
15+
- name: '@timestamp'
16+
external: ecs

‎packages/proofpoint_essentials/data_stream/clicks_blocked/fields/beats.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- name: input.type
2+
type: keyword
3+
description: Type of filebeat input.
4+
- name: log.offset
5+
type: long
6+
description: Log offset.

‎packages/proofpoint_essentials/data_stream/clicks_blocked/fields/ecs.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Define ECS constant fields as constant_keyword
2+
- name: observer.vendor
3+
type: constant_keyword
4+
external: ecs

‎packages/proofpoint_essentials/data_stream/clicks_blocked/fields/fields.yml‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
- name: proofpoint_essentials
2+
type: group
3+
fields:
4+
- name: threat
5+
type: group
6+
fields:
7+
- name: cc_addresses
8+
type: keyword
9+
description: 'A list of email addresses contained within the CC: header, excluding friendly names.'
10+
- name: classification
11+
type: keyword
12+
description: The threat category of the malicious URL.
13+
- name: click_ip
14+
type: ip
15+
description: The external IP address of the user who clicked on the link. If the user is behind a firewall performing network address translation, the IP address of the firewall will be shown.
16+
- name: click_time
17+
type: date
18+
description: The time the user clicked on the URL.
19+
- name: completely_rewritten
20+
type: keyword
21+
description: The rewrite status of the message.
22+
- name: customer_eid
23+
type: keyword
24+
description: The customer's entity ID.
25+
- name: customer_name
26+
type: keyword
27+
description: The customer's name, as configured in Essentials.
28+
- name: event_type
29+
type: keyword
30+
- name: from_address
31+
type: keyword
32+
description: 'The email address contained in the From: header, excluding friendly name.'
33+
- name: guid
34+
type: keyword
35+
description: The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
36+
- name: header_from
37+
type: keyword
38+
description: 'The full content of the From: header, including any friendly name.'
39+
- name: header_reply_to
40+
type: keyword
41+
description: 'If present, the full content of the Reply-To: header, including any friendly names.'
42+
- name: id
43+
type: keyword
44+
description: The unique id of the click.
45+
- name: impostor_score
46+
type: long
47+
description: The impostor score of the message. Higher scores indicate higher certainty.
48+
- name: malware_score
49+
type: long
50+
description: The malware score of the message. Higher scores indicate higher certainty.
51+
- name: message_details_url
52+
type: keyword
53+
description: A permalink to the messages' details page.
54+
- name: message_id
55+
type: keyword
56+
description: Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique.
57+
- name: message_parts
58+
type: group
59+
fields:
60+
- name: content_type
61+
type: keyword
62+
description: The true, detected Content-Type of the message_part. This may differ from the o_content_type value.
63+
- name: disposition
64+
type: keyword
65+
description: If the value is "inline", the message_part is a message body. If the value is "attached", the message_part is an attachment.
66+
- name: filename
67+
type: keyword
68+
description: The filename of the message_part.
69+
- name: md5
70+
type: keyword
71+
description: The MD5 hash of the message_part contents.
72+
- name: o_content_type
73+
type: keyword
74+
description: The declared Content-Type of the message_part.
75+
- name: sandbox_status
76+
type: keyword
77+
description: The verdict returned by the sandbox during the scanning process.
78+
- name: sha256
79+
type: keyword
80+
description: The SHA256 hash of the message_part contents.
81+
- name: message_size
82+
type: long
83+
description: The size in bytes of the message, including headers and attachments.
84+
- name: message_time
85+
type: date
86+
description: When the message was delivered to the user or quarantined by PPS.
87+
- name: parent_eid
88+
type: keyword
89+
description: The parent's EID.
90+
- name: parent_name
91+
type: keyword
92+
description: The parent's name, as configured in Essentials.
93+
- name: phish_score
94+
type: long
95+
description: The phish score of the message. Higher scores indicate higher certainty.
96+
- name: quarantine_rule
97+
type: keyword
98+
description: The name of the rule which quarantined the message. This appears only for messages_blocked events.
99+
- name: recipient
100+
type: keyword
101+
description: An array containing the email addresses of the SMTP (envelope) recipients.
102+
- name: reply_to_address
103+
type: keyword
104+
description: 'The email address contained in the Reply-To: header, excluding friendly name.'
105+
- name: sender
106+
type: keyword
107+
description: The email address of the SMTP (envelope) sender. The user-part is hashed. The domain-part is cleartext.
108+
- name: sender_ip
109+
type: ip
110+
description: The IP address of the sender.
111+
- name: spam_score
112+
type: long
113+
description: The spam score of the message. Higher scores indicate higher certainty.
114+
- name: stack_name
115+
type: keyword
116+
description: The name of the Essentials stack which processed the message.
117+
- name: subject
118+
type: keyword
119+
description: The subject line of the message, if available.
120+
- name: threat_id
121+
type: keyword
122+
description: The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.
123+
- name: threat_time
124+
type: date
125+
description: Proofpoint identified the URL as a threat at this time.
126+
- name: threat_status
127+
type: keyword
128+
description: The current state of the threat.
129+
- name: threats_info_map
130+
type: group
131+
fields:
132+
- name: actors
133+
type: nested
134+
description: An array of structures which contain details about the actors associated with a threat.
135+
- name: classification
136+
type: keyword
137+
description: The category of threat found in the message.
138+
- name: detection_type
139+
type: keyword
140+
- name: threat
141+
type: keyword
142+
description: The artifact which was condemned by Proofpoint. The malicious URL, hash of the attachment threat, or email address of the impostor sender.
143+
- name: threat_id
144+
type: keyword
145+
description: The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.
146+
- name: threat_status
147+
type: keyword
148+
description: The current state of the threat.
149+
- name: threat_time
150+
type: date
151+
description: Proofpoint assigned the threat_status at this time.
152+
- name: threat_type
153+
type: keyword
154+
description: Whether the threat was an attachment, URL, or message type.
155+
- name: to_addresses
156+
type: keyword
157+
description: 'A list of email addresses contained within the To: header, excluding friendly names.'
158+
- name: url
159+
type: keyword
160+
description: The malicious URL which was clicked.
161+
- name: user_agent
162+
type: keyword
163+
description: The User-Agent header from the clicker's HTTP request.
164+
- name: xmailer
165+
type: keyword
166+
description: 'The content of the X-Mailer: header, if present.'

‎packages/proofpoint_essentials/data_stream/clicks_blocked/manifest.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
title: Proofpoint Essentials Clicks Blocked Events
2+
dataset: proofpoint_essentials.clicks_blocked
3+
type: logs
4+
elasticsearch:
5+
dynamic_dataset: true
6+
dynamic_namespace: true
7+
index_template:
8+
mappings:
9+
dynamic: true

‎packages/proofpoint_essentials/data_stream/clicks_permitted/fields/base-fields.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
- name: data_stream.type
2+
external: ecs
3+
- name: data_stream.dataset
4+
external: ecs
5+
- name: data_stream.namespace
6+
external: ecs
7+
- name: event.module
8+
external: ecs
9+
type: constant_keyword
10+
value: proofpoint_essentials
11+
- name: event.dataset
12+
external: ecs
13+
type: constant_keyword
14+
value: proofpoint_essentials.clicks_permitted
15+
- name: '@timestamp'
16+
external: ecs

0 commit comments

Comments
 (0)