Skip to content

Commit 21263fe

Browse files
zmoogzmoog
committed
Add OTel to ECS adapter for Azure Logs integration
1 parent 1ddb689 commit 21263fe

File tree

10 files changed

+290
-0
lines changed

10 files changed

+290
-0
lines changed

‎packages/otel_ecs_adapter/LICENSE.txt‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
Elastic License 2.0
2+
3+
URL: https://www.elastic.co/licensing/elastic-license
4+
5+
## Acceptance
6+
7+
By using the software, you agree to all of the terms and conditions below.
8+
9+
## Copyright License
10+
11+
The licensor grants you a non-exclusive, royalty-free, worldwide,
12+
non-sublicensable, non-transferable license to use, copy, distribute, make
13+
available, and prepare derivative works of the software, in each case subject to
14+
the limitations and conditions below.
15+
16+
## Limitations
17+
18+
You may not provide the software to third parties as a hosted or managed
19+
service, where the service provides users with access to any substantial set of
20+
the features or functionality of the software.
21+
22+
You may not move, change, disable, or circumvent the license key functionality
23+
in the software, and you may not remove or obscure any functionality in the
24+
software that is protected by the license key.
25+
26+
You may not alter, remove, or obscure any licensing, copyright, or other notices
27+
of the licensor in the software. Any use of the licensor’s trademarks is subject
28+
to applicable law.
29+
30+
## Patents
31+
32+
The licensor grants you a license, under any patent claims the licensor can
33+
license, or becomes able to license, to make, have made, use, sell, offer for
34+
sale, import and have imported the software, in each case subject to the
35+
limitations and conditions in this license. This license does not cover any
36+
patent claims that you cause to be infringed by modifications or additions to
37+
the software. If you or your company make any written claim that the software
38+
infringes or contributes to infringement of any patent, your patent license for
39+
the software granted under these terms ends immediately. If your company makes
40+
such a claim, your patent license ends immediately for work on behalf of your
41+
company.
42+
43+
## Notices
44+
45+
You must ensure that anyone who gets a copy of any part of the software from you
46+
also gets a copy of these terms.
47+
48+
If you modify the software, you must include in any modified copies of the
49+
software prominent notices stating that you have modified the software.
50+
51+
## No Other Rights
52+
53+
These terms do not imply any licenses other than those expressly granted in
54+
these terms.
55+
56+
## Termination
57+
58+
If you use the software in violation of these terms, such use is not licensed,
59+
and your licenses will automatically terminate. If the licensor provides you
60+
with a notice of your violation, and you cease all violation of this license no
61+
later than 30 days after you receive that notice, your licenses will be
62+
reinstated retroactively. However, if you violate these terms after such
63+
reinstatement, any additional violation of these terms will cause your licenses
64+
to terminate automatically and permanently.
65+
66+
## No Liability
67+
68+
*As far as the law allows, the software comes as is, without any warranty or
69+
condition, and the licensor will not be liable to you for any damages arising
70+
out of these terms or the use or nature of the software, under any kind of
71+
legal claim.*
72+
73+
## Definitions
74+
75+
The **licensor** is the entity offering these terms, and the **software** is the
76+
software the licensor makes available under these terms, including any portion
77+
of it.
78+
79+
**you** refers to the individual or entity agreeing to these terms.
80+
81+
**your company** is any legal entity, sole proprietorship, or other kind of
82+
organization that you work for, plus all organizations that have control over,
83+
are under the control of, or are under common control with that
84+
organization. **control** means ownership of substantially all the assets of an
85+
entity, or the power to direct its management and policies by vote, contract, or
86+
otherwise. Control can be direct or indirect.
87+
88+
**your licenses** are all the licenses granted to you for the software under
89+
these terms.
90+
91+
**use** means anything you do with the software requiring one of your licenses.
92+
93+
**trademark** means trademarks, service marks, and similar rights.

‎packages/otel_ecs_adapter/changelog.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# newer versions go on top
2+
- version: "0.2.0"
3+
changes:
4+
- description: Introduce OTel ECS adapter to map OTel attributes to ECS fields.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/999999

‎packages/otel_ecs_adapter/data_stream/events/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
description: Pipeline for processing Azure Events logs
3+
processors:
4+
5+
# ------------------------------------------------------
6+
# Fallback pipeline for processing logs not supported by
7+
# the `azurelogs` translator.
8+
# ------------------------------------------------------
9+
- pipeline:
10+
if: "ctx.body?.structured != null"
11+
name: '{{ IngestPipeline "fallback" }}'
12+
description: "Fallback pipeline for processing logs not supported by the `azurelogs` translator"
13+
14+
on_failure:
15+
- set:
16+
field: event.kind
17+
value: pipeline_error
18+
- append:
19+
field: error.message
20+
value: '{{{ _ingest.on_failure_message }}}'

‎packages/otel_ecs_adapter/data_stream/events/elasticsearch/ingest_pipeline/fallback.yml‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
---
2+
description: |
3+
This pipelines handles log events not fully supported by the `azurelogs`
4+
translator.
5+
6+
The pipeline assumes that log events contain the `body.structured` field
7+
are not fully supported by the `azurelogs` translator.
8+
9+
The pipeline adapts the unsupported log events to look like the native
10+
Azure resource logs, so they can be rerouted to the Azure Logs integration.
11+
12+
processors:
13+
14+
# ------------------------------------------------------
15+
# Expand all fields into objects.
16+
# ------------------------------------------------------
17+
- dot_expander:
18+
path: resource.attributes
19+
field: "*"
20+
if: ctx.resource?.attributes != null
21+
22+
- dot_expander:
23+
path: body.structured
24+
field: "*"
25+
if: ctx.body?.structured != null
26+
27+
# ------------------------------------------------------
28+
# Rebuild the `time` and `resourceId` fields
29+
# ------------------------------------------------------
30+
- set:
31+
field: body.structured.time
32+
copy_from: "@timestamp"
33+
if: ctx.body?.structured != null
34+
35+
- rename:
36+
ignore_missing: true
37+
field: resource.attributes.cloud.resource_id
38+
target_field: body.structured.resourceId
39+
40+
# ------------------------------------------------------
41+
# Rebuild the OTel specific fields to the native Azure
42+
# resource logs schema.
43+
# ------------------------------------------------------
44+
- rename:
45+
ignore_missing: true
46+
field: body.structured.operation.name
47+
target_field: body.structured.operationName
48+
- rename:
49+
ignore_missing: true
50+
field: body.structured.operation.version
51+
target_field: body.structured.operationVersion
52+
- rename:
53+
ignore_missing: true
54+
field: body.structured.correlation.id
55+
target_field: body.structured.correlationId
56+
- rename:
57+
ignore_missing: true
58+
field: body.structured.result.description
59+
target_field: body.structured.resultDescription
60+
- rename:
61+
ignore_missing: true
62+
field: body.structured.result.signature
63+
target_field: body.structured.resultSignature
64+
- rename:
65+
ignore_missing: true
66+
field: body.structured.result.type
67+
target_field: body.structured.resultType
68+
- rename:
69+
ignore_missing: true
70+
field: body.structured.tenant.id
71+
target_field: body.structured.tenantId
72+
- rename:
73+
ignore_missing: true
74+
field: body.structured.cloud.region
75+
target_field: body.structured.location
76+
- rename:
77+
ignore_missing: true
78+
field: body.structured.network.peer.address
79+
target_field: body.structured.callerIpAddress
80+
81+
# ------------------------------------------------------
82+
# Clean up the empty OTel specific fields after renaming.
83+
# ------------------------------------------------------
84+
- remove:
85+
field:
86+
- body.structured.network
87+
- body.structured.correlation
88+
- body.structured.result
89+
- body.structured.operation
90+
- body.structured.tenant
91+
ignore_missing: true
92+
description: Clean up the empty OTel specific fields after renaming.
93+
- remove:
94+
field:
95+
- resource.attributes
96+
ignore_missing: true
97+
description: Removes the OTel specific fields that are not supported by the ECS schema.
98+
99+
on_failure:
100+
- append:
101+
field: error.message
102+
value:
103+
- "{{ _ingest.on_failure_message }} {{ _ingest.on_failure_processor_type }}"

‎packages/otel_ecs_adapter/data_stream/events/fields/base-fields.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
- name: data_stream.type
2+
type: constant_keyword
3+
description: Data stream type.
4+
- name: data_stream.dataset
5+
type: constant_keyword
6+
description: Data stream dataset.
7+
- name: data_stream.namespace
8+
type: constant_keyword
9+
description: Data stream namespace.
10+
- name: '@timestamp'
11+
type: date
12+
description: Event timestamp.

‎packages/otel_ecs_adapter/data_stream/events/manifest.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
title: "OTel to ECS"
2+
type: logs
3+
dataset: azure.resourcelogs.otel
4+
elasticsearch:
5+
dynamic_dataset: true
6+
dynamic_namespace: true

‎packages/otel_ecs_adapter/data_stream/events/routing_rules.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
- source_dataset: azure.resourcelogs.otel
2+
rules:
3+
- target_dataset: azure.events
4+
if: ctx.body?.structured != null
5+
namespace:
6+
- "{{data_stream.namespace}}"
7+
- default

‎packages/otel_ecs_adapter/docs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# OTel To ECS Adapter

‎packages/otel_ecs_adapter/img/otel_ecs.svg‎

Lines changed: 21 additions & 0 deletions
Loading

‎packages/otel_ecs_adapter/manifest.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
format_version: 3.4.1
2+
name: otel_ecs_adapter
3+
title: "OTel To ECS Adapter"
4+
version: 0.2.0
5+
description: "Adapter to convert OTel data to ECS data"
6+
type: integration
7+
categories:
8+
- cloud
9+
- azure
10+
- observability
11+
conditions:
12+
kibana:
13+
version: "^8.15.1 || ^9.0.0"
14+
owner:
15+
github: elastic/obs-ds-hosted-services
16+
type: elastic
17+
icons:
18+
- src: /img/otel_ecs.svg
19+
title: OTel To ECS Adapter logo
20+
size: 64x64
21+
type: image/svg+xml

0 commit comments

Comments
 (0)