Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,27 @@ Use Observability Pipelines' Amazon Data Firehose source to receive logs from Am

Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.

1. Optionally, select an AWS authentication option. If you select **Assume role**:
1. Enter the ARN of the IAM role you want to assume.
1. Optionally, enter the assumed role session name and external ID.
1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
- Enter the identifier for your Amazon Data Firehose address.
- **Note**: Only enter the identifier for the address. Do **not** enter the actual address.
- If left blank, the default is used: `SOURCE_AWS_DATA_FIREHOSE_ADDRESS`.

### Optional settings

#### AWS authentication

Select an **AWS authentication** option. If you select **Assume role**:
1. Enter the ARN of the IAM role you want to assume.
1. Optionally, enter the assumed role session name and external ID.

#### Enable TLS

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your Amazon Data Firehose key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

## Set the environment variables

Expand Down
28 changes: 21 additions & 7 deletions content/en/observability_pipelines/sources/amazon_s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,28 @@ Use Observability Pipelines' Amazon S3 source to receive logs from Amazon S3. Se

Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.

1. Enter the identifier for your Amazon S3 URL.
- **Note**: Only enter the identifier for the URL. Do **not** enter the actual URL.
- If left blank, the default is used: `SOURCE_AWS_S3_SQS_URL`.
1. Enter the AWS region.
1. Optionally, select an AWS authentication option. If you select **Assume role**:
1. Enter the ARN of the IAM role you want to assume.
1. Optionally, enter the assumed role session name and external ID.
1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

### Optional settings

#### AWS authentication

Select an **AWS authentication** option. If you select **Assume role**:
1. Enter the ARN of the IAM role you want to assume.
1. Optionally, enter the assumed role session name and external ID.

#### Enable TLS

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your Amazon S3 key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_AWS_S3_KEY_PASS`.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

## Set the environment variables

Expand Down
11 changes: 10 additions & 1 deletion content/en/observability_pipelines/sources/fluent.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,16 @@ Use Observability Pipelines' Fluentd or Fluent Bit source to receive logs from t

Select and set up this source when you [set up a pipeline][1]. The information below are for the source settings in the pipeline UI.

Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- 1. Enter the identifier for your Fluent address.
- **Note**: Only enter the identifier for the address. Do **not** enter the actual address.
- If left blank, the default is used: `SOURCE_FLUENT_ADDRESS`.

### Optional settings

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your Fluent key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_FLUENT_KEY_PASS`.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Select and set up this source when you [set up a pipeline][1]. The information b
1. Enter the subscription name.
1. Select the decoder you want to use (Bytes, GELF, JSON, syslog).
1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][3] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format.

Expand Down
30 changes: 25 additions & 5 deletions content/en/observability_pipelines/sources/http_client.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,36 @@ Select and set up this source when you [set up a pipeline][1]. The information b

To configure your HTTP/S Client source:

1. Select your authorization strategy.
2. Select the decoder you want to use on the HTTP messages. Logs pulled from the HTTP source must be in this format.
3. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
1. Enter the identifier for your HTTP Client endpoint URL.
- **Note**: Only enter the identifier for the endpoint URL. Do **not** enter the actual endpoint URL.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_ENDPOINT_URL`.
1. Select your authorization strategy. If you selected:
- **Basic**:
- Enter the identifier for your HTTP Client username.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_USERNAME`.
Enter the identifier for your HTTP Client password.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_PASSWORD`.
- **Bearer**: Enter the identifier for your bearer token.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_BEARER_TOKEN`.
1. Select the decoder you want to use on the HTTP messages. Logs pulled from the HTTP source must be in this format.

### Optional settings

#### Enable TLS
Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your HTTP Client key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_KEY_PASS`
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
4. Enter the interval between scrapes.

#### Scrape settings

- Enter the interval between scrapes.
- Your HTTP Server must be able to handle GET requests at this interval.
- Since requests run concurrently, if a scrape takes longer than the interval given, a new scrape is started, which can consume extra resources. Set the timeout to a value lower than the scrape interval to prevent this from happening.
5. Enter the timeout for each scrape request.
- Enter the timeout for each scrape request.

## Set the environment variables

Expand Down
23 changes: 18 additions & 5 deletions content/en/observability_pipelines/sources/http_server.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,25 @@ Select and set up this source when you [set up a pipeline][1]. The information b

To configure your HTTP/S Server source, enter the following:

1. Select your authorization strategy.
1. Enter the identifier for your HTTP Server address.
- **Note**: Only enter the identifier for the address. Do **not** enter the actual address.
- If left blank, the default is used: `SOURCE_HTTP_SERVER_ADDRESS`.
1. Select your authorization strategy. If you selected **Basic**:
- Enter the identifier for your HTTP Server username.
- If left blank, the default is used: `SOURCE_HTTP_SERVER_USERNAME`.
Enter the identifier for your HTTP Server password.
- If left blank, the default is used: `SOURCE_HTTP_SERVER_PASSWORD`.
1. Select the decoder you want to use on the HTTP messages. Your HTTP client logs must be in this format. **Note**: If you select `bytes` decoding, the raw log is stored in the `message` field.
1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format.

### Optional settings

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your HTTP Server key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_HTTP_SERVER_KEY_PASS`.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format.

## Set the environment variables

Expand Down
43 changes: 33 additions & 10 deletions content/en/observability_pipelines/sources/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,41 @@ You can also [send Azure Event Hub logs to Observability Pipelines using the Kaf

Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.

<div class="alert alert-danger">Only enter the identifiers for the Kafka servers, username, and password. Do <b>not</b> enter the actual values.</a></div>

1. Enter the identifier for your Kafka servers.
- If left blank, the default is used: `SOURCE_KAFKA_BOOTSTRAP_SERVERS`.
1. Enter the identifier for your Kafka username.
- If left blank, the default is used: `SOURCE_KAFKA_SASL_USERNAME`.
1. Enter the identifier for your Kafka password.
- If left blank, the default is used: `SOURCE_KAFKA_SASL_PASSWORD`.
1. Enter the group ID.
1. Enter the topic name. If there is more than one, click **Add Field** to add additional topics.
1. Optionally, toggle the switch to enable SASL Authentication and select the mechanism (**PLAIN**, **SCHRAM-SHA-256**, or **SCHRAM-SHA-512**) in the dropdown menu.
1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][5] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
1. Optionally, click **Advanced** and click **Add Option** to add additional [librdkafka options](#librdkafka-options).
1. Select an option in the dropdown menu.
1. Enter a value for that option.
1. Check your values against the [librdkafka documentation][4] to make sure they have the correct type and are within the set range.
1. Click **Add Option** to add another librdkafka option.

### Optional settings

#### Enable SASL Authentication

1. Toggle the switch to enable **SASL Authentication**
1. Select the mechanism (**PLAIN**, **SCHRAM-SHA-256**, or **SCHRAM-SHA-512**) in the dropdown menu.

#### Enable TLS

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][5] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your Kafka key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_KAFKA_KEY_PASS`.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

#### Add additional librdkafka options

1. Click **Advanced** and then **Add Option**.
1. Select an option in the dropdown menu.
1. Enter a value for that option.
1. Check your values against the [librdkafka documentation][4] to make sure they have the correct type and are within the set range.
1. Click **Add Option** to add another librdkafka option.

## Set the environment variables

Expand Down
Loading
Loading