diff --git a/content/en/observability_pipelines/destinations/amazon_opensearch.md b/content/en/observability_pipelines/destinations/amazon_opensearch.md index e729df71e7a..e1d995a71bc 100644 --- a/content/en/observability_pipelines/destinations/amazon_opensearch.md +++ b/content/en/observability_pipelines/destinations/amazon_opensearch.md @@ -16,18 +16,48 @@ Set up the Amazon OpenSearch destination and its environment variables when you ### Set up the destination -1. Optionally, enter the name of the Amazon OpenSearch index. See [template syntax][3] if you want to route logs to different indexes based on specific fields in your logs. -1. Select an authentication strategy, **Basic** or **AWS**. For **AWS**, enter the AWS region. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +
Only enter the identifiers for the Amazon OpenSearch endpoint URL, and if applicable, username and password. Do not enter the actual values.
+ +1. Enter the identifier for your Amazon OpenSearch endpoint URL. If you leave it blank, the [default](#set-secrets) is used. +1. (Optional) Enter the name of the Amazon OpenSearch index. See [template syntax][3] if you want to route logs to different indexes based on specific fields in your logs. +1. Select an authentication strategy, **Basic** or **AWS**. If you selected: + - **Basic**: + - Enter the identifier for your Amazon OpenSearch username. If you leave it blank, the [default](#set-secrets) is used. + - Enter the identifier for your Amazon OpenSearch password. If you leave it blank, the [default](#set-secrets) is used. + - **AWS**: + 1. Enter the AWS region. + 1. (Optional) Select an AWS authentication option. The **Assume role** option should only be used if the user or role you created earlier needs to assume a different role to access the specific AWS resource and that permission has to be explicitly defined.
If you select **Assume role**: + 1. Enter the ARN of the IAM role you want to assume. + 1. Optionally, enter the assumed role session name and external ID. +1. (Optional) Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - If left disabled, the maximum size for buffering is 500 events. - If enabled: 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Amazon OpenSearch endpoint URL identifier: + - The default identifier is `DESTINATION_AMAZON_OPENSEARCH_ENDPOINT_URL`. +- Amazon OpenSearch authentication username identifier: + - The default identifier is `DESTINATION_AMAZON_OPENSEARCH_USERNAME`. +- Amazon OpenSearch authentication password identifier: + - The default identifier is `DESTINATION_AMAZON_OPENSEARCH_PASSWORD`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/amazon_opensearch %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/amazon_s3.md b/content/en/observability_pipelines/destinations/amazon_s3.md index 5aa90534ad0..f9e7313ada5 100644 --- a/content/en/observability_pipelines/destinations/amazon_s3.md +++ b/content/en/observability_pipelines/destinations/amazon_s3.md @@ -90,10 +90,24 @@ Then these are the values you enter for configuring the S3 bucket for Log Archiv {{< img src="observability_pipelines/setup/amazon_s3_archive.png" alt="The log archive configuration with the example values" style="width:70%;" >}} -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +There are no secret identifiers to configure. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/destination_env_vars/datadog_archives_amazon_s3 %}} +{{% /tab %}} +{{< /tabs >}} + ## Route logs to Snowflake using the Amazon S3 destination You can route logs from Observability Pipelines to Snowflake using the Amazon S3 destination by configuring Snowpipe in Snowflake to automatically ingest those logs. To set this up: diff --git a/content/en/observability_pipelines/destinations/amazon_security_lake.md b/content/en/observability_pipelines/destinations/amazon_security_lake.md index 5c86289b070..ff23f5d5ded 100644 --- a/content/en/observability_pipelines/destinations/amazon_security_lake.md +++ b/content/en/observability_pipelines/destinations/amazon_security_lake.md @@ -20,32 +20,62 @@ You need to do the following before setting up the Amazon Security Lake destinat Set up the Amazon Security Lake destination and its environment variables when you [set up a pipeline][1]. The information below is configured in the pipelines UI. +**Notes**: +- When you add the Amazon Security Lake destination, the OCSF processor is automatically added so that you can convert your logs to Parquet before they are sent to Amazon Security Lake. See [Remap to OCSF documentation][3] for setup instructions. +- Only logs formatted by the OCSF processor are converted to Parquet. + ### Set up the destination 1. Enter your S3 bucket name. 1. Enter the AWS region. 1. Enter the custom source name. -1. Optionally, select an [AWS authentication][5] option. - 1. Enter the ARN of the IAM role you want to assume. - 1. Optionally, enter the assumed role session name and external ID. -1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][4] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. - - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - - If left disabled, the maximum size for buffering is 500 events. - - If enabled: - 1. Select the buffer type you want to set (**Memory** or **Disk**). - 1. Enter the buffer size and select the unit. -**Notes**: -- When you add the Amazon Security Lake destination, the OCSF processor is automatically added so that you can convert your logs to Parquet before they are sent to Amazon Security Lake. See [Remap to OCSF documentation][3] for setup instructions. -- Only logs formatted by the OCSF processor are converted to Parquet. +#### Optional settings + +##### AWS authentication + +1. Select an [AWS authentication][5] option. +1. Enter the ARN of the IAM role you want to assume. +1. Optionally, enter the assumed role session name and external ID. + +##### Enable TLS -### Set the environment variables +Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required. + +**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][4] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. +- Enter the identifier for your Amazon Security Lake key pass. If you leave it blank, the [default](#set-secrets) is used. + - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. +- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). +- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. + +##### Buffering options + +Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +- If left disabled, the maximum size for buffering is 500 events. +- If enabled: + 1. Select the buffer type you want to set (**Memory** or **Disk**). + 1. Enter the buffer size and select the unit. + +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Amazon Security Lake TLS passphrase identifier (when TLS is enabled): + - The default identifier is `DESTINATION_AWS_SECURITY_LAKE_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/amazon_security_lake %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### AWS Authentication diff --git a/content/en/observability_pipelines/destinations/azure_storage.md b/content/en/observability_pipelines/destinations/azure_storage.md index 39736694870..e501a5a1c2a 100644 --- a/content/en/observability_pipelines/destinations/azure_storage.md +++ b/content/en/observability_pipelines/destinations/azure_storage.md @@ -22,21 +22,48 @@ You need to have Datadog's [Azure integration][3] installed to set up Datadog Lo Set up the Azure Storage destination and its environment variables when you [set up an Archive Logs pipeline][4]. The information below is configured in the pipelines UI. +1. Enter the identifier for your Azure connection string. If you leave it blank, the [default](#set-secrets) is used. + - **Note**: Only enter the identifier for the connection string. Do **not** enter the actual connection string. 1. Enter the name of the Azure container you created earlier. -1. Optionally, enter a prefix. - - Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in `/` to act as a directory path; a trailing `/` is not automatically added. - - See [template syntax][6] if you want to route logs to different object keys based on specific fields in your logs. - - **Note**: Datadog recommends that you start your prefixes with the directory name and without a lead slash (`/`). For example, `app-logs/` or `service-logs/`. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - - If left disabled, the maximum size for buffering is 500 events. - - If enabled: - 1. Select the buffer type you want to set (**Memory** or **Disk**). - 1. Enter the buffer size and select the unit. - -### Set the environment variables + +### Optional settings + +#### Prefix to apply to all key objects + +Enter a prefix that you want to apply to all key objects. + +- Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in `/` to act as a directory path; a trailing `/` is not automatically added. +- See [template syntax][6] if you want to route logs to different object keys based on specific fields in your logs. + - **Note**: Datadog recommends that you start your prefixes with the directory name and without a lead slash (`/`). For example, `app-logs/` or `service-logs/`. + +#### Buffering options + +Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +- If left disabled, the maximum size for buffering is 500 events. +- If enabled: + 1. Select the buffer type you want to set (**Memory** or **Disk**). + 1. Enter the buffer size and select the unit. + +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Azure connection string identifier: + - References the connection string that gives the Worker access to your Azure Storage bucket. + - The default identifier is `DESTINATION_DATADOG_ARCHIVES_AZURE_BLOB_CONNECTION_STRING`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/datadog_archives_azure_storage %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/cloudprem.md b/content/en/observability_pipelines/destinations/cloudprem.md index ccc48e7c992..d51d827bff2 100644 --- a/content/en/observability_pipelines/destinations/cloudprem.md +++ b/content/en/observability_pipelines/destinations/cloudprem.md @@ -32,7 +32,23 @@ Optionally, toggle the switch to enable **Buffering Options** (Preview).
**No {{< img src="observability_pipelines/destinations/cloudprem_settings.png" alt="The CloudPrem destination settings" style="width:35%;" >}} -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- CloudPrem endpoint URL identifier: + - References the intake endpoint to which Observability Pipelines sends logs. + - In your secrets manager: + - Define the cluster URL, such as `http://cloudprem.acme.internal:7280`. **Note**: The URL must include the port. + - The Worker appends `/api/v2/logs` and `/api/v1/validate` to the endpoint URL, so these endpoints must be allowed if you are using forwarding or firewall rules. + - The default identifier is `DESTINATION_CLOUDPREM_ENDPOINT_URL`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{< img src="observability_pipelines/destinations/cloudprem_env_vars.png" alt="The install page showing the CloudPrem environment variable field" style="width:75%;" >}} @@ -41,6 +57,9 @@ Optionally, toggle the switch to enable **Buffering Options** (Preview).
**No - The Worker appends `/api/v2/logs` and `/api/v1/validate` to the endpoint URL, so these endpoints must be allowed if you are using forwarding or firewall rules. - Stored as the environment variable: `DD_OP_DESTINATION_CLOUDPREM_ENDPOINT_URL`. +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/crowdstrike_ng_siem.md b/content/en/observability_pipelines/destinations/crowdstrike_ng_siem.md index 62d4d6ef5d3..1f3f751b0b1 100644 --- a/content/en/observability_pipelines/destinations/crowdstrike_ng_siem.md +++ b/content/en/observability_pipelines/destinations/crowdstrike_ng_siem.md @@ -18,22 +18,62 @@ Set up the CrowdStrike NG-SIEM destination and its environment variables when yo To use the CrowdStrike NG-SIEM destination, you need to set up a CrowdStrike data connector using the HEC/HTTP Event Connector. See [Step 1: Set up the HEC/HTTP event data connector][3] for instructions. When you set up the data connector, you are given a HEC API key and URL, which you use when you configure the Observability Pipelines Worker later on. +
Only enter the identifiers for the CrowdStrike NG-SIEM endpoint URL, token, and if applicable, the pass key. Do not enter the actual values.
+ +1. Enter the identifier for your CrowdStrike NG-SIEM endpoint URL. If you leave it blank, the [default](#set-secrets) is used. +1. Enter the identifier for your CrowdStrike NG-SIEM token. If you leave it blank, the [default](#set-secrets) is used. 1. Select **JSON** or **Raw** encoding in the dropdown menu. -1. Optionally, enable compressions and select an algorithm (**gzip** or **zlib**) in the dropdown menu. -1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][4] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. - - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - - If left disabled, the maximum size for buffering is 500 events. - - If enabled: - 1. Select the buffer type you want to set (**Memory** or **Disk**). - 1. Enter the buffer size and select the unit. - -### Set the environment variables + +#### Optional settings + +##### Enable compressions + +1. Toggle the switch to **Enable compressions**. +1. Select an algorithm (**gzip** or **zlib**) in the dropdown menu. + +##### Enable TLS + +Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required. + +**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][4] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. + +- Enter the identifier for your CrowdStrike NG-SIEM key pass. If you leave it blank, the [default](#set-secrets) is used. +- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). +- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. + +##### Buffering options + +Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +- If left disabled, the maximum size for buffering is 500 events. +- If enabled: + 1. Select the buffer type you want to set (**Memory** or **Disk**). + 1. Enter the buffer size and select the unit. + +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- CrowdStrike NG-SIEM endpoint URL identifier: + - In your secrets manager, do **not** include the suffix `/services/collector` in the URL. The URL must follow this format: `https://.ingest.us-1.crowdstrike.com`. + - The default identifier is `DESTINATION_CROWDSTRIKE_NEXT_GEN_SIEM_ENDPOINT_URL`. +- CrowdStrike NG-SIEM token identifier: + - The default identifier is `DESTINATION_CROWDSTRIKE_NEXT_GEN_SIEM_TOKEN`. +- CrowdStrike NG-SIEM TLS passphrase identifier (when TLS is enabled): + - The default identifier is `DESTINATION_CROWDSTRIKE_NEXT_GEN_SIEM_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/crowdstrike_ng_siem %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/datadog_logs.md b/content/en/observability_pipelines/destinations/datadog_logs.md index 3ff73cea23e..53ea1f2dbcb 100644 --- a/content/en/observability_pipelines/destinations/datadog_logs.md +++ b/content/en/observability_pipelines/destinations/datadog_logs.md @@ -22,10 +22,24 @@ Set up the Datadog Logs destination and its environment variables when you [set 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +**Note**: If you entered identifiers for yours secrets and then choose to use environment variables, the environment variable is the identifier entered prepended with `DD_OP`. For example, if you entered `PASSWORD_1` for the a password identifier, the environment variable for the password is `DD_OP_PASSWORD_1`. + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +No secret identifiers are required for the Datadog Logs destination. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/datadog %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/datadog_metrics.md b/content/en/observability_pipelines/destinations/datadog_metrics.md index c40b0d21b81..881fe355d97 100644 --- a/content/en/observability_pipelines/destinations/datadog_metrics.md +++ b/content/en/observability_pipelines/destinations/datadog_metrics.md @@ -27,10 +27,24 @@ Optionally, toggle **Buffering Options** to configure how events are buffered be - Select the buffer type you want to set (Memory or Disk). - Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +There are no secret identifiers to configure. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/datadog %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works A batch of events is flushed when one of these parameters is met. See [event batching][2] for more information. diff --git a/content/en/observability_pipelines/destinations/elasticsearch.md b/content/en/observability_pipelines/destinations/elasticsearch.md index 40b07159833..ae86af78716 100644 --- a/content/en/observability_pipelines/destinations/elasticsearch.md +++ b/content/en/observability_pipelines/destinations/elasticsearch.md @@ -16,6 +16,11 @@ Set up the Elasticsearch destination and its environment variables when you [set ### Set up the destination +
Only enter the identifiers for the Elasticsearch endpoint URL, username, and password. Do not enter the actual values.
+ +1. Enter the identifier for your Elasticsearch endpoint URL. If you leave it blank, the [default](#set-secrets) is used. + (#set-secrets) is used. +1. Enter the identifier for your Elasticsearch password. If you leave it blank, the [default](#set-secrets) is used. 1. In the **Mode** dropdown menu, select **Bulk** or **Data streams**. - **Bulk** mode - Uses Elasticsearch's [Bulk API][5] to send batched events directly into a standard index. @@ -32,16 +37,35 @@ Set up the Elasticsearch destination and its environment variables when you [set - In the UI, there is a preview of the data stream name you configured. With the above example inputs, the data stream name that the Worker writes to is `logs-apache-production`. 1. Enter the name for the Elasticsearch index. See [template syntax][3] if you want to route logs to different indexes based on specific fields in your logs. 1. Enter the Elasticsearch version. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +1. (Optional) Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - If left disabled, the maximum size for buffering is 500 events. - If enabled: 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Elasticsearch endpoint URL identifier: + - The default identifier is `DESTINATION_ELASTICSEARCH_ENDPOINT_URL`. +- Elasticsearch authentication username identifier: + - The default identifier is `DESTINATION_ELASTICSEARCH_USERNAME`. +- Elasticsearch authentication password identifier: + - The default identifier is `DESTINATION_ELASTICSEARCH_PASSWORD`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/elasticsearch %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/google_chronicle.md b/content/en/observability_pipelines/destinations/google_chronicle.md index 14bc9b5e9bc..678ada13e16 100644 --- a/content/en/observability_pipelines/destinations/google_chronicle.md +++ b/content/en/observability_pipelines/destinations/google_chronicle.md @@ -20,6 +20,8 @@ Set up the Google Chronicle destination and its environment variables when you [ To set up the Worker's Google Chronicle destination: +1. Enter the identifier for your Google Chronicle endpoint URL. If you leave it blank, the [default](#set-secrets) is used. + - **Note**: Only enter the identifier for the endpoint URL. Do **not** enter the actual URL. 1. Enter the customer ID for your Google Chronicle instance. 1. If you have a credentials JSON file, enter the path to your credentials JSON file. The credentials file must be placed under `DD_OP_DATA_DIR/config`. Alternatively, you can use the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to provide the credential path. - If you're using [workload identity][6] on Google Kubernetes Engine (GKE), the `GOOGLE_APPLICATION_CREDENTIALS` is provided for you. @@ -34,10 +36,25 @@ To set up the Worker's Google Chronicle destination: **Note**: Logs sent to the Google Chronicle destination must have ingestion labels. For example, if the logs are from a A10 load balancer, it must have the ingestion label `A10_LOAD_BALANCER`. See Google Cloud's [Support log types with a default parser][5] for a list of available log types and their respective ingestion labels. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Google Chronicle endpoint URL identifier: + - The default identifier is `DESTINATION_GOOGLE_CHRONICLE_UNSTRUCTURED_ENDPOINT_URL`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/chronicle %}} +{{% /tab %}} +{{< /tabs >}} + ### How the destination works #### Event batching diff --git a/content/en/observability_pipelines/destinations/google_cloud_storage.md b/content/en/observability_pipelines/destinations/google_cloud_storage.md index 63b4976ac4b..6865004d34b 100644 --- a/content/en/observability_pipelines/destinations/google_cloud_storage.md +++ b/content/en/observability_pipelines/destinations/google_cloud_storage.md @@ -45,10 +45,24 @@ Set up the Google Cloud Storage destination and its environment variables when y 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +**Note**: If you entered identifiers for yours secrets and then choose to use environment variables, the environment variable is the identifier entered prepended with `DD_OP`. For example, if you entered `PASSWORD_1` for the a password identifier, the environment variable for the password is `DD_OP_PASSWORD_1`. + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +There are no secret identifiers to configure. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/datadog_archives_google_cloud_storage %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/google_pubsub.md b/content/en/observability_pipelines/destinations/google_pubsub.md index 1f0d8650fc8..156d9d10176 100644 --- a/content/en/observability_pipelines/destinations/google_pubsub.md +++ b/content/en/observability_pipelines/destinations/google_pubsub.md @@ -99,8 +99,8 @@ Set up the Google Pub/Sub destination and its environment variables when you [se #### Optional settings - Toggle the switch to **Enable TLS** if your organization requires secure connections with custom certificates. - - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). + - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). + - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. - Toggle the switch to enable **Buffering Options** (Preview).
**Note**: Contact your account manager to request access to the Preview. - If disabled (default): Up to 500 events are buffered before flush. @@ -113,7 +113,23 @@ Set up the Google Pub/Sub destination and its environment variables when you [se {{< img src="observability_pipelines/destinations/google_pubsub_settings.png" alt="The google pub/sub destination with sample values" style="width:30%;" >}} -### Set environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- (Optional) Google Pub/Sub endpoint URL identifier: + - By default the Worker sends data to the global endpoint: `https://pubsub.googleapis.com`. + - If your Pub/Sub topic is region-specific, configure the Google Pub/Sub alternative endpoint URL with the regional endpoint. See [About Pub/Sub endpoints][10240] for more information. Enter the configured endpoint URL into your secrets manager. + - The default identifier is `DESTINATION_GCP_PUBSUB_ENDPOINT_URL`. +- Google Pub/Sub TLS passphrase identifier (when TLS is enabled): + - The default identifier is `DESTINATION_GCP_PUBSUB_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} #### Optional alternative Pub/Sub endpoints @@ -121,6 +137,9 @@ Set up the Google Pub/Sub destination and its environment variables when you [se {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/google_pubsub %}} +{{% /tab %}} +{{< /tabs >}} + ## Troubleshooting Common issues and fixes: diff --git a/content/en/observability_pipelines/destinations/http_client.md b/content/en/observability_pipelines/destinations/http_client.md index 73955c889c0..4776c938ae4 100644 --- a/content/en/observability_pipelines/destinations/http_client.md +++ b/content/en/observability_pipelines/destinations/http_client.md @@ -16,25 +16,71 @@ Use Observability Pipelines' HTTP Client destination to send logs to an HTTP cli Set up the HTTP Client destination and its environment variables when you [set up a pipeline][1]. The information below is configured in the pipelines UI. -1. Select your authorization strategy (**None**, **Basic**, or **Bearer**). +
Only enter the identifiers for the HTTP Client URI and, if applicable, username and password for basic authorization. Do not enter the actual values.
+ +1. Enter the identifier for your HTTP Client URI. If you leave it blank, the [default](#set-secrets) is used. +1. Select your authorization strategy (**None**, **Basic**, or **Bearer**). If you selected: + - **Basic**: + - Enter the identifier for your HTTP Client username. If you leave it blank, the [default](#set-secrets) is used. + - Enter the identifier for your HTTP Client password. If you leave it blank, the [default](#set-secrets) is used. + - **Bearer**: + - Enter the identifier for your HTTP Client token. If you leave it blank, the [default](#set-secrets) is used. 1. JSON is the only available encoder. -1. Optionally, toggle the switch to enable compression. If enabled: - 1. GZIP is the only available compression algorithm. - 1. Select the compression level you want to use. -1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required: - - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - - If left disabled, the maximum size for buffering is 500 events. - - If enabled: - 1. Select the buffer type you want to set (**Memory** or **Disk**). - 1. Enter the buffer size and select the unit. - -## Set the environment variables + +### Optional settings + +#### Enable compression + +Toggle the switch to **Enable Compression**. If enabled: +1. GZIP is the only available compression algorithm. +1. Select the compression level you want to use. + +#### Enable TLS + +Toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required: +- Enter the identifier for your HTTP Client key pass. If you leave it blank, the [default](#set-secrets) is used. + - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. +- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). +- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. + +#### Buffering options + +Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +- If left disabled, the maximum size for buffering is 500 events. +- If enabled: + 1. Select the buffer type you want to set (**Memory** or **Disk**). + 1. Enter the buffer size and select the unit. + +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- HTTP Client URI endpoint identifier: + - The default identifier is `DESTINATION_HTTP_CLIENT_URI`. +- HTTP Client TLS passphrase identifier (when TLS is enabled): + - The default identifier is `DESTINATION_HTTP_CLIENT_KEY_PASS`. +- If you are using basic authentication: + - HTTP Client username identifier: + - The default identifier is `DESTINATION_HTTP_CLIENT_USERNAME`. + - HTTP Client password identifier: + - The default identifier is `DESTINATION_HTTP_CLIENT_PASSWORD`. +- If you are using bearer authentication: + - HTTP Client bearer token identifier: + - The default identifier is `DESTINATION_HTTP_CLIENT_BEARER_TOKEN`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/http_client %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/kafka.md b/content/en/observability_pipelines/destinations/kafka.md index 906fda42dcb..4cd3fd47192 100644 --- a/content/en/observability_pipelines/destinations/kafka.md +++ b/content/en/observability_pipelines/destinations/kafka.md @@ -40,8 +40,8 @@ Set up the Kafka destination and its environment variables when you [set up a pi ##### Enable TLS Toggle the switch to enable **TLS**. The following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][6] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. ##### Enable SASL authentication @@ -83,12 +83,36 @@ Click **Advanced** if you want to set any of the following fields: 1. Check your values against the [librdkafka documentation][7] to make sure they have the correct type and are within the set range. 1. Click **Add Option** to add another librdkafka option. -### Set environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Kafka bootstrap servers identifier: + - References the bootstrap server that the client uses to connect to the Kafka cluster and discover all the other hosts in the cluster. + - In your secrets manager, the host and port must be entered in the format of `host:port`, such as `10.14.22.123:9092`. If there is more than one server, use commas to separate them. + - The default identifier is `DESTINATION_KAFKA_BOOTSTRAP_SERVERS`. +- Kafka TLS passphrase identifier (when TLS is enabled): + - The default identifier is `DESTINATION_KAFKA_KEY_PASS`. +- SASL authentication (when enabled): + - Kafka SASL username identifier: + - The default identifier is `DESTINATION_KAFKA_SASL_USERNAME`. + - Kafka SASL password identifier: + - The default identifier is `DESTINATION_KAFKA_SASL_PASSWORD`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{< img src="observability_pipelines/destinations/kafka_env_var.png" alt="The install page showing the Kafka environment variable field" style="width:70%;" >}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/kafka %}} +{{% /tab %}} +{{< /tabs >}} + ## librdkafka options These are the available librdkafka options: diff --git a/content/en/observability_pipelines/destinations/microsoft_sentinel.md b/content/en/observability_pipelines/destinations/microsoft_sentinel.md index 24acef0b962..4f0f88786ba 100644 --- a/content/en/observability_pipelines/destinations/microsoft_sentinel.md +++ b/content/en/observability_pipelines/destinations/microsoft_sentinel.md @@ -65,6 +65,10 @@ The table below summarizes the Azure and Microsoft Sentinel information you need To set up the Microsoft Sentinel destination in Observability Pipelines: +
Only enter the identifiers for the Microsoft Sentinel client secret and Data Collection Endpoint. Do not enter the actual values.
+ +1. Enter the identifier for your Microsoft Sentinel client secret. If you leave it blank, the [default](#set-secrets) is used. +1. Enter the identifier for your Microsoft Sentinel Data Collection endpoint. If you leave it blank, the [default](#set-secrets) is used. 1. Enter the client ID for your application, such as `550e8400-e29b-41d4-a716-446655440000`. 1. Enter the directory ID for your tenant, such as `72f988bf-86f1-41af-91ab-2d7cd011db47`. This is the Azure AD tenant ID. 1. Enter the full table name to which you are sending logs. An example table name: `Custom-MyOPWLogs_CL`. @@ -75,10 +79,29 @@ To set up the Microsoft Sentinel destination in Observability Pipelines: 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Microsoft Sentinel client secret identifier: + - References the DCE endpoint URL shown as the **Logs Ingestion Endpoint** or **Data Collection Endpoint** on the DCR Overview page. An example URL: `https://.ingest.monitor.azure.com`. + - The default identifier is `DESTINATION_MICROSOFT_SENTINEL_CLIENT_SECRET`. +- Microsoft Sentinel Data Collection endpoint identifier: + - References the Azure AD application's client secret, such as `550e8400-e29b-41d4-a716-446655440000`. + - The default identifier is `DESTINATION_MICROSOFT_SENTINEL_DCE_URI`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/microsoft_sentinel %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/new_relic.md b/content/en/observability_pipelines/destinations/new_relic.md index dd6b1493220..d82cb5eb08f 100644 --- a/content/en/observability_pipelines/destinations/new_relic.md +++ b/content/en/observability_pipelines/destinations/new_relic.md @@ -16,6 +16,10 @@ Set up the New Relic destination and its environment variables when you [set up ### Set up the destination +
Only enter the identifiers for the account ID and license. Do not enter the actual values.
+ +1. Enter the identifier for your account ID. If you leave it blank, the [default](#set-secrets) is used. +1. Enter the identifier for your license. If you leave it blank, the [default](#set-secrets) is used. 1. Select the data center region (**US** or **EU**) of your New Relic account. 1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - If left disabled, the maximum size for buffering is 500 events. @@ -23,10 +27,27 @@ Set up the New Relic destination and its environment variables when you [set up 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- New Relic account ID identifier: + - The default identifier is `DESTINATION_NEW_RELIC_ACCOUNT_ID`. +- New Relic license identifier: + - The default identifier is `DESTINATION_NEW_RELIC_LICENSE_KEY`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/new_relic %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/opensearch.md b/content/en/observability_pipelines/destinations/opensearch.md index 2c4bea3d3a7..c404a8cf822 100644 --- a/content/en/observability_pipelines/destinations/opensearch.md +++ b/content/en/observability_pipelines/destinations/opensearch.md @@ -16,17 +16,41 @@ Set up the OpenSearch destination and its environment variables when you [set up ### Set up the destination -1. Optionally, enter the name of the OpenSearch index. See [template syntax][3] if you want to route logs to different indexes based on specific fields in your logs. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +
Only enter the identifiers for the OpenSearch endpoint URL, username, and password. Do not enter the actual values.
+ +1. Enter the identifier for your OpenSearch endpoint URL. If you leave it blank, the [default](#set-secrets) is used. +1. Enter the identifier for your OpenSearch username. If you leave it blank, the [default](#set-secrets) is used. +1. Enter the identifier for your OpenSearch password. If you leave it blank, the [default](#set-secrets) is used. +1. (Optional) Enter the name of the OpenSearch index. See [template syntax][3] if you want to route logs to different indexes based on specific fields in your logs. +1. (Optional) Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - If left disabled, the maximum size for buffering is 500 events. - If enabled: 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- OpenSearch endpoint URL identifier: + - The default identifier is `DESTINATION_OPENSEARCH_ENDPOINT_URL`. +- OpenSearch authentication username identifier: + - The default identifier is `DESTINATION_OPENSEARCH_USERNAME`. +- OpenSearch authentication password identifier: + - The default identifier is `DESTINATION_OPENSEARCH_PASSWORD`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opensearch %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/sentinelone.md b/content/en/observability_pipelines/destinations/sentinelone.md index 15943cd72b9..9aa35c2a955 100644 --- a/content/en/observability_pipelines/destinations/sentinelone.md +++ b/content/en/observability_pipelines/destinations/sentinelone.md @@ -20,6 +20,9 @@ Set up the SentinelOne destination and its environment variables when you [set u ### Set up the destination +
Only enter the identifier for the token. Do not enter the actual value.
+ +1. Enter the identifier for your token. If you leave it blank, the [default](#set-secrets) is used. 1. Select your SentinelOne logs environment in the dropdown menu. 1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - If left disabled, the maximum size for buffering is 500 events. @@ -27,10 +30,25 @@ Set up the SentinelOne destination and its environment variables when you [set u 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- SentinelOne write access token identifier: + - The default identifier is `DESTINATION_SENTINEL_ONE_TOKEN`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/sentinelone %}} +{{% /tab %}} +{{< /tabs >}} + ## View logs in a SentinelOne cluster After you've set up the pipeline to send logs to the SentinelOne destination, you can view the logs in a SentinelOne cluster: diff --git a/content/en/observability_pipelines/destinations/socket.md b/content/en/observability_pipelines/destinations/socket.md index 041f758fb87..5f9ca077cb9 100644 --- a/content/en/observability_pipelines/destinations/socket.md +++ b/content/en/observability_pipelines/destinations/socket.md @@ -16,22 +16,53 @@ Set up the Socket destination and its environment variables when you [set up a p ### Set up the destination +
Only enter the identifier for the socket address and, if appliable, the key pass. Do not enter the actual values.
+ +1. Enter the identifier for your address. If you leave it blank, the [default](#set-secrets) is used. 1. In the **Mode** dropdown menu, select the socket type to use. 1. In the **Encoding** dropdown menu, select either `JSON` or `Raw message` as the output format. -1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required: - - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - - If left disabled, the maximum size for buffering is 500 events. - - If enabled: - 1. Select the buffer type you want to set (**Memory** or **Disk**). - 1. Enter the buffer size and select the unit. - -### Set the environment variables + +#### Optional settings + +##### Enable TLS + +If you enabled **TCP** mode, you can toggle the switch to **Enable TLS**. The following certificate and key files are required for TLS: +- Enter the identifier for your socket key pass. If you leave it blank, the [default](#set-secrets) is used. + +- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). +- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. + +#### Buffering options + +Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +- If left disabled, the maximum size for buffering is 500 events. +- If enabled: + 1. Select the buffer type you want to set (**Memory** or **Disk**). + 1. Enter the buffer size and select the unit. + +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Socket address identifier: + - References the address to which the Observability Pipelines Worker sends processed logs. + - The default identifier is `DESTINATION_SOCKET_ADDRESS`. +- Socket TLS passphrase identifier (when TLS is enabled): + - The default identifier is `DESTINATION_SOCKET_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/socket %}} +{{% /tab %}} +{{< /tabs >}} + ### How the destination works #### Event batching diff --git a/content/en/observability_pipelines/destinations/splunk_hec.md b/content/en/observability_pipelines/destinations/splunk_hec.md index 944741743c1..7d0ba50bc96 100644 --- a/content/en/observability_pipelines/destinations/splunk_hec.md +++ b/content/en/observability_pipelines/destinations/splunk_hec.md @@ -16,9 +16,13 @@ Set up the Splunk HEC destination and its environment variables when you [set up ### Set up the destination -
Observability Pipelines compresses logs with the gzip (level 6) algorithm.
+
Observability Pipelines compresses logs with the gzip (level 6) algorithm.
Only enter the identifiers for the Splunk HEC token and endpoint. Do not enter the actual values.
+ +1. Enter the identifier for your token. If you leave it blank, the [default](#set-secrets) is used. +1. Enter the identifier for your endpoint URL. If you leave it blank, the [default](#set-secrets) is used. + +#### Optional settings -The following fields are optional: 1. Enter the name of the Splunk index you want your data in. This has to be an allowed index for your HEC. See [template syntax][3] if you want to route logs to different indexes based on specific fields in your logs. 1. Select whether the timestamp should be auto-extracted. If set to `true`, Splunk extracts the timestamp from the message with the expected format of `yyyy-mm-dd hh:mm:ss`. 1. Optionally, set the `sourcetype` to override Splunk's default value, which is `httpevent` for HEC data. See [template syntax][3] if you want to route logs to different source types based on specific fields in your logs. @@ -28,10 +32,30 @@ The following fields are optional: 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Splunk HEC token identifier: + - References the Splunk HEC token for the Splunk indexer. + - The default identifier is `DESTINATION_SPLUNK_HEC_TOKEN`. +- Splunk HEC endpoint URL identifier: + - References the Splunk HTTP Event Collector endpoint your Observability Pipelines Worker sends processed logs to. For example, `https://hec.splunkcloud.com:8088`. + - **Note**: `/services/collector/event` path is automatically appended to the endpoint. + - The default identifier is `DESTINATION_SPLUNK_HEC_ENDPOINT_URL`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/splunk_hec %}} +{{% /tab %}} +{{< /tabs >}} + ### How the destination works #### Event batching diff --git a/content/en/observability_pipelines/destinations/sumo_logic_hosted_collector.md b/content/en/observability_pipelines/destinations/sumo_logic_hosted_collector.md index 94049ef8e04..418df3407a8 100644 --- a/content/en/observability_pipelines/destinations/sumo_logic_hosted_collector.md +++ b/content/en/observability_pipelines/destinations/sumo_logic_hosted_collector.md @@ -16,7 +16,11 @@ Set up the Sumo Logic destination and its environment variables when you [set up ### Set up the destination -The following fields are optional: +
Only enter the identifier for the Sumo Logic endpoint URL. Do not enter the actual value.
+ +- Enter the identifier for your endpoint URL. If you leave it blank, the [default](#set-secrets) is used. + +#### Optional settings 1. In the **Encoding** dropdown menu, select whether you want to encode your pipeline's output in `JSON`, `Logfmt`, or `Raw` text. If no decoding is selected, the decoding defaults to JSON. 1. Enter a **source name** to override the default `name` value configured for your Sumo Logic collector's source. @@ -29,10 +33,28 @@ The following fields are optional: 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. -### Set the environment variables +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Sumo Logic HTTP Collector URL identifier: + - References the Sumo Logic HTTP Source endpoint. The Observability Pipelines Worker sends processed logs to this endpoint. For example, `https://.collection.sumologic.com/receiver/v1/http/`, where: + - `` is your Sumo collection endpoint. + - `` is the string that follows the last forward slash (`/`) in the upload URL for the HTTP source. + - The default identifier is `DESTINATION_SUMO_LOGIC_HTTP_COLLECTOR_URL`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/sumo_logic %}} +{{% /tab %}} +{{< /tabs >}} + ## How the destination works ### Event batching diff --git a/content/en/observability_pipelines/destinations/syslog.md b/content/en/observability_pipelines/destinations/syslog.md index 715c6df4211..217646ffc02 100644 --- a/content/en/observability_pipelines/destinations/syslog.md +++ b/content/en/observability_pipelines/destinations/syslog.md @@ -31,23 +31,56 @@ The rsyslog and syslog-ng destinations match these log fields to the following S | log["host"] | HOSTNAME | `NIL` | | log["timestamp"]| TIMESTAMP | Current UTC time. | -The following destination settings are optional: +To set up the syslog destination: -1. Toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required: - - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). - - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -1. Enter the number of seconds to wait before sending TCP keepalive probes on an idle connection. -1. Optionally, toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. - - If left disabled, the maximum size for buffering is 500 events. - - If enabled: - 1. Select the buffer type you want to set (**Memory** or **Disk**). - 1. Enter the buffer size and select the unit. +
Only enter the identifiers for the syslog endpoint URL and, if applicable, the key pass. Do not enter the actual values.
-### Set the environment variables +- Enter the identifier for your endpoint URL. If you leave it blank, the [default](#set-secrets) is used. + +#### Optional settings + +##### Enable TLS + +Toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required: +- Enter the identifier for your syslog key pass. If you leave it blank, the [default](#set-secrets) is used. +- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). +- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. + +##### Wait time for TCP keepalive probes + +Enter the number of seconds to wait before sending TCP keepalive probes on an idle connection. + +##### Buffering options + +Toggle the switch to enable **Buffering Options**.
**Note**: Buffering options is in Preview. Contact your account manager to request access. +- If left disabled, the maximum size for buffering is 500 events. +- If enabled: + 1. Select the buffer type you want to set (**Memory** or **Disk**). + 1. Enter the buffer size and select the unit. + +### Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- rsyslog or syslog-ng endpoint URL identifier: + - References address and port to which Observability Pipelines Worker sends logs. For example, `127.0.0.1:9997`. + - The default identifier is `DESTINATION_SYSLOG_ENDPOINT_URL`. +- rsyslog or syslog-ng TLS passphrase identifier (when TLS is enabled): + - The default identifier is `DESTINATION_SYSLOG_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/syslog %}} +{{% /tab %}} +{{< /tabs >}} + ### How the destination works #### Event batching diff --git a/content/en/observability_pipelines/sources/kafka.md b/content/en/observability_pipelines/sources/kafka.md index b4a7096e189..2093a00a2e1 100644 --- a/content/en/observability_pipelines/sources/kafka.md +++ b/content/en/observability_pipelines/sources/kafka.md @@ -20,7 +20,7 @@ You can also [send Azure Event Hub logs to Observability Pipelines using the Kaf Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -
Only enter the identifiers for the Kafka servers, username, and password. Do not enter the actual values.
+
Only enter the identifiers for the Kafka servers, username, and password. Do not enter the actual values.
1. Enter the identifier for your Kafka servers. - If left blank, the default is used: `SOURCE_KAFKA_BOOTSTRAP_SERVERS`.