Understanding datadog_agent source internal metrics and logs

[techministrator]

Question

Hi vector team and everyone,

May I check and learn more about the internal metrics and logs related to datadog_agent source (ref).

A small context is that I've just recently rolled out the Vector to receive and handle a large amount of metrics traffic from the internal datadog agents (configuring using the observability_pipelines_worker option).

I've also configured quite an extensive dashboards to monitor vector and noticed periodic metric data of component_discarded_events_total with intentional tag/label as false (ref). Usually accompanying by:

• an internal ERROR log of Events dropped at datadog_agent source
• some internal WARN logs of HTTP error and Retrying after error (with some suppressions) at datadog_metrics sink
• memory buffer and usage increased expectedly
• sometimes Adaptive Concurrency RTT gets worse/spiked as well

So I'm trying to understand what was the real reason of the Events dropped issue in this case. In documentation, it just mentions there's due to errors but is it errors from Vector Sink or is it from the Source Datadog Agents in this case? Metrics Events dropped means all the data is permanently lost because I don't enable acknowledgement feature? Any recommendations to mitigate or avoid this issue?

I put some of the most important configurations below to avoid too much configs. Generally, I don't see any errors or discarded events at the transforms stage thus I remove it from this context to avoid noise.

Thank you for spending the time, I know this is a lot of context.

Vector Config

There are many context for the below configs though I'm open for recommendations:

• An average metric event size in our system is 4-5KB thus I have to decrease the max batch events to improve RTT from Datadog API Endpoint. If keeping the default size, it would take >30s and surely dropped along the way.
• Memory events buffer also is being used the maximum usually go up to 70-80% with this 400,000 events config
• Concurrency settings I tried to make it scale out as fast and as many as possible since this is Datadog Platform. They don't limit metrics submissions in their endpoint (ref). But this doesn't prevent 500 error msg returned sometimes and decrease concurrency (though back up quite fast).
• Also due to the above reasons, I decrease max backoff durations.

data_dir: "/var/lib/vector"

sources:
  datadog_agents:
    type: datadog_agent
    address: 0.0.0.0:9000
    disable_logs: true
    disable_traces: true
    store_api_key: true

sinks:
  datadog_platform:
    inputs:
      - datadog_agents
    type: datadog_metrics
    default_api_key: "${DD_API_KEY:?missing DD_API_KEY as default}"
    buffer:
      - type: memory
        max_events: 400000
        when_full: block
    batch:
      max_events: 1000
      timeout_secs: 1
    request:
      retry_attempts: 10
      retry_initial_backoff_secs: 1
      retry_max_duration_secs: 10
      retry_jitter_mode: Full
      timeout_secs: 60
      concurrency: adaptive
      adaptive_concurrency: 
        initial_concurrency: 600
        max_concurrency_limit: 1500
        decrease_ratio: 1.0
        rtt_deviation_scale: 3.0
        ewma_alpha: 0.4

Vector Logs

{
	"id": <obscured>
	"content": {
		"timestamp": "2025-10-01T10:52:09.060Z",
		"tags": [
			"source:undefined",
			"datadog.submission_auth:api_key",
			"datadog.api_key_uuid:504665cb-9c53-496d-b3c6-552e2eb73468"
		],
		"host": "<obscured>",
		"service": "<obscured>",
		"message": "Events dropped",
		"attributes": {
			"reason": "Source send cancelled.",
			"metadata": {
				"kind": "event",
				"module_path": "vector_common::internal_event::component_events_dropped",
				"target": "vector_common::internal_event::component_events_dropped"
			},
			"intentional": false,
			"count": 3044,
			"internal_log_rate_limit": true,
			"pid": 998,
			"source_type": "internal_logs",
			"env": "staging",
			"hostname": "<obscured>",
			"appname": "<obscured>",
			"service": "<obscured>",
			"vector": {
				"component_type": "datadog_agent",
				"component_id": "datadog_agents",
				"component_kind": "source"
			},
			"status": "ERROR",
			"timestamp": 1759315929060
		}
	}
}