System.InvalidOperationException: The metric size exceeds the internal buffer capacity 512 observed frequently in ASP.NET app with Datadog tracing

[ansonpinhero]

Tracer Version(s)

3.15.0.0

Operating system and platform

Linux X64

TFM

NET8.0

Description

While investigating high CPU usage in an Azure Web App (containerized ASP.NET Core application) instrumented with Datadog, we observed a large number of exceptions being thrown within a short period.

During a 10-minute trace capture, the following exception occurred over 300k times:

| Throw(System.InvalidOperationException) The metric size exceeds the internal buffer capacity 512: runtime.dotnet.gc.pause_time:6.8234|ms|#lang:.NET,lang_interpreter:.NET,lang_version:8.0.20,tracer_version:3.15.0.0

Environment

App type: ASP.NET Core Web App

Hosting: Azure Web App (Containerized)

.NET Runtime: .NET 8.0.20

Datadog Tracer Version: 3.15.0.0

[pablomartinezbernardo]

Thank you for the report. Could you open an issue with our support team linking to this GH issue sharing your full logs?

[ansonpinhero]

@pablomartinezbernardo I've opened an issue: Request #2250393

[pablomartinezbernardo]

Having the debug logs will help us determine if this is actually the case, it looks like this might be related to the number of tags in the metric.

[simon-trup]

We have experienced extremely high CPU in our pods after upgrading from v3.25.0 to v3.27.0. We will attempt to collect debug logs to confirm asap.

Our setup is dontnet core services running on Linux containers in AKS.