Skip to content

Commit 466dddb

Browse files
committed
Document Azure Functions span parenting issue (APMSVLS-58)
Added detailed documentation of the span parenting issue that occurs with isolated Azure Functions using ASP.NET Core integration. The issue: Worker process spans are incorrectly parented to the root host span instead of the HTTP client span that makes the host-to-worker call. Documentation includes: - Visual comparison of current vs expected span hierarchy - Root cause analysis of context propagation flow - Technical details about HTTP proxying and gRPC integration - Step-by-step reproduction instructions - Example code and API queries for verification - Files that need to be modified for the fix
1 parent 9a9525e commit 466dddb

File tree

1 file changed

+76
-0
lines changed

1 file changed

+76
-0
lines changed

docs/development/AzureFunctions.md

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,82 @@ Whether a Functions app uses this new mode is subtle:
9595

9696
and the project will have a reference to [Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore](https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore/) package.
9797

98+
## Known Issues and Fixes
99+
100+
### Span Parenting Issue with ASP.NET Core Integration (APMSVLS-58)
101+
102+
**Problem:**
103+
When using isolated Azure Functions with ASP.NET Core Integration, spans created in the worker process are incorrectly parented to the root host span instead of the HTTP client span that makes the host→worker call.
104+
105+
**Current (Incorrect) Behavior:**
106+
```
107+
ROOT: azure_functions.invoke: GET /api/httptest [PID 27 - HOST]
108+
├─ http.request: GET localhost:40521/api/HttpTest [HOST → WORKER HTTP call]
109+
└─ azure_functions.invoke: Http HttpTest [PID 56 - WORKER] ❌ WRONG PARENT
110+
├─ test_span [WORKER]
111+
└─ http.request: GET jsonplaceholder... [WORKER]
112+
```
113+
114+
**Expected (Correct) Behavior:**
115+
```
116+
ROOT: azure_functions.invoke: GET /api/httptest [PID 27 - HOST]
117+
└─ http.request: GET localhost:40521/api/HttpTest [HOST → WORKER HTTP call]
118+
└─ azure_functions.invoke: Http HttpTest [PID 56 - WORKER] ✓ CORRECT PARENT
119+
├─ test_span [WORKER]
120+
└─ http.request: GET jsonplaceholder... [WORKER]
121+
```
122+
123+
**Root Cause:**
124+
In `GrpcMessageConversionExtensionsToRpcHttpIntegration.cs`, the code injects the Azure Functions host span context into the gRPC message (line 87-88). However, when HTTP proxying is enabled with ASP.NET Core integration:
125+
126+
1. The host process creates an HTTP client call to the worker process (`GET localhost:40521/api/HttpTest`)
127+
2. The HTTP client instrumentation automatically creates a span for this call
128+
3. The HTTP client automatically propagates its trace context in the HTTP headers
129+
4. However, the gRPC integration has already injected the *parent* span's context (the Azure Functions host span) into the gRPC message
130+
5. The worker process extracts context from both sources, but prefers the gRPC context
131+
6. This causes worker spans to be parented to the wrong span (the host root instead of the HTTP call)
132+
133+
**Technical Details:**
134+
- When `isHttpProxying` is true, `func.exe` makes an actual HTTP call to the worker instead of only using gRPC
135+
- The HTTP client span (e.g., `GET localhost:40521`) is created *after* `ToRpcHttp()` runs
136+
- Currently, `ToRpcHttp()` injects the Azure Functions span context regardless of proxying mode
137+
- The HTTP client instrumentation then injects its own context into HTTP headers
138+
- The worker receives both contexts and needs to choose the correct one
139+
140+
**Solution:**
141+
When HTTP proxying is enabled (`isHttpProxying && !requiresRouteParameters`), we should not inject trace context into the gRPC message. Instead, let the HTTP client instrumentation handle context propagation naturally through HTTP headers. This ensures worker spans are properly parented to the HTTP call span.
142+
143+
**How to Reproduce:**
144+
1. Deploy an isolated Azure Functions app with ASP.NET Core integration
145+
2. Create a function that makes an outbound HTTP call with a custom span:
146+
```csharp
147+
[Function(nameof(HttpTest))]
148+
public async Task<IActionResult> HttpTest([HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequest request)
149+
{
150+
using (var scope = Tracer.Instance.StartActive("test_span"))
151+
{
152+
using var httpClient = new HttpClient();
153+
await httpClient.GetStringAsync("https://jsonplaceholder.typicode.com/users/1");
154+
return new OkObjectResult(new { message = "success" });
155+
}
156+
}
157+
```
158+
3. Trigger the function via HTTP: `curl https://<function-app>.azurewebsites.net/api/HttpTest`
159+
4. Query traces via Datadog API:
160+
```bash
161+
curl -G "https://api.datadoghq.com/api/v2/spans/events" \
162+
--data-urlencode "filter[query]=service:<function-app> resource_name:*HttpTest*" \
163+
--data-urlencode "filter[from]=now-10m" \
164+
--data-urlencode "page[limit]=10" \
165+
-H "DD-API-KEY: <key>" \
166+
-H "DD-APPLICATION-KEY: <app-key>"
167+
```
168+
5. Examine the `parent_id` of worker spans - they will incorrectly point to the root host span
169+
170+
**Files Involved:**
171+
- `tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Azure/Functions/Isolated/GrpcMessageConversionExtensionsToRpcHttpIntegration.cs`
172+
- `tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Azure/Functions/AzureFunctionsCommon.cs`
173+
98174
## Debugging
99175

100176
To debug Azure Functions locally ensure that you have the following:

0 commit comments

Comments
 (0)