Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 143 additions & 1 deletion src/content/docs/distributed-tracing/ui-data/trace-details.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -347,12 +347,154 @@ Check out ways to display your traces' spans:

When you select a span, a pane opens up with span details. These details can be helpful for troubleshooting performance issues. This page has three tabs:

* <DNT>**Performance**</DNT>: Youll see charts showing the average duration and throughput for that span operation, as well as how the performance of this specific span compares to the average.
* <DNT>**Performance**</DNT>: You'll see charts showing the average duration and throughput for that span operation, as well as how the performance of this specific span compares to the average.
* <DNT>**Attributes**</DNT>: You can learn more about attributes from our [data dictionary](/attribute-dictionary/?dataSource=Distributed+Tracing&event=Span).
* <DNT>**Details**</DNT>: You can view details, such as the [span's full name](/docs/distributed-tracing/ui-data/understand-use-distributed-tracing-ui/#prettified-span-names) and data source.
* <DNT>**Span links**</DNT>: If the span has [span links](#span-links), you'll see this tab. It shows causal relationships with spans from other traces. [Learn more about span links](#span-links).

What a span displays is based on its span type. For example, the datastore span's details will include the database query. For more on the trace structure and how span properties are determined, see [Trace structure](/docs/understand-dependencies/distributed-tracing/get-started/how-new-relic-distributed-tracing-works#trace-structure).

## Understanding span links [#span-links]

Span links help you understand causal relationships between spans across different traces. When traces get split due to asynchronous communication or long-running processes, span links allow you to navigate between related traces and see the complete picture of your distributed system.

### When traces get split [#why-traces-split]

In modern distributed systems, traces can appear as separate, disconnected traces in the following scenarios:

* <DNT>**Asynchronous message queues**</DNT>: A service publishes messages to a queue (like AWS SQS, RabbitMQ, or Kafka), and another service consumes them later. The producing trace and consuming trace are separate, but causally related.
* <DNT>**Long-running workflows**</DNT>: Background jobs or workflows that exceed trace duration limits or run for extended periods.
* <DNT>**Batch processing**</DNT>: Workers that process multiple messages in batches, where each message originated from a different trace.
* <DNT>**Fan-in patterns**</DNT>: Multiple services independently produce messages that one consumer aggregates. Each producer creates its own trace, and the consumer creates a new trace that processes all of them.

When traces are split like this, traditional parent-child relationships don't capture the full picture. Span links fill this gap by creating explicit causal connections.

### What are span links? [#what-are-span-links]

Span links are OpenTelemetry attributes that create causal relationships between spans that don't have a direct parent-child connection. They allow you to:

* Navigate from a consuming span back to the producing span
* Understand the complete request flow even when traces are split
* Debug issues that span across asynchronous boundaries
* Visualize dependencies in event-driven architectures

A span link includes the trace ID and span ID of the related span, allowing New Relic to connect the traces for you.

### Identifying traces with span links [#identify-span-links]

In the trace details UI, traces with span links are marked in several ways:

1. <DNT>**Span links filter badge**</DNT>: In the trace filter bar, you'll see a badge showing the number of spans with links (for example, `span links (2)`).
2. <DNT>**Span count indicator**</DNT>: When viewing the list of spans, each span shows a link count if it has span links.

<img
title="Distributed tracing span links filter"
alt="Screenshot showing the span links filter badge in the trace details UI"
src="/images/distributed-tracing-span-links-filter-badge.webp"
/>

<figcaption>
The span links filter badge appears in the trace filter bar when a trace contains spans with links.
</figcaption>

### Viewing span links [#view-span-links]

To view and navigate span links:

1. From the trace details page, select a span that has span links (look for the span links indicator).
2. In the span details pane, click the <DNT>**Span links**</DNT> tab.
3. You'll see a list of linked traces with the following information:
* <DNT>**Direction badge**</DNT>: `Forward` indicates a span that occurred later in time (successor), while `Backward` indicates a span that occurred earlier (predecessor).
* <DNT>**Trace ID**</DNT>: The ID of the linked trace (truncated for display).
* <DNT>**Timestamp**</DNT>: When the linked span occurred.
* <DNT>**Duration**</DNT>: How long the linked span took.
* <DNT>**Errors**</DNT>: Number of errors in the linked trace.

4. Click on a linked trace to navigate to it. Use your browser's back button to return to the original trace.

<img
title="Span links tab showing linked traces"
alt="Screenshot showing the Span links tab with forward and backward linked traces"
src="/images/distributed-tracing-span-links-tab-detail.webp"
/>

<figcaption>
The Span links tab displays linked traces with direction badges (Forward/Backward), trace IDs, timestamps, durations, and error counts.
</figcaption>

<Callout variant="tip">
<DNT>**Understanding forward vs. backward links**</DNT>

* <DNT>**Forward links**</DNT>: Point to traces that happened as a result of the current span. Example: A message was published by this span and processed later by another trace.
* <DNT>**Backward links**</DNT>: Point to traces that caused the current span. Example: This span is processing a message that was published by an earlier trace.
</Callout>

### When span links don't appear [#span-links-troubleshooting]

If you expect to see span links but they're not appearing, check the following:

<img
title="Empty state when trace has no span links"
alt="Screenshot showing the empty state message when a trace has no span links"
src="/images/distributed-tracing-span-links-empty-state.webp"
/>

<figcaption>
When a trace has no span links, you'll see this message in the Span links tab.
</figcaption>

<CollapserGroup>
<Collapser
id="trace-context-not-propagated"
title="Trace context not propagated"
>
Span links require proper trace context propagation through your messaging system:

* Ensure W3C trace context headers (`traceparent`, `tracestate`) are included in message headers or metadata
* Verify your messaging library or framework supports context propagation
* When instrumenting manually, make sure you're extracting context from messages and creating span links programmatically

For implementation examples, see [OpenTelemetry span links best practices](/docs/opentelemetry/best-practices/opentelemetry-best-practices-traces/#span-links).
</Collapser>

<Collapser
id="cross-account-visibility"
title="Cross-account visibility issues"
>
If your linked traces are in different New Relic accounts:

* Verify that trusted account IDs are configured correctly for cross-account tracing
* Check that all sub-accounts share the same parent account ID
* Ensure you have access to both accounts

Without proper cross-account configuration, you may see span link data but won't be able to navigate to the linked trace.
</Collapser>

<Collapser
id="sampling"
title="Sampling considerations"
>
Both the original trace and the linked trace must be sampled for span links to appear:

* If either trace is dropped by sampling, the span link won't be visible
* Consider using head-based or tail-based sampling strategies that preserve important traces
* For critical workflows with span links, you may want to adjust sampling rates
</Collapser>

<Collapser
id="data-processing"
title="Data not yet processed"
>
There may be a delay between when trace data is sent and when span links appear:

* Allow 1-2 minutes for data to be processed and indexed
* Check that both linked traces have been successfully ingested
* Verify that you're not hitting any data limits or rate limits
</Collapser>
</CollapserGroup>

For more information on implementing span links in your instrumentation, see our [OpenTelemetry best practices guide for traces](/docs/opentelemetry/best-practices/opentelemetry-best-practices-traces/#span-links).

## Span attributes [#span-attributes]

If you'd like to learn more about `Span` data:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@ If you're looking to troubleshoot errors in a transaction that spans many servic
3. On the [trace details page](/docs/distributed-tracing/ui-data/trace-details/), review the span along the request route that originated the error.
4. Noting the error class and message, navigate to the service from its span in the trace so you can see that the error is occurring at a high rate.

<Callout variant="tip">
<DNT>**Working with asynchronous traces?**</DNT>

If your traces get split across asynchronous boundaries like message queues or event streams, you can use [span links](/docs/distributed-tracing/ui-data/trace-details/#span-links) to navigate between causally related traces. Span links help you understand the complete request flow even when traces are disconnected. Learn more about [understanding span links](/docs/distributed-tracing/ui-data/trace-details/#span-links).
</Callout>

Read on to explore the options in the distributed tracing UI.

## Open the distributed tracing UI [#open-dt-ui]
Expand Down
Loading
Loading