Proposal to Adopt Tokio Tracing as the OTel Tracing API

[cijothomas]

Introduction

This issue builds upon Option 2 from the discussion OTel Tracing vs Tokio-Tracing, proposing a strategic shift in our approach to traces in OpenTelemetry Rust.

Summary

We propose deprecating the OpenTelemetry (OTel) Tracing API in favor of adopting Tokio Tracing as the official Tracing API, requiring re-instrumentation for those apps already using OTel Tracing API. The functionalities already provided by the OpenTelemetry Tracing SDK, such as Sampling, SpanProcessors, and Exporters, would remain largely unchanged.

Details

• Default Behavior: tracing macros act as no-ops when no subscriber is configured, consistent with the current OTel API.
• Instrumentation: Users will transition from the OTel API to tracing macros for span generation, requiring re-instrumentation.
• Context Propagation: Users will adopt tracing's methods for in-process context propagation (#instrument attribute or using span.in_scope(closure)), necessitating re-instrumentation. For out-of-process context propagation, OpenTelemetry's abstractions will continue to be used, leveraging the strengths of both tracing for in-process and OpenTelemetry for out-of-process propagation.
• SDK Enablement: Installation of the opentelemetry-sdk crate and configuration of the TracerProvider will remain mostly unchanged.
• Telemetry Backends: No changes to exported telemetry are expected, nor are changes required in telemetry backends or vendors.
• Changes to Existing Crates: Once the OpenTelemetry SDK officially recognizes tracing as the API, there will no longer be a need for tracing-opentelemetry, which currently bridges tracing to OTel. Additionally, the opentelemetry-appender-tracing could be integrated into the OTel SDK itself, allowing users to decide if events from tracing should be converted to SpanEvents or LogRecords resolving issues like this.

Advantages

• Simplification: Reduces the learning curve by consolidating on a single, familiar, and widely adopted tracing mechanism.
• Community Focus: Concentrates community efforts on one primary tracing ecosystem. Without consolidation, libraries and applications in the Rust ecosystem will be forced to choose one over the other. This often leads to abstractions on top of these choices, effectively leading to further fragmentation, as humorously illustrated in this xkcd comic.
• Eliminate interop issues: With only a single choice for instrumentation, an entire class of issues arising from lack of interoperability are eliminated, even resolving issues related to Log Correlation as well.
• Unblocks OTel Rust GA: Resolving this fundamental issue is a prerequisite for progressing OTel Rust towards a GA release.

Challenges

• Adaptation by tracing: Necessary modifications in tracing to support OpenTelemetry scenarios are crucial. An initial set of required changes is listed below.
• Community and Governance Acceptance: Endorsement from the OpenTelemetry Technical and Governance Committees is required for OTel Rust to abandon its Tracing API.
• Migration Help: Since this involves re-instrumentation efforts, we will need to provide the necessary support and guidance (docs etc.) to minimize disruption.
• OTel SDK Refactoring: Significant portions of the OpenTelemetry tracing SDK code need refactoring.
• Deviation from other languages: Since tracing was never written with OTel specs in mind, there would always be some differences compared to other languages OTel based Tracing API. For example, Span macros in tracing has the notion of Level, which is not yet present in OTel.

Required Changes in tracing

1. Introduction of Span Kind, an enum with 5 variants Server,Client,Producer,Consumer,Internal. The tracing-opentelemetry bridge currently handles this through specially named attributes, serving as a short-term workaround.
2. Span links - the follow_from in tracing might be the equivalent, but this requires more prototyping.
3. The notion of "target" in tracing is equivalent to instrumentation scope, and could serve as a reasonable alternative, though it lacks support for version, schema URL, and additional descriptive key-value pairs.
4. tracing macros require attribute keys to be declared at compile time. This is probably manageable as shown by implementations such as reqwest-tracing and axum-tracing-opentelemetry.
5. tracing is not a 1.0 crate yet. This is not necessarily an issue, but OTel Rust and Tokio Tracing should require some co-ordination when planning future releases.

Prototype

A minimal prototype is available here: https://github.com/cijothomas/opentelemetry-tracing/tree/main/src

Please share your thoughts, concerns, and issues we may have overlooked. We recognize that re-instrumentation is a considerable effort and not something anyone looks forward to. However, this small investment now is expected to yield significant long-term benefits, setting OpenTelemetry Rust up for greater success in the future.

[mladedav]

Required Changes in tracing
Introduction of Span Kind

Using span kinds is somewhat niche use case that pretty much only middle wares (be it server or client) really care about.

What I've been thinking about lately was to create tracing-like macros to use these magic strings safely. E.g. something like

otel::info_span!(kind: server, otel_parent: &extracted_context, target: "target", "span_name", ?tracing.field, %another.field);

which would get translated to

tracing::info_span!(target: "target", "span_name", ?tracing.field, %another.field, otel.kind = "server", otel.parent.traceId = &extracted_context.trace_id, otel.parent.spanId = &extracted_context.span_id);

This would guarantee that people don't mistype the names and the macro can also check for correct types.

This can also be used to provide parent context at the time of span creation so that a spans always either have the same from their creation or they're root spans, but it will not change as it can now. I think that caused some issues with pairing contexts to logs in the appender, right?

[cijothomas]

Using span kinds is somewhat niche use case that pretty much only middle wares (be it server or client) really care about.

Not sure if that makes it niche! Queueing/Messaging scenarios also need to convey the SpanKind.
Of course it is possible to workaround by storing it as specially named attributes, but I'd hope that tracing can make it a first class concept.
SpanKind is also an input to the OTel Sampler. Sampling is executed in hot-path, so Samplers that use Span.Kind to make decision would need it to be a first class thing, as opposed to examining the attributes to find the specially named one.

A lot of users would be just fine with "internal (the default)" Span Kind. So the existing macros can continue to work as-is, but new ones on top can be written for use by those users who need a diff. Span Kind than the default. Very similar to how target is treated - it is a first class thing, but optional in the macros. (Anyway this is up-to the tracing team to decide!)

[cijothomas]

Tagging @open-telemetry/rust-approvers
@jtescher as otel rust maintainer and tracing-opentelemetry maintainer (from which the prototype shared above is inspired from!)
@davidbarsky as tokio-tracing maintainer

Please share your comments/thoughts.

[austinlparker]

I've added this to an upcoming GC meeting agenda.

[tedsuo]

How is this expected to work regarding feature development? Is the Tokio community willing to join the OTel spec going forwards?

[austinlparker]

I would also point out that OpenTelemetry isn't just a tracing API; How is this going to work with metrics, logs, profiles, events, and any other future API surface area the spec defines?

[cijothomas]

I would also point out that OpenTelemetry isn't just a tracing API; How is this going to work with metrics, logs, profiles, events, and any other future API surface area the spec defines?

@austinlparker This is strictly limited to the tracing API (tracer/span one).

[cijothomas]

How is this expected to work regarding feature development? Is the Tokio community willing to join the OTel spec going forwards?

As I have mentioned in the issue itself, none of this would work without tokio-tracing willing to make the changes required. (not just now, but in the future). The prototype is to explore if we really need changes in tokio-tracing, and if yes, what are the exact changes, and then tokio-tracing can make a decision.

Tokio community willing to join the OTel spec going forwards?

I am not so sure if I understand this part of the question? Could you clarify? Did you mean if tokio-tracing maintainers should join OTel spec as approver and/or attend spec meetings? From what I can tell, that is not required. If there is a new requirement in OTel Tracing API, the OTel Rust community can propose the same for tokio-tracing.
(This is similar to what OpenTelemetry .NET SIG does. eg: Recent addition to .NET for support AddLink after Span creation.)

[cijothomas]

I've added this to an upcoming GC meeting agenda.

Thanks. I've briefly mentioned about this informally to Trask, Jack (in a different context). Did not want to bother TC/GC officially, until OTel Rust community and Tokio-tracing maintainers have expressed willingness to move in this direction, but happy to get an early bless/block on the idea.

[austinlparker]

I've added this to an upcoming GC meeting agenda.

Thanks. I've briefly mentioned about this informally to Trask, Jack (in a different context). Did not want to bother TC/GC officially, until OTel Rust community and Tokio-tracing maintainers have expressed willingness to move in this direction, but happy to get an early bless/block on the idea.

Speaking personally, I would much rather the GC gets something like this sooner rather than later so that we can provide feedback and understand it better. :)

[cijothomas]

I've added this to an upcoming GC meeting agenda.

Thanks. I've briefly mentioned about this informally to Trask, Jack (in a different context). Did not want to bother TC/GC officially, until OTel Rust community and Tokio-tracing maintainers have expressed willingness to move in this direction, but happy to get an early bless/block on the idea.

Speaking personally, I would much rather the GC gets something like this sooner rather than later so that we can provide feedback and understand it better. :)

Thanks. Noted!

By the way, there is no harm if GC/TC or Tracing or OTel Rust rejects this proposal! It'd be a very valuable learning for us. Also this is just option2 from the original issue: #1571 and I volunteered to write down option2 in detail, while other maintainers volunteered to write down option3 in detail. None has signed up for option1 and option4 yet - most likely I'll take a stab at them.

[lalitb]

None has signed up for option1 and option4 yet - most likely I'll take a stab at them.

I can take a stab at Option 4, if no one has yet started on this.