diff --git a/.chloggen/gen-ai-sampling-relevant.yaml b/.chloggen/gen-ai-sampling-relevant.yaml new file mode 100644 index 0000000000..43246da3ce --- /dev/null +++ b/.chloggen/gen-ai-sampling-relevant.yaml @@ -0,0 +1,4 @@ +change_type: enhancement +component: gen_ai +note: Added `sampling-relevant` flag to relevant GenAI span attributes to indicate their importance for sampling decisions. +issues: [2994] diff --git a/docs/gen-ai/aws-bedrock.md b/docs/gen-ai/aws-bedrock.md index e53f8dcdd0..39f57f90a0 100644 --- a/docs/gen-ai/aws-bedrock.md +++ b/docs/gen-ai/aws-bedrock.md @@ -204,6 +204,15 @@ Since this attribute could be large, it's NOT RECOMMENDED to populate it by default. Instrumentations MAY provide a way to enable populating this attribute. +The following attributes can be important for making sampling decisions +and SHOULD be provided **at span creation time** (if provided at all): + +* [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.provider.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.request.model`](/docs/registry/attributes/gen-ai.md) +* [`server.address`](/docs/registry/attributes/server.md) +* [`server.port`](/docs/registry/attributes/server.md) + --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. diff --git a/docs/gen-ai/azure-ai-inference.md b/docs/gen-ai/azure-ai-inference.md index 7b71311edc..2f14ca7b61 100644 --- a/docs/gen-ai/azure-ai-inference.md +++ b/docs/gen-ai/azure-ai-inference.md @@ -205,6 +205,14 @@ Since this attribute could be large, it's NOT RECOMMENDED to populate it by default. Instrumentations MAY provide a way to enable populating this attribute. +The following attributes can be important for making sampling decisions +and SHOULD be provided **at span creation time** (if provided at all): + +* [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.request.model`](/docs/registry/attributes/gen-ai.md) +* [`server.address`](/docs/registry/attributes/server.md) +* [`server.port`](/docs/registry/attributes/server.md) + --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. diff --git a/docs/gen-ai/gen-ai-agent-spans.md b/docs/gen-ai/gen-ai-agent-spans.md index 7696cb8135..4f0303da5c 100644 --- a/docs/gen-ai/gen-ai-agent-spans.md +++ b/docs/gen-ai/gen-ai-agent-spans.md @@ -114,6 +114,15 @@ Instrumentations SHOULD document the list of errors they report. **[6] `server.address`:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent the server address behind any intermediaries, for example proxies, if it's available. +The following attributes can be important for making sampling decisions +and SHOULD be provided **at span creation time** (if provided at all): + +* [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.provider.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.request.model`](/docs/registry/attributes/gen-ai.md) +* [`server.address`](/docs/registry/attributes/server.md) +* [`server.port`](/docs/registry/attributes/server.md) + --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. @@ -354,6 +363,14 @@ Since this attribute could be large, it's NOT RECOMMENDED to populate it by default. Instrumentations MAY provide a way to enable populating this attribute. +The following attributes can be important for making sampling decisions +and SHOULD be provided **at span creation time** (if provided at all): + +* [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.provider.name`](/docs/registry/attributes/gen-ai.md) +* [`server.address`](/docs/registry/attributes/server.md) +* [`server.port`](/docs/registry/attributes/server.md) + --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index 6e0e8217df..92b03c1143 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -8,8 +8,8 @@ linkTitle: Events -- [Event: `event.gen_ai.client.inference.operation.details`](#event-eventgen_aiclientinferenceoperationdetails) -- [Event: `event.gen_ai.evaluation.result`](#event-eventgen_aievaluationresult) +- [Event: `gen_ai.client.inference.operation.details`](#event-gen_aiclientinferenceoperationdetails) +- [Event: `gen_ai.evaluation.result`](#event-gen_aievaluationresult) @@ -39,7 +39,7 @@ GenAI instrumentations MAY capture user inputs sent to the model and responses r > Note: > Events are in-development and not yet available in some languages. Check [spec-compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.50.0/spec-compliance-matrix.md#logs) to see the implementation status in corresponding language. -## Event: `event.gen_ai.client.inference.operation.details` +## Event: `gen_ai.client.inference.operation.details` @@ -223,7 +223,7 @@ populating this attribute. -## Event: `event.gen_ai.evaluation.result` +## Event: `gen_ai.evaluation.result` diff --git a/docs/gen-ai/gen-ai-spans.md b/docs/gen-ai/gen-ai-spans.md index cb04d8a4e9..b8d9e4d864 100644 --- a/docs/gen-ai/gen-ai-spans.md +++ b/docs/gen-ai/gen-ai-spans.md @@ -217,6 +217,15 @@ Since this attribute could be large, it's NOT RECOMMENDED to populate it by default. Instrumentations MAY provide a way to enable populating this attribute. +The following attributes can be important for making sampling decisions +and SHOULD be provided **at span creation time** (if provided at all): + +* [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.provider.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.request.model`](/docs/registry/attributes/gen-ai.md) +* [`server.address`](/docs/registry/attributes/server.md) +* [`server.port`](/docs/registry/attributes/server.md) + --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. @@ -307,27 +316,56 @@ The `gen_ai.operation.name` SHOULD be `embeddings`. | Key | Stability | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Value Type | Description | Example Values | |---|---|---|---|---|---| | [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Required` | string | The name of the operation being performed. [1] | `chat`; `generate_content`; `text_completion` | -| [`error.type`](/docs/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` if the operation ended in an error | string | Describes a class of error the operation ended with. [2] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | -| [`gen_ai.request.model`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Conditionally Required` If available. | string | The name of the GenAI model a request is being made to. [3] | `gpt-4` | -| [`server.port`](/docs/registry/attributes/server.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If `server.address` is set. | int | GenAI server port. [4] | `80`; `8080`; `443` | +| [`gen_ai.provider.name`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Required` | string | The Generative AI provider as identified by the client or server instrumentation. [2] | `openai`; `gcp.gen_ai`; `gcp.vertex_ai` | +| [`error.type`](/docs/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` if the operation ended in an error | string | Describes a class of error the operation ended with. [3] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | +| [`gen_ai.request.model`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Conditionally Required` If available. | string | The name of the GenAI model a request is being made to. [4] | `gpt-4` | +| [`server.port`](/docs/registry/attributes/server.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If `server.address` is set. | int | GenAI server port. [5] | `80`; `8080`; `443` | | [`gen_ai.embeddings.dimension.count`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Recommended` | int | The number of dimensions the resulting output embeddings should have. | `512`; `1024` | -| [`gen_ai.request.encoding_formats`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Recommended` | string[] | The encoding formats requested in an embeddings operation, if specified. [5] | `["base64"]`; `["float", "binary"]` | +| [`gen_ai.request.encoding_formats`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Recommended` | string[] | The encoding formats requested in an embeddings operation, if specified. [6] | `["base64"]`; `["float", "binary"]` | | [`gen_ai.usage.input_tokens`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Recommended` | int | The number of tokens used in the GenAI input (prompt). | `100` | -| [`server.address`](/docs/registry/attributes/server.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Recommended` | string | GenAI server address. [6] | `example.com`; `10.1.2.80`; `/tmp/my.sock` | +| [`server.address`](/docs/registry/attributes/server.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Recommended` | string | GenAI server address. [7] | `example.com`; `10.1.2.80`; `/tmp/my.sock` | **[1] `gen_ai.operation.name`:** If one of the predefined values applies, but specific system uses a different name it's RECOMMENDED to document it in the semantic conventions for specific GenAI system and use system-specific name in the instrumentation. If a different name is not documented, instrumentation libraries SHOULD use applicable predefined value. -**[2] `error.type`:** The `error.type` SHOULD match the error code returned by the Generative AI provider or the client library, +**[2] `gen_ai.provider.name`:** The attribute SHOULD be set based on the instrumentation's best +knowledge and may differ from the actual model provider. + +Multiple providers, including Azure OpenAI, Gemini, and AI hosting platforms +are accessible using the OpenAI REST API and corresponding client libraries, +but may proxy or host models from different providers. + +The `gen_ai.request.model`, `gen_ai.response.model`, and `server.address` +attributes may help identify the actual system in use. + +The `gen_ai.provider.name` attribute acts as a discriminator that +identifies the GenAI telemetry format flavor specific to that provider +within GenAI semantic conventions. +It SHOULD be set consistently with provider-specific attributes and signals. +For example, GenAI spans, metrics, and events related to AWS Bedrock +should have the `gen_ai.provider.name` set to `aws.bedrock` and include +applicable `aws.bedrock.*` attributes and are not expected to include +`openai.*` attributes. + +**[3] `error.type`:** The `error.type` SHOULD match the error code returned by the Generative AI provider or the client library, the canonical name of exception that occurred, or another low-cardinality error identifier. Instrumentations SHOULD document the list of errors they report. -**[3] `gen_ai.request.model`:** The name of the GenAI model a request is being made to. If the model is supplied by a vendor, then the value must be the exact name of the model requested. If the model is a fine-tuned custom model, the value should have a more specific name than the base model that's been fine-tuned. +**[4] `gen_ai.request.model`:** The name of the GenAI model a request is being made to. If the model is supplied by a vendor, then the value must be the exact name of the model requested. If the model is a fine-tuned custom model, the value should have a more specific name than the base model that's been fine-tuned. -**[4] `server.port`:** When observed from the client side, and when communicating through an intermediary, `server.port` SHOULD represent the server port behind any intermediaries, for example proxies, if it's available. +**[5] `server.port`:** When observed from the client side, and when communicating through an intermediary, `server.port` SHOULD represent the server port behind any intermediaries, for example proxies, if it's available. -**[5] `gen_ai.request.encoding_formats`:** In some GenAI systems the encoding formats are called embedding types. Also, some GenAI systems only accept a single format per request. +**[6] `gen_ai.request.encoding_formats`:** In some GenAI systems the encoding formats are called embedding types. Also, some GenAI systems only accept a single format per request. -**[6] `server.address`:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent the server address behind any intermediaries, for example proxies, if it's available. +**[7] `server.address`:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent the server address behind any intermediaries, for example proxies, if it's available. + +The following attributes can be important for making sampling decisions +and SHOULD be provided **at span creation time** (if provided at all): + +* [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.provider.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.request.model`](/docs/registry/attributes/gen-ai.md) +* [`server.address`](/docs/registry/attributes/server.md) +* [`server.port`](/docs/registry/attributes/server.md) --- @@ -351,6 +389,34 @@ Instrumentations SHOULD document the list of errors they report. | `invoke_agent` | Invoke GenAI agent | ![Development](https://img.shields.io/badge/-development-blue) | | `text_completion` | Text completions operation such as [OpenAI Completions API (Legacy)](https://platform.openai.com/docs/api-reference/completions) | ![Development](https://img.shields.io/badge/-development-blue) | +--- + +`gen_ai.provider.name` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `anthropic` | [Anthropic](https://www.anthropic.com/) | ![Development](https://img.shields.io/badge/-development-blue) | +| `aws.bedrock` | [AWS Bedrock](https://aws.amazon.com/bedrock) | ![Development](https://img.shields.io/badge/-development-blue) | +| `azure.ai.inference` | Azure AI Inference | ![Development](https://img.shields.io/badge/-development-blue) | +| `azure.ai.openai` | [Azure OpenAI](https://azure.microsoft.com/products/ai-services/openai-service/) | ![Development](https://img.shields.io/badge/-development-blue) | +| `cohere` | [Cohere](https://cohere.com/) | ![Development](https://img.shields.io/badge/-development-blue) | +| `deepseek` | [DeepSeek](https://www.deepseek.com/) | ![Development](https://img.shields.io/badge/-development-blue) | +| `gcp.gemini` | [Gemini](https://cloud.google.com/products/gemini) [8] | ![Development](https://img.shields.io/badge/-development-blue) | +| `gcp.gen_ai` | Any Google generative AI endpoint [9] | ![Development](https://img.shields.io/badge/-development-blue) | +| `gcp.vertex_ai` | [Vertex AI](https://cloud.google.com/vertex-ai) [10] | ![Development](https://img.shields.io/badge/-development-blue) | +| `groq` | [Groq](https://groq.com/) | ![Development](https://img.shields.io/badge/-development-blue) | +| `ibm.watsonx.ai` | [IBM Watsonx AI](https://www.ibm.com/products/watsonx-ai) | ![Development](https://img.shields.io/badge/-development-blue) | +| `mistral_ai` | [Mistral AI](https://mistral.ai/) | ![Development](https://img.shields.io/badge/-development-blue) | +| `openai` | [OpenAI](https://openai.com/) | ![Development](https://img.shields.io/badge/-development-blue) | +| `perplexity` | [Perplexity](https://www.perplexity.ai/) | ![Development](https://img.shields.io/badge/-development-blue) | +| `x_ai` | [xAI](https://x.ai/) | ![Development](https://img.shields.io/badge/-development-blue) | + +**[8]:** Used when accessing the 'generativelanguage.googleapis.com' endpoint. Also known as the AI Studio API. + +**[9]:** May be used when specific backend is unknown. + +**[10]:** Used when accessing the 'aiplatform.googleapis.com' endpoint. + @@ -421,6 +487,11 @@ It's expected to be an object - in case a serialized string is available to the instrumentation, the instrumentation SHOULD do the best effort to deserialize it to an object. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise. +The following attributes can be important for making sampling decisions +and SHOULD be provided **at span creation time** (if provided at all): + +* [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) + --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. diff --git a/docs/gen-ai/openai.md b/docs/gen-ai/openai.md index 8721b718e6..819b3e30af 100644 --- a/docs/gen-ai/openai.md +++ b/docs/gen-ai/openai.md @@ -210,6 +210,14 @@ Since this attribute could be large, it's NOT RECOMMENDED to populate it by default. Instrumentations MAY provide a way to enable populating this attribute. +The following attributes can be important for making sampling decisions +and SHOULD be provided **at span creation time** (if provided at all): + +* [`gen_ai.operation.name`](/docs/registry/attributes/gen-ai.md) +* [`gen_ai.request.model`](/docs/registry/attributes/gen-ai.md) +* [`server.address`](/docs/registry/attributes/server.md) +* [`server.port`](/docs/registry/attributes/server.md) + --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. diff --git a/model/gen-ai/spans.yaml b/model/gen-ai/spans.yaml index c930f920a9..0cfc4ae787 100644 --- a/model/gen-ai/spans.yaml +++ b/model/gen-ai/spans.yaml @@ -119,6 +119,15 @@ groups: - ref: gen_ai.provider.name # TODO: Not adding to common attributes because of https://github.com/open-telemetry/weaver/issues/479 requirement_level: required + sampling_relevant: true + - ref: gen_ai.operation.name + sampling_relevant: true + - ref: server.address + sampling_relevant: true + - ref: server.port + sampling_relevant: true + - ref: gen_ai.request.model + sampling_relevant: true - ref: gen_ai.request.top_k requirement_level: recommended @@ -143,6 +152,14 @@ groups: Additional output format details may be recorded in the future in the `gen_ai.output.{type}.*` attributes. + - ref: server.address + sampling_relevant: true + - ref: server.port + sampling_relevant: true + - ref: gen_ai.request.model + sampling_relevant: true + - ref: gen_ai.operation.name + sampling_relevant: true - id: span.openai.inference.client extends: attributes.gen_ai.inference.openai_based @@ -210,6 +227,17 @@ groups: **Span name** SHOULD be `{gen_ai.operation.name} {gen_ai.request.model}`. extends: attributes.gen_ai.common.client attributes: + - ref: gen_ai.provider.name + requirement_level: required + sampling_relevant: true + - ref: gen_ai.operation.name + sampling_relevant: true + - ref: server.address + sampling_relevant: true + - ref: server.port + sampling_relevant: true + - ref: gen_ai.request.model + sampling_relevant: true - ref: gen_ai.request.encoding_formats requirement_level: recommended - ref: gen_ai.usage.input_tokens @@ -233,6 +261,15 @@ groups: attributes: - ref: gen_ai.provider.name requirement_level: required + sampling_relevant: true + - ref: gen_ai.operation.name + sampling_relevant: true + - ref: server.address + sampling_relevant: true + - ref: server.port + sampling_relevant: true + - ref: gen_ai.request.model + sampling_relevant: true - ref: gen_ai.agent.id requirement_level: conditionally_required: if applicable. @@ -272,6 +309,9 @@ groups: attributes: - ref: gen_ai.provider.name requirement_level: required + sampling_relevant: true + - ref: gen_ai.operation.name + sampling_relevant: true - ref: gen_ai.agent.id requirement_level: conditionally_required: if applicable. @@ -287,6 +327,9 @@ groups: - ref: server.address requirement_level: recommended: when span kind is `CLIENT`. + sampling_relevant: true + - ref: server.port + sampling_relevant: true - ref: gen_ai.system_instructions requirement_level: opt_in - ref: gen_ai.input.messages @@ -312,6 +355,7 @@ groups: attributes: - ref: gen_ai.operation.name requirement_level: required + sampling_relevant: true - ref: gen_ai.tool.name requirement_level: recommended - ref: gen_ai.tool.call.id