I'm trying to instrument SQS and I don't know what I'm doing wrong

[theotherian]

Hello community,

First, some context: our applications are already instruments with Open Telemetry and we are getting traces for the out-of-the-box supported frameworks. We run Spring Boot applications on Java 21 and have a little bit of customization in place that unfortunately requires us to stay on 1.41 (or maybe slightly later; the dependency is on the EventLogger stuff and I know 1.55 definitely doesn't work).

The main thing we've been missing is trying to get trace data for the consumption side of interacting with SQS. We can see trace data when producing messages but that's far less useful than seeing what happens when it's consumed.

The happy path would be having distributed tracing on both production and consumption of a message, but for the time being I'm trying to walk before I run.

As far as I can tell, getting instrumentation working with SQS requires additional steps per this page:
https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/aws-sdk/aws-sdk-2.2/library

Also of note: another engineer on my team was playing around with this example with an AI tool and the tool called out that the code as written in the example was missing the execution intercepter configuration which looks to be the part that actually detects the consumption side. The code it suggested adding was:

.overrideConfiguration(config -> config.addExecutionInterceptor(awsSdkTelemetry.newExecutionInterceptor()))

Due to the dependency reasons I stated above with our internal tooling, I've been trying to get things to work with version 1.41 of the open telemetry library, 2.7.0 of the Java instrumentation, and 2.7.0-alpha of the instrumentation extensions. This process has largely left me in Maven dependency hell with no clear way out.

When I first added this I encountered this issue:
#13724

I tried to solve this by adding the opentelemetry-javaagent-tooling library to my project but then I encountered a new problem with byte-buddy-dep being on the classpath since we use both Mockito and assertj for our tests which pull in byte-buddy. Having both present results in a bunch of instrumentation errors that are unsolvable.

It also appears I'm not the only person to have encountered this issue, but there's no comment or solution on the other instance I found: mockito/mockito#3342

Suffice to say, I feel like I'm completely misunderstanding something or doing something egregiously wrong here for reasons that aren't obvious to me. Any guidance on how to properly get SQS instrumentation in place on both the send and receive side would be very much appreciated.

[laurit]

As far as I can tell, getting instrumentation working with SQS requires additional steps per this page:
https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/aws-sdk/aws-sdk-2.2/library
Also of note: another engineer on my team was playing around with this example with an AI tool and the tool called out that the code as written in the example was missing the execution intercepter configuration which looks to be the part that actually detects the consumption side.

The instructions in https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/aws-sdk/aws-sdk-2.2/library include:
To register instrumentation only on a specific SDK client, register the interceptor when creating it.

AwsSdkTelemetry telemetry = AwsSdkTelemetry.create(openTelemetry).build();
DynamoDbClient client = DynamoDbClient.builder()
  .overrideConfiguration(ClientOverrideConfiguration.builder()
    .addExecutionInterceptor(telemetry.newExecutionInterceptor()))
    .build())
  .build();

which, as far as I can tell, looks about the same as what your AI tool suggested. Just replace DynamoDbClient with SqsClient. Could yo elaborate what should be added there? Note that there are additional steps for wrapping the SQS client in the instructions that you should follow.

When I first added this I encountered this issue:
#13724

it means that you are mixing dependencies from incompatible versions. Use mvn dependenies:tree to determine what versions of otel libraries you have and where they come from.

I tried to solve this by adding the opentelemetry-javaagent-tooling library to my project but then I encountered a new problem with byte-buddy-dep being on the classpath since we use both Mockito and assertj for our tests which pull in byte-buddy. Having both present results in a bunch of instrumentation errors that are unsolvable.

opentelemetry-javaagent-tooling belongs to the agent. Since you are using library instrumentation adding it won't help you in any way.

Suffice to say, I feel like I'm completely misunderstanding something or doing something egregiously wrong here for reasons that aren't obvious to me. Any guidance on how to properly get SQS instrumentation in place on both the send and receive side would be very much appreciated.

You could try examining our tests and create a toy application based on the test. If you get that working then perhaps you can figure out what is different in your real application. You could also try the agent instead of the library instrumentation to see whether it yields a better result. The test you could start from is

opentelemetry-java-instrumentation/instrumentation/aws-sdk/aws-sdk-2.2/testing/src/main/java/io/opentelemetry/instrumentation/awssdk/v2_2/AbstractAws2SqsBaseTest.java

Lines 153 to 167 in 61af4bd

	void testSimpleSqsProducerConsumerServicesSync() throws URISyntaxException {
	SqsClientBuilder builder = SqsClient.builder();
	configureSdkClient(builder);
	SqsClient client = configureSqsClient(builder.build());
	client.createQueue(createQueueRequest);
	client.sendMessage(sendMessageRequest);
	ReceiveMessageResponse response = client.receiveMessage(receiveMessageRequest);
	assertThat(response.messages().size()).isEqualTo(1);
	response.messages().forEach(message -> getTesting().runWithSpan("process child", () -> {}));
	assertSqsTraces(false, false);
	}

the trace it produces can be seen in

opentelemetry-java-instrumentation/instrumentation/aws-sdk/aws-sdk-2.2/testing/src/main/java/io/opentelemetry/instrumentation/awssdk/v2_2/AbstractAws2SqsSuppressReceiveSpansTest.java

Lines 47 to 58 in 61af4bd

	List<Consumer<TraceAssert>> traceAsserts =
	new ArrayList<>(
	Arrays.asList(
	trace -> trace.hasSpansSatisfyingExactly(span -> createQueueSpan(span)),
	trace ->
	trace.hasSpansSatisfyingExactly(
	span -> publishSpan(span, queueUrl, "SendMessage"),
	span -> processSpan(span, trace.getSpan(0)),
	span ->
	span.hasName("process child")
	.hasParent(trace.getSpan(1))
	.hasAttributes(Attributes.empty()))));

Note that the consumer span is created when you iterate over the messages in response.messages().forEach().

[theotherian]

An additional question on this for you @laurit after looking at the examples more closely.

I haven't attempted to get the instrumentation running locally yet, but I'm curious if maybe the problem is a little deeper given what you said about response.messages{}.forEach().

We're currently using the AWS Spring Cloud library and are consuming messages via the @SqsListener annotation rather than interacting with the client directly. Do you know if that can interfere with instrumentation?

[laurit]

It does. The approach used in sqs instrumentation that produces processing spans as the messages list is iterated is hacky. It works only when the collection is traversed through and iterator. Internally spring-cloud-aws accesses it through a stream so the spans are not created. Even if they were created it wouldn't be what you want because then you'd get spans for when spring aws is converting the messages not when your @SqsListener method is called. To handle this correctly a dedicated spring-cloud-aws instrumentation would be needed. Note that this works with the agent.

[theotherian]

@laurit I saw this: #12314

and realized it's in a version that we can use (1.43/2.9.0) so I'm going to see if that helps unless you're aware of other reasons why it wouldn't.

[theotherian]

To register instrumentation only on a specific SDK client, register the interceptor when creating it.

Ah, this was my mistake. I saw DynamoDbClient and SqsAsyncClient separately and didn't follow through. Shame on me for this one. I did both wrap and add the execution interceptor so it matches what you're suggesting.

it means that you are mixing dependencies from incompatible versions. Use mvn dependenies:tree to determine what versions of otel libraries you have and where they come from.

opentelemetry-javaagent-tooling belongs to the agent. Since you are using library instrumentation adding it won't help you in any way.

This is where I'm maybe most confused then. I only started getting this after adding the aws-sdk instrumentation and added the tooling dependency because that's where I found AgentInstrumentationConfig to address the NoSuchMethodError. Bootstrapping our application failed when starting the JDBC pool, which seems related to the fact that we're using IAM authentication for RDS. This isn't the full stacktrace (which is enormous) but the reference to Flyway.migrate is related to our JDBC configuration:

Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: 'io.opentelemetry.instrumentation.api.incubator.config.internal.InstrumentationConfig io.opentelemetry.javaagent.bootstrap.internal.AgentInstrumentationConfig.get():147)
  at org.flywaydb.core.Flyway.migrate(Flyway.java:124)
  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at io.opentelemetry.instrumentation.awssdk.v2_2.autoconfigure.AwsSdkSingletons.getBoolean(AwsSdkSingletons.java:72)
  at io.opentelemetry.instrumentation.awssdk.v2_2.autoconfigure.AwsSdkSingletons.captureExperimentalSpanAttributes(AwsSdkSingletons.java:48)
  at io.opentelemetry.instrumentation.awssdk.v2_2.autoconfigure.AwsSdkSingletons.<clinit>(AwsSdkSingletons.java:23)
  at io.opentelemetry.instrumentation.awssdk.v2_2.autoconfigure.TracingExecutionInterceptor.<init>(TracingExecutionInterceptor.java:29)

AFAICT, version 1.41 of the main open telemetry libraries and 2.7.0/2.7.0-alpha of the instrumentation libraries appear to be compatible with each other if I look at the dependencies the poms declare.

Is there a different dependency I should be using to have the aws-sdk-2.2 configuration in my classpath?

[laurit]

Firstly you appear to be using https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/aws-sdk/aws-sdk-2.2/library-autoconfigure this automatically set up the execution interceptor but it won't handle the wrapping of the sqs client that is described in https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/aws-sdk/aws-sdk-2.2/library#usage Secondly you are also using the agent or have included classes from the agent in your application. In later versions we have reworked the code so that this exception doesn't happen. For now you could try removing the opentelemetry-aws-sdk-2.2-autoconfigure dependency and use opentelemetry-aws-sdk-2.2.

[theotherian]

Thank you for pointing this out to me. I removed the autoconfigure library from the classpath and now the service is stabilizing.

I still don't have any traces coming through on the receive side yet but I need to do more work on my end with our observability platform support and our infra team to understand more about what's happening. Something is still wrong somewhere in the chain but I don't know what yet.

Thanks for helping me get past this hurdle!

EDIT: Hmm, looks like you were the one who added support last year for @SqsListener actually in #12314

EDIT 2: Hmm, maybe I need the otel.instrumentation.aws-sdk.experimental-use-propagator-for-messaging setting in https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/aws-sdk/README.md on the agent side?

EDIT 3: Maybe this is where I'm going wrong. I noticed you also wrote:

When using java agent you don't need to include opentelemetry-aws-sdk-2.2-autoconfigure.
We are running the agent so maybe what I did adding it to the classpath is unnecessary.