NegativeArraySizeException in Hk2RequestScope.release on Jersey 3.1.11

[lbnetrom]

After upgrading Jersey, we observed a NegativeArraySizeException being thrown during request scope cleanup.

This happens in version 3.1.11 (3.1.10...3.1.11) in org.glassfish.jersey.inject.hk2.Hk2RequestScope$Instance.release, where the implementation was changed from:
new HashSet<>(store.keySet()).forEach(this::remove);
to
ArrayList<ForeignDescriptor> reverse = new ArrayList<>(store.keySet());

The new ArrayList<>(store.keySet()) constructor immediately calls store.keySet().size(), which can return a negative number if the store (a LinkedHashMap) has been corrupted due to concurrent access, resulting in the following error:

java.lang.NegativeArraySizeException: -6
    at java.base/java.util.LinkedHashMap$LinkedKeySet.toArray(LinkedHashMap.java:713)
    at java.base/java.util.ArrayList.<init>(ArrayList.java:181)
    at org.glassfish.jersey.inject.hk2.Hk2RequestScope$Instance.release(Hk2RequestScope.java:144)

[paulrutter]

This regression has been introduced in a1c8068#diff-fe1935c893ed8d887610559535bacae3eb73ae6e101ff4e45d4e5acf9c5c05a4 (#5906).

@jansupol what do you think?
It happens often enough that we're going to revert to 3.1.10 for now.

[jansupol]

What JDK do you have? It is odd that Set#size() returns <= 0.
new HashSet<>(store.keySet())
can be translated into

map = new HashMap<>(Math.max((int) (store.keySet().size()/.75f) + 1, 16));
for (ForeignDescriptor fd : store.keySet()) {
   map.add(fd);
}

Which we can rewrite to

int size = store.size();
ArrayList<ForeignDescriptor> reverse = new ArrayList<>(size >= 0 ? size : 8);
for (ForeignDescriptor fd : store.keySet()) {
    reverse.add(fd);
}

We never had ConcurrentModificationException or any similar issue with the 3.10 code. But my concern is that moving from HashSet to LinkedHashSet might cause some during the iteration, if the size is <=0.

[paulrutter]

We are using Java 21 from AWS, Corretto.
The exact version i can share tomorrow.

[jansupol]

has been corrupted due to concurrent access

Is this an assumption or are you sure about the concurrency happening?

The Hk2RequestScope should be a scope/instance per request, and request-scoped instances are handled by a threadlocal... hence no synchronization...

Is the stack trace from the server or client side? Can you share more of it? If from server, do you use more threads for computation, such as with @AsyncResponse?

[paulrutter]

We haven't been able to reproduce, but it seems that the affected endpoints indeed use @AsyncResponse and writing to the output stream via multiple concurrent threads (using StreamingOutput. It's indeed serverside.

I can share the full stacktrace tomorrow.

[paulrutter]

This is all the background information i have at the moment.
A few stacktraces:

Caused by: java.lang.NegativeArraySizeException: -1
	at java.base/java.util.LinkedHashMap$LinkedKeySet.toArray(LinkedHashMap.java:713)
	at java.base/java.util.ArrayList.<init>(ArrayList.java:181)
	at org.glassfish.jersey.inject.hk2.Hk2RequestScope$Instance.release(Hk2RequestScope.java:144)
	at org.glassfish.jersey.process.internal.RequestScope.release(RequestScope.java:246)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:268)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:398)

And:

Caused by: java.lang.NegativeArraySizeException: -4
	at java.base/java.util.LinkedHashMap$LinkedKeySet.toArray(LinkedHashMap.java:713)
	at java.base/java.util.ArrayList.<init>(ArrayList.java:181)
	at org.glassfish.jersey.inject.hk2.Hk2RequestScope$Instance.release(Hk2RequestScope.java:144)
	at org.glassfish.jersey.process.internal.RequestScope.release(RequestScope.java:246)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:268)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:398)

Java version: Corretto 1:21.0.8+9-1.amzn2023.1.
Jersey version: 3.1.11
Servlet API version: 6.0
Jetty version: 12.0.25

I was mistaken that we're using StreamingOutput, as the affected endpoints all use ChunkedOutput to write to the outputstream with multiple concurrent threads.
The code calling write on the ChunkedOutput instance is not synchronized (except for the first and last chunk).
The resource method that returns ChunkedOutput is annotated with @Suspended final AsyncResponse asyncResponse.

Since we haven't been able to reproduce, the current analysis is an assumption, not a given.
But given the fact that the only affected endpoints are the ones that write to the outputstream with concurrent threads, that seems very plausible.

[jansupol]

I assume this is done by releasing the request context multiple times, for instance in ChunkedOutput and then in RequestScope itself.

[paulrutter]

So is that a Jersey bug then, or would that be an error in our application code using ChunkedOutput?

[jansupol]

I tried to create a reproducer, and hopefully we have one that simulates your condition. That would fix it in Jersey.

[paulrutter]

I checked the reproducer and i can map that to what we're seeing in production, although we haven't been able to reproduce. I'm confident that this fixes the issue.