"IllegalReferenceCountException Occurs with HTTP/2 Due to Race Condition Between HttpServerOperation and SimpleCompressionHandler" #3366
Comments
ojh3636
added
status/need-triage
|
The Full stack trace is here. |
violetagg
removed
the
status/need-triage
|
I think the second trace is not related to this issue but related other code that release nettyRequest (maybe other request decoder, I'll do further investigation) |
|
@ojh3636 If I prepare a change, will you be able to test the SNAPSHOT version? |
|
Thanks for your attention. |
|
@ojh3636 Depending on the version that you use, can you try - |
|
@violetagg Wow! Thank you for the quick follow-up! |
|
thanks |
|
The error still occurs 😢 Seconds one is here. |
|
I don't know if this helps, but when I patched the SimpleCompressionHandler as shown below, the error did not occur for about 2 days (approximately 800 million requests) void decode(ChannelHandlerContext ctx, HttpRequest msg) {
List<Object> out = new ArrayList<>();
HttpRequest request = msg;
try {
// CUSTOM: when FullHttpRequest in come, always make DefaultHttpRequest
if (msg instanceof FullHttpRequest) {
request = new DefaultHttpRequest(msg.protocolVersion(), msg.method(), msg.uri(), msg.headers());
}
super.decode(ctx, request, out);
}
catch (DecoderException e) {
throw e;
}
catch (Exception e) {
throw new DecoderException(e);
}
finally {
// ReferenceCountUtil.retain(...) is invoked in decode(...) so release(...) here
ReferenceCountUtil.release(request);
out.clear();
}
} |
|
@ojh3636 The problem might be different now. The change ensures that |
|
@violetagg Do you have any ideas on how this can be resolved? |
|
@ojh3636 Actually it can (do you have any caching in the gateway?): |
|
@violetagg Below is our reactor checkpoint log. The logging filter only logs the path and headers, and AddContext filter only mutate headers or write reactor context. |
|
@violetagg @Override
public NettyOutbound send(Publisher<? extends ByteBuf> dataStream, Predicate<ByteBuf> predicate) {
requireNonNull(predicate, "predicate");
if (!channel().isActive()) {
return then(Mono.error(AbortedException.beforeSend()));
}
if (dataStream instanceof Mono) {
return then(((Mono<?>) dataStream).flatMap(m -> FutureMono.from(channel().writeAndFlush(m)))
.doOnDiscard(ByteBuf.class, ByteBuf::release));
}
return then(MonoSendMany.byteBufSource(dataStream, channel(), predicate));
}
|
I'll investigate our application code... thank you |
violetagg
added
the
for/user-attention
|
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed. |
|
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open. |
violetagg
removed
for/user-attention
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I use Spring Cloud Gateway. After changing the inbound protocol from HTTP/1.1 to HTTP/2, we occasionally encounter
IllegalReferenceCountException.I observed that the error occurs only in two specific locations: the
finallyrelease block inSimpleCompressionHandler#decodeand therequest.release()block inHttpServerOperations#onInboundNext.This issue happens only when an HTTP/2 frame is decoded into a
FullHttpRequest(Header Frame with END_STREAM flgas that can be created by a simple GET request).In the code below, the
stateChangehook triggers Spring Cloud Gateway to proxy the downstream request to upstream on other thread. Subsequently,request.release()is called.However, if the upstream response arrives and meets the compression condition before request.release() is called, the decode function in SimpleCompressionHandler is invoked. Since HttpServerOperations' request.release() hasn't been called yet, the code does not create a new DefaultHttpRequest. Instead, it proceeds with the decode process and tries to retain the FullHttpRequest. Simultaneously, HttpServerOperations releases the same content. After HttpServerOperations releases the content (which causes the byteBuf to be freed), the retain operation at decoder throws an IllegalReferenceCountException. Then the finally block attempts to release the byteBuf again, leading to the exception.
Expected Behavior
There should be no
IllegalReferenceCountExceptioninside Reactor Netty logic.Actual Behavior
IllegalReferenceCountExceptioncan occur.Steps to Reproduce
Possible Solution
Move the hook timing of
listener().onStateChange(this, HttpServerState.REQUEST_RECEIVED);afterrequest.release()or use some other external synchronization to avoid double release.Your Environment
netty, ...):java -version): OpenJDK Runtime Environment Temurin-21.0.1+12uname -a): Linux 6.5.3-1.el8.elrepo.x86_64The text was updated successfully, but these errors were encountered: