Skip to content

vertx-http-proxy gets stuck when the original client is unavailable with a pending request. #117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
edu-gimenez opened this issue Mar 28, 2025 · 7 comments
Open

vertx-http-proxy gets stuck when the original client is unavailable with a pending request. #117

edu-gimenez opened this issue Mar 28, 2025 · 7 comments
Assignees
@tsegismont
Labels
bug Something isn't working
Milestone
4.5.16

Comments

@edu-gimenez
Copy link

edu-gimenez commented Mar 28, 2025 •
edited
Loading

I am using vertx-http-proxy:4.5.13 in a Vert.x server, and sometimes the server becomes unresponsive. I have realized that when the original client dies while there are pending requests in progress, the HTTP client responsible for the proxy gets stuck.

I believe the issue comes from the interaction between ProxiedRequest and PipeImpl, but I am not 100% sure.
These are the exceptions that I get before the client gets stuck.

2025-03-28 08:08:28.939 [vert.x-eventloop-thread-2] INFO  ProxyServer.lambda$start$3:48 - ProxyVerticle deployed successfully
2025-03-28 08:08:58.360 [vert.x-eventloop-thread-0] TRACE HttpServerImpl.: - Connection failure
io.netty.channel.StacklessClosedChannelException: null
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
2025-03-28 08:08:58.370 [vert.x-eventloop-thread-0] TRACE ReverseProxy.: - Error in sending the response
java.lang.IllegalArgumentException: Stream no longer exists: 403
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.requireStream(DefaultHttp2ConnectionEncoder.java:418) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeData(DefaultHttp2ConnectionEncoder.java:124) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeData(DecoratingHttp2FrameWriter.java:37) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.writeData(VertxHttp2ConnectionHandler.java:233) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2Stream.doWriteData(VertxHttp2Stream.java:267) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2Stream.writeData(VertxHttp2Stream.java:236) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerResponse.write(Http2ServerResponse.java:469) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerResponse.write(Http2ServerResponse.java:354) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerResponse.write(Http2ServerResponse.java:48) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.streams.impl.PipeImpl.lambda$to$1(PipeImpl.java:81) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.HttpEventHandler.handleChunk(HttpEventHandler.java:51) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:239) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ClientConnection$StreamImpl.handleData(Http2ClientConnection.java:501) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2Stream.lambda$new$1(VertxHttp2Stream.java:76) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:327) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:158) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2Stream.onData(VertxHttp2Stream.java:123) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ConnectionBase.onDataRead(Http2ConnectionBase.java:316) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:34) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:34) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onDataRead(Http2EmptyDataFrameListener.java:49) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:320) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:409) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:244) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:164) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:186) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:391) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelRead(VertxHttp2ConnectionHandler.java:405) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at java.lang.Thread.run(Thread.java:1583) [?:?]
2025-03-28 08:08:58.373 [vert.x-eventloop-thread-0] TRACE ReverseProxy.: - Error in sending the request
java.lang.IllegalStateException: Request has already been read
        at io.vertx.core.http.impl.Http2ServerRequest.checkEnded(Http2ServerRequest.java:205) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerRequest.pause(Http2ServerRequest.java:257) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerRequest.pause(Http2ServerRequest.java:48) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.streams.ReadStream.pipe(ReadStream.java:114) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.httpproxy.impl.ProxiedRequest.sendRequest(ProxiedRequest.java:209) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.httpproxy.impl.ProxiedRequest.send(ProxiedRequest.java:242) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:38) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.HttpClientImpl.lambda$doRequest$3(HttpClientImpl.java:392) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:342) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:163) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ClientConnection.createStream(Http2ClientConnection.java:157) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.HttpClientImpl.lambda$doRequest$4(HttpClientImpl.java:372) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.Endpoint.lambda$getConnection$0(Endpoint.java:52) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.SharedClientHttpStreamEndpoint$Request.handle(SharedClientHttpStreamEndpoint.java:162) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.SharedClientHttpStreamEndpoint$Request.handle(SharedClientHttpStreamEndpoint.java:123) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:342) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:335) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool$LeaseImpl.emit(SimpleConnectionPool.java:714) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool$Recycle$1.run(SimpleConnectionPool.java:735) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.Task.runNextTasks(Task.java:43) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.CombinerExecutor.submit(CombinerExecutor.java:91) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool.execute(SimpleConnectionPool.java:244) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool.recycle(SimpleConnectionPool.java:751) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool.access$3300(SimpleConnectionPool.java:79) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool$LeaseImpl.recycle(SimpleConnectionPool.java:710) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.HttpClientImpl.lambda$doRequest$2(HttpClientImpl.java:376) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ClientConnection$Stream.onClose(Http2ClientConnection.java:393) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ConnectionBase.onStreamClosed(Http2ConnectionBase.java:157) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler$1.onStreamClosed(VertxHttp2ConnectionHandler.java:93) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:355) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1034) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:990) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:515) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:521) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.doCloseStream(Http2ConnectionHandler.java:919) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:627) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStreamRemote(Http2ConnectionHandler.java:619) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:323) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:409) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:244) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:164) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:186) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:391) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelRead(VertxHttp2ConnectionHandler.java:405) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at java.lang.Thread.run(Thread.java:1583) [?:?]
2025-03-28 08:08:58.374 [vert.x-eventloop-thread-0] TRACE ReverseProxy.: - Error in sending the response
java.lang.IllegalStateException: Request has already been read
        at io.vertx.core.http.impl.Http2ServerRequest.checkEnded(Http2ServerRequest.java:205) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerRequest.fetch(Http2ServerRequest.java:271) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerRequest.resume(Http2ServerRequest.java:265) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerRequest.resume(Http2ServerRequest.java:48) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.httpproxy.impl.ProxiedRequest.release(ProxiedRequest.java:154) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.httpproxy.impl.ReverseProxy.lambda$handle$1(ReverseProxy.java:81) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.Composition.onFailure(Composition.java:50) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:278) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:40) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.HttpClientImpl.lambda$doRequest$3(HttpClientImpl.java:392) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:342) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:163) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ClientConnection.createStream(Http2ClientConnection.java:157) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.HttpClientImpl.lambda$doRequest$4(HttpClientImpl.java:372) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.Endpoint.lambda$getConnection$0(Endpoint.java:52) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.SharedClientHttpStreamEndpoint$Request.handle(SharedClientHttpStreamEndpoint.java:162) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.SharedClientHttpStreamEndpoint$Request.handle(SharedClientHttpStreamEndpoint.java:123) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:342) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:335) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool$LeaseImpl.emit(SimpleConnectionPool.java:714) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool$Recycle$1.run(SimpleConnectionPool.java:735) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.Task.runNextTasks(Task.java:43) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.CombinerExecutor.submit(CombinerExecutor.java:91) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool.execute(SimpleConnectionPool.java:244) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool.recycle(SimpleConnectionPool.java:751) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool.access$3300(SimpleConnectionPool.java:79) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.net.impl.pool.SimpleConnectionPool$LeaseImpl.recycle(SimpleConnectionPool.java:710) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.HttpClientImpl.lambda$doRequest$2(HttpClientImpl.java:376) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ClientConnection$Stream.onClose(Http2ClientConnection.java:393) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ConnectionBase.onStreamClosed(Http2ConnectionBase.java:157) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler$1.onStreamClosed(VertxHttp2ConnectionHandler.java:93) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:355) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1034) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:990) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:515) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:521) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.doCloseStream(Http2ConnectionHandler.java:919) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:627) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStreamRemote(Http2ConnectionHandler.java:619) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:323) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:409) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:244) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:164) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:186) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:391) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelRead(VertxHttp2ConnectionHandler.java:405) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at java.lang.Thread.run(Thread.java:1583) [?:?]

Do you have a reproducer?

I have created a small project to reproduce this error.
The project contains 3 main classes:

  • Client with a jetty client (without vertx) that generates traffic, configurable with several options. This client terminates before receiving all responses
  • BackendServer with a vertx server that receives the proxied requests
  • ProxyServer, with a vertx server and client that proxies the requests. The client has been configured with Http2MaxSize and Http2MultiplexingLimit set to 1, to be able to reproduce the error as quickly as possible.

http-proxy-vertx-issue.zip

Steps to reproduce

  1. Compile the fat-jar -> mvn package
  2. Run the backend server -> java -cp target/http-proxy-vertx-issue-1.0-SNAPSHOT.jar com.http.proxy.vertx.issue.backend.BackendServer
  3. Run the proxy server -> java -cp target/http-proxy-vertx-issue-1.0-SNAPSHOT.jar com.http.proxy.vertx.issue.proxy.ProxyServer
  4. Run the client -> java -cp target/http-proxy-vertx-issue-1.0-SNAPSHOT.jar com.http.proxy.vertx.issue.client.App 2 200 0 100 0 0 GET http://localhost:20000/some/path 0 none (Client can be configured, use java -cp target/http-proxy-vertx-issue-1.0-SNAPSHOT.jar com.http.proxy.vertx.issue.client.App --help` to see all options)
  5. Run the client again. The ProxyServer will no longer accept requests. (Since this appears to be a race condition, it may be necessary to run the client multiple times.)
@edu-gimenez edu-gimenez added the bug Something isn't working label Mar 28, 2025
@vietj vietj added this to the 4.5.14 milestone Mar 28, 2025
@tsegismont tsegismont self-assigned this Mar 28, 2025
@tsegismont
Copy link
Contributor

Thank you for reporting, I'll look into it

@tsegismont
Copy link
Contributor

@edu-gimenez this should be fixed with #118

Any chance for you to try a snapshot build with the fix?

@edu-gimenez
Copy link
Author

Hi @tsegismont, it seems that the issue is resolved with #118.
I still think there are some strange behaviors, but I need more time to analyze them.
For now, this ticket can be closed.
Thanks!

@tsegismont
Copy link
Contributor

Great, thanks for the update. Please go ahead and file new issues if required.

@edu-gimenez
Copy link
Author

edu-gimenez commented Apr 11, 2025 •
edited
Loading

Hi @tsegismont ,
I'm sorry, but I have not been able to spend time on this topic.
I'm afraid the issue is still present in 4.5.14-SNAPSHOT. Could you please test it using the project I shared earlier, just by changing the vertx-http-proxy version?

That said, I believe there’s more than one issue here. While I think there are some unhandled corner cases in ReverseProxy, I also suspect there’s a different corner case in vertx-core.

In particular, there's a scenario where the HttpClientRequest is created, but the Http2ClientConnection stream is never initialized (for example, when the HttpServerRequest is reset when it is going to be piped).

2025-04-11 11:39:21.096 [vert.x-eventloop-thread-0] TRACE ReverseProxy.: - Error in sending the request
java.lang.IllegalStateException: Request has already been read
        at io.vertx.core.http.impl.Http2ServerRequest.checkEnded(Http2ServerRequest.java:206) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerRequest.pause(Http2ServerRequest.java:277) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.http.impl.Http2ServerRequest.pause(Http2ServerRequest.java:49) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]
        at io.vertx.core.streams.ReadStream.pipe(ReadStream.java:114) ~[http-proxy-vertx-issue-1.0-SNAPSHOT.jar:?]

In this case, the handler in HttpClientImpl#doRequest is never invoked:

stream.closeHandler(v -> {
    lease.recycle();
});

I guess that if the recycle method is not called, the slot is not released and the connection becomes blocked.

@tsegismont
Copy link
Contributor

No problem to reopen, I thought it was fixed as per you previous comment. I'll take a look at the reproducer using the latest 4.5 version

@tsegismont tsegismont modified the milestones: 4.5.14, 4.5.15, 4.5.16 Apr 11, 2025
@tsegismont tsegismont reopened this Apr 11, 2025
@edu-gimenez
Copy link
Author

I also thought the issue was fixed with the previous commit, but since it's a strange race condition, it seems I wasn't able to reproduce it in my initial tests.
The thing is, the issue is related to vertx-core and not to vertx-http-proxy, so if you think I should open the ticket directly in vertx-core, please let me know.

@vietj vietj modified the milestones: 4.5.15, 4.5.16 May 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tsegismont tsegismont

Labels
bug Something isn't working
Projects
None yet
Milestone
4.5.16
Development

No branches or pull requests
3 participants
@vietj @tsegismont @edu-gimenez