Allow configuring Max Frame Size in Vert.x HTTP Client

[vmuzikar]

Is your enhancement related to a problem? Please describe

In some cases, it is necessary to transmit larger amount of data via the K8s Client. The particular use case for this is an Operator with a large CR. E.g. in the Keycloak Operator, we are seeing a lot of error messages like:

[io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-1) Max frame length of 65536 has been exceeded.

This is not happening with e.g. OKHttp Client. However, Quarkus and the Operator SDK is moving to Vert.x HTTP Client, so using Vert.x will be inevitable.

Describe the solution you'd like

Max Frame Size needs to be configurable.

Describe alternatives you've considered

No response

Additional context

No response

[manusa]

/cc @vietj @shawkins

[manusa]

Similar to what we decided about MAX_THREADS, timeouts, and so on, we might want to make all HttpClient(s) compliant to a common baseline.

Besides making the Max Frame Size configurable (as suggested), is there any consequence to provide a default higher limit?

[vietj]

can you describe what kind of frame we are talking about ? I'm assuming it is WebSocket, can you show some code that reproduces the behaviour to get an idea ?

[vmuzikar]

Thank you for your response. Unifying defaults with other clients sounds like a great idea.

@vietj Yes, it is WebSocket and sorry, I forgot to attach the full stack trace in the issue:

ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-3) Max frame length of 65536 has been exceeded.: io.netty.handler.codec.http.websocketx.CorruptedWebSocketFrameException: Max frame length of 65536 has been exceeded.
        at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:427)
        at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:288)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)

On top of that, I was able to create a simple reproducer:

client.configMaps().withName("large-configmap").watch(new Watcher<ConfigMap>() {
    @Override
    public void eventReceived(Action action, ConfigMap resource) {}

    @Override
    public void onClose(WatcherException cause) {}
});

The used ConfigMap can be found here.

[shawkins]

Besides making the Max Frame Size configurable (as suggested), is there any consequence to provide a default higher limit?

There's not much risk as long as we also make appropriate changes to the max message size as well. This does appear to warrant an upstream kubernetes issue as they should be able to provide guidance on the max frame, and possibly message size as well. My best guess is that they simply default all frames to be final and transmit whatever is provided fully rather than to break the message up into smaller frames: https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/vendor/golang.org/x/net/websocket/hybi.go#L252 - so we'll need similar settings for max frame and max message.

Jetty appears to have a similar issue with large frames, but there's no indication of why the message is not received without turning up the logging - that is because both jetty (defaulting to 64KB) and vertx enforce a max message length above the max frame length. That is expressed as an exception that is passed to our websocket listener - which just logs at a debug level and starts a reconnect

kubernetes-client/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/WatcherWebSocketListener.java

Line 46 in b3af4a7

logger.debug("WebSocket error received", t);

We probably want this at an info or warn instead so that this case is not silent.

We're also assuming that the exception is a terminal state - but that doesn't appear to be the case at least for vertx. We'll need to improve our handling of simply assuming that we can reconnect - if it's not a connection related problem, we need to terminate instead.