On Spring 3.2.7 and above it throws `INTERNAL: Encountered end-of-stream mid-frame` #862

namero999 · 2024-07-03T15:08:55Z

Describe the bug

A simple worker activation produces the error INTERNAL: Encountered end-of-stream mid-frame

To Reproduce

Use spring >= 3.2.7 with latest version of Spring-Zeebe. In 3.2.7, netty is upgraded. Maybe it's related to that...

Expected behavior

No errors in logs

Log/Stacktrace

Full Stacktrace

2024-07-03T17:04:40.089+02:00  WARN 13712 --- [ault-executor-0] io.camunda.zeebe.client.job.poller       : Failed to activate jobs for worker kakfaProducerWorker#produce and job type kafka-producer

io.grpc.StatusRuntimeException: INTERNAL: Encountered end-of-stream mid-frame
	at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-api-1.62.2.jar:1.62.2]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.62.2.jar:1.62.2]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574) ~[grpc-core-1.62.2.jar:1.62.2]
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72) ~[grpc-core-1.62.2.jar:1.62.2]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742) ~[grpc-core-1.62.2.jar:1.62.2]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723) ~[grpc-core-1.62.2.jar:1.62.2]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.62.2.jar:1.62.2]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.62.2.jar:1.62.2]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

Environment:

OS: MacOs
Deployment: SM
Spring-Zeebe Version: 8.5.7
SpringBoot Version: 3.2.7
Configuration: Only ES exporter, no connectors

The text was updated successfully, but these errors were encountered:

sephiroth-j · 2024-07-11T15:20:03Z

We observed something similar with Zeebe 8.4.9.

stacktrace for http2 exception

io.grpc.StatusRuntimeException: INTERNAL: http2 exception
	at io.grpc.Status.asRuntimeException(Status.java:537)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.netty.handler.codec.http2.Http2Exception: Received frame of type 1 while processing headers on stream 139.
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.verifyNotProcessingHeaders(DefaultHttp2FrameReader.java:763)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.verifyHeadersFrame(DefaultHttp2FrameReader.java:291)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.verifyFrameState(DefaultHttp2FrameReader.java:203)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:163)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:39)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:186)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:391)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 common frames omitted

stacktrace for Encountered end-of-stream mid-frame

Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: INTERNAL: Encountered end-of-stream mid-frame
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:72)
	... 4 more
Caused by: io.grpc.StatusRuntimeException: INTERNAL: Encountered end-of-stream mid-frame
	at io.grpc.Status.asRuntimeException(Status.java:537)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

This is due to an incompatibility of the old version of grpc-java with Netty version 4.1.111, which is shipped with Spring Boot 3.2.7. The versions 1.64.1 and 1.65.0 of grpc-java contain the necessary compatibility fixes for Netty 4.1.111 - see https://github.com/grpc/grpc-java/releases/tag/v1.64.1 and https://github.com/grpc/grpc-java/releases/tag/v1.65.0.

As a workaround, you could use a newer version of grpc by adding the following parts to the dependencyManagement section of you pom. I am pretty sure that this will also work for Zeebe 8.5.

			<dependency>
				<groupId>io.grpc</groupId>
				<artifactId>grpc-stub</artifactId>
				<version>1.65.0</version>
			</dependency>
			<dependency>
				<groupId>io.grpc</groupId>
				<artifactId>grpc-core</artifactId>
				<version>1.65.0</version>
			</dependency>
			<dependency>
				<groupId>io.grpc</groupId>
				<artifactId>grpc-api</artifactId>
				<version>1.65.0</version>
			</dependency>
			<dependency>
				<groupId>io.grpc</groupId>
				<artifactId>grpc-netty</artifactId>
				<version>1.65.0</version>
			</dependency>
			<dependency>
				<groupId>io.grpc</groupId>
				<artifactId>grpc-util</artifactId>
				<version>1.65.0</version>
			</dependency>
			<dependency>
				<groupId>io.grpc</groupId>
				<artifactId>grpc-protobuf</artifactId>
				<version>1.65.0</version>
			</dependency>

CC: @jonathanlukas

jonathanlukas · 2024-07-11T19:32:50Z

Hello @sephiroth-j ,
thank you for raising awareness and providing a workaround. As spring-zeebe does not manage the grpc-* dependencies, this should be raised with the zeebe repo.

I will take this to them internally and link the issue once created.

billbarni · 2024-08-19T13:41:26Z

This issue is still happening.

Spring Zeebe 8.5.9 and Spring Boot 3.3.2

jonathanlukas · 2024-08-19T14:11:08Z

@billbarni which grpc dependency versions are pulled in for your project?

billbarni · 2024-08-19T14:23:56Z

gRPC 1.62.2 it seems

namero999 · 2024-08-19T14:50:24Z

We fixed with implementation(platform("io.grpc:grpc-bom:1.66.0")) till the deps in spring-zeebe get updated.

jonathanlukas · 2024-08-19T16:26:19Z

Just to manage expectations here: spring-zeebe does not pull in any of these dependency versions. I would advise you to manage the effective dependency version in your own project.

billbarni · 2024-08-19T17:03:44Z

Possibly just add this information to the README.

jonathanlukas · 2024-08-19T17:18:23Z

@billbarni good idea, I will do that.

namero999 · 2024-08-22T09:42:32Z

Just to manage expectations here: spring-zeebe does not pull in any of these dependency versions. I would advise you to manage the effective dependency version in your own project.

Transitively it comes from here https://mvnrepository.com/artifact/io.camunda/zeebe-client-java/8.5.6

jonathanlukas · 2024-08-23T07:22:57Z

@namero999 this is the official zeebe client, located here: https://github.com/camunda/camunda/tree/main/clients/java

jonathanlukas mentioned this issue Jul 12, 2024

Introduce more detailed dependency management #868

Merged

jonathanlukas closed this as completed in #868 Jul 12, 2024

jonathanlukas reopened this Aug 19, 2024

jonathanlukas mentioned this issue Aug 23, 2024

add comment about dependency management #918

Merged

stephanpelikan mentioned this issue Aug 27, 2024

Encountered end-of-stream mid-frame camunda-community-hub/vanillabp-camunda8-adapter#57

Open

jonathanlukas closed this as completed in #918 Sep 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

On Spring 3.2.7 and above it throws `INTERNAL: Encountered end-of-stream mid-frame` #862

On Spring 3.2.7 and above it throws `INTERNAL: Encountered end-of-stream mid-frame` #862

namero999 commented Jul 3, 2024

sephiroth-j commented Jul 11, 2024 •

edited

Loading

jonathanlukas commented Jul 11, 2024

billbarni commented Aug 19, 2024

jonathanlukas commented Aug 19, 2024

billbarni commented Aug 19, 2024

namero999 commented Aug 19, 2024

jonathanlukas commented Aug 19, 2024

billbarni commented Aug 19, 2024

jonathanlukas commented Aug 19, 2024

namero999 commented Aug 22, 2024

jonathanlukas commented Aug 23, 2024

On Spring 3.2.7 and above it throws INTERNAL: Encountered end-of-stream mid-frame #862

On Spring 3.2.7 and above it throws INTERNAL: Encountered end-of-stream mid-frame #862

Comments

namero999 commented Jul 3, 2024

sephiroth-j commented Jul 11, 2024 • edited Loading

jonathanlukas commented Jul 11, 2024

billbarni commented Aug 19, 2024

jonathanlukas commented Aug 19, 2024

billbarni commented Aug 19, 2024

namero999 commented Aug 19, 2024

jonathanlukas commented Aug 19, 2024

billbarni commented Aug 19, 2024

jonathanlukas commented Aug 19, 2024

namero999 commented Aug 22, 2024

jonathanlukas commented Aug 23, 2024

On Spring 3.2.7 and above it throws `INTERNAL: Encountered end-of-stream mid-frame` #862

On Spring 3.2.7 and above it throws `INTERNAL: Encountered end-of-stream mid-frame` #862

sephiroth-j commented Jul 11, 2024 •

edited

Loading