Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using OpenTelemetry gRPC instrumentation produces NullPointerException when running quarkusBuild #22864

Closed
XSAM opened this issue Jan 13, 2022 · 7 comments · Fixed by #22866
Closed
Assignees
Milestone

Comments

@XSAM
Copy link
Contributor

XSAM commented Jan 13, 2022

Describe the bug

Using io.opentelemetry.instrumentation:opentelemetry-grpc-1.6:1.9.2-alpha and io.quarkus:quarkus-opentelemetry-exporter-otlp with Quarkus 2.6.2.Final produces NullPointerException when packaging image by quarkusBuild.

Expected behavior

It should produce an image without error.

Actual behavior

$ ./gradlew quarkusBuild

> Task :quarkusGenerateCode
preparing quarkus application
Base image 'foo' does not use a specific image digest - build may not be reproducible

> Task :quarkusBuild FAILED
building quarkus jar

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':quarkusBuild'.
> io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkus.container.image.jib.deployment.JibProcessor#buildFromJar threw an exception: java.lang.RuntimeException: Unable to create container image
        at io.quarkus.container.image.jib.deployment.JibProcessor.containerize(JibProcessor.java:197)
        at io.quarkus.container.image.jib.deployment.JibProcessor.buildFromJar(JibProcessor.java:136)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:887)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads.JBossThread.run(JBossThread.java:501)
  Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
        at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88)
        at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$obtainBaseImagesLayers$6(StepsRunner.java:313)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
  Caused by: java.lang.NullPointerException
        at io.quarkus.opentelemetry.runtime.QuarkusContextStorage.getVertxContext(QuarkusContextStorage.java:62)
        at io.quarkus.opentelemetry.runtime.QuarkusContextStorage.current(QuarkusContextStorage.java:54)
        at io.opentelemetry.context.Context.current(Context.java:86)
        at io.opentelemetry.instrumentation.grpc.v1_6.internal.ContextStorageBridge.current(ContextStorageBridge.java:75)
        at io.grpc.override.ContextStorageOverride.current(ContextStorageOverride.java:31)
        at io.grpc.Context.current(Context.java:172)
        at io.opencensus.trace.CurrentSpanUtils.getCurrentSpan(CurrentSpanUtils.java:37)
        at io.opencensus.trace.Tracer.spanBuilder(Tracer.java:308)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:865)
        at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:349)
        at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:266)
        at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:138)
        at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:114)
        at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint(RegistryClient.java:625)
        at com.google.cloud.tools.jib.registry.RegistryClient.pullManifest(RegistryClient.java:436)
        at com.google.cloud.tools.jib.registry.RegistryClient.pullManifest(RegistryClient.java:441)
        at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.pullBaseImages(PullBaseImageStep.java:290)
        at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:157)
        at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:69)
        ... 6 more

How to Reproduce?

  1. Create a new project
mvn io.quarkus.platform:quarkus-maven-plugin:2.6.2.Final:create \
    -DprojectGroupId=my-groupId \
    -DprojectArtifactId=my-artifactId \
    -DprojectVersion=my-version \
    -DclassName="org.my.group.MyResource" \
    -Dextensions="resteasy,resteasy-jackson" \
    -DbuildTool=gradle
  1. Add dependencies
    implementation 'io.quarkus:quarkus-container-image-jib'
    implementation 'io.opentelemetry.instrumentation:opentelemetry-grpc-1.6:1.9.2-alpha'
    implementation 'io.quarkus:quarkus-opentelemetry-exporter-otlp'
  1. Add properties to gradle.properties file
quarkus.container-image.build=true
quarkus.jib.base-jvm-image=ghcr.io/foo/bar

quarkus.container-image.group=foo
quarkus.container-image.name=bar
  1. Run ./gradlew quarkusBuild

Output of uname -a or ver

21.2.0

Output of java -version

openjdk version "16.0.1" 2021-04-20 OpenJDK Runtime Environment (build 16.0.1+9-24) OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.6.2.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 7.3.3

Additional information

No response

@quarkus-bot
Copy link

quarkus-bot bot commented Jan 13, 2022

@geoand geoand self-assigned this Jan 13, 2022
geoand added a commit to geoand/quarkus that referenced this issue Jan 13, 2022
geoand added a commit to geoand/quarkus that referenced this issue Jan 13, 2022
geoand added a commit to geoand/quarkus that referenced this issue Jan 13, 2022
geoand added a commit to geoand/quarkus that referenced this issue Jan 13, 2022
geoand added a commit that referenced this issue Jan 13, 2022
Ensure that the presence of OpenTelemetry does not break Jib builds
@quarkus-bot quarkus-bot bot added this to the 2.7 - main milestone Jan 13, 2022
@gsmet gsmet modified the milestones: 2.7 - main, 2.6.3.Final Jan 15, 2022
gsmet pushed a commit to gsmet/quarkus that referenced this issue Jan 15, 2022
@XSAM
Copy link
Contributor Author

XSAM commented Jan 21, 2022

I found a similar issue when running a test that will use gRPC client even using Quarkus 2.6.3.

Error logs:

$ ./gradlew test
io.grpc.StatusRuntimeException: UNKNOWN
	at app//io.grpc.Status.asRuntimeException(Status.java:526)
	at app//com.ibm.etcd.client.GrpcClient.waitFor(GrpcClient.java:782)
	at app//com.ibm.etcd.client.GrpcClient.waitForCall(GrpcClient.java:729)
	at app//com.ibm.etcd.client.FluentRequest$AbstractFluentRequest.sync(FluentRequest.java:103)
	at app//io.foo.etcd.EtcdWatcherTest.testBasics(EtcdWatcherTest.java:23)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
	at app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at app//org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at app//org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at app//org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at app//org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1541)
	at app//org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1541)
	at app//org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at app//org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at app//org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at app//org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at app//org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at app//org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at app//org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at app//org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy5.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.NullPointerException
	at io.quarkus.opentelemetry.runtime.QuarkusContextStorage.getVertxContext(QuarkusContextStorage.java:62)
	at io.quarkus.opentelemetry.runtime.QuarkusContextStorage.current(QuarkusContextStorage.java:54)
	at io.opentelemetry.context.Context.current(Context.java:86)
	at io.opentelemetry.instrumentation.grpc.v1_6.internal.ContextStorageBridge.current(ContextStorageBridge.java:75)
	at io.grpc.override.ContextStorageOverride.current(ContextStorageOverride.java:31)
	at io.grpc.Context.current(Context.java:172)
	at io.grpc.internal.ManagedChannelImpl$RealChannel.newCall(ManagedChannelImpl.java:1007)
	at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:920)
	at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63)
	at com.ibm.etcd.client.GrpcClient.fuCall(GrpcClient.java:332)
	at com.ibm.etcd.client.GrpcClient.lambda$fuCall$5(GrpcClient.java:326)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleAsyncTask.runInterruptibly(TrustedListenableFutureTask.java:160)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleAsyncTask.runInterruptibly(TrustedListenableFutureTask.java:143)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:387)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

@XSAM
Copy link
Contributor Author

XSAM commented Jan 21, 2022

Maybe using #22865 could fix these kinds of issues once for all? It could stop getVertxContext throw NullPointerException while vertx is not initialized.

@geoand
Copy link
Contributor

geoand commented Jan 21, 2022

That sounds similar to #22866. #22865 is not the proper solution

@XSAM
Copy link
Contributor Author

XSAM commented Jan 21, 2022

That sounds similar to #22866.

Yeah, but #22866 is not working for this new situation by disable otel while pulling image.

#22865 is not the proper solution.

I am curious why #22865 is not the proper solution. It fixed a potential nullpointer error, and it can also fix the old issue. (Not testing the new one)

@geoand
Copy link
Contributor

geoand commented Jan 21, 2022

That sounds similar to #22866.

Yeah, but #22866 is not working for this new situation by disable otel while pulling image.

I know, I never said it would fix your latest issue. What I am saying is that the root cause is similar.

I am curious why #22865 is not the proper solution. It fixed a potential nullpointer error, and it can also fix the old issue. (Not testing the new one)

Because it's just a bandaid. We don't want arbitrary Vert.x contexts popping up out of nowhere.

@XSAM
Copy link
Contributor Author

XSAM commented Jan 28, 2022

Adding @QuarkusTest annotation to the test case could fix this latest issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment