Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure to build container-image via s2i in some clusters #7454

Closed
iocanel opened this issue Feb 27, 2020 · 6 comments · Fixed by #7600
Closed

Failure to build container-image via s2i in some clusters #7454

iocanel opened this issue Feb 27, 2020 · 6 comments · Fixed by #7600
Assignees
@iocanel
Labels
kind/bug Something isn't working
Milestone
1.3.0.CR1

Comments

@iocanel
Copy link
Contributor

iocanel commented Feb 27, 2020

Describe the bug

S2i binary builds fails in some clusters, with stream was reset and timeout errors.

Expected behavior
S2i build should work normally.

Actual behavior
During the s2i build, which happens using the java client we receive the following excpetion:

[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:999-SNAPSHOT:build (default) on project openshift-quickstart: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR]     [error]: Build step io.quarkus.container.image.s2i.deployment.S2iProcessor#s2iBuildFromJar threw an exception: java.lang.RuntimeException: Execution of s2i build failed. See s2i output for more details
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iException(S2iProcessor.java:291)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuild(S2iProcessor.java:272)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.lambda$s2iBuild$10(S2iProcessor.java:252)
[ERROR]     at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
[ERROR]     at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
[ERROR]     at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
[ERROR]     at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
[ERROR]     at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
[ERROR]     at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
[ERROR]     at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
[ERROR]     at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
[ERROR]     at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[ERROR]     at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuild(S2iProcessor.java:252)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.createContainerImage(S2iProcessor.java:220)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuildFromJar(S2iProcessor.java:143)
[ERROR]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR]     at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:938)
[ERROR]     at io.quarkus.builder.BuildContext.run(BuildContext.java:273)
[ERROR]     at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[ERROR]     at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2027)
[ERROR]     at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1551)
[ERROR]     at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1442)
[ERROR]     at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR]     at org.jboss.threads.JBossThread.run(JBossThread.java:479)
[ERROR] Caused by: io.dekorate.deps.kubernetes.client.KubernetesClientException: Can't instantiate binary build, due to error reading/writing stream. Can be caused if the output stream was closed by the server.
[ERROR]     at io.dekorate.deps.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl$1.writeTo(BuildConfigOperationsImpl.java:237)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:72)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.openshift.client.internal.OpenShiftOAuthInterceptor.intercept(OpenShiftOAuthInterceptor.java:96)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
[ERROR]     at io.dekorate.deps.okhttp3.RealCall.execute(RealCall.java:93)
[ERROR]     at io.dekorate.deps.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:469)
[ERROR]     at io.dekorate.deps.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:448)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromInputStream(BuildConfigOperationsImpl.java:251)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromFile(BuildConfigOperationsImpl.java:208)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromFile(BuildConfigOperationsImpl.java:65)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuild(S2iProcessor.java:266)
[ERROR]     ... 26 more
[ERROR] Caused by: io.dekorate.deps.okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR
[ERROR]     at io.dekorate.deps.okhttp3.internal.http2.Http2Stream.checkOutNotClosed(Http2Stream.java:644)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http2.Http2Stream$FramingSink.emitFrame(Http2Stream.java:578)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http2.Http2Stream$FramingSink.write(Http2Stream.java:558)
[ERROR]     at io.dekorate.deps.okio.ForwardingSink.write(ForwardingSink.java:35)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.CallServerInterceptor$CountingSink.write(CallServerInterceptor.java:149)
[ERROR]     at io.dekorate.deps.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:179)
[ERROR]     at io.dekorate.deps.okio.RealBufferedSink.writeAll(RealBufferedSink.java:107)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl$1.writeTo(BuildConfigOperationsImpl.java:235)
[ERROR]     ... 56 more
[ERROR] -> [Help 1]

The build then falls back to the use of the oc binary that sometime fails with a timeout error.

To Reproduce
Steps to reproduce the behavior:

  1. Login to an openshift cluster

oc login --token=<some token> --server=<some server>

  1. Create a new quarkus project, using the kubernetes extension:
mvn io.quarkus:quarkus-maven-plugin:{quarkus-version}:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=openshift-quickstart \
    -DclassName="org.acme.rest.GreetingResource" \
    -Dpath="/greeting" \
    -Dextensions="kubernetes"
  1. Then perform a build and deploy:
mvn clean package -Dquarkus.kubernetes.deploy=true -Dquarkus.kubernetes.deployment-target=openshift

Configuration

# Depending on the cluster you may or may not need this:
quarkus.kubernetes-client.trust-certs=true
@iocanel iocanel added the kind/bug Something isn't working label Feb 27, 2020
@iocanel iocanel self-assigned this Feb 27, 2020
@iocanel iocanel mentioned this issue Feb 27, 2020
Closed
4 tasks
@iocanel
Copy link
Contributor Author

iocanel commented Feb 27, 2020

Additional information after ~5 hours, the build eventually fails, with the following logs

Receiving source from STDIN as archive ...
Caching blobs under "/var/cache/blobs".
Getting image source signatures
Copying blob sha256:1fdb7be5b7e04d78c06271c1fac140923afd7fceba99e670b43a4caeb9e12fe7
Copying blob sha256:7dc0dca2b1516961d6b3200564049db0a6e0410b370bb2189e2efae0d368616f
Copying blob sha256:d6fff4bb0996493ee898ab513fb1570c797d91a2d77365cb8ce60ac995d61bef
Copying blob sha256:e39e0acd05a132f9851ab89aa9c003304acaea15239d98a49e66e1a9c82b897f
Copying blob sha256:2fab1038512362e3ecc53bd088d15b573554228e40c3a8c2b6a35dd034319d2e
Copying blob sha256:4b44292fec1d5442d3e34b2c3adb430711d3c231d70954a9d42d2217f959a3a5
Copying blob sha256:edd9a9ea806e66bc81595225cd1eb3587288e91c3f0e7a0e050d090529da899c
Copying blob sha256:279f9e9dbd81ffde465c3613c8d42668fc7f90fcdd0ea2fc096dbe00d422ad3d
Copying blob sha256:3b0e4c1fe20d89e41b1a408582ee80141da58e972e1e85bb67a3bbb723ac5ba9
Copying blob sha256:444e42197d037bb691bfd39d27dadfddf260c117e648287ffdc9d06b0deb9d9b
Copying blob sha256:d4feb2e228a99db66740d8a9716e7b425433cf76c6c96ace4a240a48db221b8a
Copying blob sha256:728478adadf64ecda279321079d1a30871f5698a434390e893f4924cc3f3772b
Copying blob sha256:e7886c5bdbb9091a5fa49fecb3b38ae3b86b4dc3d898f35868ea304a07858fb1
Copying blob sha256:05562722fbe3d386d80195a54e38230f27521c800957f861bbe1f4aa16cee114
Copying blob sha256:8700ab630acc9c991fd8c8d432f57f8e667269a84cd6c4af7d4bb526d39a7fbd
Copying blob sha256:5d7ab3a9fe79144e76610e53f4ca88facbaa68d78a4d652dd5ec01b502a15eef
Copying blob sha256:6df3de21051156aa46c745270ffeaad903d7457901761412c7fdcc175425b11c
Copying blob sha256:86b908ea994135af1d1bebc0376f820b833c710d54db966a6248bfce32a6b0f0
Copying blob sha256:b1e6f56c9412d73422f3d05a420d8060910afe9df98eab26b64f2419be72b41c
Copying blob sha256:5530204593f325c8521f77e253694b97cc5390d1e14d0493c35b0eb75f4bc57e
Copying blob sha256:dd32d2faf9fa4e68f8a7195033339c20be7c43096b490f7e39f9778f2ba56e8b
Copying blob sha256:dde9d98ac0071a3ace2bd04b41e99cc440183f7a26083253bb612b9e5955f8ee
Copying blob sha256:2bea995a93a4fb35fffb148c149f340652271706e452e5f49f7e30988b29fe33
Copying blob sha256:94b14ee1a4420484af51a0ed5b9457c83523b69effdbfa57c0704501f19431e2
Copying blob sha256:854b48faa47da5e0268a141d13643359a3c01bbf40ef42361d2ee884715ecdf8
Copying blob sha256:b66cdd67f702702aa7ed8edbac84cd97c42e8357a5a519e57d94dc23ad144a64
Copying config sha256:10a0b9c43308245e87a0fef6a30e759ff18af8412788b28161afe04d02c990aa
Writing manifest to image destination
Storing signatures
Generating dockerfile with builder image fabric8/s2i-java@sha256:a6ba8a6e76ebd75f99f2431fe0ec39a82985bc21c0f9bd56021deca1ffde520c
STEP 1: FROM fabric8/s2i-java@sha256:a6ba8a6e76ebd75f99f2431fe0ec39a82985bc21c0f9bd56021deca1ffde520c
STEP 2: LABEL "io.openshift.build.source-location"="/tmp/build/inputs" "io.openshift.build.image"="fabric8/s2i-java@sha256:a6ba8a6e76ebd75f99f2431fe0ec39a82985bc21c0f9bd56021deca1ffde520c"
STEP 3: ENV OPENSHIFT_BUILD_NAME="kubernetes-quickstart-1" OPENSHIFT_BUILD_NAMESPACE="quarkus"
STEP 4: USER root
STEP 5: COPY upload/src /tmp/src
STEP 6: RUN chown -R 1000:0 /tmp/src
time="2020-02-27T19:14:44Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah191141109/mnt/rootfs\": error checking if \"/tmp/buildah191141109/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2020-02-27T19:14:44Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah191141109/mnt/rootfs\": error checking if \"/tmp/buildah191141109/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
STEP 7: USER 1000
STEP 8: RUN /usr/local/s2i/assemble
==================================================================
Starting S2I Java Build .....
S2I source build with plain binaries detected
Copying binaries from /tmp/src to /deployments ...
cp: cannot stat '/tmp/src/*': No such file or directory
Aborting due to error code 1 for copying /tmp/src to /deployments
subprocess exited with status 1
subprocess exited with status 1
error: build error: error building at STEP "RUN /usr/local/s2i/assemble": exit status 1

@iocanel iocanel mentioned this issue Mar 5, 2020
Merged
@gsmet gsmet added this to the 1.3.0.CR1 milestone Mar 5, 2020
@sbreault
Copy link

s2i-java-11 ClusterTask build (pipeline) in OCP v.4.8 ends in same failure as above:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
subprocess exited with status 1
subprocess exited with status 1
error building at STEP "RUN /usr/local/s2i/assemble": exit status 1
level=error msg="exit status 1"

However, this thread reads as though the problem is fixed.

I've taken the s2i-java-11 ClusterTask and made a custom Task within my project/namespace, and even added Quarkus build args:

      echo "MAVEN_S2I_ARTIFACT_DIRS=target" >> env-file

      echo "S2I_SOURCE_DEPLOYMENTS_FILTER='*-runner.jar lib'" >> env-file

      echo "JAVA_OPTIONS=-Dquarkus.native.container-build='true'
      -Dquarkus.container-image.build=s2i
      -Dquarkus.s2i.jar-file-name=getting-started-1.0.0-SNAPSHOT.jar
      -Dquarkus.native.debug.enabled='true'" >> env-file

      echo "AB_JOLOKIA_OFF=true" >> env-file

      echo "JAVA_APP_JAR=/deployments/quarkus-run.jar" >> env-file

But, it still ends in error. Please advise.

@iocanel
Copy link
Contributor Author

iocanel commented Jun 15, 2022

@sbreault I think that you are mixing concepts that are not meant to be mixed together.
More specifically, you are using s2i-java-11 which perfroms a full blown s2i build, but you are passing parameters that are meant to trigger internally an s2i binary build via quarkus-openshift. So, in other words you running an s2i build from within an s2i build. I would use one or the other.

@sbreault
Copy link

I added the Quarkus args to the build task because without them I get a NPE with very little info to troubleshoot the root cause:

Caused by: java.lang.NullPointerException
at io.quarkus.kubernetes.deployment.KubernetesDeployer.determineDeploymentTarget (KubernetesDeployer.java:152)
at io.quarkus.kubernetes.deployment.KubernetesDeployer.selectDeploymentTarget (KubernetesDeployer.java:68)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at io.quarkus.deployment.ExtensionLoader$2.execute (ExtensionLoader.java:920)
at io.quarkus.builder.BuildContext.run (BuildContext.java:277)
at org.jboss.threads.EnhancedQueueExecutor$Task.run (EnhancedQueueExecutor.java:2415)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run (EnhancedQueueExecutor.java:1452)
at java.lang.Thread.run (Thread.java:834)
at org.jboss.threads.JBossThread.run (JBossThread.java:501)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
subprocess exited with status 1
subprocess exited with status 1
error building at STEP "RUN /usr/local/s2i/assemble": exit status 1

@omeryounus
Copy link

STEP 4/9: USER root
STEP 5/9: COPY upload/src /tmp/src
STEP 6/9: RUN chown -R 1000:0 /tmp/src
STEP 7/9: USER 1000
STEP 8/9: RUN /usr/local/s2i/assemble

Starting S2I Java Build .....
S2I source build with plain binaries detected
Copying binaries from /tmp/src to /deployments ...
cp: cannot stat '/tmp/src/*': No such file or directory
Aborting due to error code 1 for copying /tmp/src to /deployments
error: build error: error building at STEP "RUN /usr/local/s2i/assemble": error while running runtime: exit status 1

@sbreault
Copy link

What are you saying with this reply? I know this is the s2i build command sequence - I see this work successfully with a s2i Java build using a BuildConfig:
image
My problem is with creating a s2i Java 11 pipeline using the OCP v. 4.8 ClusterTask because it ends in failure (NPE error; exist status 1).
I've created a Task based on this ClusterTask so I can modify it, but I haven't found a fix for the NPE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iocanel iocanel

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
1.3.0.CR1
Development

Successfully merging a pull request may close this issue.

Improvements around s2i and kubernetes deployment iocanel/quarkus
4 participants
@iocanel @gsmet @omeryounus @sbreault