Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Triggering s2i binary builds sometimes fails with stream was reset #2032

Closed
4 tasks
iocanel opened this issue Feb 27, 2020 · 4 comments
Closed
4 tasks

Triggering s2i binary builds sometimes fails with stream was reset #2032

iocanel opened this issue Feb 27, 2020 · 4 comments
Assignees

Comments

@iocanel
Copy link
Member

iocanel commented Feb 27, 2020

Using 4.7.0 version of the client.
In some cases the following code:

client.buildConfigs().withName("my-build")
                               .instantiateBinary()
                               .withTimeoutInMillis(5*60*1000)
                               .fromFile(binaryFile);

Seems to trigger the following error:

[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:999-SNAPSHOT:build (default) on project openshift-quickstart: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR]     [error]: Build step io.quarkus.container.image.s2i.deployment.S2iProcessor#s2iBuildFromJar threw an exception: java.lang.RuntimeException: Execution of s2i build failed. See s2i output for more details
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iException(S2iProcessor.java:291)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuild(S2iProcessor.java:272)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.lambda$s2iBuild$10(S2iProcessor.java:252)
[ERROR]     at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
[ERROR]     at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
[ERROR]     at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
[ERROR]     at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
[ERROR]     at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
[ERROR]     at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
[ERROR]     at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
[ERROR]     at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
[ERROR]     at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[ERROR]     at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuild(S2iProcessor.java:252)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.createContainerImage(S2iProcessor.java:220)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuildFromJar(S2iProcessor.java:143)
[ERROR]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR]     at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:938)
[ERROR]     at io.quarkus.builder.BuildContext.run(BuildContext.java:273)
[ERROR]     at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[ERROR]     at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2027)
[ERROR]     at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1551)
[ERROR]     at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1442)
[ERROR]     at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR]     at org.jboss.threads.JBossThread.run(JBossThread.java:479)
[ERROR] Caused by: io.dekorate.deps.kubernetes.client.KubernetesClientException: Can't instantiate binary build, due to error reading/writing stream. Can be caused if the output stream was closed by the server.
[ERROR]     at io.dekorate.deps.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl$1.writeTo(BuildConfigOperationsImpl.java:237)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:72)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.openshift.client.internal.OpenShiftOAuthInterceptor.intercept(OpenShiftOAuthInterceptor.java:96)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]     at io.dekorate.deps.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
[ERROR]     at io.dekorate.deps.okhttp3.RealCall.execute(RealCall.java:93)
[ERROR]     at io.dekorate.deps.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:469)
[ERROR]     at io.dekorate.deps.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:448)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromInputStream(BuildConfigOperationsImpl.java:251)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromFile(BuildConfigOperationsImpl.java:208)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromFile(BuildConfigOperationsImpl.java:65)
[ERROR]     at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuild(S2iProcessor.java:266)
[ERROR]     ... 26 more
[ERROR] Caused by: io.dekorate.deps.okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR
[ERROR]     at io.dekorate.deps.okhttp3.internal.http2.Http2Stream.checkOutNotClosed(Http2Stream.java:644)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http2.Http2Stream$FramingSink.emitFrame(Http2Stream.java:578)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http2.Http2Stream$FramingSink.write(Http2Stream.java:558)
[ERROR]     at io.dekorate.deps.okio.ForwardingSink.write(ForwardingSink.java:35)
[ERROR]     at io.dekorate.deps.okhttp3.internal.http.CallServerInterceptor$CountingSink.write(CallServerInterceptor.java:149)
[ERROR]     at io.dekorate.deps.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:179)
[ERROR]     at io.dekorate.deps.okio.RealBufferedSink.writeAll(RealBufferedSink.java:107)
[ERROR]     at io.dekorate.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl$1.writeTo(BuildConfigOperationsImpl.java:235)
[ERROR]     ... 56 more
[ERROR] -> [Help 1]

The error seems to happen often in certain clusters, while others seem completely unaffected.

The same code run perfrectly fine in some clusters:

  • v3.11
    -v4.3
    -v4.4

while in some others it didn't:
-v3.11
-v4.2

So, I am thinking that its more like an environment issue rather a version issue.

Important: In some cases falling back to oc start-build --from-archive does seem to work. While in other cases oc it fails with a timeout error. So, there seems to be some room of improvement here.

Summing up the tasks needed for this one:

  • Identify why the stream is getting closed in some cases
  • Improve the client so that it works, at least as robust as oc.
  • Improve error message and delegate to admins (if they have to).
  • Create a FAQ about this issue

Issue has been originally reported at: quarkusio/quarkus#7454

@rohanKanojia rohanKanojia added bug platform/openshift Specific to OpenShift labels Feb 27, 2020
@iocanel
Copy link
Member Author

iocanel commented Feb 27, 2020

@rhuss: If I remember correctly, we've seen this issue before on FMP and it was then when we added Can be caused if the output stream was closed by the server.. Did you ever figured out why Openshift was closing the stream?

@rhuss
Copy link
Contributor

rhuss commented Mar 1, 2020

@iocanel oh, the good old times ;-) Yes, I remember we had a similar issue but I can't really remember the cause or fix :(. I think it could be a myriad of errors, hidden by a bad error message. For one of those errors causing this message, e.g. see https://issues.redhat.com/browse/ENTESB-8791

@stale
Copy link

stale bot commented May 30, 2020

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

@stale stale bot added the status/stale label May 30, 2020
@stale stale bot removed the status/stale label May 30, 2020
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Jun 18, 2020
…abric8io#2032)

+ Add BuildConfig's name and namespace in query parameters in order to align with `oc`
+ Use BufferedInputStream while uploading archive file to sink.
+ Add recent events to exception message and some logging in case of upload failure of archive file
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Jun 18, 2020
…abric8io#2032)

+ Add BuildConfig's name and namespace in query parameters in order to align with `oc`
+ Use BufferedInputStream while uploading archive file to sink.
+ Add recent events to exception message and some logging in case of upload failure of archive file
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Jul 6, 2020
…abric8io#2032)

+ Add BuildConfig's name and namespace in query parameters in order to align with `oc`
+ Use BufferedInputStream while uploading archive file to sink.
+ Add recent events to exception message and some logging in case of upload failure of archive file
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Jul 13, 2020
…abric8io#2032)

+ Add BuildConfig's name and namespace in query parameters in order to align with `oc`
+ Use BufferedInputStream while uploading archive file to sink.
+ Add recent events to exception message and some logging in case of upload failure of archive file
rohanKanojia added a commit that referenced this issue Jul 13, 2020
…2032)

+ Add BuildConfig's name and namespace in query parameters in order to align with `oc`
+ Use BufferedInputStream while uploading archive file to sink.
+ Add recent events to exception message and some logging in case of upload failure of archive file
@rohanKanojia
Copy link
Member

Shall we close this? PR with supposedly improved error handling seems to be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants