Flakey Build Failures in CI

[cgardens]

Expected Behavior

• Build outcome should be the same each time the build is run for the same commit (assuming no externalities, e.g. can't fetch a pip lib)

Current Behavior

• Build outcome varies. Perhaps due to caching?
• this commit: ba423ab is an example. though we have seen this in numerous instances now.
  • potentially relevant information:
    • It seems like waiting or the success of the build of some other commit seems to allow subsequent runs of commits like this to succeed
    • We have seen our github action instance run out of disk space before. potentially that could be related, which is why we commented out the python standard tests on master for now (commit). Maybe related here too? SO.
• Logs are attached of two runs, one success, one failure. It seems to be that either docker is not correctly building an image or able to find an image. This happens intermittently.

logs_4853_failure.zip
logs_4863_success.zip

most relevant part of error logs:

2020-11-28T04:41:00.4572852Z > Task :airbyte-integrations:connectors:source-marketo-singer:compileTestJava NO-SOURCE
2020-11-28T04:41:00.4575829Z > Task :airbyte-integrations:connectors:source-marketo-singer:processTestResources NO-SOURCE
2020-11-28T04:41:00.4578764Z > Task :airbyte-integrations:connectors:source-marketo-singer:testClasses UP-TO-DATE
2020-11-28T04:41:00.4581460Z > Task :airbyte-integrations:connectors:source-marketo-singer:test NO-SOURCE
2020-11-28T04:41:00.4584664Z > Task :airbyte-integrations:connectors:source-marketo-singer:jacocoTestReport SKIPPED
2020-11-28T04:41:00.4587443Z > Task :airbyte-integrations:connectors:source-marketo-singer:check UP-TO-DATE
2020-11-28T04:41:00.4590147Z > Task :airbyte-integrations:connectors:source-marketo-singer:build
2020-11-28T04:41:03.2570054Z > Task :airbyte-integrations:connectors:source-mysql:compileJava
2020-11-28T04:41:03.9570263Z > Task :airbyte-integrations:connectors:source-mysql:processResources
2020-11-28T04:41:03.9572848Z > Task :airbyte-integrations:connectors:source-mysql:classes
2020-11-28T04:41:04.0568953Z > Task :airbyte-integrations:connectors:source-mysql:jar
2020-11-28T04:41:04.2607964Z > Task :airbyte-integrations:connectors:source-mysql:startScripts
2020-11-28T04:41:04.3589130Z > Task :airbyte-integrations:connectors:source-mysql:distTar
2020-11-28T04:41:04.3591586Z > Task :airbyte-integrations:connectors:source-mssql:test
2020-11-28T04:41:05.8568473Z > Task :airbyte-integrations:connectors:source-mysql:distZip
2020-11-28T04:41:05.8571017Z > Task :airbyte-integrations:connectors:source-mysql:assemble
2020-11-28T04:41:07.8570143Z Error response from daemon: manifest for localhost:5000/airbyte/source-mysql:dev not found: manifest unknown: manifest unknown
2020-11-28T04:41:07.8571834Z 
2020-11-28T04:41:07.8573559Z #1 [internal] load .dockerignore
2020-11-28T04:41:07.8575495Z #1 transferring context: 34B done
2020-11-28T04:41:07.8578105Z > Task :airbyte-integrations:connectors:source-mysql:airbyteDocker
2020-11-28T04:41:09.6569529Z #1 DONE 1.9s
2020-11-28T04:41:09.6572854Z 
2020-11-28T04:41:09.6575010Z #2 [internal] load build definition from Dockerfile
2020-11-28T04:41:09.6577086Z #2 transferring dockerfile: 37B done
2020-11-28T04:41:09.7570092Z #2 DONE 2.0s
2020-11-28T04:41:10.3568541Z 
2020-11-28T04:41:10.3573763Z #3 [internal] load metadata for docker.io/airbyte/integration-base-java:dev
2020-11-28T04:41:10.3575600Z #3 DONE 0.0s
2020-11-28T04:41:10.3576356Z 
2020-11-28T04:41:10.3578288Z #4 [1/4] FROM docker.io/airbyte/integration-base-java:dev
2020-11-28T04:41:10.3579972Z #4 DONE 0.0s
2020-11-28T04:41:10.3580740Z 
2020-11-28T04:41:10.3582985Z #9 importing cache manifest from localhost:5000/airbyte/source-mysql:dev
2020-11-28T04:41:10.3585246Z #9 ERROR: localhost:5000/airbyte/source-mysql:dev not found
2020-11-28T04:41:10.3586296Z 
2020-11-28T04:41:10.3587740Z #6 [internal] load build context
2020-11-28T04:41:10.3589365Z #6 transferring context: 66B done
2020-11-28T04:41:10.3590841Z #6 DONE 0.0s
2020-11-28T04:41:10.3591605Z 
2020-11-28T04:41:10.3592942Z #5 [2/4] WORKDIR /airbyte
2020-11-28T04:41:10.3594346Z #5 CACHED
2020-11-28T04:41:10.3595218Z 
2020-11-28T04:41:10.3602918Z #7 [3/4] COPY build/distributions/source-mssql*.tar source-mssql.tar
2020-11-28T04:41:16.9567553Z #7 DONE 6.7s
2020-11-28T04:41:17.1568566Z 
2020-11-28T04:41:17.1571588Z #8 [4/4] RUN tar xf source-mssql.tar --strip-components=1
2020-11-28T04:41:17.4572018Z #8 0.416 tar: source-mssql.tar: Cannot open: No such file or directory
2020-11-28T04:41:17.4574498Z #8 0.416 tar: Error is not recoverable: exiting now
2020-11-28T04:41:17.8574121Z #8 ERROR: executor failed running [/bin/sh -c tar xf $***APPLICATION***.tar --strip-components=1]: runc did not terminate sucessfully
2020-11-28T04:41:17.8577357Z ------
2020-11-28T04:41:17.8578140Z 
2020-11-28T04:41:17.8580878Z  > importing cache manifest from localhost:5000/airbyte/source-mysql:dev:
2020-11-28T04:41:17.8584154Z > Task :airbyte-integrations:connectors:source-mysql:airbyteDocker FAILED
2020-11-28T04:41:17.8586282Z ------
2020-11-28T04:41:17.8587868Z ------
2020-11-28T04:41:17.8589863Z  > [4/4] RUN tar xf source-mssql.tar --strip-components=1:
2020-11-28T04:41:17.8591801Z ------
2020-11-28T04:41:17.8594864Z failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c tar xf $***APPLICATION***.tar --strip-components=1]: runc did not terminate sucessfully
2020-11-28T04:41:36.4567463Z 
2020-11-28T04:41:36.4570968Z > Task :airbyte-integrations:connectors:source-mssql:jacocoTestReport
2020-11-28T04:41:36.4572240Z 
2020-11-28T04:41:36.4574140Z Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
2020-11-28T04:41:36.4576586Z Use '--warning-mode all' to show the individual deprecation warnings.
2020-11-28T04:41:36.4579030Z See https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings
2020-11-28T04:41:36.4581404Z 358 actionable tasks: 357 executed, 1 up-to-date
2020-11-28T04:41:36.4584002Z FAILURE: Build failed with an exception.
2020-11-28T04:41:36.4585193Z 
2020-11-28T04:41:36.4586683Z * What went wrong:
2020-11-28T04:41:36.4589194Z Execution failed for task ':airbyte-integrations:connectors:source-mysql:airbyteDocker'.
2020-11-28T04:41:36.4594383Z > Process 'command '/home/runner/work/airbyte/airbyte/tools/bin/build_image.sh'' finished with non-zero exit value 1
2020-11-28T04:41:36.4595673Z 
2020-11-28T04:41:36.4596931Z * Try:
2020-11-28T04:41:36.4599354Z Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
2020-11-28T04:41:36.4600751Z 
2020-11-28T04:41:36.4602331Z * Get more help at https://help.gradle.org
2020-11-28T04:41:36.4603366Z 
2020-11-28T04:41:36.4604700Z BUILD FAILED in 14m 46s

[cgardens]

another example of this here: https://github.com/airbytehq/airbyte/runs/1472846366 25e1378

[cgardens]

@jrhizor - i haven't seen any of these in awhile. did you just hack around this or is there a sustainable fix here so we can close it? can't remember.

[jrhizor]

I think this can be marked as fixed. There were several improvements to dependencies and removal of various flaky optimizations for the build. So unfortunately the base build is slower now...

I'll mark this as closed. I'm going to try turning back on caching this morning. If it is flaky we can just revert it and accept the times.