Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: Parallelize the CI image builds #25564

Conversation

sebastianliebscher
Copy link
Contributor

@sebastianliebscher sebastianliebscher commented Oct 8, 2023

SUMMARY

This PR replaces the custom script docker_build_push.sh that runs the docker builds sequentially with GitHub action jobs running in parallel.

  • set tags with docker/metadata-action GitHub Action
  • build images with docker/build-push-action GitHub Action
  • on release tag docker image with docker/metadata-action tags: type=pep440,pattern={{version}}

No functionality changed, except:

  • removes docker buildx local cache usage as every build runs on its own job hence on different machines where local cache does not apply
  • docker buildx cache will be a follow-up PR to use GitHub's container registry to speed up builds

TESTING INSTRUCTIONS

Use case: PR from fork (can be verified by the successful jobs from this PR)

  • detect that no secrets are present
  • images build without failures
  • the subsequent steps to push the images are skipped as intended (PR from fork)
  • tags: (no tags)

Use case: PR from Superset repo

Use case: commit to master

Use case: release


ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

- replaces custom script to set tags with docker/metadata-action GitHub Action
- replaces custom script to sequentially build images with docker/build-push-action GitHub Action
- moves docker-release.yml logic into docker.yml by utilizing docker/metadata-action 'tags: type=pep440,pattern={{version}}'
- removes docker buildx local cache usage as every build runs on its own job hence on different machines (docker buildx registry cache will be a follow-up PR)
@@ -21,33 +24,253 @@ jobs:
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
echo "has secrets!"
else
echo "has-secrets=0" >> "$GITHUB_OUTPUT"
echo "no secrets!"
fi
Copy link
Contributor Author

@sebastianliebscher sebastianliebscher Oct 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work as expected. The check if: needs.config.outputs.has-secrets returns true in both cases.

Either check for if: needs.config.outputs.has-secrets == 1 or don't set has-secrets when there are no secrets. I went with the latter.

@sebastianliebscher sebastianliebscher changed the title [WIP] build: Parallelize the CI image builds build: Parallelize the CI image builds Oct 8, 2023
@sebastianliebscher sebastianliebscher marked this pull request as ready for review October 8, 2023 13:01
@craig-rueda
Copy link
Member

Any updates here? Looks like there's conflicts :)

@mistercrunch
Copy link
Member

I pull this, rebased and reopened here #26697

@rusackas
Copy link
Member

Closing this since #26698 merged

@rusackas rusackas closed this Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants