Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WX-719 modernize cromwell-publish image #7013

Merged
merged 5 commits into from
Feb 21, 2023
Merged

WX-719 modernize cromwell-publish image #7013

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
adb44c5
Hack
aednichols Feb 16, 2023
f5aff19
Hack
aednichols Feb 16, 2023
11711df
Hack
aednichols Feb 16, 2023
c62fdba
Hack
aednichols Feb 16, 2023
bf88e1c
Make `latest` after successful test
aednichols Feb 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion processes/release_processes/README.MD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Make or copy the following files into a temporary `release/` directory outside t
"publish_workflow.githubToken": "<<GITHUB TOKEN VALUE>>",
"publish_workflow.majorRelease": true,
"publish_workflow.publishHomebrew": false,
"publish_workflow.publishDocker": "broadinstitute/cromwell-publish:latest",
"publish_workflow.publishDocker": "broadinstitute/cromwell-publish:ubuntu-nobrew",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to do the naming shuffle and formally call this the latest version of this image?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I wanted to make sure the release well and truly works before doing so - e.g. make sure we don't need to put the SBT path in the workflow.

"publish_workflow.organization": "broadinstitute"
}
```
Expand Down
5 changes: 2 additions & 3 deletions publish/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# Alternatively instead of `FROM linuxbrew/brew` we could run all of the steps used to install brew in docker-setup.sh
# https://github.com/Homebrew/brew/blob/0ff2afdfa8c5943a0e55d9bfe3cdb5d11da8342a/Dockerfile
FROM linuxbrew/brew
# Latest stable release including non-LTS
FROM ubuntu:rolling

WORKDIR /cromwell-publish/
COPY docker-setup.sh git-setup.sh ./
Expand Down
10 changes: 8 additions & 2 deletions publish/docker-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@

set -euo pipefail

export DOCKER_CLI_EXPERIMENTAL=enabled

docker buildx rm cromwell-multi-arch-builder || true
docker buildx create --use --name cromwell-multi-arch-builder

build_root="$( dirname "${BASH_SOURCE[0]}" )"
docker build "${build_root}" -t broadinstitute/cromwell-publish
docker push broadinstitute/cromwell-publish
docker buildx build "${build_root}" --platform linux/amd64,linux/arm64 -t broadinstitute/cromwell-publish:ubuntu-nobrew --push
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and other changes in this file necessary for Apple CPU support


docker buildx rm cromwell-multi-arch-builder
26 changes: 11 additions & 15 deletions publish/docker-setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,36 +4,32 @@

set -eou pipefail

apt-get update
apt-get install \
apt update
apt install \
apt-transport-https \
curl \
git \
gnupg \
wget \
ca-certificates \
unzip \
zip \
-y --no-install-recommends

# setup install for adoptopenjdk
# https://adoptopenjdk.net/installation.html#linux-pkg-deb
wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | apt-key add -
echo "deb https://adoptopenjdk.jfrog.io/adoptopenjdk/deb $(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This listing is abandoned since 2021 and has no recent releases, replacement steps below are from current instructions

grep UBUNTU_CODENAME /etc/os-release | cut -d = -f 2
) main" |
tee /etc/apt/sources.list.d/adoptopenjdk.list
mkdir -p /etc/apt/keyrings
wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public | tee /etc/apt/keyrings/adoptium.asc
echo "deb [signed-by=/etc/apt/keyrings/adoptium.asc] https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list
apt update
apt install -y temurin-11-jdk

# Install jq 1.6 to ensure --rawfile is supported
curl -L https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 -o /usr/bin/jq
chmod +x /usr/bin/jq

apt-get update
apt-get install \
adoptopenjdk-11-hotspot \
-y --no-install-recommends

# sbt launcher non-deb package installation instructions adapted from
# - https://github.com/sbt/sbt/releases/tag/v1.4.9
# - https://github.com/broadinstitute/scala-baseimage/pull/4/files
curl --location --fail --silent --show-error "https://github.com/sbt/sbt/releases/download/v1.5.5/sbt-1.5.5.tgz" |
curl --location --fail --silent --show-error "https://github.com/sbt/sbt/releases/download/v1.8.2/sbt-1.8.2.tgz" |
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to automatically get latest but using official SBT instructions for apt resulted in 1.8.2 on x86 and 1.4.9 on ARM.

tar zxf - -C /usr/share
update-alternatives --install /usr/bin/sbt sbt /usr/share/sbt/bin/sbt 1

Expand Down
2 changes: 1 addition & 1 deletion publish/publish_inputs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"publish_workflow.githubToken": "<<GITHUB TOKEN VALUE>>",
"publish_workflow.majorRelease": true,
"publish_workflow.publishHomebrew": false,
"publish_workflow.publishDocker": "broadinstitute/cromwell-publish:latest",
"publish_workflow.publishDocker": "broadinstitute/cromwell-publish:ubuntu-nobrew",
"publish_workflow.organization": "broadinstitute"
}
2 changes: 1 addition & 1 deletion publish/publish_workflow.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -633,7 +633,7 @@ workflow publish_workflow {
String organization
Boolean majorRelease = true
Boolean publishHomebrew = true
String publishDocker = "broadinstitute/cromwell-publish:latest"
String publishDocker = "broadinstitute/cromwell-publish:ubuntu-nobrew"
}

parameter_meta {
Expand Down