Skip to content

How to release GATK4

Jump to bottom
droazen edited this page Sep 18, 2018 · 31 revisions

One-time setup tasks

In order to be able to release GATK4, you first need to perform the following one-time setup tasks:

  • Set up your sonatype account

    • If you don't already have a sonatype account, create one by going to https://issues.sonatype.org/secure/Signup!default.jspa.

    • Check whether your sonatype account already has Deploy and UI Upload permissions for repo target org.broadinstitute by going to https://oss.sonatype.org/#stagingProfiles while logged in.

    • If you don't have Deploy and UI Upload permissions for org.broadinstitute, you need to request them. Ask someone who already has deployment permission to open a ticket on your behalf at https://issues.sonatype.org requesting permissions for you, and comment on the ticket yourself from your sonatype account once it's opened.

  • Set up your signing keys

    • Download GnuPG from http://www.gnupg.org/download/ if you don't already have it installed. The instructions below apply to GnuPG 1.x, so download the latest 1.x version if possible. The procedure for GnuPG 2.x is likely different.

    • Run gpg --gen-key to generate a new keypair (NOTE: on some systems, the gpg command will be gpg1 or gpg2). Select the default values when asked for the kind and the size of the keys, and be sure to specify a passphrase for the keypair.

    • Run gpg --list-keys and gpg --list-secret-keys. The output should look something like this:

      $ gpg --list-keys
/Users/YOUR_USERNAME/.gnupg/pubring.gpg
---------------------------------
pub   4096R/4C1B2A36 2016-05-17
uid                  YOUR NAME <YOUR@EMAIL>
sub   4096R/26A37141 2016-05-17

    • Take note of the path to your secret keyring file (eg., /Users/YOUR_USERNAME/.gnupg/secring.gpg) and the public key id (4C1B2A36 in this case).

    • Publish your public key by running gpg --keyserver hkp://pool.sks-keyservers.net --send-keys YOUR_PUBLIC_KEY_ID, replacing YOUR_PUBLIC_KEY_ID with the key ID for your public key noted above (4C1B2A36 in the example).

  • Set up your credentials in gradle

    • Open ~/.gradle/gradle.properties in a text editor (creating it if necessary).

    • Add the following lines:

      signing.keyId=ID_FOR_YOUR_PUBLIC_KEY
signing.password=PASSPHRASE_FOR_YOUR_KEYPAIR
signing.secretKeyRingFile=PATH_TO_YOUR_SECRET_KEYRING_FILE
sonatypeUsername=YOUR_SONATYPE_USERNAME
sonatypePassword=YOUR_SONATYPE_PASSWORD

    • Save the file.

  • Set up docker and your dockerhub account

    • Install docker from docker.com, and start the docker daemon.

    • Test that you can pull and run the latest GATK4 docker image:

      docker pull broadinstitute/gatk:latest
docker run -it broadinstitute/gatk:latest

    • If you don't already have a dockerhub account, create one by going to hub.docker.com

    • Make sure that you have push access to thehttps://hub.docker.com/r/broadinstitute/gatk/ repository on dockerhub, as well as the us.gcr.io/broad-gatk/gatk repository on GCR. Ask to join the gatk-release google group, which is used to manage these permissions (you may need to verify that you have been granted access).

    • Enable "Experimental Features" in your Docker settings (Docker -> Preferences -> Daemon).

Releasing GATK4

After you've performed the one-time setup steps above, here's what you need to do each time you want to release GATK4:

  • Before releasing

    • Check with other GATK developers to confirm that it's a good time to release. In particular:

      • Tools and arguments not ready for production use should be marked as @BetaFeature

      • If a developer has added unstable functionality to a previously-stable tool, their new functionality should be isolated into a separate code path that's off by default and activated using an argument marked as experimental.

    • Make sure that all tests are passing on master in travis!

  • Writing release notes

    • Write release notes in advance and save them in a text editor, otherwise you'll introduce a significant delay in the middle of the release process.

    • Use a markdown-formatted bulleted list, and include a bullet point for each new feature or bug fix in the release, with a brief summary of the major changes at the top. Consult with other GATK developers if necessary to understand new features/patches. Try to include links to the relevant github issue(s) for each item.

    • You can get a list of the commits since the last release by going to the tag for the previous release in https://github.com/broadinstitute/gatk/releases and clicking the X commits to master since this release link. This can form the basis for your release notes.

    • A good way to quickly auto-generate a skeleton for markdown-formatted release notes is by running this command from your GATK4 clone with the latest master checked out:

      git log --pretty=oneline PREVIOUS_VERSION..HEAD | grep -v "Merge pull request" | awk '{ $1=""; print "* " $0; }'

      Replacing PREVIOUS_VERSION with the tag for the previous release

      You'll have to tweak the output of this command to produce the final release notes, as not all commit messages will be sufficiently descriptive, and you'll need to add links to the appropriate github issues.

  • Preparing your git clone

    • In your git clone of GATK4, save any uncommitted work and run git checkout -f master. Running git diff HEAD should produce no output.

    • Run git pull --ff-only from your master branch to bring it up to date. Check the commit at the top of the git log output and make sure that it matches the most recent commit at https://github.com/broadinstitute/gatk/commits/master.

    • Run ./gradlew clean to clean your build directory.

  • Releasing to github

    • Decide on a version number for the new GATK4 release (for example, 4.beta.3).

    • Be sure that you still have the latest master branch commit checked out, as instructed above.

    • Tag the release by running git tag -a VERSION_NUMBER, where VERSION_NUMBER is the version you've chosen for the new release (eg., 4.beta.3). Your text editor will pop up and ask you to write a commit message for the new tag (this can just be the release number for now). After doing so, save and then exit. git describe should now print the new version.

    • Push the new tag to github by running git push origin VERSION_NUMBER, where VERSION_NUMBER is the tag you just created (for example, git push origin 4.beta.3).

    • Run ./gradlew clean bundle -Drelease=true. This will create a zip for the release in build/

    • Go to https://github.com/broadinstitute/gatk/releases, click on the new tag you just pushed, then click "Edit Release".

    • Attach the GATK4 release zip file you just created by clicking on the Attach binaries by dropping them here or selecting them link, and navigating to the zip file in the build/ subdirectory within your GATK4 clone.

    • Paste the release notes you wrote earlier into the text box. Look at the markdown preview and make sure that they look reasonable.

    • Once the zip file has finished uploading to github, click the Publish Release button.

    • Verify that the new release looks ok in https://github.com/broadinstitute/gatk/releases, and has an attached (non-source-code) GATK4 zip file.

  • Releasing to dockerhub

    • Ensure that the docker daemon is running, and that you still have the latest master checked out, as instructed above. You'll also need to have done the steps for the github release already.

    • If you've done a docker release in the past, be sure to delete any old docker staging directories(s) left over in your GATK4 clone. You'll have to use sudo to do the deletion.

    • From the root of your GATK clone, run:

      bash build_docker.sh -e ${GITHUB_TAG} -p -u -d ${STAGING_DIR}

      Replace ${GITHUB_TAG} with the tag for your new release that you created above (eg., 4.beta.3), and ${STAGING_DIR} with the name of the directory that should be used to build the image. This directory should either not exist or be empty!

    • When prompted by the script, enter your dockerhub username and password.

    • After the script completes (may take 15-20 minutes depending on your upload speed), go to https://hub.docker.com/r/broadinstitute/gatk/tags/ and confirm that the new image is there.

    • Add a link to the new image in the release notes you wrote on github.

  • Releasing to maven central

    • Be sure that you still have the latest master branch commit checked out, as instructed above. You'll also need to have done the steps for the github release already.

    • Check GATK4's dependencies in build.gradle. If there are any dependencies on a SNAPSHOT of a library, you can't release to maven central, and should skip the remainder of this section.

    • Run ./gradlew clean uploadArchives -Drelease=true. This builds the release and uploads it to sonatype's staging area.

    • Go to https://oss.sonatype.org/#stagingRepositories, logging in if necessary. If you don't see anything, click "refresh".

    • Find the release you just uploaded. It will probably be at the bottom with a name like orgbroadinstitute-1027, with your user ID listed as owner.

    • Check the box next to your release, then select "close". Press the refresh button repeatedly until it updates and its status says "closed".

    • Select your release again and click "release". Select the box to "automatically drop" in the pop-up confirmation dialog.

    • Wait ~30-180 minutes for the maven central release to happen. Once it happens, it will show up on http://maven.org. Due to caching on maven's website it's often available by directly navigating to the coordinates sooner than it shows up in search.

Clone this wiki locally