-
Notifications
You must be signed in to change notification settings - Fork 4.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
We're on a quest to reduce our pipeline execution time to both enhance
our developer productivity but also to reduce the overall cost of the CI
pipeline. The strategy we use here reduces workflow execution time and
network I/O cost by reducing our module cache size and using binary
external tools when possible. We no longer download modules and build
many of the external tools thousands of times a day.
Our previous process of installing internal and external developer tools
was scattered and inconsistent. Some tools were installed via `go
generate -tags tools ./tools/...`,
others via various `make` targets, and some only in Github Actions
workflows. This process led to some undesirable side effects:
* The modules of some dev and test tools were included with those
of the Vault project. This leads to us having to manage our own
Go modules with those of external tools. Prior to Go 1.16 this
was the recommended way to handle external tools, but now
`go install tool@version` is the recommended way to handle
external tools that need to be build from source as it supports
specific versions but does not modify the go.mod.
* Due to Github cache constraints we combine our build and test Go
module caches together, but having our developer tools as deps in
our module results in a larger cache which is downloaded on every
build and test workflow runner. Removing the external tools that were
included in our go.mod reduced the expanded module cache by size
by ~300MB, thus saving time and network I/O costs when downloading
the module cache.
* Not all of our developer tools were included in our modules. Some were
being installed with `go install` or `go run`, so they didn't take
advantage of a single module cache. This resulted in us downloading
Go modules on every CI and Build runner in order to build our
external tools.
* Building our developer tools from source in CI is slow. Where possible
we can prefer to use pre-built binaries in CI workflows. No more
module download or tool compiles if we can avoid them.
I've refactored how we define internal and external build tools
in our Makefile and added several new targets to handle both building
the developer tools locally for development and verifying that they are
available. This allows for an easy developer bootstrap while also
supporting installation of many of the external developer tools from
pre-build binaries in CI. This reduces our network IO and run time
across nearly all of our actions runners.
While working on this I caught and resolved a few unrelated issue:
* Both our Go and Proto format checks we're being run incorrectly. In
CI they we're writing changes but not failing if changes were
detected. The Go was less of a problem as we have git hooks that
are intended to enforce formatting, however we drifted over time.
* Our Git hooks couldn't handle removing a Go file without failing. I
moved the diff check into the new Go helper and updated it to handle
removing files.
* I combined a few separate scripts and into helpers and added a few
new capabilities.
* I refactored how we install Go modules to make it easier to download
and tidy all of the projects go.mod's.
* Refactor our internal and external tool installation and verification
into a tools.sh helper.
* Combined more complex Go verification into `scripts/go-helper.sh` and
utilize it in the `Makefile` and git commit hooks.
* Add `Makefile` targets for executing our various tools.sh helpers.
* Update our existing `make` targets to use new tool targets.
* Normalize our various scripts and targets output to have a consistent
output format.
* In CI, install many of our external dependencies as binaries wherever
possible. When not possible we'll build them from scratch but not mess
with the shared module cache.
* [QT-641] Remove our external build tools from our project Go modules.
* [QT-641] Remove extraneous `go list`'s from our `set-up-to` composite
action.
* Fix formatting and regen our protos
Signed-off-by: Ryan Cragun <[email protected]>- Loading branch information
1 parent
fd52fc7
commit da714f3
Showing
54 changed files
with
977 additions
and
424 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: BUSL-1.1 | ||
|
|
||
| --- | ||
| name: Install external tools for CI | ||
| description: Install external tools CI | ||
|
|
||
| # When possible, prefer installing pre-built external tools for speed. This allows us to avoid | ||
| # downloading modules and compiling external tools on CI runners. | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - uses: ./.github/actions/set-up-buf | ||
| with: | ||
| version: v1.25.0 # This should match the version in tools/tool.sh | ||
| - uses: ./.github/actions/set-up-gofumpt | ||
| - uses: ./.github/actions/set-up-gotestsum | ||
| - uses: ./.github/actions/set-up-misspell | ||
| - uses: ./.github/actions/set-up-staticcheck | ||
| # We assume that the Go toolchain will be managed by the caller workflow so we don't set one | ||
| # up here. | ||
| - run: go install google.golang.org/protobuf/cmd/protoc-gen-go@latest | ||
| shell: bash | ||
| - run: go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest | ||
| shell: bash | ||
| - run: go install github.com/favadi/protoc-go-inject-tag@latest | ||
| shell: bash | ||
| - run: go install golang.org/x/tools/cmd/goimports@latest | ||
| shell: bash | ||
| - run: go install github.com/golangci/revgrep/cmd/revgrep@latest | ||
| shell: bash |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: BUSL-1.1 | ||
|
|
||
| --- | ||
| name: Set up buf from Github releases | ||
| description: Set up buf from Github releases | ||
|
|
||
| inputs: | ||
| destination: | ||
| description: "Where to install the buf binary (default: $HOME/bin/buf)" | ||
| type: boolean | ||
| default: "$HOME/bin/buf" | ||
| version: | ||
| description: "The version to install (default: latest)" | ||
| type: string | ||
| default: Latest | ||
|
|
||
| outputs: | ||
| destination: | ||
| description: Where the installed buf binary is | ||
| value: ${{ steps.install.outputs.destination }} | ||
| destination-dir: | ||
| description: The directory where the installed buf binary is | ||
| value: ${{ steps.install.outputs.destination-dir }} | ||
| version: | ||
| description: The installed version of buf | ||
| value: ${{ steps.install.outputs.version }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - id: install | ||
| shell: bash | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| VERSION=$(gh release list -R bufbuild/buf --exclude-drafts --exclude-pre-releases | grep ${{ inputs.version }} | cut -f1) | ||
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" | ||
| mkdir -p $(dirname ${{ inputs.destination }}) | ||
| DESTINATION="$(readlink -f "${{ inputs.destination }}")" | ||
| echo "destination=$DESTINATION" >> "$GITHUB_OUTPUT" | ||
| DESTINATION_DIR="$(dirname "$DESTINATION")" | ||
| echo "$DESTINATION_DIR" >> "$GITHUB_PATH" | ||
| echo "destination-dir=$DESTINATION_DIR" >> "$GITHUB_OUTPUT" | ||
| ARCH="$(echo "$RUNNER_ARCH" | tr '[:upper:]' '[:lower:]')" | ||
| OS="$RUNNER_OS" | ||
| if [ "$ARCH" = "x64" ]; then | ||
| export ARCH="x86_64" | ||
| fi | ||
| if [ "$ARCH" = "arm64" ] && [ "$OS" = "Linux" ]; then | ||
| export ARCH="aarch64" | ||
| fi | ||
| if [ "$OS" = "macOS" ]; then | ||
| export OS="Darwin" | ||
| fi | ||
| mkdir -p tmp | ||
| gh release download "$VERSION" -p "buf-${OS}-${ARCH}.tar.gz" -O tmp/buf.tgz -R bufbuild/buf | ||
| pushd tmp && tar -xvf buf.tgz && popd | ||
| mv tmp/buf/bin/buf "$DESTINATION" | ||
| rm -rf tmp |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: BUSL-1.1 | ||
|
|
||
| --- | ||
| name: Set up gofumpt from Github releases | ||
| description: Set up gofumpt from Github releases | ||
|
|
||
| inputs: | ||
| destination: | ||
| description: "Where to install the gofumpt binary (default: $HOME/bin/gofumpt)" | ||
| type: boolean | ||
| default: "$HOME/bin/gofumpt" | ||
| version: | ||
| description: "The version to install (default: latest)" | ||
| type: string | ||
| default: Latest | ||
|
|
||
| outputs: | ||
| destination: | ||
| description: Where the installed gofumpt binary is | ||
| value: ${{ steps.install.outputs.destination }} | ||
| destination-dir: | ||
| description: The directory where the installed gofumpt binary is | ||
| value: ${{ steps.install.outputs.destination-dir }} | ||
| version: | ||
| description: The installed version of gofumpt | ||
| value: ${{ steps.install.outputs.version }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - id: install | ||
| shell: bash | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| VERSION=$(gh release list -R mvdan/gofumpt --exclude-drafts --exclude-pre-releases | grep ${{ inputs.version }} | cut -f1) | ||
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" | ||
| mkdir -p $(dirname ${{ inputs.destination }}) | ||
| DESTINATION="$(readlink -f "${{ inputs.destination }}")" | ||
| echo "destination=$DESTINATION" >> "$GITHUB_OUTPUT" | ||
| DESTINATION_DIR="$(dirname "$DESTINATION")" | ||
| echo "$DESTINATION_DIR" >> "$GITHUB_PATH" | ||
| echo "destination-dir=$DESTINATION_DIR" >> "$GITHUB_OUTPUT" | ||
| ARCH="$(echo "$RUNNER_ARCH" | tr '[:upper:]' '[:lower:]')" | ||
| OS="$(echo "$RUNNER_OS" | tr '[:upper:]' '[:lower:]')" | ||
| if [ "$ARCH" = "x64" ]; then | ||
| export ARCH="amd64" | ||
| fi | ||
| if [ "$OS" = "macos" ]; then | ||
| export OS="darwin" | ||
| fi | ||
| gh release download "$VERSION" -p "gofumpt_*_${OS}_${ARCH}" -O gofumpt -R mvdan/gofumpt | ||
| chmod +x gofumpt | ||
| mv gofumpt "$DESTINATION" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: BUSL-1.1 | ||
|
|
||
| --- | ||
| name: Set up misspell from Github releases | ||
| description: Set up misspell from Github releases | ||
|
|
||
| inputs: | ||
| destination: | ||
| description: "Where to install the misspell binary (default: $HOME/bin/misspell)" | ||
| type: boolean | ||
| default: "$HOME/bin/misspell" | ||
| version: | ||
| description: "The version to install (default: latest)" | ||
| type: string | ||
| default: Latest | ||
|
|
||
| outputs: | ||
| destination: | ||
| description: Where the installed misspell binary is | ||
| value: ${{ steps.install.outputs.destination }} | ||
| destination-dir: | ||
| description: The directory where the installed misspell binary is | ||
| value: ${{ steps.install.outputs.destination-dir }} | ||
| version: | ||
| description: The installed version of misspell | ||
| value: ${{ steps.install.outputs.version }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - id: install | ||
| shell: bash | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| VERSION=$(gh release list -R golangci/misspell --exclude-drafts --exclude-pre-releases | grep ${{ inputs.version }} | cut -f1) | ||
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" | ||
| mkdir -p $(dirname ${{ inputs.destination }}) | ||
| DESTINATION="$(readlink -f "${{ inputs.destination }}")" | ||
| echo "destination=$DESTINATION" >> "$GITHUB_OUTPUT" | ||
| DESTINATION_DIR="$(dirname "$DESTINATION")" | ||
| echo "$DESTINATION_DIR" >> "$GITHUB_PATH" | ||
| echo "destination-dir=$DESTINATION_DIR" >> "$GITHUB_OUTPUT" | ||
| ARCH="$(echo "$RUNNER_ARCH" | tr '[:upper:]' '[:lower:]')" | ||
| OS="$(echo "$RUNNER_OS" | tr '[:upper:]' '[:lower:]')" | ||
| if [ "$ARCH" = "x64" ]; then | ||
| export ARCH="64bit" | ||
| fi | ||
| if [ "$OS" = "macos" ]; then | ||
| export OS="mac" | ||
| fi | ||
| mkdir -p tmp | ||
| gh release download "$VERSION" -p "misspell_*_${OS}_${ARCH}.tar.gz" -O tmp/misspell.tgz -R golangci/misspell | ||
| pushd tmp && tar -xvf misspell.tgz && popd | ||
| mv tmp/misspell "$DESTINATION" | ||
| rm -rf tmp |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: BUSL-1.1 | ||
|
|
||
| --- | ||
| name: Set up staticcheck from Github releases | ||
| description: Set up staticcheck from Github releases | ||
|
|
||
| inputs: | ||
| destination: | ||
| description: "Where to install the staticcheck binary (default: $HOME/bin/staticcheck)" | ||
| type: boolean | ||
| default: "$HOME/bin/staticcheck" | ||
| version: | ||
| description: "The version to install (default: latest)" | ||
| type: string | ||
| default: Latest | ||
|
|
||
| outputs: | ||
| destination: | ||
| description: Where the installed staticcheck binary is | ||
| value: ${{ steps.install.outputs.destination }} | ||
| destination-dir: | ||
| description: The directory where the installed staticcheck binary is | ||
| value: ${{ steps.install.outputs.destination-dir }} | ||
| version: | ||
| description: The installed version of staticcheck | ||
| value: ${{ steps.install.outputs.version }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - id: install | ||
| shell: bash | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| VERSION=$(gh release list -R dominikh/go-tools --exclude-drafts --exclude-pre-releases | grep ${{ inputs.version }} | cut -d " " -f2) | ||
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" | ||
| mkdir -p $(dirname ${{ inputs.destination }}) | ||
| DESTINATION="$(readlink -f "${{ inputs.destination }}")" | ||
| echo "destination=$DESTINATION" >> "$GITHUB_OUTPUT" | ||
| DESTINATION_DIR="$(dirname "$DESTINATION")" | ||
| echo "$DESTINATION_DIR" >> "$GITHUB_PATH" | ||
| echo "destination-dir=$DESTINATION_DIR" >> "$GITHUB_OUTPUT" | ||
| ARCH="$(echo "$RUNNER_ARCH" | tr '[:upper:]' '[:lower:]')" | ||
| OS="$(echo "$RUNNER_OS" | tr '[:upper:]' '[:lower:]')" | ||
| if [ "$ARCH" = "x64" ]; then | ||
| export ARCH="amd64" | ||
| fi | ||
| if [ "$OS" = "macos" ]; then | ||
| export OS="darwin" | ||
| fi | ||
| mkdir -p tmp | ||
| gh release download "$VERSION" -p "staticcheck_${OS}_${ARCH}.tar.gz" -O tmp/staticcheck.tgz -R dominikh/go-tools | ||
| pushd tmp && tar -xvf staticcheck.tgz && popd | ||
| mv tmp/staticcheck/staticcheck "$DESTINATION" | ||
| rm -rf tmp |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.