Skip to content

Commit

Permalink
feat: Coverage trend checker (#17)
Browse files Browse the repository at this point in the history 
Bring in the `coverage-notifier` check from HUGR:

https://github.com/CQCL/hugr/blob/8d9da5f6371469055c75378748a06d7290098aaf/.github/workflows/notify-coverage.yml

So it can be reused elsewhere.

drive-by: Alphabetise and cleanup the README
  • Loading branch information
@aborgna-q
aborgna-q authored Oct 25, 2024
1 parent 25836b4 commit 3d83760
Show file tree
Hide file tree
Showing 2 changed files with 213 additions and 44 deletions.
124 changes: 124 additions & 0 deletions .github/workflows/coverage-trend.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
name: Check coverage trend on Codecov
on:
workflow_call:
secrets:
CODECOV_GET_TOKEN:
description: 'The token to query the codecov API.'
required: true
outputs:
should_notify:
description: 'Returns true if the coverage has changed.'
value: ${{ jobs.check-coverage.outputs.should_notify }}
msg:
description: 'The message describing the change. Suitable to post on Slack.'
value: ${{ jobs.check-coverage.outputs.msg }}

jobs:
check-coverage:
runs-on: ubuntu-latest
outputs:
msg: ${{ steps.make_msg.outputs.msg }}
should_notify: ${{ steps.get_coverage.outputs.should_notify }}
steps:
- name: Download commit sha of the most recent successful run
uses: dawidd6/action-download-artifact@v6
with:
# Downloads the artifact from the most recent successful run
workflow: 'coverage-trend.yml'
name: head-sha.txt
if_no_artifact_found: ignore
- name: Get today's and last run's coverage trends from codecov
id: get_coverage
# API reference: https://docs.codecov.com/reference/repos_totals_retrieve
run: |
# Get the previous commit coverage, if the last sha is available
if [ ! -f head-sha.txt ]
then
echo "No previous coverage found."
# Update the head-sha.txt file with the current sha,
# so next time we campare against the current coverage.
echo ${{ github.sha }} > head-sha.txt
echo "should_notify=false" >> "$GITHUB_OUTPUT"
exit 0
fi
PREV_SHA=$( cat head-sha.txt )
echo "Previous sha: \"$PREV_SHA\""
# Check if the sha has changed
if [ "$PREV_SHA" == "${{ github.sha }}" ]
then
echo "No new commits since last run."
echo "should_notify=false" >> "$GITHUB_OUTPUT"
exit 0
fi
# Query the previous coverage from codecov
curl --request GET \
--url "https://api.codecov.io/api/v2/github/${{ github.repository_owner }}/repos/${{ github.event.repository.name }}/totals/?sha=$PREV_SHA" \
--header 'accept: application/json' \
--header "authorization: Bearer ${{ secrets.CODECOV_GET_TOKEN }}" \
> coverage-prev.json
cat coverage-prev.json | jq ".totals.coverage" > coverage-prev.txt
echo "Previous coverage query result:"
cat coverage-prev.json | jq "del(.files)"
echo
# Query the current coverage from codecov
curl --request GET \
--url "https://api.codecov.io/api/v2/github/${{ github.repository_owner }}/repos/${{ github.event.repository.name }}/totals/?sha=${{ github.sha }}" \
--header 'accept: application/json' \
--header "authorization: Bearer ${{ secrets.CODECOV_GET_TOKEN }}" \
> coverage.json
cat coverage.json | jq ".totals.coverage" > coverage.txt
echo "Current coverage query result:"
cat coverage.json | jq "del(.files)"
echo
echo
echo "Previous coverage: `cat coverage-prev.txt`%"
echo "Current coverage: `cat coverage.txt`%"
# A `null` in either coverage means that the coverage is not available,
# so we don't want to notify about that.
if [ "$( cat coverage-prev.txt )" == "null" ]
then
echo "Previous coverage not available."
echo ${{ github.sha }} > head-sha.txt
echo "should_notify=false" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "$( cat coverage.txt )" == "null" ]
then
echo "Current coverage not available."
# Note that we don't update the head-sha.txt file here,
# so next time we compare against the one that had coverage data.
echo "should_notify=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo ${{ github.sha }} > head-sha.txt
echo "should_notify=true" >> "$GITHUB_OUTPUT"
- name: Compare with previous summary and make message
id: make_msg
if: steps.get_coverage.outputs.should_notify == 'true'
run: |
prev=`cat coverage-prev.txt`
current=`cat coverage.txt`
change=`printf "%.2f%% --> %.2f%%" $prev $current`
codecov="https://codecov.io/gh/${{ github.repository }}?search=&trend=7%20days"
if (( $(echo "$prev < $current + 0.04" | bc -l) ))
then
MSG="msg=Coverage check for ${{ github.repository }} shows no regression (${change}). ✅ ${codecov}"
else
MSG="msg=Coverage check for ${{ github.repository }} shows regression (${change}). ❌ ${codecov}"
fi
echo $MSG
echo $MSG >> "$GITHUB_OUTPUT"
- name: Upload current HEAD sha
uses: actions/upload-artifact@v4
with:
name: head-sha.txt
path: head-sha.txt
133 changes: 89 additions & 44 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,99 @@ access the GitHub API. For these we [generate fine-grained access
tokens](https://github.com/settings/personal-access-tokens/new) with the
@hugrbot bot account, which must be stored in the repository secrets.

## Workflows

The following workflows are available:

- [`add-to-project`](#add-to-project): Adds new issues to a GitHub project board when they are created.
- [`coverage-trend`](#coverage-trend): Checks the coverage trend for the project, and produces a summary that can be posted to slack.
- [`drop-cache`](#drop-cache): Drops the cache for a branch when a pull request is closed.
- [`pr-title`](#pr-title): Checks the title of pull requests to ensure they follow the conventional commits format.
- [`rs-semver-checks`](#rs-semver-checks): Runs `cargo-semver-checks` on a PR against the base branch, and reports back if there are breaking changes.
- [`add-to-project`](#add-to-project): Adds new issues to a GitHub project board when they are created.

### [`drop-cache`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/drop-cache.yml)
## [`add-to-project`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/add-to-project.yml)

Adds new issues to a GitHub project board when they are created.

### Usage
```yaml
name: Add issues to project board
on:
issues:
types:
- opened

jobs:
add-to-project:
uses: CQCL/hugrverse-actions/.github/workflows/add-to-project.yml@main
with:
project-url: https://github.com/orgs/{your-org}/projects/{project-id}
secrets:
GITHUB_PAT: ${{ secrets.ADD_TO_PROJECT_PAT }}
```
### Token Permissions
The fine-grained `GITHUB_PAT` secret must include the following permissions:

| Permission | Access |
| --- | --- |
| Projects | Read and write |
| Pull requests | Read |

Note that fine-grained access tokens cannot grant permissions to projects and repositories in different organisations simultaneously.
In those cases, you will need an unrestricted _classical_ github token instead.

## [`coverage-trend`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/coverage-trend.yml)

Compares the project coverage on [Codecov](https://codecov.io/) against the last workflow run,
and produces a summary of the changes that can be posted to slack.

If the project didn't have new commits that changed the coverage since the last run,
the `should_notify` output will be set to `false` and the `msg` output will be empty.

### Usage
```yaml
name: Notify coverage changes
on:
schedule:
# 04:00 every Monday
- cron: '0 4 * * 1'
workflow_dispatch: {}
jobs:
coverage-trend:
uses: CQCL/hugrverse-actions/.github/workflows/coverage-trend.yml@main
secrets:
CODECOV_GET_TOKEN: ${{ secrets.CODECOV_GET_TOKEN }}
# Post the result somewhere.
notify-slack:
needs: coverage-trend
runs-on: ubuntu-latest
if: needs.coverage-trend.outputs.should_notify == 'true'
steps:
- name: Send notification
uses: slackapi/[email protected]
with:
channel-id: "SOME CHANNEL ID"
slack-message: ${{ needs.coverage-trend.outputs.msg }}
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
```

### Outputs

- `should_notify`: Whether there has been a change in coverage since the last run, which we can post about.
- `msg`: A message summarising the coverage changes. This is intended to be posted to slack.

### Token Permissions

`CODECOV_GET_TOKEN` is a token generated by Codecov to access the repository's coverage data.

## [`drop-cache`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/drop-cache.yml)

Drops the cache for a branch when a pull request is closed. This helps to avoid
cache pollution by freeing up some of github's limited cache space.

#### Usage
### Usage
```yaml
name: cleanup caches by a branch
on:
Expand All @@ -37,14 +115,14 @@ jobs:
uses: CQCL/hugrverse-actions/.github/workflows/drop-cache.yml@main
```

### [`pr-title`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/pr-title.yml)
## [`pr-title`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/pr-title.yml)

Checks the title of pull requests to ensure they follow the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) format. If the title
does not follow the conventional commits, a comment is posted on the PR to help
the user fix it.

#### Usage
### Usage
```yaml
name: Check Conventional Commits format
on:
Expand All @@ -67,21 +145,21 @@ jobs:
GITHUB_PAT: ${{ secrets.GITHUB_PAT }}
```

#### Token Permissions
### Token Permissions

The fine-grained `GITHUB_PAT` secret must include the following permissions:

| Permission | Access |
| --- | --- |
| Pull requests | Read and write |

### [`rs-semver-checks`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/rs-semver-checks.yml)
## [`rs-semver-checks`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/rs-semver-checks.yml)

Runs `cargo-semver-checks` on a PR against the base branch, and reports back if
there are breaking changes.
Suggests adding a breaking change flag to the PR title if necessary.

#### Usage
### Usage
```yaml
name: Rust Semver Checks
on:
Expand All @@ -98,44 +176,11 @@ jobs:

The workflow compares against the base branch of the PR by default. Use the `baseline-rev` input to specify a different base commit.

#### Token Permissions
### Token Permissions

The fine-grained `GITHUB_PAT` secret must include the following permissions:

| Permission | Access |
| --- | --- |
| Pull requests | Read and write |


### [`add-to-project`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/add-to-project.yml)

Adds new issues to a GitHub project board when they are created.

#### Usage
```yaml
name: Add issues to project board
on:
issues:
types:
- opened
jobs:
add-to-project:
uses: CQCL/hugrverse-actions/.github/workflows/add-to-project.yml@main
with:
project-url: https://github.com/orgs/{your-org}/projects/{project-id}
secrets:
GITHUB_PAT: ${{ secrets.ADD_TO_PROJECT_PAT }}
```

#### Token Permissions

The fine-grained `GITHUB_PAT` secret must include the following permissions:

| Permission | Access |
| --- | --- |
| Projects | Read and write |
| Pull requests | Read |

Note that fine-grained access tokens cannot grant permissions to projects and repositories in different organisations simultaneously.
In those cases, you will need an unrestricted _classical_ github token instead.

0 comments on commit 3d83760

Please sign in to comment.