Suggestion: Set up release automation and changelog

[sambhav]

It would be great if the hera release process could be automated.

Some examples of how it can be done are -

• https://github.com/cruft/cruft/blob/master/.github/workflows/pypi_publish.yml and https://github.com/cruft/cruft/blob/master/.github/workflows/test_pypi_publish.yml

The above releases the package on PyPI automatically through the Github UI by creating a release.

You can also couple it with release drafter which will give you nice changelogs like https://github.com/buildpacks/libcnb/releases

[sambhav]

Currently it looks like hera publish workflow fails on every commit unless it is a new version since pypi doesn't allow version overrides. See https://github.com/argoproj-labs/hera-workflows/actions/runs/3347332597/jobs/5545275302

The above may be a better solution.

[flaviuvadan]

Currently it looks like hera publish workflow fails on every commit unless it is a new version since pypi doesn't allow version overrides

Hera currently uses twine publish ... --skip-existing to avoid this, specifically 🤔 the CICD failure is new, and we probably need to investigate why the --skip-existing does not seem to do what is expected. Hera used to use poetry publish but poetry publish does not have a --skip-existing flag (I think?). Also, Hera used to publish to Test PyPi, download that version, run tests against that, then publish the official version to PyPy. That was removed with the migration to tox for multi-Python version testing, which install Hera already, so the "test" of installation is performed by tox itself. See this PR for more details

The auto-generated release is nice! We can certainly take a look at that!

[flaviuvadan]

The CICD failures happens because we're attempting to upload hera-workflows-4.0.1.tar.gz while PyPi already contains hera_workflows-4.0.1.tar.gz. I am unsure what caused the different format, especially since the .whl file is correctly formatted and correctly skipped for 4.0.1. I hypothesize that the issue of failing CI will disappear post 4.1.0 so I will wait until then before further investigation

[elliotgunton]

I am confused whether Hera is supposed to be auto-publishing to PyPi? Is CICD broken as 4.2.0 was from 31st Oct? Would love to play with the new features (hooks in particular)!

[flaviuvadan]

@elliotgunton Hera publishes to PyPi. You can see the block here. It is not broken as of 4.2.0, AFAICT, since the build succeeds by successfully skipping existing versions (via --skip-existing here). When a new version is published (via changing the pyproject.toml version here) CI successfully pushes a new version (as evidences by the 4.2.0) release

[elliotgunton]

Thank you Flav! I see, then Hera only releases to PyPi when pyproject.toml is manually updated. So is it possible to do that automatically per commit to main with a version bump commit from CICD? Otherwise releases are currently adhoc?

Possibly introducing a dev branch for a release cut would also be good!

[flaviuvadan]

Could definitely do that! Are you thinking about one release per PR, almost? 🤔 We could set something up such as:

• force users to add [PATCH | MINOR | MAJOR | DOCS] spec in a PR title
• user the PR title on merge to main to determine the version bump
• release that version

I think no matter what avenue we pursue there is a "manual" step of specifying "I want to release a new version":

• set pyproject.yoml version
• PR title
• Git tag
• Git release
• etc

So, I think this really becomes a problem of clarity - which way to release is the clearest? At the moment the release cycle is, really, only clear to me.. I make a hera/x.y.z branch in which I enrich the CHANGELOG, set the version, and merge.

[elliotgunton]

Are you thinking about one release per PR, almost?

Potentially! I really like your suggestion of "add [PATCH | MINOR | MAJOR | DOCS] spec in a PR title" as it lets contributors "request" a new version with their contribution to be released, and I think as project maintainer you can change PR titles? So you can decide before merging how to bump - especially also because with a major version bump we expect breaking changes so you'd want to use it carefully, possibly restricting who can use it, or with your approval?

Another alternative is using Git PR labels?

there is a "manual" step of specifying "I want to release a new version"

Agreed! I think it's too restrictive to do full automation of minor/patch bumps on a merge to main - we should at least be able to say whether it's a new feature (minor) or a bugfix (patch).

which way to release is the clearest?

I think there are differences between those ways in terms of who can release - for Github tags and releases I think only you can do those?

I think updating pyproject.toml or using the PR title/label would be best - my vote would be optionally specifying the bump type in the PR title (what if you wanted to group multiple PRs into a new minor or major feature release? Or maybe that would suit a second "stable" feature branch until that it merged to main?). Also, either way would be a little opaque without a section in the README for the "when" and "how" of releasing, and allowing a manual version bump on the PR-level might cause some back-and-forth on PRs to determine the final version bump, but most importantly it allows contributors to create a new version with their new change. They can also provide a summary of the change in the changelog at the same time and a PR validation check can be used to say if the version is being bumped then the changelog must also be updated?

There are some other approaches and ideas in https://py-pkgs.org/07-releasing-versioning.html#automatic-version-bumping but enforcing those <type> commit messages might be difficult.