Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs releases #766

Merged
merged 14 commits into from
Mar 11, 2024
8 changes: 4 additions & 4 deletions .github/workflows/release-libs.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# This workflow is used to publish SV2 crates to cratesio
# The workflow try to update all the library crates so if a crate is not to updated, the step will fail
# This workflow is used to publish SV2 crates to crates.io
# The workflow tries to update all the library crates, so if a crate is not to updated, the step will fail
# for that each step have continue-on-error set to true.
# Since each step can fail the output ot the action must be manually check to macke sure that all
# Since each step can fail, the output ot the action must be manually checked to make sure that all
# the library intended to be published are published.
# Runing cargo release in the various workspace help to prepare the version number and everything.
# Running cargo release in the various workspace help to prepare the version number and everything.
# ATTENTION
# Is very important to check the output manually cause when too many crates are updated crates.io could fail
# and ask to rerun the action later
Expand Down
111 changes: 62 additions & 49 deletions RELEASE.md
Original file line number Diff line number Diff line change
@@ -1,66 +1,79 @@
# Releasing Roles Binaries and Publishing
# Versioning

The github binary releases of the roles and the publishing of the SRI crates are both handled in the `release.yaml` workflow. This workflow must be manually
started by navigating to the "Actions" tab in the SRI repo, then navigating to the Release workflow section, and clicking "Run Workflow".
SRI releases follow [SemVer 2.0.0](https://semver.org/).

## Crates Publishing
Given a version number `MAJOR.MINOR.PATCH`, we increment the:
- `MAJOR` version when incompatible API changes are introduced
- `MINOR` version when functionality is added in a backward compatible manner
- `PATCH` version when backward compatible bug fixes are introduced

For publishing crates we use the `cargo-release` tools because it easily handles the order in which crates can be published. Crates are published
in the `crates_publish` job. Currently, crates.io rate limits the publishing of more than 5 NEW crates. This means that if we have more than 5 crates
to publish, that are not currently published to crates.io, this workflow will not publish anything. To bypass this we will need to add the `--exclude [crate_name]`
flag for as many crates as needed to the `cargo-release` command in the "Publish" step. For example, if there are 7 new crates to publish, we will need to add
`--exclude [crate_1] --exclude [crate_2]` to the command. This is not an issue for publishing version updates since the crates are not NEW publishes.
SRI has a global version, which uses git tags and keeps track of how the codebase evolves across time as a whole.

### Current Successful Publishes
Every internal SRI crate also follows SemVer 2.0.0, but each crate version is only set on the respective `Cargo.toml`,
(no git tags), and it evolves independently of the global release version.

Due to either github dependencies or a crate failing the build stage during publish not all crates are being published.
Whenever a `PATCH` is introduced, it is applied to all the latest `MAJOR` releases.
For example: imagine there's releases `v1.0.0`, `v1.1.0`, and `v2.0.0`. A bug is found, dating back all the way to `v1.0.0`.
A patch is applied such that the following new tags are introduced: `v1.1.1` and `v2.0.1`.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we do it? I guess that we should have a branch (or a tag?) for all the last x majior version of each libs.
This seems complicate. Another way could be to create the branch only when needed. We fix a bug for lib1-2.0.0 we find that also lib1-1.1.0 is affected and we want to backport the fix. We create a new branch from whatever was the state of the repo when lib1 have been go from v1 ti v2 and we apply the fix. If we do that only for last majior version, only when we have a bug that affect also the last majior version I do not think that we will end up with 'too many' branches, considering that most library in this repo will do not get a new majior version in many years. If this is a concern we can just say that we support the last majior version and that is not safe to use any version that is not the majior, or we can think about it when we will have more then one majior version (0 wont be maintained cause is unstable). Maybe we can just leave the various possibility here, for now we can say that we support only last majior so we do not make promise and when we will have more majior we will think about it.

Copy link
Collaborator Author

@plebhash plebhash Mar 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On lines 10 and 12 of my proposal, there is a very important distinction we need to keep in mind:

SRI has a global version, which uses git tags and keeps track of how the codebase evolves across time as a whole.

Every internal SRI crate also follows SemVer 2.0.0, but each crate version is only set on the respective `Cargo.toml`,
(no git tags), and it evolves independently of the global release version.

Let's imagine this scenario, where both crate-A and crate-B are affected by a bug on the following versions:

Global Release Version v2.0.0 v1.1.0
crate-A v1.0.0 v0.9.0
crate-B v2.0.0 v2.0.0

Here, Global Release Versions would be indicated as git tags.

Crate versions however, would only be indicated by Cargo.toml and crates.io. Otherwise, we would end up with too many tags.

After we patch the bug, we would have the following:

Global Release Version v2.0.1 (new git tag) v1.1.1 (new git tag)
crate-A v1.0.1 (only bump Cargo.toml) v0.9.1 (only bump Cargo.toml)
crate-B v2.0.1 (only bump Cargo.toml) v2.0.1 (only bump Cargo.toml)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok make sense. Only thing I wouldn't maintain version 0 and 1. I would just bump majior version of everyting to 2. And start from there 2 reasons here:

  1. We didn't followed semver so if you assume semver for what we have now you risk to end up with not compatible libraries.
  2. There several 0 library that now are 1 that at 0 were very very experimental and it do not make sense to to support them

An example of that is the noise crate.

Also is startum v2 so it can make sense to start from 2. Otherwise we should remove the old library from crates.io but I dont' know if is something possible or not.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with what you're saying @Fi3, the only thing that bugs me is starting with 2.0.0, afaik we didn't really have an official release yet, and in public we always referred to things as MVP and later update. 1.0.0 is really a nice clean milestone, I am worried 2.0.0 may communicate certain level of maturity and stability which I am not sure we have yet.

Otherwise we should remove the old library from crates.io but I don't know if is something possible or not.

Doesn't sound like something impossible, I doubt there can be any consequences to this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With regards to 2.0.0, we should keep the SRI vs SV2 distinction in mind.

Sure, SV2 implies v2, but only for the Specifications.

That doesn't necessarily mean that as a Rust codebase, SRI should start at v2.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With regards to crates.io removal, I found this:

rust-lang/crates.io#1506 (comment)

which is discouraging... it basically means we will not be able to remove the old libraries from crates.io, and we cannot "start from scratch".

I did understand your point on starting on 2.0.0 @Fi3 . It would be a way to put all SRI crates at the same starting point. But I still think starting on 2.0.0 would be bad, for the reasons I explained in my previous comment.

I wonder if we could do the same, but with all crates starting at 1.0.0?
I did a quick search on all workspaces and I didn't see any crate that is above 1.0.0.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from @Fi3 (over discord):

main issue is noise crate already 1.1.0 and is not semver 1.0.0 is not compatible with 1.1.0 but jakub told me in the call that you can mark a crate on crates.io as not usable so this could be a solution we could mark all crates like this one as not usable and in that case start from 1.1.0 and mark 1.0.0 as not usable

so assuming we will set all crates to v1.0.0 (except noise-sv2), I crated this PR: #785

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. All the 0s will become 1.0.0. All the one already to 1 are yanked and we start from the very next minor. And we say that 0 is not supported anymore.

- [x] buffer_sv2
- [x] binary_codec_sv2
- [x] binary_sv2
- [x] common_messages_sv2
- [x] const_sv2
- [x] derive_codec_sv2
- [x] framing_sv2
- [x] serde_sv2
- [x] sv1_api
- [] noise_sv2 - github depependency (ed25519-dalek)
- [] codec_sv2 - noise dependency
- [] template_distribution_sv2 - build failure
- [] mining_sv2 - publishes after template_distribution_sv2
- [] job-declaration_sv2 - publishes after template_distribution_sv2
- [] roles_logic_sv2 - publishes after template_distribution_sv2
- [] network_helpers - publishes after template_distribution_sv2
- [] error_handling - had to exclude to bypass rate limit but will probably publish now
- [] sv2_ffi - had to exclude to bypass rate limit but will probably publish now
- [] all roles - noise dependency
# Git Branching Strategy

## Github Roles Releases
We follow a simplified [gitflow](https://nvie.com/posts/a-successful-git-branching-model/) branching strategy.

The roles binaries are released in github in all jobs other than the `crates_publish` job. To be able to publish,
the job must be able to find a new tag (one that hasnt already been released), matching the crate name of the role to be
released. For example, imagine there is currently a release for pool_sv2-v1.0.0. For the jobs releasing a new pool version to succeed,
it must be able to find a tag containing the string "pool_sv2" with a version greater than 1.0.0. This should not be an issue since the tags
are automatically created by the `autoversion.yaml` workflow, and moved into main when the generated PR is merged.
Although our strategy is very similar to the classic gitflow model, we do not keep release branches.

### Github Release Issue
![](git-branching.png)

Currently we do not support windows releases because we are unable to run the step to return the last tag for a given crate name. See the section above
for an explaination of the process.
## Principal Branches

The SRI repo holds two principal branches with an infinite lifetime:
- `main`
- `dev`

We consider `main` to be the branch where the source code of `HEAD` always reflects a production-ready state.

## Github Binary Releases
We consider `dev` to be the branch where the source code of `HEAD` always reflects a state with the latest delivered development changes for the next release.

# Versioning
The SRI team will decide the appropriate time when the changes to `dev` are merged back into `main` and then tagged with a release number while bumping `MAJOR` and/or `MINOR`.

## Feature Branches

New features are developed into separate branches that only live in the contributor's forks.

- branch off from: `dev`
- merge back into: `dev`
- naming convention: `feat-x`, where `x` describes the feature

## Patch Branches

Bugs are patched into separate branches that only live in the contributor's forks.

- branch off from: `main`
- merge back into: `main` + `dev`
- naming convention: `patch-x`, where `x` describes the bug/patch

# Releasing Roles Binaries

The [release page of SRI repo](https://github.com/stratum-mining/stratum/releases) provides executable binaries for all SRI roles, targeting popular hardware architectures.

The GitHub binary releases of the roles are handled in the `release-bin.yaml` workflow.

This workflow is manually started by navigating to the "Actions" tab in the SRI repo, then navigating to the Release workflow section, and clicking "Run Workflow".

Note: in order to be able to manually trigger the "Run Workflow" button, the user needs to have "Write" permissions on the repository, otherwise the button will not show up on the UI.

# Publishing Library Crates

Although SRI has a global release cycle, which is attached to the binaries, each internal crate also has its own versioning history.

Lib crates are published to crates.io in the `release-lib.yaml` workflow. The workflow tries to update all the library crates.

If a crate is not updated successfully, the step will fail, but since all steps have `continue-on-error` set to true, the workflow will continue.

Since steps can fail, the output of the action must be manually checked.

Versioning is handled by the `autorelease.yaml` workflow. The workflow will will auto detect changes in crates and auto bump the patch versions
in the target crate, as well as any crate that uses the target crate as a dependency, if changes are detected. It will then push a new commit with the
versioning changes to the `bot/versioning` branch, and Lastly, it will auto create a PR (bot/versioning -> main). Since cargo-smart-release does not yet
support autodetection of MAJOR and MINOR changes with regards to Semver, any MAJOR/MINOR versioning will need to be manually changed in the package Cargo.toml.
Every PR to `main` (either coming from `dev`, or a patch) needs to increase the version of whatever crate it is touching. Otherwise, we will mess up the dependency chain of whoever is fetching from crates.io

## Versioning Notes
Manually running `cargo release` in the various workspaces helps to prepare the version number before the releases are published.

This workflow should not run if the push is resulting from this workflow, but this not necessarily guaranteed since the
a developer could make a minor change to the bot/versioning PR, and the latest author would no longer be "Github Actions Bot".
If the workflow does run again, there will just be no changes, and the PR can be closed.
Every time we bump some crate's version, `release-libs.yaml` needs to be manually triggered in order to update crates.io.
Binary file added git-branching.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading