Skip to content

Crates publishing automation

JP edited this page Feb 7, 2023 · 20 revisions

TOC

How it works

Short version

The automation detects which crates have changed by comparing their local content to the contents of the highest version from crates.io. If the content has changed, the crate is published again.

Once the automation is run for a given commit, it should not be run for commits before that one, because in that case it would republish crates with outdated source code.

In-depth version

  • Define an order of crates from least to most dependents. For each crate in the order:
    • Determine the highest version based on the current version from Cargo.toml [1] in addition to the versions endpoint of crates.io (https://crates.io/api/v1/crates/$CRATE/versions)
    • Set the crate's .version to the highest version computed in the previous step
    • Strip dev-dependencies, then generate a .crate file for the crate with cargo package
    • Download the .crate file for the highest version from crates.io
    • Compare the generated .crate file with the one downloaded from crates.io. If they differ [2], it means that the crate has changed and needs to be published again [3].
    • If the crate needs to be published:
      • If necessary [4], bump its version for a breaking change [5]
      • Strip dev-dependencies
      • Publish the crate

[1] - If the crate doesn't have a version in it's Cargo.toml, as in https://github.com/paritytech/polkadot/pull/6095, then the automation relies solely on crates.io, as described in https://github.com/paritytech/polkadot/pull/6095#issuecomment-1332097303.

[2] - If the crate hasn't been published before, i.e. it has no .crate file from crates.io, then it's considered different. Note:

[3] - Once the automation is run for a given commit, it should not be run for commits before that one, because in that case it would republish crates with outdated source code.

[4] - If the crate isn't published yet, there's no need to change its version except for removing some extraneous pre-release suffix such as -dev

[5] - Bumping for a breaking change is the simplest and crude way available at the moment. It's possible to refine the version bump heuristic in future versions, but currently there's no commitment for when such a thing would be implemented.

How it is set up

The automation runs from GitLab CI jobs.

  • First there's a publish-crates-locally job which sets up a local crates.io instance and runs the publishing automation against that instance. It is meant to be a trial step to validate that the crates are in a publishable state and the automation works as intended.

    This job runs on both pull requests and master.

  • After publish-crates-locally has worked, if we're on master, next comes the publish-crates job, which publishes crates to the official crates.io server. It is scheduled to run automatically every week, but it can also be triggered manually at any time.

    See the "How to run" and "How to restart" sections for how to use it.

Who maintains it

  • release-engineering maintains the automation's implementation.
  • CI/CD maintains the infrastructure in which the automation runs on.

How to report problems

What to do if publishing fails

  • If the failure is intermittent, for example a sporadic network failure or temporary server error, either restart the automation's CI job (see "How to restart") OR wait for the next scheduled run of the publishing pipeline (happens weekly on Sunday).
  • In any other circumstances, follow the instructions in the "How to report problems" section.

How to run it

The automation is scheduled to run automatically on every Sunday for master. If you wish to run it manually:

1 - Go to the project's Pipeline Schedules page (e.g. https://gitlab.parity.io/parity/mirrors/substrate/-/pipeline_schedules for Substrate)

2 - Trigger the "Scheduled pipeline for crate publishing automation" schedule by clicking the play button

image

If you do not see the play button, please request to be granted "Maintainer" access to that project by the CI/CD team.

How to restart it

⚠️ Restart is only suitable for situations where the publishing job has failed and you wish to continue from where it left off; otherwise, you should trigger a new run.


1 - Go to the project's Pipeline Schedules page (e.g. https://gitlab.parity.io/parity/mirrors/substrate/-/pipeline_schedules for Substrate)

2 - Go to the latest pipeline for the "Scheduled pipeline for crate publishing automation" schedule

image

3 - Find the publish-crates job within that pipeline and restart it by clicking the retry button

image

If you do not see that button, please request to be granted "Maintainer" access to that project by the CI/CD team.

How to exclude crates from being published

Through the source code: set publish = false in the crate's Cargo.toml.

Ad-hoc: Provide a $SPUB_EXCLUDE variable to the automation's CI job. The content of that variable should be a newline-delimited string where each line defines a crate to be excluded. Note that dependents of those crates also won't be published.

How to publish only a few crates

Provide a $SPUB_PUBLISH_ONLY variable to the automation's CI job. The content of that variable should be a newline-delimited string where each line defines a crate to be published. Note that dependencies of those crates will also be selected.