[Backport release-24.11] Parallel GH actions workflow for Nixpkgs eval #357652
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…ttrNamesOnly The attrNamesOnly feature is used by pkgs/top-level/release-attrpaths-superset.nix to return a superset of all attributes that might be built by Hydra. Before this change it would include all attribute paths to derivations that could be found recursively, ignoring the recurseForDerivations and recurseForRelease attributes that control Hydra's recursion. This had the effect that it would end up with ~266000 attributes, most of which definitely won't be built by Hydra. We can remove those while staying true to the superset notion to end up with just ~97000, a reduction of ~63.6%! This also comes with an eval time reduction from 31.7s to 18.7s (-41.0%)! As an example, all derivations in pypy310Packages don't get included anymore, because it doesn't have a `.recurseForDerivations` set. As a nice side effect, with `--arg enableWarnings false`, no more warnings are printed, because things like `checkpointBuildTools.mkCheckpointedBuild` (which is deprecated) isn't being recursed to anymore. (cherry picked from commit 72f462b)
Motivated by ofborg struggling [1] and its evaluations taking too long, inspired by Jörg's initial PR [2] and Adam's previous attempt to parallelise Nixpkgs evaluation [3], this PR contains initial work to relief ofborg from its evaluation duty by using GitHub Actions to evaluate Nixpkgs. For now this doesn't take care of all of what ofborg does, such as requesting appropriate reviewers or labeling mass rebuilds, but this can be follow-up work. [1]: https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025?u=infinisil [2]: #352808 [3]: #269403 Co-Authored-By: Jörg Thalheim <[email protected]> Co-Authored-By: Adam Joseph <[email protected]> (cherry picked from commit fbbe972)
2 tasks
github-actions
bot
added
6.topic: haskell
6.topic: policy discussion
6.topic: lib
nix-owners
bot
requested review from
LeSuisse,
maralorn,
mweinelt,
philiptaron,
risicle,
sternenseemann and
zowoq
|
This should now also contain the follow up fixes otherwise it's misleading as it tests the wrong commit. |
1 task
Although matrix.system is supposed to be generated from trusted code, we'd better follow [Github Actions good practices][1]. [1]: https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable (cherry picked from commit f807208)
- `env.mergedSha` is empty, so it checked out the master version by default - The process step used `needs.attrs.outputs.mergedSha`, but apparently that's empty unless `attrs` is declared as a `needs`, even though `outputs` implicitly depends on `attrs` (cherry picked from commit 19db54e)
(cherry picked from commit bb19bea)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bot-based backport to
release-24.11, triggered by a label in #356023.