Skip to content

Commit

Permalink
buildNimSbom: init a new package builder for Nim
Browse files Browse the repository at this point in the history
  • Loading branch information
ehmry committed Nov 14, 2024
1 parent 76612b1 commit 3be81ac
Show file tree
Hide file tree
Showing 6 changed files with 249 additions and 5 deletions.
1 change: 1 addition & 0 deletions .github/labeler.yml
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@
- any-glob-to-any-file:
- doc/languages-frameworks/nim.section.md
- pkgs/build-support/build-nim-package.nix
- pkgs/build-support/build-nim-sbom.nix
- pkgs/by-name/ni/nim*
- pkgs/top-level/nim-overrides.nix

Expand Down
1 change: 1 addition & 0 deletions ci/OWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,7 @@ pkgs/development/python-modules/buildcatrust/ @ajs124 @lukegb @mweinelt
# nim
/doc/languages-frameworks/nim.section.md @ehmry
/pkgs/build-support/build-nim-package.nix @ehmry
/pkgs/build-support/build-nim-sbom.nix @ehmry
/pkgs/top-level/nim-overrides.nix @ehmry

# terraform providers
Expand Down
44 changes: 39 additions & 5 deletions doc/languages-frameworks/nim.section.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Nim {#nim}

The Nim compiler and a builder function is available.
Nim programs are built using `buildNimPackage` and a lockfile containing Nim dependencies.
Nim programs are built using a lockfile and either `buildNimPackage` or `buildNimSbom`.

## buildNimPackage {#buildNimPackage}

The following example shows a Nim program that depends only on Nim libraries:
```nix
Expand All @@ -15,7 +17,7 @@ buildNimPackage (finalAttrs: {
owner = "inv2004";
repo = "ttop";
rev = "v${finalAttrs.version}";
hash = "sha256-oPdaUqh6eN1X5kAYVvevOndkB/xnQng9QVLX9bu5P5E=";
hash = lib.fakeHash;
};
lockFile = ./lock.json;
Expand All @@ -26,7 +28,7 @@ buildNimPackage (finalAttrs: {
})
```

## `buildNimPackage` parameters {#buildnimpackage-parameters}
### `buildNimPackage` parameters {#buildnimpackage-parameters}

The `buildNimPackage` function takes an attrset of parameters that are passed on to `stdenv.mkDerivation`.

Expand All @@ -41,7 +43,7 @@ The following parameters are specific to `buildNimPackage`:
Use this to specify defines with arguments in the form of `-d:${name}=${value}`.
* `nimDoc` ? false`: Build and install HTML documentation.

## Lockfiles {#nim-lockfiles}
### Lockfiles {#nim-lockfiles}
Nim lockfiles are created with the `nim_lk` utility.
Run `nim_lk` with the source directory as an argument and it will print a lockfile to stdout.
```sh
Expand All @@ -50,9 +52,41 @@ $ nix build -f . ttop.src
$ nix run -f . nim_lk ./result | jq --sort-keys > pkgs/by-name/tt/ttop/lock.json
```

## buildNimSbom {#buildNimSbom}

An alternative to `buildNimPackage` is `buildNimSbom` which builds packages from [CycloneDX SBOM](https://cyclonedx.org/) files.
`buildNimSbom` resolves Nim dependencies to [fixed-output derivations](https://nixos.org/manual/nix/stable/glossary#gloss-fixed-output-derivation) using the [nix:fod namespace](#sec-interop.cylonedx-fod).

In the following minimal example only the source code checkout and a `buildInput` are specified.
The SBOM file provides metadata such as `pname` and `version` as well as the sources to Nim dependencies.
```nix
# pkgs/by-name/ni/nim_lk/package.nix
{
lib,
buildNimSbom,
fetchFromSourcehut,
openssl,
}:
buildNimSbom (finalAttrs: {
src = fetchFromSourcehut {
owner = "~ehmry";
repo = "nim_lk";
rev = finalAttrs.version;
hash = lib.fakeHash;
};
buildInputs = [ openssl ];
}) ./sbom.json
```

### Generating SBOMs {#generating-nim-sboms}

The [nim_lk](https://git.sr.ht/~ehmry/nim_lk) utility can generate SBOMs from [Nimble](https://github.com/nim-lang/nimble) package metadata.
See the [nim_lk documentation](https://git.sr.ht/~ehmry/nim_lk#nimble-to-cyclonedx-sbom) for more information.

## Overriding Nim packages {#nim-overrides}

The `buildNimPackage` function generates flags and additional build dependencies from the `lockFile` parameter passed to `buildNimPackage`. Using [`overrideAttrs`](#sec-pkg-overrideAttrs) on the final package will apply after this has already been generated, so this can't be used to override the `lockFile` in a package built with `buildNimPackage`. To be able to override parameters before flags and build dependencies are generated from the `lockFile`, use `overrideNimAttrs` instead with the same syntax as `overrideAttrs`:
The `buildNimPackage` and `buildNimSbom` functions generate flags and additional build dependencies from the `lockFile` parameter passed to `buildNimPackage`. Using [`overrideAttrs`](#sec-pkg-overrideAttrs) on the final package will apply after this has already been generated, so this can't be used to override the `lockFile` in a package built with `buildNimPackage`. To be able to override parameters before flags and build dependencies are generated from the `lockFile`, use `overrideNimAttrs` instead with the same syntax as `overrideAttrs`:

```nix
pkgs.nitter.overrideNimAttrs {
Expand Down
2 changes: 2 additions & 0 deletions nixos/doc/manual/release-notes/rl-2411.section.md
Original file line number Diff line number Diff line change
Expand Up @@ -852,6 +852,8 @@

- `qgis` and `qgis-ltr` are now built without `grass` by default. `grass` support can be enabled with `qgis.override { withGrass = true; }`.

- `buildNimSbom` was added as an alternative to `buildNimPackage`. `buildNimSbom` uses [SBOMs](https://cyclonedx.org/) to generate packages whereas `buildNimPackage` uses a custom JSON lockfile format.

## Detailed Migration Information {#sec-release-24.11-migration}

### `sound` options removal {#sec-release-24.11-migration-sound}
Expand Down
205 changes: 205 additions & 0 deletions pkgs/build-support/build-nim-sbom.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
{
lib,
stdenv,
fetchgit,
fetchzip,
runCommand,
xorg,
nim,
nimOverrides,
}:

let
fetchers = {
fetchzip =
{ url, sha256, ... }:
fetchzip {
name = "source";
inherit url sha256;
};
fetchgit =
{
fetchSubmodules ? false,
leaveDotGit ? false,
rev,
sha256,
url,
...
}:
fetchgit {
inherit
fetchSubmodules
leaveDotGit
rev
sha256
url
;
};
};

filterPropertiesToAttrs =
prefix: properties:
lib.pipe properties [
(builtins.filter ({ name, ... }: (lib.strings.hasPrefix prefix name)))
(map (
{ name, value }:
{
name = lib.strings.removePrefix prefix name;
inherit value;
}
))
builtins.listToAttrs
];

buildNimCfg =
{ backend, components, ... }:
let
componentSrcDirs = map (
{ properties, ... }:
let
fodProps = filterPropertiesToAttrs "nix:fod:" properties;
fod = fetchers.${fodProps.method} fodProps;
srcDir = fodProps.srcDir or "";
in
if srcDir == "" then fod else "${fod}/${srcDir}"
) components;
in
runCommand "nim.cfg"
{
outputs = [
"out"
"src"
];
nativeBuildInputs = [ xorg.lndir ];
}
''
cat << EOF >> $out
backend:${backend}
path:"$src"
EOF
mkdir -p "$src"
${lib.strings.concatMapStrings (d: ''
lndir "${d}" "$src"
'') componentSrcDirs}
'';

buildCommands = lib.attrsets.mapAttrsToList (
output: input: ''
nim compile $nimFlags --out:${output} ${input}
''
);

installCommands = lib.attrsets.mapAttrsToList (
output: input: ''
install -Dt $out/bin ${output}
''
);

applySbom =
sbom:
{
nimFlags ? [ ],
nimRelease ? true,
passthru ? { },
...
}@prevAttrs:
let
properties = # SBOM metadata.component.properties as an attrset.
lib.attrsets.recursiveUpdate (builtins.listToAttrs sbom.metadata.component.properties)
passthru.properties or { };

nimBin = # A mapping of Nim module file paths to names of programs.
lib.attrsets.recursiveUpdate (lib.pipe properties [
(lib.attrsets.filterAttrs (name: value: lib.strings.hasPrefix "nim:bin:" name))
(lib.attrsets.mapAttrs' (
name: value: {
name = lib.strings.removePrefix "nim:bin:" name;
value = "${properties."nim:binDir" or (properties."nim:srcDir" or ".")}/${value}";
}
))
]) passthru.nimBin or { };
in
{
strictDeps = true;

pname = prevAttrs.pname or sbom.metadata.component.name;
version = prevAttrs.version or sbom.metadata.component.version or null;

nimFlags =
nimFlags
++ (lib.optional nimRelease "-d:release")
++ (
let
srcDir = properties."nim:srcDir" or "";
in
lib.optional (srcDir != "") "--path:${srcDir}"
);

configurePhase =
prevAttrs.configurePhase or ''
runHook preConfigure
echo "nim.cfg << $nimCfg"
cat $nimCfg >> nim.cfg
cat << EOF >> nim.cfg
nimcache:"$NIX_BUILD_TOP/nimcache"
parallelBuild:$NIX_BUILD_CORES
EOF
runHook postConfigure
'';

buildPhase =
prevAttrs.buildPhase or ''
runHook preBuild
${lib.strings.concatLines (buildCommands nimBin)}
runHook postBuild
'';

installPhase =
prevAttrs.installPhase or ''
runHook preInstall
${lib.strings.concatLines (installCommands nimBin)}
runHook postInstall
'';

nativeBuildInputs = (prevAttrs.nativeBuildInputs or [ ]) ++ [ nim ];

nimCfg =
prevAttrs.nimCfg or (buildNimCfg {
backend = prevAttrs.nimBackend or properties."nim:backend" or "c";
inherit (sbom) components;
});

passthru = {
inherit sbom properties nimBin;
};
};

applyOverrides =
prevAttrs:
builtins.foldl' (
prevAttrs:
{ name, ... }@component:
if (builtins.hasAttr name nimOverrides) then
let
result = nimOverrides.${name} component prevAttrs;
in
prevAttrs // (if builtins.isAttrs result then result else result { })
else
prevAttrs
) prevAttrs prevAttrs.passthru.sbom.components;

compose =
callerArg: sbom: finalAttrs:
let
callerAttrs = if builtins.isAttrs callerArg then callerArg else callerArg finalAttrs;
sbomAttrs = callerAttrs // (applySbom sbom callerAttrs);
overrideAttrs = sbomAttrs // (applyOverrides sbomAttrs);
in
overrideAttrs;
in
callerArg: sbomArg:
let
sbom = if builtins.isAttrs sbomArg then sbomArg else builtins.fromJSON (builtins.readFile sbomArg);
overrideSbom = f: stdenv.mkDerivation (compose callerArg (sbom // (f sbom)));
in
(stdenv.mkDerivation (compose callerArg sbom)) // { inherit overrideSbom; }
1 change: 1 addition & 0 deletions pkgs/top-level/all-packages.nix
Original file line number Diff line number Diff line change
Expand Up @@ -15181,6 +15181,7 @@ with pkgs;
nim-unwrapped-2 = nim-unwrapped-2_2;

buildNimPackage = callPackage ../build-support/build-nim-package.nix { };
buildNimSbom = callPackage ../build-support/build-nim-sbom.nix { };
nimOverrides = callPackage ./nim-overrides.nix { };

neko = callPackage ../development/compilers/neko { };
Expand Down

0 comments on commit 3be81ac

Please sign in to comment.