INF-629: Update common to get split builds (#200)

* INF-629: Update common to get split builds Updates the `common` repo to have split builds: * shell build * format build * lint + doc build * debug build * release build Includes the following commits: commit b35ba23941c5f3b555eba33c144f2cf9e5036cfb Author: Joachim Breitner <[email protected]> Date: Wed Nov 20 11:57:28 2019 +0100 Do not use lib.inNixShell (#69) * Do not use lib.inNixShell this does not just mean “is the curren derivation the target of `nix-shell`”, but it also turns true when this is a _dependency_ of some `nix-shell` (e.g. Motoko’s). This is bad, because it changes the derivation of `drun`, so although `drun` in in the nix cache, when trying to enter `nix-shell`, it now depends on a _different_ `drun` (one built with `-D warnings`), so our poor people have to watch rust compile stuff. Debuggig this was also tricky, and required help from the nix IRC channel. They said that using `lib.inNixShell` in a derivation like this is a bad idea. So we now omit `-D warnings` simply for the `debug` and `shell` build, where that makes more sense, and do not use `lib.inNixShell`. commit b4e92e6010ad7caee863cb8e74b8c1426f92794a Merge: d862d5d f19329c Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 18:16:18 2019 +0100 Merge pull request #67 from dfinity-lab/nm-always-touch Touch rust files in preBuild commit f19329c71b275d58f19109ac923110963bfbc2d3 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 18:15:09 2019 +0100 Run preBuild and postBuild in lint commit 639866f9ec57523c1f4a53c14205841503f4f8ae Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 17:49:31 2019 +0100 s/preBuild/preCheck commit 5dfef852592e35e88c86d96280584f4d631d3269 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 17:36:10 2019 +0100 Fix standalone test evaluation commit 3df1411493bb6edad6e54664fcc819e28715ed31 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 17:21:00 2019 +0100 Touch rust files in preBuild commit d862d5d1c492a2077364d3771645c72e0233ce93 Merge: 8387b5b 0367c94 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 16:51:43 2019 +0100 Merge pull request #66 from dfinity-lab/nm-fix-standalone-builder Port standalone builder to new rust builder commit 0367c948b70603b29b0f4ceff45df2c29045fd80 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 16:31:21 2019 +0100 Port standalone builder to new rust builder Before this commit, building standalone rust on Linux leads to evaluation errors. commit 8387b5bfd0cd8df0acccdca53feb1c38010eecd3 Merge: 68ddd96 bc868d9 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 16:04:10 2019 +0100 Merge pull request #65 from dfinity-lab/nm-split-dfn-further INF-484: INF-588: INF-581: Split rust jobs commit bc868d9fc02230a5e6a0833833b9af26edcd7325 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 16:02:03 2019 +0100 Create empty directory for lint job commit 68ddd96f7c6171cb5316b6ca51e137ff8d478e06 Merge: a9d886e d8b3e93 Author: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Date: Thu Nov 14 14:46:30 2019 +0000 Merge pull request #33 from dfinity-lab/update-niv Update niv commit d8b3e93d9bc8ad7ebe4eacea2bed47bd39c41f93 Author: Nikola Knezevic <[email protected]> Date: Tue Oct 29 10:36:01 2019 +0100 Update niv This is to help with a problem in other repos where we had an infinite recursion due to setting up nixpkgs in nix-shell shebang line. commit 765754201b3b899899af232a8c6f8fd4a01d7594 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 15:23:36 2019 +0100 Update naersk for faster builds commit 0b7f62cad98e0a6374af1e2496ca86dd7868f485 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 15:22:52 2019 +0100 Make doc overridable commit 5d40a3800d3eb2591330884784c2f3e6ac384494 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 15:22:38 2019 +0100 Remove unnecessary phase commit 8385c76857644474749f454d032029b556133265 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 15:22:18 2019 +0100 Use releaseTools.aggregate to aggregate commit 54e2e13973d1603500185936212bdb78ab4d837b Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 14:39:33 2019 +0100 INF-484: INF-588: INF-581: split rust jobs This splits the rust jobs further, exporting a single interface for rust packages, exporting the following: * `build`: the release build * `debug`: the debug build * `doc`: the built documentation * `lint`: the lint checks * `fileChecks`: extra file checks * `fmt`: the formatting checks * `shell`: a shell derivation Moreover a target called `allChecks` is added, which builds all of the above. commit 548f4e78dd896ac1cef990a20daf00fbb93a7655 Author: Nicolas Mattia <[email protected]> Date: Thu Nov 14 14:38:10 2019 +0100 lib: inline moreutils commit a9d886e463e3df12439c4e6888d26e4a6902e757 Author: Akhi Singhania <[email protected]> Date: Thu Nov 14 12:30:49 2019 +0100 Add libfuse commit 2f236bd89793c2816bed76832f996ec80173d620 Merge: f77e818 a73dfb9 Author: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Date: Wed Nov 13 16:02:53 2019 +0000 Merge pull request #63 from dfinity-lab/nm-split-checks INF-581: INF-588: Split buildDfinityRustPackage jobs commit a73dfb98a9a9e1d477a0f8ac05a0613a0f79e465 Author: Nicolas Mattia <[email protected]> Date: Wed Nov 13 15:55:02 2019 +0100 INF-581: INF-588: Extract fmt check from rust workspace commit 99b261dcc45c25d50b5721bd3d52747ebbb39303 Author: Nicolas Mattia <[email protected]> Date: Wed Nov 13 13:25:20 2019 +0100 INF-581: INF-588: Extract file checks from rust workspace commit f77e818eb07ca5731fe97004c4b7524132429128 Merge: bbf39c3 63bd2a5 Author: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Date: Wed Nov 13 10:45:50 2019 +0000 Merge pull request #62 from dfinity-lab/nm-refactor-build-dfn-rs-pkg INF-581: INF-588: Refactor buildDfinityRustPackage commit 63bd2a5d6ebfb2885a66bfde431de157057fee6f Author: Nicolas Mattia <[email protected]> Date: Wed Nov 13 10:55:45 2019 +0100 INF-581: INF-588: Refactor buildDfinityRustPackage * Simplify rust-shell.nix * Don't build shells twice
dfinity · Nov 25, 2019 · 824280a · 824280a
1 parent 7ee8cc4
commit 824280a
Show file tree

Hide file tree

Showing 4 changed files with 53 additions and 63 deletions.
diff --git a/nix/default.nix b/nix/default.nix
@@ -21,7 +21,7 @@ let
     else builtins.fetchGit {
       name = "common-sources";
       url = "ssh://[email protected]/dfinity-lab/common";
-      rev = "bbf39c31e08b4ab7db82f799a3e3c693a0fdced6";
+      rev = "b35ba23941c5f3b555eba33c144f2cf9e5036cfb";
     };
 in import commonSrc {
   inherit system crossSystem config;

diff --git a/nix/overlays/dfinity-sdk.nix b/nix/overlays/dfinity-sdk.nix
@@ -1,23 +1,21 @@
 self: super:
 let
   mkRelease = super.callPackage ./mk-release.nix {};
-
+  rust-package' = super.callPackage ../rust-workspace.nix {};
+  # remove some stuff leftover by callPackage
+  rust-package = removeAttrs rust-package'
+    [ "override" "overrideDerivation" ];
+  rust-workspace = rust-package.build;
 in {
   dfinity-sdk = rec {
-    packages = rec {
+    packages =
+      # remove the shell since it's being built below in "shells"
+      removeAttrs rust-package [ "shell" ] // rec {
+        inherit rust-workspace;
+        rust-workspace-debug = rust-package.debug;
         js-user-library = super.callPackage ../../js-user-library/package.nix {
           inherit (self) napalm;
         };
-        rust-workspace = super.callPackage ../rust-workspace.nix {};
-        rust-workspace-debug = (rust-workspace.override (_: {
-          release = false;
-          doClippy = true;
-          doFmt = true;
-          doDoc = true;
-        })).overrideAttrs (oldAttrs: {
-          name = "${oldAttrs.name}-debug";
-        });
-        rust-workspace-doc = rust-workspace-debug.doc;
 
         rust-workspace-standalone = super.lib.standaloneRust
           { drv = rust-workspace;
@@ -136,7 +134,7 @@ in {
     # of all the shells here.
     shells = {
       js-user-library = import ../../js-user-library/shell.nix { pkgs = self; };
-      rust-workspace = import ../rust-shell.nix { pkgs = self; };
+      rust-workspace = import ../rust-shell.nix { pkgs = self; shell = rust-package.shell; };
     };
 
     licenses = {

diff --git a/nix/rust-shell.nix b/nix/rust-shell.nix
@@ -1,21 +1,9 @@
-{ pkgs ? (import ./. {}).pkgs }:
+{ pkgs ? (import ./. {}).pkgs, shell }:
 pkgs.mkCompositeShell {
   name = "dfinity-sdk-rust-env";
   buildInputs = [pkgs.rls];
-  inputsFrom = [
-
-    (pkgs.dfinity-sdk.packages.rust-workspace-debug.overrideAttrs (oldAttrs: {
-      # _oldAttrs.configurePhase refers to the dfinity-application-and-others-deps
-      # derivation which is the build of all 3rd-party Rust dependencies. Since in this
-      # nix-shell we use cargo locally to build all dependencies we don't need to depend
-      # on this derivation saving a lot of time downloading/building.
-      configurePhase = "";
-
-      # for some odd reason this is needed in the shell:
-      # https://dfinity.atlassian.net/browse/INF-542
-      nativeBuildInputs = oldAttrs.nativeBuildInputs ++ [ pkgs.stdenv.cc ];
-    })) ];
-
+  nativeBuildInputs = [ pkgs.stdenv.cc ];
+  inputsFrom = [ shell ];
   shellHook = ''
     # Set CARGO_HOME to minimize interaction with any environment outside nix
     export CARGO_HOME=${pkgs.lib.dfinityRoot}/.cargo-home

diff --git a/nix/rust-workspace.nix b/nix/rust-workspace.nix
@@ -1,8 +1,4 @@
-{ release ? true
-, doClippy ? false
-, doFmt ? false
-, doDoc ? false
-, motoko
+{ motoko
 , buildDfinityRustPackage
 , cargo-graph
 , darwin
@@ -15,7 +11,7 @@
 , dfinity-sdk
 }:
 let
-  drv = buildDfinityRustPackage {
+  workspace = buildDfinityRustPackage {
     name = "dfinity-sdk-rust";
     srcDir = ../.;
     regexes = [
@@ -26,35 +22,43 @@ let
       ".*Cargo\.lock$"
       "^.cargo/config$"
     ];
-    inherit release doClippy doFmt doDoc;
   };
-in
-drv.overrideAttrs (oldAttrs: {
-  DFX_ASSETS = runCommandNoCC "dfx-assets" {} ''
-    mkdir -p $out
-    cp ${dfinity.nodemanager}/bin/nodemanager $out
-    cp ${dfinity.ic-client}/bin/client $out
-    cp ${motoko.moc-bin}/bin/moc $out
-    cp ${motoko.mo-ide}/bin/mo-ide $out
-    cp ${motoko.didc}/bin/didc $out
-    cp ${motoko.rts}/rts/mo-rts.wasm $out
-    mkdir $out/stdlib && cp -R ${motoko.stdlib}/. $out/stdlib
-    mkdir $out/js-user-library && cp -R ${dfinity-sdk.packages.js-user-library}/. $out/js-user-library
-  '';
+  workspace' = (workspace //
+    { lint = workspace.lint.overrideAttrs (oldAttrs: {
+      nativeBuildInputs = oldAttrs.nativeBuildInputs ++ [
+        cargo-graph
+        graphviz
+      ];
 
-  nativeBuildInputs = oldAttrs.nativeBuildInputs ++ lib.optionals doDoc [
-    cargo-graph
-    graphviz
-  ];
+      postDoc = oldAttrs.postDoc + ''
+        pushd dfx
+        cargo graph | dot -Tsvg > ../target/doc/dfx/cargo-graph.svg
+        popd
+      '';
 
-  postDoc = oldAttrs.postDoc + ''
-    pushd dfx
-    cargo graph | dot -Tsvg > ../target/doc/dfx/cargo-graph.svg
-    popd
-  '';
+      postInstall = oldAttrs.postInstall + ''
+        echo "report cargo-graph-dfx $doc dfx/cargo-graph.svg" >> \
+          $doc/nix-support/hydra-build-products
+      '';
+    });
+  });
+in
 
-  postInstall = oldAttrs.postInstall + lib.optionalString doDoc ''
-    echo "report cargo-graph-dfx $doc dfx/cargo-graph.svg" >> \
-      $doc/nix-support/hydra-build-products
-  '';
-})
+# override all derivations and add DFX_ASSETS as an environment variable
+(lib.mapAttrs (k: drv:
+  if !lib.isDerivation drv then drv else
+    drv.overrideAttrs (_: {
+      DFX_ASSETS = runCommandNoCC "dfx-assets" {} ''
+        mkdir -p $out
+        cp ${dfinity.nodemanager}/bin/nodemanager $out
+        cp ${dfinity.ic-client}/bin/client $out
+        cp ${motoko.moc-bin}/bin/moc $out
+        cp ${motoko.mo-ide}/bin/mo-ide $out
+        cp ${motoko.didc}/bin/didc $out
+        cp ${motoko.rts}/rts/mo-rts.wasm $out
+        mkdir $out/stdlib && cp -R ${motoko.stdlib}/. $out/stdlib
+        mkdir $out/js-user-library && cp -R ${dfinity-sdk.packages.js-user-library}/. $out/js-user-library
+      '';
+    })
+) workspace')
+(throw "this argument is used to trigger the functor and shouldn't actually be evaluated.")