foliage build fail on macOS (aarch64-darwin)

[yvan-sraka]

So, I tried to follow the getting started example …

% mkdir -p _sources/typed-protocols/0.1.0.0

% cat _sources/typed-protocols/0.1.0.0/meta.toml
github = { repo = "input-output-hk/ouroboros-network", rev = "fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa" }
subdir = "typed-protocols"

% nix run ".#" -- build
warning: Using saved setting for 'extra-substituters = https://cache.iog.io https://foliage.cachix.org https://cache.zw3rk.com' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= foliage.cachix.org-1:kAFyYLnk8JcRURWReWZCatM9v3Rk24F5wNMpEj14Q/g= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=' from ~/.local/share/nix/trusted-settings.json.
warning: ignoring untrusted substituter 'https://foliage.cachix.org'
error: flake 'git+file:///Users/yvan/GitHub/foliage' does not provide attribute 'apps.aarch64-darwin.default', 'defaultApp.aarch64-darwin', 'packages.aarch64-darwin.default' or 'defaultPackage.aarch64-darwin'

Let's then add aarch64-darwin to the list …

diff --git a/flake.nix b/flake.nix
index 4c9834d..c3c327a 100644
--- a/flake.nix
+++ b/flake.nix
@@ -13,7 +13,7 @@
 
   outputs = { self, nixpkgs, flake-utils, haskell-nix, ... }:
 
-    flake-utils.lib.eachSystem [ "x86_64-linux" ] (system:
+    flake-utils.lib.eachSystem [ "x86_64-linux" "aarch64-darwin" ] (system:
       let
         pkgs = import nixpkgs {
           inherit system;

Then it fails because Linux and macOS cp options differs …

% nix run ".#" -- build                         
...
🌿 Foliage
# buildAction
You don't seem to have created a set of TUF keys. I will create one in _keys
Current time set to 2023-03-29T14:11:52Z. You can set a fixed time using the --current-time option.
# _sources/typed-protocols/0.1.0.0/meta.toml
# curl (for RemoteAsset https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa)
tar xzf _cache/https/github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa -C /tmp/extra-dir-1029921117
cp --recursive --no-target-directory --dereference /tmp/extra-dir-1029921117/input-output-hk-ouroboros-network-fa10cb4/typed-protocols _cache/typed-protocols/0.1.0.0
cp: illegal option -- -
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file ... target_directory
foliage: Error when running Shake build system:
  at want, called at app/Foliage/CmdBuild.hs:45:7 in main:Foliage.CmdBuild
* Depends on: buildAction
  at apply1, called at app/Foliage/PrepareSource.hs:32:31 in main:Foliage.PrepareSource
* Depends on: PrepareSourceRule (PackageIdentifier {pkgName = PackageName "typed-protocols", pkgVersion = mkVersion [0,1,0,0]}) (PackageVersionSpec {packageVersionTimestamp = Nothing, packageVersionSource = GitHubSource {githubRepo = "input-output-hk/ouroboros-network", githubRev = "fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa", subdir = Just "typed-protocols"}, packageVersionRevisions = [], packageVersionForce = False})
  at cmd_, called at app/Foliage/PrepareSource.hs:99:17 in main:Foliage.PrepareSource
* Raised the exception:
Development.Shake.cmd, system command failed
Command line: cp --recursive --no-target-directory --dereference /tmp/extra-dir-1029921117/input-output-hk-ouroboros-network-fa10cb4/typed-protocols _cache/typed-protocols/0.1.0.0
Exit code: 64
Stderr:
cp: illegal option -- -
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file ... target_directory

I tried my best to fix it (cp -T option doesn't seem to exist on macOS) …

diff --git a/app/Foliage/PrepareSource.hs b/app/Foliage/PrepareSource.hs
index 34443a6..18e4820 100644
--- a/app/Foliage/PrepareSource.hs
+++ b/app/Foliage/PrepareSource.hs
@@ -72,7 +72,7 @@ addPrepareSourceRule inputDir cacheDir = addBuiltinRule noLint noIdentity run
                     fix2 = case mSubdir of Just s -> (</> s); _ -> id
                     tdir = fix2 $ fix1 tmpDir
 
-                cmd_ "cp --recursive --no-target-directory --dereference" [tdir, srcDir]
+                cmd_ "cp -rL" [tdir, srcDir]
             --
             -- This is almost identical to the above but we get to keep the
             -- metadata.
@@ -96,7 +96,7 @@ addPrepareSourceRule inputDir cacheDir = addBuiltinRule noLint noIdentity run
                     fix2 = case mSubdir of Just s -> (</> s); _ -> id
                     tdir = fix2 $ fix1 tmpDir
 
-                cmd_ "cp --recursive --no-target-directory --dereference" [tdir, srcDir]
+                cmd_ "cp -rL" [tdir, srcDir]
 
           let patchesDir = inputDir </> unPackageName pkgName </> prettyShow pkgVersion </> "patches"
           hasPatches <- doesDirectoryExist patchesDir

And then Shake complains it can't find _cache/typed-protocols/0.1.0.0/typed-protocols.cabal …

% nix run ".#" -- build
...
🌿 Foliage
# buildAction
Current time set to 2023-03-29T14:14:12Z. You can set a fixed time using the --current-time option.
# curl (for RemoteAsset https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa)
tar xzf _cache/https/github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa -C /tmp/extra-dir-33940425217
cp -rL /tmp/extra-dir-33940425217/input-output-hk-ouroboros-network-fa10cb4/typed-protocols _cache/typed-protocols/0.1.0.0
# _cache/typed-protocols/0.1.0.0/typed-protocols.cabal
foliage: Error when running Shake build system:
  at want, called at app/Foliage/CmdBuild.hs:45:7 in main:Foliage.CmdBuild
* Depends on: buildAction
  at need, called at app/Foliage/Shake.hs:38:3 in main:Foliage.Shake
* Depends on: _cache/typed-protocols/0.1.0.0/typed-protocols.cabal
  at error, called at src/Development/Shake/Internal/Rules/File.hs:179:58 in shake-0.19.7-GFPtSLQxwCcJDXDlNacG7C:Development.Shake.Internal.Rules.File
* Raised the exception:
Error, file does not exist and no rule available:
  _cache/typed-protocols/0.1.0.0/typed-protocols.cabal

Searching through the Shake codebase doesn't seem really helpful https://github.com/ndmitchell/shake/blob/v0.19.7/src/Development/Shake/Internal/Rules/File.hs#L179 … Maybe the error is more in https://github.com/andreabedini/foliage/blob/main/app/Foliage/CmdBuild.hs#L45 or https://github.com/andreabedini/foliage/blob/main/app/Foliage/Shake.hs#L38, do you have any pointers on how to go further from here? 🙂

[andreabedini]

@yvan-sraka Thanks for reporting this! and apologies if this doesn't work (yet!) on macOS. The options --recursive --no-target-directory --dereference are important. Can you check whether cp -rL results in the following structure?

❯ ls _cache/typed-protocols/0.1.0.0/
LICENSE  NOTICE  src  typed-protocols.cabal

If it does, then I have a problem with shake. I already ran into issues calling need before.

If it doesn't, then we need to have a way to do the same thing in a portable way, e.g. rewriting it in Haskell.

[yvan-sraka]

Thanks! I tested it again on my macOS M1 machine, and it seems to works as expected:

yvan@Butternut ~/GitHub/foliage (git)-[main] % nix run ".#" -- build
🌿 Foliage
# buildAction
You don't seem to have created a set of TUF keys. I will create one in _keys
root keys:
  da1395e06b0be4c0b6020f0423e49bc3eb6bc971f4405e9b382b2fba203543e1
  69134d69b80da17534fd0bf097f352e4cdd48445bae9141c72c0df72a2376cf0
  264cae55c2db8f8b40162c06e930afd716c8ca8f25f6d5b5a94106e5d15a902a
Current time set to 2023-07-11T14:04:09Z. You can set a fixed time using the --current-time option.
# _sources/typed-protocols/0.1.0.0/meta.toml
# curl (for RemoteAsset https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa)
tar xzf _cache/https/github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa -C /tmp/extra-dir-77615494607
cp --recursive --no-target-directory --dereference /tmp/extra-dir-77615494607/input-output-hk-ouroboros-network-fa10cb4/typed-protocols _cache/typed-protocols/0.1.0.0
# _cache/typed-protocols/0.1.0.0/typed-protocols.cabal
# cabal sdist (for PrepareSdistRule "_cache/typed-protocols/0.1.0.0")
Wrote /Users/yvan/GitHub/foliage/_repo/package/typed-protocols-0.1.0.0.tar.gz (new hash 5f3d78b9b1d79e00e681965fa2c0074cb790caa739edb6f4f8e3615cf08fdfe5)
# webpages / index (for buildAction)
# webpages / all-packages (for buildAction)
# webpages / all-package-versions (for buildAction)
# webpages / package / typed-protocols-0.1.0.0 (for buildAction)
Copying from _cache/typed-protocols/0.1.0.0/typed-protocols.cabal to _repo/package/typed-protocols-0.1.0.0/revision/0.cabal
Copying from _cache/typed-protocols/0.1.0.0/typed-protocols.cabal to _repo/package/typed-protocols-0.1.0.0/typed-protocols.cabal
# _keys/target/abf5b4426f84e66b75cb6ea1efd9f507bd16cb90d76423bde460d743c2cd3926.json
# _keys/target/40a36de36f6786722d43c86b6fecd4e73d2efcc1eda30ef31084733f740bdb63.json
# _keys/target/1c31a4dc0f794e4f4c2a11125317b0db36044686597d368d261312beb6741e24.json
# Writing index (for buildAction)
# _keys/root/af5259dfe4d0965cc6f45e15a418b202cfd62cdaf9d45e4a5aede52eee1d119e.json
# _keys/root/4bc482dbb0ed09fbbde5d62a619343309ad9d6f2a6c48b5f4cd86073550ff745.json
# _keys/root/0245ac705d744838302055350d83740d9b6b3d5a4a5fd6e57a55c942198ddb9b.json
# _keys/snapshot/cce3ce14a6d045d6698ee644534ee3efdeaddfb4d6269009a1244dc9d69686b4.json
# _keys/snapshot/b055006304cedf0c0d28f6f675fd1481d7ecab2b15cd9d1b9d5bc8edde609db7.json
# _keys/snapshot/3a09adc9ae4cf925a1007797eef1810f186671f83e3943d0c103ab4655ce2f6d.json
# _keys/timestamp/e400ff095ba92ff9b9cc66ea11c583072f6941df0f5396adf12eb703da252fb5.json
# _keys/timestamp/b1f10d5c96930067141fe9255a698410511c9b70418de39800df5922c1e2bc71.json
# _keys/timestamp/83bc62c5df49e534bcb66b417300c792768eeaffb2cd653aceae5c3fc5747d17.json
# _keys/mirrors/c9b2b46256d797b4535194ffee02e5cd1ba731af07bf526bd6b3195a6f64bb02.json
# _keys/mirrors/963cfa2d79990fef4fea77d1bcbd1f4f02c81774d7e46d4d088a315ffb432473.json
# _keys/mirrors/4f33e4fd38126f93a3c1f14fafc67dfddca836ded17a89eee7883706ce8938a0.json
# /Users/yvan/GitHub/foliage/_repo/root.json
# /Users/yvan/GitHub/foliage/_repo/mirrors.json
# /Users/yvan/GitHub/foliage/_repo/01-index.tar
# /Users/yvan/GitHub/foliage/_repo/01-index.tar.gz
# /Users/yvan/GitHub/foliage/_repo/snapshot.json