Build a graph of file trees before generating the tar stream

[nlewo]

Before this commit, nix2container read the file tree of store paths and write all file in the order they are walked by the os.Walk Go function.

In this commit, the build process contains now two steps:

1. file trees are read and a graph representing these files is built
2. this graph is walk to write files to the tar stream

This allows to easily transform the file tree (filename rewritting for instance) and also allow to easily detect duplicated files.

[nktpro]

I tested with some of the big, troublesome images (e.g. github-runner and gitlab-runner with nested docker) and confirmed this finally allowed us to skopeo copy ... directly from a local nix store on macOS to a container registry, just like on Linux.

This is huge for the community! Thanks @nlewo for your phenomenal work.

[nlewo]

@nktpro Nice, thx for your tests!
I'm still waiting for few more real life tests before merging this.

[nktpro]

Ugh, I just noticed there's a regression with copyToRoot, looks like it no longer merges all the paths together, e.g combining /bin from different packages.

For example:

nix2container.buildImage
{
  name = "netcat";
  tag = netcat.version;
  copyToRoot = [ dumb-init netcat bash ];
  config = {
    env = [
      "PATH=/bin"
    ];
    entrypoint = [ "dumb-init" "--" "nc" ];
  };
}

Will fail (on a typical Linux env) with:

The file '/bin' already exists in the tar with source path /nix/store/hx04v6jvr03kqw20ipyqfg63xv09mjpl-dumb-init-1.2.5/bin
but is added again with the source path /nix/store/529yr34pwqixjzna8j0f1zpp0x4r7bhg-libressl-3.5.3-nc/bin

[nlewo]

@nktpro it should be better now :)

[nktpro]

@nlewo I'm not sure which branch I should test again with that has the combined result of all your recent work?

The tar-graph branch still has the digest calculation issue on macOS, and the case-hack branch doesn't seem to have your latest fixes in the tar-graph branch?

[nlewo]

@nlewo I'm not sure which branch I should test again with?

Ah sorry, i didn't have rebased the case-hack branch onto the new tar-graph branch. That's now done.

[nktpro]

Thanks @nlewo, that did it. Happy to report that I found no other issue or regression at this point.

[blaggacao]

Unfortunately, it looks like testing this at work currently doesn't rally align with other urgencies. 😏

[nlewo]

@blaggacao Thx for the update. So, let's move forward then!