Build within nix2container image: "Digest did not match" from Skopeo

[kolloch]

Problem

When building docker images with nix2container within an image build with nix2container, I am getting weird errors like this:

Running skopeo --insecure-policy copy nix:/nix/store/47g6iqca8nhldrrin16imczai5fcj929-image-nix-ci.json $@
Getting image source signatures
Copying blob 440aaf385e0f done  
Copying blob afaeec1f88a2 done  
Copying blob 59f7db87f4ab done  
Copying blob 00a004aa7f8b done  
Copying blob 20ea6e6cb56a done  
FATA[0000] writing blob: happened during read: Digest did not match, expected sha256:20ea6e6cb56aa5dc118ae4b4fbeddbb221ddca60a5e13677fee3c44f079ee3fe, got sha256:17ce2732d9ad185f1ad8e3705211fd7a94dea9888d63a57c4e1e1f1f48c67c0a 

Reproduction

This repo contains everything to reproduce the error for me.

With docker and local copy:

1. nix run -L .\#containers.x86_64-linux.nix-ci.copyToDockerDaemon
2. docker run --privileged -v $PWD:/workspace -v ~/.docker:/home/user/.docker --workdir /workspace -it $(nix eval --raw ".#containers.x86_64-linux.nix-ci.imageRefUnsafe") nix run .\#containers.x86_64-linux.nix-ci.copyTo oci:oci_sample_inside

Ideas

I wonder if it is some sort of store path corruption?

I build it with and without #96:

• with Fixes #94: do not include config file in nix database. #96, I got the error shown
• without Fixes #94: do not include config file in nix database. #96, I couldn't build the image inside
• with Fixes #94: do not include config file in nix database. #96 outside, and without Fixes #94: do not include config file in nix database. #96 inside, it was building fine

[nlewo]

I will take a look later, but to debug such kind of issues, i will do something such as explained in this comment: #23 (comment)

This would allow you to compare files in the layer produced at build time to the files in the layer produced at push time.

[kolloch]

Hi @nlewo,

Tried to debug this some more and wrote my findings in here https://gitlab.com/nexxiot-labs/nix2container-checksum/-/blob/465dd01ba20887fea1457c4205713b7ae291b99a/README.md

Everything is hopefully reproducible by executing commands such as

nix run .\#x86_64-linux.gitlab.runnables.debug-building-nix-ci-in-nix-ci

Which also contain debug info.

Maybe the most surprising result first: All layers except the top-level layer are actually the same between working and non-working builds. Only the top-level image layer also containing the nix-database differs.

If you add a package to that top-level layer (I did that with hello), then the build for that container succeeds.

I assume that some incorrect substitution happens otherwise, not sure.

The non-working solution also have content-addressable hashes in the closuregraph for some reason.

[kolloch]

I'd be happy to debug that together. I am in the CET time zone and are rather flexible.

[kolloch]

Hi @nlewo,

I updated my PR quite significantly:

#96 (comment)

With this PR in place, most things are deterministic.

Very weirdly, if you set reproducible=false, then the container builds. If not, the old error.

I stared long and hard at functions such as newLayers and the core piece of the digest/tar implementation: TarPaths

But they honestly look quite nicely written and good to me 🤷

[kolloch]

I also updated the README/code of https://gitlab.com/nexxiot-labs/nix2container-checksum so that it should be quite straight-forward to repo. At least for me, it is fully reproducible.

[kolloch]

Within the nix-ci container with nix running as user, I can reproduce the issue by:

nix run .#aarch64-linux.gitlab.containers.only-stdenv.copyTo oci:test-oci

from nix2container-checksum which is build by:

only-stdenv = nix2container.buildImage {
    name = "only-stdenv";
    copyToRoot = [nixpkgs.stdenv];
  };

With my fixes from #96 and without stdenv, it actually works now! :)

[kolloch]

I cannot reproduce the issue anymore 🤷