arion: init at 0.1.0.0

[roberth]

Motivation for this change

Package Arion, a tool for working with Docker through Nix!

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
  • arion: 124.0MB
  • of which docker-compose-1.24.1: 115.6 MB
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[cdepillabout]

@roberth Thanks for sending this PR.

I don't quite understand why arion-compose can't just be used from haskellPackages.arion-compose. Why is this additional arion derivation needed? It also seems somewhat fishy to rely on a "magic" file ./cabal2nix-arion-compose.nix that makes the derivation do something slightly different depending on whether or not it exists.

[roberth]

@cdepillabout

Why is this additional arion derivation needed?

• To make it more lightweight by building a static executable as far as haskell libraries are concerned
• To provide the arion.build and arion.eval functions (although this is technically an answer to why an addition expression is needed)
  • These functions require builtins.fetchTarball rather than the usual fetchers because some people do not want to use IFD that doesn't work well on Hydra
• To rename the product to its actual name. It's called Arion, but that name turned out to be already taken on hackage
• To include a known-compatible version of docker compose. Arion should package its own version of docker compose. Docker compose itself can not be used with arion projects, so if the user wants to use both programs they should be able to install a different version in the profile (or shell or whatever)

It also seems somewhat fishy to rely on a "magic" file ./cabal2nix-arion-compose.nix that makes the derivation do something slightly different depending on whether or not it exists.

I do agree in general, but I try to keep the number of variations of the expressions low. This way I can backport the expression without having to maintain another variation of this file. So it does serve a purpose.
Do you think this is an acceptable reason to keep it?

[cdepillabout]

Instead of hard-coding the path ./cabal2nix-arion-compose.nix, why not take optionalCabal2nixFile as an argument to this file, so you'd have something like:

{ pkgs
, haskellPackages ? pkgs.haskellPackages
, optionalCabal2nixFile ? null
}:

Then, the arion-compose expression would become:

  arion-compose =
    # Test suite needs nixpkgs as dependency which would make it quite heavy.
    if isNull optionalCabal2nixFile then haskellPackages.arion-compose else pkgs.haskell.lib.dontCheck (haskellPackages.callPackage optionalCabal2nixFile {});

This should make it more natural to use for most users who are used to the callPackage style of packages.

[cdepillabout]

Instead of taking pkgs as an argument here, could you only declare the things you need from pkgs? That would make it more natural to use as a callPackage-style package.

I'm specifically thinking of things like haskellPackages (which you do declare), haskell (for haskell.lib, or even just the specific things you use from haskell.lib).

[cdepillabout]

The same thing here about turning this into a callPackage-style package.

The current way it is written, it is not obvious that arion depends on things like justStaticExecutables, overrideCabal, docker-compose, makeWrapper, etc. It would also be more difficult for someone to override those arguments, since they are taken directly from pkgs.

[cdepillabout]

Could you add comments explaining what these are supposed to be?

[cdepillabout]

Could you add comments on what eval and build are supposed to be? Also, an example of how they are intended to be used would be helpful.

[cdepillabout]

I don't think this necessarily needs to be changed, but it might be easier for other people to run if this update script just pulled arion from the latest version on Hackage and used that. Then you wouldn't need arion checked out locally (and have to match versions by hand).

[cdepillabout]

This file is generated automatically, so if you edit it, there is a chance it will cause a build conflict.

Can you remove this change? It will be marked as unbroken next time the file is automatically generated (which happens quite frequently).

You might be interested in this video which explains the process:

https://discourse.nixos.org/t/video-tutorial-how-to-fix-broken-haskell-packages-in-nix/3968

[roberth]

How can you tell I didn't run it?

Tbh I got a linker error in cabal2nix. I wrote an issue here to make that process reproducible if you're interested: NixOS/cabal2nix#431

[cdepillabout]

How can you tell I didn't run it?

As far as I know, in general hackage2nix is run automatically by some sort of CI setup by peti. So when people send PRs to nixpkgs, they are often asked to not change pkgs/development/haskell-modules/hackage-packages.nix by hand, because it will sometimes cause merge conflicts.

Also, hackage2nix will always(?) remove both the broken = true; and hydraPlatforms = stdenv.lib.platforms.none; lines.

[cdepillabout]

@roberth Thanks for the additional explanation.

I think one of the things I'm having a hard time following is why this is split into a default.nix file and an arion.nix file? Would it be possible to just put everything in one file?

If not, could you add some documentation explaining the reason for the split? I think I'm having a tough time following what is going on in this derivation.

[roberth]

@cdepillabout, thanks for the extensive review!

The reason for the split is that I can use the same arion.nix both here and in the arion source project. I'll try to make it more like a callPackage package, but since most things are Haskell related, it's not going to be a big difference. For instance I can only inject pkgs.haskell, not pkgs.haskell.lib directly, let alone pkgs.haskell.lib.justStaticExecutables.

[cdepillabout]

One more thing I forgot: since you've changed the pkgs/development/haskell-modules/configuration-hackage2nix.yaml, could you target the haskell-updates branch instead of master?

[roberth]

@cdepillabout I've dropped the ability to move files between here and the arion project. It probably wasn't worth optimizing for that.

[cdepillabout]

@roberth Thanks for updating this. This looks good.

I think this should be able to be merged as-is. However, I'm still not 100% sure how the eval and build functions are supposed to be used. Maybe you're using them on the upstream arion repo? Are they meant to be used by normal nix users (people other than you and Domen)?

[roberth]

@cdepillabout yes, they are intended for arion users. We will publish documentation very soon and add a link in the inline docs here.

I'll merge it so it's there just in time for NixCon.

[jbmusso]

Experimenting building on Mac OS X with a simple derivation - nixpkgs master as of this morning. Roughly:

{
  arionSrc = pkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "ce14f092eab6efcb2915c1db189558c1d5a469e8"; # master as of 20191025 ~Friday morning
    sha256 = "1w901v41gfnxd9xaanxns5qxrjnhlpfgsn6g7bhf76jg11lggxf8";
  };

  arion = (pkgs.callPackage arionSrc {}).arion;
}

Install tests are failing with an OSError: AF_UNIX path too long error.
Log details are a bit verbose, so they're collapsed below (click to expand):

OSError: AF_UNIX path too long

cannot build derivation '/nix/store/cqamkpf3gw05bk7sw163ysm53yqyfh2z-docker-compose-1.24.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ml4dz80glkkxgv1nv1ndi8rzp08hyl7r-arion-0.1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/ml4dz80glkkxgv1nv1ndi8rzp08hyl7r-arion-0.1.0.0.drv' failed

running install tests
============================= test session starts ==============================
platform darwin -- Python 3.7.5, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
rootdir: /private/var/folders/8c/cjp8r0355mbdsygc0_mg00l80000gn/T/nix-build-python3.7-docker-4.1.0.drv-0/docker-4.1.0
collected 547 items

tests/unit/api_build_test.py .........s...                               [  2%]
tests/unit/api_container_test.py ....................................... [  9%]
.........................................................                [ 19%]
tests/unit/api_exec_test.py .....                                        [ 20%]
tests/unit/api_image_test.py .........................                   [ 25%]
tests/unit/api_network_test.py ......                                    [ 26%]
tests/unit/api_test.py .......................F.............             [ 33%]
tests/unit/api_volume_test.py ..........                                 [ 35%]
tests/unit/auth_test.py ................................................ [ 43%]
........                                                                 [ 45%]
tests/unit/client_test.py ..........                                     [ 47%]
tests/unit/dockertypes_test.py ......................................... [ 54%]
.....x............                                                       [ 57%]
tests/unit/errors_test.py ..................                             [ 61%]
tests/unit/models_containers_test.py ................................... [ 67%]
...                                                                      [ 68%]
tests/unit/models_images_test.py ................                        [ 71%]
tests/unit/models_networks_test.py ......                                [ 72%]
tests/unit/models_resources_test.py ..                                   [ 72%]
tests/unit/models_services_test.py .                                     [ 72%]
tests/unit/ssladapter_test.py ......                                     [ 73%]
tests/unit/swarm_test.py ...                                             [ 74%]
tests/unit/types_containers_test.py .                                    [ 74%]
tests/unit/utils_build_test.py ...s.......................s.........F... [ 82%]
...                                                                      [ 82%]
tests/unit/utils_config_test.py .....s....                               [ 84%]
tests/unit/utils_json_stream_test.py .......                             [ 85%]
tests/unit/utils_proxy_test.py .....                                     [ 86%]
tests/unit/utils_test.py ...........................................s... [ 95%]
..........................                                               [100%]

=================================== FAILURES ===================================
_______________ UnixSocketStreamTest.test_early_stream_response ________________

self = <tests.unit.api_test.UnixSocketStreamTest testMethod=test_early_stream_response>

    def setUp(self):
        socket_dir = tempfile.mkdtemp()
        self.build_context = tempfile.mkdtemp()
        self.addCleanup(shutil.rmtree, socket_dir)
        self.addCleanup(shutil.rmtree, self.build_context)
        self.socket_file = os.path.join(socket_dir, 'test_sock.sock')
>       self.server_socket = self._setup_socket()

tests/unit/api_test.py:372:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.unit.api_test.UnixSocketStreamTest testMethod=test_early_stream_response>

    def _setup_socket(self):
        server_sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>       server_sock.bind(self.socket_file)
E       OSError: AF_UNIX path too long

tests/unit/api_test.py:387: OSError
_________________________ TarTest.test_tar_socket_file _________________________

self = <tests.unit.utils_build_test.TarTest testMethod=test_tar_socket_file>

    @pytest.mark.skipif(IS_WINDOWS_PLATFORM, reason='No UNIX sockets on Win32')
    def test_tar_socket_file(self):
        base = tempfile.mkdtemp()
        self.addCleanup(shutil.rmtree, base)
        for d in ['foo', 'bar']:
            os.makedirs(os.path.join(base, d))
        sock = socket.socket(socket.AF_UNIX)
        self.addCleanup(sock.close)
>       sock.bind(os.path.join(base, 'test.sock'))
E       OSError: AF_UNIX path too long

tests/unit/utils_build_test.py:463: OSError
============= 2 failed, 539 passed, 5 skipped, 1 xfailed in 4.68s ==============
builder for '/nix/store/0j7mc6zhfq7x70n3cbl4wzplbn424y71-python3.7-docker-4.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/cqamkpf3gw05bk7sw163ysm53yqyfh2z-docker-compose-1.24.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ml4dz80glkkxgv1nv1ndi8rzp08hyl7r-arion-0.1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/ml4dz80glkkxgv1nv1ndi8rzp08hyl7r-arion-0.1.0.0.drv' failed

It appears that the path length limit for a Unix domain socket is around 104 on Mac OS. This could be related.

Please let me know if you would rather want me to open a distinct issue so this merged PR doesn't get hijacked.

[roberth]

@jbmusso This looks like a build failure in the python docker package in docker-compose's closure.

It's probably best to open a Nixpkgs issue and @ mention jonringer there.