purescript: build from source

[ghost]

Description of changes

This is still a bit rough... but hey, at long last we're not patchelfing Debian binaries. I would like to solicit feedback from @sternenseemann and @cdepillabout in particular (as well as anybody else who is interested).

Obviously the optimal solution would be to get purescript working within the hackage2nix-generated pkgs/development/haskell-modules system, but I'm not sure that's feasible. I assert that this solution (and its fairly bloaty 212-package closure) may not be ideal, but is better than rebundling binaries.

An earlier draft of this PR kept the old (binary) expression while adding the new one. I switched to deleting the old expression simply to make the PR more readable; if people want to keep both versions just let me know (including guidance on how to organize them) and I'll update the PR.

Big thank-you to @rhendric for so patiently explaining to me the final trick that made this all work.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
• Tested basic functionality of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

[ghost]

Stepping back a bit, is there a way to run cabal2nix on every package in a cabal.project.freeze file, using the versions therein? This amounts to allowing the "rust policy" (version constraints solved independently for each executable/non-library package) in the rare circumstances where it is acceptable.

I realize that this runs strongly against the overall philosophy behind nixpkgs' curated haskell packageset. However I don't think any other approach is going to yield a long-term robust built-from-source purescript, based on comments from the purescript developers:

NOTE: Please do not edit these version constraints manually. They are deliberately made narrow because changing the dependency versions in use can often result in changes in the compiler's behaviour. The PureScript compiler is an executable first and foremost, and only incidentally a library, and supporting a wide range of dependencies is not a goal.

Purescript appears to be about to do a major upgrade of their dependencies, so we might soon enter a brief period where it's possible to build purescript from the same curated set as the rest of nixpkgs' haskell packages. However purescript deliberately does not do this often, so the happy situation will be brief and purescript will break again at some point.

I do sympathize a bit with purescript here: it's a very large project with a lot of dependencies, and bumping those dependencies at the same rate that pkgs/top-level/haskell-packages.nix does would burn up a lot of their time chasing down subtle behavior changes.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/packages-marked-as-broken-should-come-with-an-explanation/19187/30

[cdepillabout]

In my opinion, we should remove this flag and either provide the jailbroken or non-jailbroken build. I think it will likely be too much work to maintain both builds, and most users will only ever use whatever we make the default.

We should probably just use the non-jailbroken build, since that is what upstream would likely prefer us to use.

[cdepillabout]

I'm sure you're on the same page, but it would be ideal if this code block could be auto-generated somehow.

[cdepillabout]

@a-m-joseph One thing that would be a MUST for me before we merge this is for you to add yourself as a maintainer, and do things like the following:

• update this purescript derivation in a timely manner whenever there is a new release upstream
• respond to issues / PRs here in Nixpkgs about PureScript (although honestly there are few issues or PRs)

If you stop updating this PureScript derivation, I'd likely revert it back to just patchelf'ing the binaries (since it seems like it may be somewhat annoying to update).

I'd also like to hear @mbbx6spp's opinion about this PR.

[cdepillabout]

Thanks for taking a stab a this!

I don't have a strong feeling for or against this, but I've left a few comments about various things.

[cdepillabout]

It'd be nice if we could generate this file automatically as well.

One question I had though about this. Are you sure that the versions of the Haskell packages in the package set produced with these overrides is identical to the package set you get when building with stack in the PureScript repo? Are there no transitive dependencies that also need to be overridden? How did you confirm this? Do you have some script or something? Or did you do it by hand?

If you did it by hand, it seems like it would be quite annoying to have to do it every time there is a new version of PureScript released.

[ghost]

Are you sure that the versions of the Haskell packages in the package set produced with these overrides is identical to the package set you get when building with stack in the PureScript repo?

On the contrary, I am quite sure that they are not identical. That list was produced, and minimized, by hand.

However I am confident that the resulting binary obeys all of purescript.cabal's version bounds, which are quite tight (and often exact) under the default useJailbreak=false.

If you did it by hand, it seems like it would be quite annoying to have to do it every time there is a new version of PureScript released.

Yes, although my bigger concern is that this expression will become broken due to changes in release-haskell.nix, since I tried to maximize the number of packages taken from there (and minimize the number for which generate.sh runs cabal2nix).

This is what brought me to write this comment. At the moment that is the only strategy I can think of that would keep this expression robust and maintainable.

[cdepillabout]

Two other random things:

• If you add purescript somewhere in

  nixpkgs/pkgs/top-level/release-haskell.nix

  Lines 157 to 160 in 879bd40

  	# top-level packages that depend on haskellPackages
  	inherit (pkgsPlatforms)
  	agda
  	arion

  , we'll get notifications when the purescript build breaks for whatever reason.
• With this PR, purescript is now a completely separate package. You can't easily obtain it from haskellPackages or haskell.packages.ghc8107 or wherever. This means that purescript can't easily be used as a Haskell library dependency for another Haskell package (like purenix for example). I don't think this should be a reason to block this PR, since the current purescript derivation of course can't be used like this either. But it is somewhat unfortunate that despite building from source, we still can't use it within the normal package set.

[cdepillabout]

Stepping back a bit, is there a way to run cabal2nix on every package in a cabal.project.freeze file, using the versions therein?

As far as I know, there is currently no tooling to generate a Nixpkgs Haskell package set based on either a cabal.project.freeze file, or a stack.yaml file.

There have been some tools in the past that do this, like https://github.com/input-output-hk/stack2nix. There have been other more recent projects that have tried to implement this (but didn't succeed?), like https://github.com/NorfairKing/stacklock2nix.

(haskell.nix does have this functionality, but it doesn't produce package sets that are compatible with the Nixpkgs Haskell stuff.)

[sternenseemann]

I’m a bit sceptical. haskell.packages.ghc8107.purescript was fixed by me last week and all it took me was about an hour, so I’m not sure all this manually orchestrated stuff is necessary. I’m away for the weekend, I’ll have a closer look next week.

…

On 9. Jun 2022, at 10:45, Dennis Gosnell ***@***.***> wrote:  Stepping back a bit, is there a way to run cabal2nix on every package in a cabal.project.freeze file, using the versions therein? As far as I know, there is currently no tooling to generate a Nixpkgs Haskell package set based on either a cabal.project.freeze file, or a stack.yaml file. There have been some tools in the past that do this, like https://github.com/input-output-hk/stack2nix. There have been other more recent projects that have tried to implement this (but didn't succeed?), like https://github.com/NorfairKing/stacklock2nix. (haskell.nix does have this functionality, but it doesn't produce package sets that are compatible with the Nixpkgs Haskell stuff.) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[Profpatsch]

Can confirm that haskell.packages.ghc8107.purescript produces a working purs binary, maybe it has some disadvantage that I’m not aware of.

But on the other hand, this also means we get the purs library as @cdepillabout noted above. So I’m in favor of leaving things as they are for now; alternatively replacing the toplevel purescript with a statically built version of the ghc8107 one, but see https://github.com/justinwoo/my-blog-posts/blob/4236c827e760a0e94a0ecdad1e7160516e6659b3/posts/2019-04-29-why-easy-purescript-nix.md#this-is-the-default-in-nixpkgs-now-for-a-reason for the historical context of using a pre-built binary.

[cdepillabout]

@sternenseemann

I’m a bit sceptical. haskell.packages.ghc8107.purescript was fixed by me last week and all it took me was about an hour, so I’m not sure all this manually orchestrated stuff is necessary.

The big problem is that PureScript sometimes has very specific version bounds that have to do with correctness. So sometimes you can get PureScript building with a different set of package versions than upstream supports, but the resulting binaries end up having bugs at runtime.

@a-m-joseph's explanation touched on this:

I don't think any other approach is going to yield a long-term robust built-from-source purescript, based on comments from the purescript developers:

NOTE: Please do not edit these version constraints manually. They are deliberately made narrow because changing the dependency versions in use can often result in changes in the compiler's behaviour. The PureScript compiler is an executable first and foremost, and only incidentally a library, and supporting a wide range of dependencies is not a goal.

It would be safest for us if we distributed PureScript either as a patchelf'd binary, or compiled from source with the exact same set of transitive dependencies as upstream. In the past I've gotten haskellPackages.purescript compiling without fully matching the blessed upstream dependency versions, but there is always a little risk in this. I've never considered making haskellPackages.purescript this the actual top-level purescript attribute.

We also don't have any guarantee that PureScript will continue tracking relatively closely to the LTS we're on.

[Profpatsch]

It would be safest for us if we distributed PureScript either as a patchelf'd binary, or compiled from source with the exact same set of transitive dependencies as upstream.

Then I see no reason to build the binary from source apart from (misguided) ideas about software purity.

[cdepillabout]

@Profpatsch

Then I see no reason to build the binary from source apart from (misguided) ideas about software purity.

Ah, I didn't want to suggest that there are no benefits from building from source.

I think my personal opinion is probably inline with yours (that software purity is not the absolute top-priority). But in the case of Nixpkgs, by building PureScript from source, we can produce builds for many different architectures (including aarch64-linux and aarch64-darwin, both of which upstream doesn't provide). And of course things like building with musl and statically-linking.

[ghost]

I’m a bit sceptical. haskell.packages.ghc8107.purescript was fixed by me last week and all it took me was about an hour, so I’m not sure all this manually orchestrated stuff is necessary.

Did you jailbreak it?

That's one reason I left the useJailbreak option in this PR. If you flip that on, all the manually orchestrated stuff goes away and it's like a two-line nix expression. But I'm not sure if ignoring purescript.cabal's version bounds is considered acceptable. The purescript developers certainly don't like it.

[ghost]

• This means that purescript can't easily be used as a Haskell library dependency for another Haskell package (like purenix for example).

Purenix and purescript-native are next on my list.

I don't think we really want random Haskell libraries depending on this custom purescript-tracking-haskellPackages rather than the broader-ecosystem-tracking-haskellPackages. As the purescript developers say, their project is mainly an executable and only incidentally a library.

[ghost]

And of course things like building with musl and statically-linking.

and overlays.

[ghost]

Okay, let me summarize where things appear to stand here. Whereas:

1. This PR is not mergeable with the current approach, mainly because it would require an impractical amount of hand maintenance. I have closed it.
2. The most viable strategy for getting a long-term-maintainable built-from-source purescript in nixpkgs seems to be implementing cabal2nix cabal.project.freeze, which would run cabal2nix cabal://package-version for each (package,version) pair in the freeze file. Then the nixpkgs purescript expression would have an update.sh script that simply fetches the source, runs cabal update; cabal freeze; cabal2nix cabal.project.freeze. This style of nixpkgs packaging would be considered explicitly less desirable than integration into haskellPackages, to be tolerated only as a last resort -- as a slightly-less-awful alternative to patchelf-binary-repacking packaging.
3. As mentioned earlier, all of the above is moot unless I personally am willing to be the maintainer of the resulting purescript expression, and take responsibility for ensuring that it continues to be buildable and up-to-date. I would be happy to do this if purescript-native and purenix are things I am using on a regular basis. However I haven't spent much (or really, any) time using these two tools because the target computing environment I have in mind has requirements (mainly honoring overlays which add patches and change stdenv) that patchelf-binary derivations can never satisfy. I wanted to make sure that a purescript expression that meets these requirements was possible before investing effort in learining purescript. Now that I know it is possible, I can invest that effort.

Now therefore:

1. It will take a month or two before I know for sure if we will be adopting purescript-family tooling for long-term use. So, that's probably the timeframe in which you'll hear from me again about this.
2. In the meantime, I would appreciate feedback on whether or not (2) is an acceptable feature for cabal2nix to have (obviously I would write the implementation).