Cabal REPL stack overflows when working around lack of support for multiple home units

[habibalamin]

Describe the bug

Since Cabal lacks multiple home units support, when using Cabal to power the REPL that GHCid sends reload commands to, you can either load the test suite component, watch for changes to the test suite only and not the library and auto-rerun the test suite on build success, or you can load the library component, watch for changes to the library only and not the test-suite and auto-rerun a library entrypoint on build success (like a web server).

I'm sure you could even run two GHCid processes so that one reruns the tests and the other reruns the server, but the server will only reload when the library changes (not a problem) and the tests will only reload when the tests change (a pretty big problem).

The workaround is to add the library hs-source-dir to the test suite component, as well as the library modules to the test suite's other-modules, so that GHCid can detect changes to library files as changes to the test suite files as well. Any exposed-modules library entrypoints that you want GHCid to also be able to run (like a server) on build success, perhaps in a second process, must be in the other-modules of the test suite.

You can do all this with a common stanza, and I haven't tried without it (way too much duplication otherwise). All good, as the common GHCid refrain goes, right? Wrong!

All good if you run ghcid directly from the Cabal store (which you can get by doing cabal v2-exec -- which ghcid or similar), but if you just run ghcid via cabal v2-exec, running the GHCid process which watches the library and auto-reruns the server right after running the GHCid process which watches the test suite and auto-reruns the test suite results in a stack overflow before GHCid can even open the REPL it will send commands to.

This then necessitates another workaround on top of the fact that I'm already only doing this to work around Cabal's lack of support for multiple home units.

To Reproduce

Check out my repo https://github.com/habibalamin/ghc-env-breaks-ghcid-server-watch-using-lib-home-and-test-suite-imports-lib-deps-from-common-stanza.

Run make watch-test, Ctrl-C it, then run make watch-server-broken.

Expected behavior

I expect Cabal to just be able to load multiple units, and this will even allow me to just do the whole thing in a single GHCid process that watches both library and test code, then on build success, either runs the test suite to completion and the server right afterwards, or just runs them both in parallel somehow (which requires handling interleaving screen output, but that'd be my responsibility, not Cabal's or GHCid's).

Barring that, with the workaround, I expect to be able to just use cabal v2-exec to run any command line tool that is installed in my build-tool-depends, like GHCid (although it's more of a dev tool than a build tool, perhaps that can be another ticket), without it causing funky interaction and state issues.

System information

• macOS Ventura 13.5.1 on Apple Silicon M1 Pro
• GHCup 0.1.19.4
• Cabal library 3.10.1.0 via aforementioned GHCup
• Cabal 3.4 package description file
• GHC 9.4.7 via aforementioned GHCup

Additional context

I found two workarounds.

The cabal v2-exec sets a GHC_ENVIRONMENT environment variable that seems to cause the breakage, which you can unset with env before running the Cabal REPL.

The other workaround is to run the Cabal REPL with the test suite component instead of the library component even when running the GHCid process that watches the library and auto-reruns the server instead of the test suite.

In the reproduction repo, see Make rules, watch-server-workaround-use-test-suite-home-unit and watch-server-workaround-no-ghc-environment, for the implementations of watch-server w/ the respective workarounds.

Basic code quality issues with these are outlined in the reproduction repo.

See ndmitchell/ghcid#320 for the GHCid ticket relating to its inability to monitor test-suite AND library changes with Cabal, due to Cabal's lack of support for multiple home units.

[fendor]

I feel like this issue is about like three separate issues but multiple home unit support in cabal has been discussed and addressed in #8491.

But see #9195 for some issues.

[habibalamin]

This ticket references other issues to give the context behind why I am trying to do what I'm trying to do, which sadly is a bit complicated, but the real issue here is the stack overflow.

The stack overflow happens when running the server watcher directly after running and quitting the test suite watcher. Regardless of multiple home units support or not, this shouldn't happen.

[habibalamin]

However, that ticket you linked, #9195, suggests that Cabal will be getting multiple home units support in 3.11, which shouldn't be too far away. I am happy to live with the workaround until then, in that case.

I'll keep this ticket open in case you want to fix this bug (or you can close it), and the workarounds I've listed above should anyone need them.

[fendor]

Please supply some logs for the stack overflow (?) error, if there is no readable output, try running the cabal commands with -v.

[habibalamin]

$ make watch-test
Loading      cabal v2-repl       test-suite:test-minimal-reproduction    ...
Build profile: -w ghc-9.4.7 -O1
In order, the following will be built (use -v for more details):
 - minimal-reproduction-0.1.0.0 (test:test-minimal-reproduction) (first run)
Preprocessing test suite 'test-minimal-reproduction' for minimal-reproduction-0.1.0.0..
Loaded package environment from /tmp/minimal-reproduction
GHCi, version 9.4.7: https://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /Users/habib/dotfiles/ghci
[1 of 5] Compiling Dotenv           ( src/lib/Dotenv.hs, interpreted )
[2 of 5] Compiling MinimalRepro.Server ( src/lib/MinimalRepro/Server.hs, interpreted )
[3 of 5] Compiling Spec             ( src/spec/Spec.hs, interpreted )
[4 of 5] Compiling Main             ( src/spec/Main.hs, interpreted )
Ok, four modules loaded.

All good (4 modules, at 14:05:30)
Running test...

Finished in 0.0001 seconds
0 examples, 0 failures

...done
^Cmake: *** [watch-test] Interrupt: 2

$ make watch-server-broken
Loading      cabal v2-repl       library:libminimalrepro      --ghc-option=-fdiagnostics-color=always    ...
Build profile: -w ghc-9.4.7 -O1
In order, the following will be built (use -v for more details):
 - minimal-reproduction-0.1.0.0 (lib:libminimalrepro) (ephemeral targets)
Preprocessing library 'libminimalrepro' for minimal-reproduction-0.1.0.0..
Loaded package environment from /tmp/minimal-reproduction
GHCi, version 9.4.7: https://www.haskell.org/ghc/  :? for help
stack overflow: use +RTS -K<size> to increase it
Error: cabal: repl failed for lib:libminimalrepro from
minimal-reproduction-0.1.0.0.

Command "     cabal v2-repl       library:libminimalrepro      --ghc-option=-fdiagnostics-color=always   " exited unexpectedly with error message:
make: *** [watch-server-broken] Error 1

By the way, the my GHCi rcfile just sets the prompt and nothing else.