cabal watch command

[gbaz]

This ticket is intended to capture ideas for a cabal watch command to give functionality along the lines of ghcid. In particular, one should be able to "watch" a component just as one new-repls a component. I would imagine that there would be a few knobs to turn on what happens when something is altered. One could recompile with no-code (related issue: #1176) or perhaps send some form of signal to a running ghci process that just causes it to execute a :r automatically. In which case cabal watch would really be an option on new-repl. It might also be configurable to run a custom command. The main idea is that since cabal knows which files are in the dependency graph, it is a good place to stick a watch command on them.

[gbaz]

One missing component is just a little tiny api to control ghci from outside -- i.e. send it commands like ":r" and ":q" over some port or the like. This would be hugely useful to ghcid too and other tools in general. Maybe @alanz or @ndmitchell have some thoughts?

[ndmitchell]

There is an API to send :r and :q to ghci from the outside - it's called System.Process. The new API would have to be simpler and more robust, which is hard given that the existing API is super simple and reasonably robust...

If you put it inside ghci/ghc itself there would be a lot of value, but having Cabal fake this on top just seems to be moving the problem around.

Perhaps one thing that would be useful is Cabal declaring what files/directories would need to to restart, and which would require a reload? Just pretty printing Depending on .cabal-metadata-file or similar would be useful.

[gbaz]

My sense is that using stdout/stdin for communication with interactive programs is a bit messy, even though ghcid manages to pull it off? What about a port or a shared fd on the system? Not sure which of these mechanisms works well across platforms either. A different division of labor that just gives a ghci watch and has cabal give it some helpful tips does sound more robust if we're ok with pushing enough fsnotify functionality directly into the ghc codebase.

[ndmitchell]

There are lots of ways the complexity of ghcid leaks out, but stdin/stdout isn't one of them, so you'd be paying a cost (shared fd or whatever) and not really reaping any benefits. I completely agree that stdin/stdout is ugly, especially with buffering, but part of that is that the underlying GHC is a single shared resource - if it were multithreaded (allowed multiple concurrent executions), then you could imagine a richer communication (http being my default favourite, since it's easily debuggable).

[gbaz]

i confess i don't understand how a single shared resources gets in the way of richer communication? suppose we had an http port -- why couldn't communication on that still only be with regards to the single shared ghc resource.

[ndmitchell]

Sorry, to rephrase better, the communication channel we have now (stdin/stdout) sucks if you had a concurrent resource, but we don't. All the other communication channels are almost no better when you have a single-access resource, but quite a lot better when you have a multiaccess resource, so if you made the resource multiaccess there would be more desire to go to richer communication.

[gbaz]

Just a note: some of the data we need exists in new-build --dry and new-repl --dry but we also should have a way to get fully-qualified components, etc

[alanz]

I'm late to this party, sorry.

I know when I looked into using ghci as a subcomponent of hie, one of the things that troubled me is that all the IO happens via stdin/stdout, including when user code is executed and it can do anything with stdio, which means any kind of tooling interaction while a program is running must pause, and also that anything can be sent by the app, since it is completely out of our control.

One of the possible workarounds for this is to use the external interpreter mode, and use specific pipes for the user stdio, which can then be routed to the host IDE in a controlled way.

And in this model putting a proper multiplexed protocol on the stdio, or exposing a different API end point becomes possible.

[ndmitchell]

The user can't do anything with stdio, in particular they can't guess at the magic cookie you use to delineate their stuff from yours. That's what ghcid relies on. However, on the flip side, they can leave a daemon running in ghci which prints to stdout randomly - but if they do they probably aren't going to have an enjoyable time.

That said, I don't think cabal watch solves any of those problems. It solves an entirely different set of problems (that I don't find very problematic...).

[alanz]

True.

And I also agree that I do not see any real use for this feature in hie, nor for me in everyday usage.

[gbaz]

So the question to me is if there is some underlying improvement to ghci that might improve life for hie and ghcid, and as a bonus could also enable a "cabal watch" feature along similar lines.

Similarly if there is a particular feature of cabal that might be useful to ghcid or hie or both in terms of e.g. emitting the --dry output in a better fashion or the like?

[ndmitchell]

The biggest problem that ghcid faces is that ghci is actually quite buggy. It doesn't reload when it should, it over-reloads, and just gets wedged every now and again. What's needed is a dedicated maintainer with lots of times to actually fix the bugs that are already in trac. Otherwise we're putting lipstick on a pig.

For the use of ghcid with Cabal, it's not really a mode I use it in (I tend to use raw ghci with a globally installed package data set) - but I've not really had any complaints from people.

[alanz]

Apart from what @ndmitchell said, my biggest need is to have something in cabal that can make cabal-helper unnecessary. There is an issue for it, and @DanielG has some concerns with what is there. So adressing those and getting it done would help.

And identifying the memory leak would help too.

[gbaz]

is that issue #3872 ?

[alanz]

That one does carry some of the discussion, but there is also #2771 and the parent of #3872, being DanielG/ghc-mod#835

[fosskers]

A first-class cabal watch that tracked changes across interconnected components would be very nice. Until then, piping a stack / cabal repl into ghcid seems to do the trick.

[doyougnu]

Not sure if this is still active but I use entr as a work around. For example, I'm working on a branch of sbv and I want to rerun a particular test every time a .hs file changes in my root directory:

[nix-shell:~/programming/sbv]$ find -name "*.hs" | entr -s 'cabal new-test sbv:SBVTest --test-options="--accept --hide-successes +RTS -N -RTS -p "genIntTest.arithmetic-clearBit.u64_0_0""'

[chrisdone]

FYI, as an alternative approach for Emacs users, I made this tiny script for my own purposes:
https://github.com/chrisdone/emacs-config/blob/master/packages/watchexec-ghci/watchexec-ghci.el

Install watchexec to your PATH.

Go to a buffer that has ghci in it (launched however you like cabal repl, stack ghci, or ghci, etc.), such as shell-mode or a repl equivalent mode. Then you run

M-x watchexec-ghci RET /the/dir/you/want/to/watch RET :r

for example and it'll run :r RET in the buffer whenever a file changes.

I tend to use: :cmd to encode e.g. ":r" followed by ":main" for running a test suite on every change.

It's an instantly fast workflow, like ghcid. The nice thing is you can still work in the REPL at the same time.

[peterbecich]

Noting this comment from the ghcid readme here:

FAQ:

I want to run arbitrary commands when arbitrary files change.

This project reloads ghci when files loaded by ghci change. If you want a more general mechanism something like Steel Overseer or Watchman will probably work better.

https://github.com/ndmitchell/ghcid#i-want-to-run-arbitrary-commands-when-arbitrary-files-change

[domenkozar]

Hitting ghcid not cleaning up threads really makes development hard, how can we help to get this going?

I've opened haskell-org/summer-of-haskell#178, happy if someone more knowledable around HIE would mentor, but I'm also happy to do it.

[sgraf812]

how can we help to get this going?

I'm lacking a lot of context here, having just skimmed through both this thread as well as ndmitchell/ghcid#191 (comment).

But what exactly is "this"? How has "this" an easier time to kill the supposed zombie processes that are described in ndmitchell/ghcid#191? Wouldn't it be easier to fix ghcid?

[domenkozar]

how can we help to get this going?

I'm lacking a lot of context here, having just skimmed through both this thread as well as ndmitchell/ghcid#191 (comment).

But what exactly is "this"? How has "this" an easier time to kill the supposed zombie processes that are described in ndmitchell/ghcid#191? Wouldn't it be easier to fix ghcid?

It would be certainly possible to fix ghcid, but in the last year, especially with HLS we've gained a much better foundations on which we can build similar functionality in more robust way.

[sgraf812]

Aha, but what role does cabal (library as well as executable, I suppose) play in that improved version of ghcid? I think it's reasonable to refactor ghcid such that it uses hie-bios as you stated in the GSoC proposal (I naively presumed that was already the case), but I don't see why cabal watch couldn't be implemented as an external command cabal-watch that simply symlinks to ghcid2.

I'm having the impression that the OP suggests to reimplement huge parts of what ghcid does reasonably well (such as interprocess communication), while not explicitly tackling any of the problems that you linked to. It just sounds like quite a bit of distracting work is necessary before actually improving on the situation we are in now, plus why not maintain cabal-watch/ghcid2 as an independent project when that is feasible?

[domenkozar]

My main use case is that I'd like a fast feedback loop when developing Haskell executables.

I'm using https://devenv.sh that would roughly look like this:

{ pkgs, ... }: {
  languages.haskell.enable = true;
  languages.haskell.package = lib.mkForce pkgs.haskell.compiler.ghc963;

  processes.bar.exec = "cabal watch --restart schema.sql --run exe:bar lib:foo -- some args";
}

l'd imagine it would do something like cabal repl lib:foo exe:bar --enable-multi-repl under the hood to load a ghci session.

Upon reloading (a module has changed), it would :r, kill all threads and call main.

Upon restarting (cabal has changed or --restart <file>) it would reenter the repl and start the main function again.

It's certainly possible to implement this outside of Cabal (as stack and ghcid have demonstrated), but I do believe it's essential feature for software development that should ship with Cabal.

[sgraf812]

Thanks, that seems like a compelling use case. Perhaps one could start by writing an external tool (ghcid2), see whether it does the job and then later merge it into Cabal? If that is not possible because ghcid2 really needs to be tightly coupled to Cabal, then we'll see soon enough.