Multi-project stack does not properly reload

[rvion]

@ndmitchell I have a problem with ghcid 0.6.1

*It is currently impossible to start ghcid so it reloads my code when some .hs file change in my multi-package project

I use stack, I have a set of ~10 packages within one same folder.
All of them have a src folder containing module tree (Foo.hs, Foo/Bar.hs, etc.)
I have one empty package at the top level importing all other packages to simplify compatibility with cabal specific stuff (like haskell-ide-atom).

• default doesn't work

  ghcid --> never reload when any file changes

  (this is probably a bug 🐛 related to this message printed in ghci: )

  * * * * * * * *
  The main module to load is ambiguous. Candidates are:
  1. Package `app' component exe:app-example with main-is file: /Users/rvion/dev/cogito/app/Main.hs
  2. Package `pghs' component exe:pg2hs with main-is file: /Users/rvion/dev/cogito/pghs/Main.hs
  You can specify which one to pick by:
   * Specifying targets to stack ghci e.g. stack ghci app:exe:app-example
  Specify main module to use (press enter to load none): Not loading any main modules, as no valid     module selected
   * Specifying what the main is e.g. stack ghci --main-is app:exe:app-example
  
   * Choosing from the candidate above [1..2]
  * * * * * * * *
  

• command to load top level package doesn't work because it doesn't watch local deps change

  `ghcid -c "stack ghci toplevelpackage"

• one reload param with list of file doesn't watch

  ghcid --test="testFunction" --reload="**/*.hs" -> loop weirdly because of "*/.hs"

• one --reload per folder doesn't work because no recursivity

  ghcid --reload pkg1 --reload pkg2 --> never reload when pkg1/src/*/.hs change because only watch for pkg1/* (direct folder, no recursive)

So far: the only 2 solutions I thought about were

1. make a giant cmd like

   ghcid ... \
      --reload pkg1/src/ \
      --reload pkg1/src/Foo \
      --reload pkg1/src/Foo/Bar \
   
      --reload pkg2/src/ \
     -- ...

   possible partial workaround:

   ghcid  $(ls **/*.hs | while read -r line; do dirname "$line"; done | uniq | sed 's/^/--reload=/' | tr '\n' ' ' | sed 's/ $//g')

2. start one ghcid cmd per package

I went with (1) but then I found a bug 🐛 in ghcid:

ghcid --reload=pkg1/src/

doens't reload anything unless I'm specifying a target.
since I have several executables, not specifying a target shows

* * * * * * * *
The main module to load is ambiguous. Candidates are:
1. Package `app' component exe:app-example with main-is file: /Users/rvion/dev/cogito/app/Main.hs
2. Package `pghs' component exe:pg2hs with main-is file: /Users/rvion/dev/cogito/pghs/Main.hs
You can specify which one to pick by:
 * Specifying targets to stack ghci e.g. stack ghci app:exe:app-example
Specify main module to use (press enter to load none): Not loading any main modules, as no valid module selected
 * Specifying what the main is e.g. stack ghci --main-is app:exe:app-example

 * Choosing from the candidate above [1..2]
* * * * * * * *

and then no reload occurs at all

so I did

ghcid -c "stack ghci toplevelpackage" --reload=pkg1/src/ --reload=pkg1/src/Foo --reload=...

but arg again: ghci don't reload files in pkg1 since I specified toplevelpackage. so toplevelpackage reload when I change pkg1, but it doesn't pick changes in pkg1

Otherwise, great project ! Thanks

[rvion]

after some more experimentation, I didn't find any way for me to reload default ghcid command when any */.hs change.
(above message has been edited several after to reflect my discoveries)

[ndmitchell]

Reload can take a directory name or a file name. It can't take patterns and it doesn't deal with recursive directories. These are deliberate limitations - things that could be fixed, but generally needing super-powerful reloading is a good sign that something else is broken. In this case, clearly something else is broken! It should be all automatic.

When you run ghcid the first line should be something like:

Loading stack ghci --test --no-load --ghci-options=-fno-code ...

What is the associated command? If you run the associated command (e.g. stack ghci --test --no-load --ghci-options=-fno-code in this case), what does Stack load? Do you have a .ghci file? If you do :show modules what does it say?

[rvion]

@ndmitchell
stack ghci --test --no-load --ghci-options=-fno-code
doens't :show modules

$ stack ghci --test --no-load --ghci-options=-fno-code
Configuring GHCi with the following packages: cogito, app, orm, pghs, types
GHCi, version 7.10.3: http://www.haskell.org/ghc/  :? for help
Prelude> :show modules
Prelude>

but...

$ ghcid
Loading stack ghci --test --ghci-options=-fno-code ...

* * * * * * * *
The main module to load is ambiguous. Candidates are:
1. Package `app' component exe:app-example with main-is file: /Users/rvion/dev/cogito/app/Main.hs
2. Package `pghs' component exe:pg2hs with main-is file: /Users/rvion/dev/cogito/pghs/Main.hs
You can specify which one to pick by:
Specify main module to use (press enter to load none): Not loading any main modules, as no valid module selected
 * Specifying targets to stack ghci e.g. stack ghci app:exe:app-example

 * Specifying what the main is e.g. stack ghci --main-is app:exe:app-example
 * Choosing from the candidate above [1..2]
* * * * * * * *

Configuring GHCi with the following packages: cogito, app, orm, pghs, types
GHCi, version 7.10.3: http://www.haskell.org/ghc/  :? for help
[ 1 of 40] Compiling PG.Types         ( /Users/rvion/dev/cogito/pghs/src/PG/Types.hs, nothing )
[ 2 of 40] Compiling PG.Dump          ( /Users/rvion/dev/cogito/pghs/src/PG/Dump.hs, nothing )
[ 3 of 40] Compiling Gen.Types        ( /Users/rvion/dev/cogito/pghs/src/Gen/Types.hs, nothing )
[ 4 of 40] Compiling Gen.Pluralize    ( /Users/rvion/dev/cogito/pghs/src/Gen/Pluralize.hs, nothing )
...

[ndmitchell]

Do you have a .ghci file?

[rvion]

No, I don't have a .ghci file (sorry, I was still writing the other part of my answer ^^)

[rvion]

Reload can take a directory name or a file name. It can't take patterns and it doesn't deal with recursive directories. These are deliberate limitations

Completely makes sense to me

---

the strange thing (bug?) is that

ghcid doens't reload anything by default
ghcid --reload="..." doens't reload anything either despite the param

but

ghcid -c "stack ghci sometarget" reload sometarget

and

ghcid -c "stack ghci sometarget" --reload="local-dep-1/src" reload sometarget

• when any of its source files changed
• when local-dep-1/src/* change,

but in both cases, it doens't reload modules of local-dep-1

---

also, sorry for editing my original message at the same time you were reading it, but I included more details in it. Thanks for being super responsive on that :)

[ndmitchell]

Let's focus on the first problem first - why doesn't just plain vanilla ghcid work. It isn't loading anything to start with, so nothing else works. The entire problem seems to come back to the --no-load on the initial command line, and reading the logic:

["--no-load" | ".ghci" `elem` files] ++

What command do you use to start ghcid? Which directory are you starting ghcid from? Can you do a ls -a on that? Is there a .ghci file or directory?

[rvion]

@ndmitchell

I made this repo for you to reproduce easilly https://github.com/rvion/ghcid-bug-repro
(please clone it as git clone https://github.com/rvion/ghcid-bug-repro ghcid-test because my github repo name doesn't match the top level ghcid-test package)

if you clone it and run ghcid, you'll see that nothing reload when you change file.
If you specify one specifc project (a, b, or ghcid-test) by running ghcid -c "stack ghci b", then ghcid will only reload modules from that project, but not reload local dependencies depended on by it, even when they are modified.

Neither ghcid -c "stack ghci b" nor ghcid -c "stack ghci b" --reload="a/src" will reload module A depended on by B.

---

to answer you questions:

What command do you use to start ghcid?

• either plain ghcid but nothing reload,
• either ghcid -c "stack ghci toplevelpackage" but modifications done in other local dependencies of toplevelpackage are not reloaded.

Which directory are you starting ghcid from?

I always launch command from the root project folder.

Can you do a ls -a on that?

(slightly edited)

toplevelpackage $ ls -la
total 96
drwxr-xr-x   16 rvion  staff   544 Apr 12 18:34 .git
-rw-r--r--    1 rvion  staff   212 Apr 11 14:05 .gitignore
drwxr-xr-x    6 rvion  staff   204 Apr 11 13:31 .stack-work
-rw-r--r--    1 rvion  staff   196 Apr 12 12:27 TopLevelPackage.hs
-rw-r--r--    1 rvion  staff  4823 Apr 11 16:02 README.md
drwxr-xr-x    9 rvion  staff   306 Apr 11 16:02 app
drwxr-xr-x    6 rvion  staff   204 Apr  6 17:09 arch
drwxr-xr-x    9 rvion  staff   306 Mar 19 11:09 auth
drwxr-xr-x   13 rvion  staff   442 Apr  7 09:49 client
-rw-r--r--    1 rvion  staff  1086 Apr 12 09:27 toplevelpackage.cabal
-rw-r--r--    1 rvion  staff   632 Apr 12 09:27 toplevelpackage.hpack.yaml
drwxr-xr-x    3 rvion  staff   102 Apr 11 12:23 doc
drwxr-xr-x    6 rvion  staff   204 Apr 11 14:06 org
drwxr-xr-x   11 rvion  staff   374 Apr 12 13:03 orm
drwxr-xr-x    8 rvion  staff   272 Apr 11 15:38 pghs
drwxr-xr-x@   7 rvion  staff   238 Apr 11 12:25 scripts
drwxr-xr-x    5 rvion  staff   170 Apr 11 14:09 sql
-rw-r--r--    1 rvion  staff   968 Apr 11 16:14 stack.yaml
drwxr-xr-x    6 rvion  staff   204 Apr 11 16:03 types

Is there a .ghci file or directory?

No

why doesn't just plain vanilla ghcid work. It isn't loading anything to start with

as I wrote in (my comment above)

here is the output of a plain ghcid command invocation at the root of the folder. I don't really know if it's loading stuff correctly or not

$ ghcid
Loading stack ghci --test --ghci-options=-fno-code ...

* * * * * * * *
The main module to load is ambiguous. Candidates are:
1. Package `app' component exe:app-example with main-is file: /Users/rvion/dev/cogito/app/Main.hs
2. Package `pghs' component exe:pg2hs with main-is file: /Users/rvion/dev/cogito/pghs/Main.hs
You can specify which one to pick by:
Specify main module to use (press enter to load none): Not loading any main modules, as no valid module selected
 * Specifying targets to stack ghci e.g. stack ghci app:exe:app-example

 * Specifying what the main is e.g. stack ghci --main-is app:exe:app-example
 * Choosing from the candidate above [1..2]
* * * * * * * *

Configuring GHCi with the following packages: cogito, app, orm, pghs, types
GHCi, version 7.10.3: http://www.haskell.org/ghc/  :? for help
[ 1 of 40] Compiling PG.Types         ( /Users/rvion/dev/cogito/pghs/src/PG/Types.hs, nothing )
[ 2 of 40] Compiling PG.Dump          ( /Users/rvion/dev/cogito/pghs/src/PG/Dump.hs, nothing )
[ 3 of 40] Compiling Gen.Types        ( /Users/rvion/dev/cogito/pghs/src/Gen/Types.hs, nothing )
[ 4 of 40] Compiling Gen.Pluralize    ( /Users/rvion/dev/cogito/pghs/src/Gen/Pluralize.hs, nothing )
...


--------------

[rvion]

@ndmitchell

ok. I finally understood the problem (or at least a good part of it):

stack ghci has a --load-local-deps flag.
ghcid should use it by default I think.

this is working fine for me:

ghcid -c "stack ghci toplevelmodule --load-local-deps" --test="..."

now, the last thing to figure out is why the default plain ghcid command (without args) doesnt' reload anything.
(is is also just the --load-local-deps flag missing ? If so, problem fixed, but I think there might be something more to it, because ghcid -c "stack ghci --load-local-deps" --test="2+2" load all local packages at command invocation, but never reload anything, and never refresh the screen neither (full redundant import warning message displayed) )

[ndmitchell]

Thanks for the analysis. I'm afraid I didn't get time to look at it in detail yet, and probably won't until next week - but do hope to take a look pretty soon.

[rvion]

Thanks :)
All of your tools are great gift to the community

[ndmitchell]

Thanks for the test case. The issue is that Stack drops us out at:

Specify main module to use (press enter to load none):

Ghcid then sends some essential configuration commands to GHCi to allow it to detect loading has finished. They get sent to that prompt, and thus do nothing. As a result, ghcid can't tell that loading finished. I tried sending an extra \n before my important commands, but Stack seemingly doesn't accept input that has already arrived - maybe that's a stack bug. Needs more investigation.

[ndmitchell]

Hmm, it actually seems that Stack is swallowing subsequent commands. If I create a file input.txt:

1
print 1

And run:

type input.txt | stack ghci --test

Then the line 1 goes to stack to select the right project, but the print 1 line gets swallowed. Maybe a changing in buffering can get Stack to only consume 1 line? Otherwise some flag to say don't read it from the terminal might be a good idea.

[mgsloan]

It's probably because we're passing over control of stdout / stdin to the ghci project, but that doesn't pass along the buffered input. Just a guess!

I suppose the fix would be to greedily read as much input as possible. I'm actually a little surprised this doesn't just work.

[ndmitchell]

@mgsloan, that was my guess too - I wondered if setting LineBuffering/NoBuffering explicitly on stdin might help. I do that on my side, but it would need to be present in the Stack side too - and even then might make no difference if the buffering happens somewhere you can't see it.

[mgsloan]

I tried NoBuffering in stack, but no such luck with printf "1\nputStrLn 2" | stack ghci with multiple executable targets. Not sure what's up with that.

One approach might be to pass in --ghci-options="--ghci-script ..." or something like that.

[ndmitchell]

I tried no buffering in a small sample program, in both Haskell and C, and in neither case could I get the child process to pick up stdin. It just seems to be "not possible" as far as I can tell.

For ghcid, I regularly need to send stuff on stdin, and get stuff on stdout. I could use ghci-script to send the setup commands, watch for them to echo back, and then send real commands to stdin. There are a few issues:

1. What if the parent process is still pulling off stdin? I don't think this happens, but it seems that I'm not getting guaranteed behaviour anymore.
2. Given an arbitrary command it can be hard to add in a ghci-script, since I don't know if I send it directly, through stack, or something else.
3. I believe ghci script can conflict with .ghci files, which would be a pain.

So an easier alternative would be to ask Stack to not put that, and pick the default. Is that an option @mgsloan?

[mgsloan]

Given an arbitrary command it can be hard to add in a ghci-script, since I don't know if I send it directly, through stack, or something else.

True, it may be necessary to make ghcid know about stack. I realize that's counter the minimalist design, but hey, if we want the functionality

I believe ghci script can conflict with .ghci files, which would be a pain.

I believe it's possible to generate ghci scripts that tend to play well with others - see commercialhaskell/stack#1888

So an easier alternative would be to ask Stack to not put that, and pick the default. Is that an option @mgsloan?

Yup, that's a viable option, lets do that. Something like --no-main?

[luigy]

I tried NoBuffering in stack, but no such luck with printf "1\nputStrLn 2" | stack ghci with multiple executable targets. Not sure what's up with that.

It would be nice for this to work!

Yup, that's a viable option, lets do that. Something like --no-main?

Yeah, looks good. We still have the (experimental) flag in stack so nbd 😂

[mgsloan]

I had a thought for how to solve this. What if ghcid left stdin to the user until it sees a line of output matching GHCi, version ...?

[ndmitchell]

Yep, that might work - will give it a whirl.

[ndmitchell]

I followed @mgsloan's suggestion, and it worked. With that, I can load up the project properly. I am not using --load-local-dep, but it doesn't seem to be necessary.

I'm still gently unwary with waiting for the GHCi, version string to go past, but let's address that if it turns out to be problematic somewhere.

[ndmitchell]

This doesn't work because:

• I have to send \n to the stdin before doing anything else, so that I can get past the interactive prompt Stack puts up.
• I don't know in advance if Stack is going to ask me a question, and if it doesn't, ghci gets the \n and then the prompt hasn't been changed so that goes wrong.

Now trying to only send \n if I see a stack specific prefix, but that's getting very ugly. Having an option to Stack to not read this preference from stdin, some kind of --main-is whatever_the_default would make it cleaner.

[ndmitchell]

I implemented something which looks for a Stack line and send the \n then. Seems to work, but yuk. @mgsloan - would you guys be happy with a flag to say "use the default, don't ask me"?

[ndmitchell]

OK, I've implemented more hacks, and I think it's now working. It is quite unpleasant, and I worry about the long-term robustness.

[mgsloan]

Cool! Yeah, not super pleasant I realize. Thanks for making it happen!

We could also add a flag, certainly.

[ndmitchell]

I've put all the stack requests into commercialhaskell/stack#1948 (comment) so closing this.