Docker: stack commands "hang" until return hit waiting for docker start to complete

[Dridus]

Disclaimer: I'm filing this even though I'm running in an odd environment and so it could be something other than stack itself, but I'm not sure and I figure best to file it and then close it if need be. Also this issue might be a bit longer than desired for issue filing. Let me know if you want it split out into gists or whatever.

This is possibly related to #547, #531, #194.

I'm running stack 0.1.3.1 with the lts-3.0 resolver on OSX with boot2docker 1.7.1 and docker 1.7.1 using a custom image.

My custom image is my best attempt to reverse engineer fpco/stack-build:lts-2.22 with adjustments to make it work with boot2docker (in particular, ignore the WORK_UID environment and read it from the .cabal directory that's mounted) and have lts-3.0 / GHC 7.10.2. I've pushed the sources for this image to GH: https://github.com/SocialBrain/docker-infrastructure/tree/master/stack-build

I've previously had this configuration working with no apparent issues other than the boot2docker warning with stack 0.1.2, GHC 7.10.1 from the haskell:7.10 docker image, and the nightly-2015-07-23 resolver.

Since upgrading to stack 0.1.3.1 and lts-3.0 though I've had some trouble. Other issues I managed to track down to the particular GHC versions and how I was installing them and have since resolved, but the final issue I can't sort out is that when I run stack commands that delegate through docker they wait for input before finishing.

For example, if I stack -v solver, I get this output (ignore *** lines - they're debug from my_init):

> stack -v solver
Version 0.1.3.1, Git revision 908b04205e6f436d4a5f420b1c6c646ed2b804d7
2015-08-14 17:28:09.39285: [debug] Checking for project config at: /Users/rmm/sb/swarm/stack.yaml @(stack-0.1.3.1:Stack.Config src/Stack/Config.hs:459:9)
2015-08-14 17:28:09.393791: [debug] Loading project config file stack.yaml @(stack-0.1.3.1:Stack.Config src/Stack/Config.hs:482:13)
2015-08-14 17:28:09.399917: [debug] Run process: docker --version @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:09.421453: [debug] Run process: id -u @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:09.427448: [debug] Run process: id -g @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:09.430322: [warn] Warning: Using boot2docker is NOT supported, and not likely to perform well. @(stack-0.1.3.1:Stack.Docker src/Stack/Docker.hs:189:12)
2015-08-14 17:28:09.430415: [debug] Run process: docker inspect sb/swarm-build:latest @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:09.629991: [debug] Run process: docker create --net=host -e STACK_IN_CONTAINER=1 -e STACK_ROOT=/Users/rmm/.stack -e WORK_UID=501 -e WORK_GID=20 -e WORK_WD=/Users/rmm/sb/swarm -e WORK_HOME=/Users/rmm/sb/swarm/.stack-work/docker/default -e WORK_ROOT=/Users/rmm/sb/swarm -v /Users/rmm/.stack:/Users/rmm/.stack -v /Users/rmm/sb/swarm:/Users/rmm/sb/swarm -v /Users/rmm/sb/swarm/.stack-work/docker/_sandbox/default:/Users/rmm/sb/swarm/.stack-work/docker -v /Users/rmm/sb/swarm/.stack-work/docker/_home:/Users/rmm/sb/swarm/.stack-work/docker/default -v /Users/rmm/.stack:/Users/rmm/sb/swarm/.stack-work/docker/default/.stack -v /Users/rmm/sb/swarm/.stack-work/docker/default/.ghc:/Users/rmm/sb/swarm/.stack-work/docker/default/.ghc -v /Users/rmm/sb/swarm/.stack-work/docker/default/.cabal:/Users/rmm/sb/swarm/.stack-work/docker/default/.cabal -v /Users/rmm/sb/swarm/.stack-work/docker/default/.ghcjs:/Users/rmm/sb/swarm/.stack-work/docker/default/.ghcjs -t -i sb/swarm-build:latest stack --internal-re-exec -v solver @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:09.907688: [debug] Run process: /usr/local/bin/docker start -a -i 5a5aeebd6403d990fa2d8fcf02cbcd2554ca42ea1b69355452e6fea4dd5036bb @(stack-0.1.3.1:System.Process.Run src/System/Process/Run.hs:92:5)
*** Running gosu stack stack --internal-re-exec -v solver...
Version 0.1.3.1, Git revision 908b04205e6f436d4a5f420b1c6c646ed2b804d7
2015-08-14 17:28:14.702005: [debug] Checking for project config at: /Users/rmm/sb/swarm/stack.yaml @(stack-0.1.3.1:Stack.Config src/Stack/Config.hs:459:9)
2015-08-14 17:28:14.702483: [debug] Loading project config file stack.yaml @(stack-0.1.3.1:Stack.Config src/Stack/Config.hs:482:13)
2015-08-14 17:28:14.719646: [debug] Run process: ghc --info @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:14.76869: [debug] Run process: ghc --numeric-version @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:14.791786: [debug] Run process: ghc-pkg --no-user-package-db field --simple-output Cabal id @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:14.844554: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:14.892092: [info] This command is not guaranteed to give you a perfect build plan @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:152:5)
2015-08-14 17:28:14.892231: [info] It's possible that even with the changes generated below, you will still need to do some manual tweaking @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:153:5)
2015-08-14 17:28:18.432504: [debug] Run process: ghc --numeric-version @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:18.463329: [info] Asking cabal to calculate a build plan, please wait @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:72:5)
2015-08-14 17:28:18.463424: [debug] Run process: cabal --config-file=/tmp/cabal-solver26/cabal.config install -v --dry-run --only-dependencies --reorder-goals --max-backjumps=-1 --package-db=clear --package-db=global /Users/rmm/sb/swarm/swarm-api-server/ /Users/rmm/sb/swarm/swarm-data-server/ /Users/rmm/sb/swarm/swarm-session-server/ /Users/rmm/sb/swarm/swarm-shared/ /Users/rmm/sb/swarm/thrift/lib/hs/ @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:28:31.670408: [info] extra-deps: @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:184:19)
2015-08-14 17:28:31.670555: [info] - ekg-0.4.0.8 @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:184:19)
2015-08-14 17:28:31.670625: [info] - ekg-json-0.1.0.0 @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:184:19)
2015-08-14 17:28:31.670668: [info] - etcd-1.0.5 @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:184:19)
*** gosu exited with status 0.
*** Killing all processes...

and no prompt. if I hit return, this then happens:

2015-08-14 17:29:40.928159: [debug] Run process: docker rm -f 5a5aeebd6403d990fa2d8fcf02cbcd2554ca42ea1b69355452e6fea4dd5036bb @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
>

while it's waiting for input if I check docker ps there's nothing running. With further investigation it seems like between stack 0.1.2 and 0.1.3 the way it invokes docker has changed and that's causing the issue for me. If I take the docker commands from stack's verbose log:

docker create --net=host -e STACK_IN_CONTAINER=1 -e STACK_ROOT=/Users/rmm/.stack -e WORK_UID=501 -e WORK_GID=20 -e WORK_WD=/Users/rmm/sb/swarm -e WORK_HOME=/Users/rmm/sb/swarm/.stack-work/docker/default -e WORK_ROOT=/Users/rmm/sb/swarm -v /Users/rmm/.stack:/Users/rmm/.stack -v /Users/rmm/sb/swarm:/Users/rmm/sb/swarm -v /Users/rmm/sb/swarm/.stack-work/docker/_sandbox/default:/Users/rmm/sb/swarm/.stack-work/docker -v /Users/rmm/sb/swarm/.stack-work/docker/_home:/Users/rmm/sb/swarm/.stack-work/docker/default -v /Users/rmm/.stack:/Users/rmm/sb/swarm/.stack-work/docker/default/.stack -v /Users/rmm/sb/swarm/.stack-work/docker/default/.ghc:/Users/rmm/sb/swarm/.stack-work/docker/default/.ghc -v /Users/rmm/sb/swarm/.stack-work/docker/default/.cabal:/Users/rmm/sb/swarm/.stack-work/docker/default/.cabal -v /Users/rmm/sb/swarm/.stack-work/docker/default/.ghcjs:/Users/rmm/sb/swarm/.stack-work/docker/default/.ghcjs -t -i sb/swarm-build:latest stack --internal-re-exec -v solver

/usr/local/bin/docker start -a -i 5a5aeebd6403d990fa2d8fcf02cbcd2554ca42ea1b69355452e6fea4dd5036bb

then the issue reproduces as it does with stack. However, if I turn off the --interactive switch from docker start or switch docker create to docker run the problem goes away and it properly exits:

> docker create --net=host -e STACK_IN_CONTAINER=1 -e STACK_ROOT=/Users/rmm/.stack -e WORK_UID=501 -e WORK_GID=20 -e WORK_WD=/Users/rmm/sb/swarm -e WORK_HOME=/Users/rmm/sb/swarm/.stack-work/docker/default -e WORK_ROOT=/Users/rmm/sb/swarm -v /Users/rmm/.stack:/Users/rmm/.stack -v /Users/rmm/sb/swarm:/Users/rmm/sb/swarm -v /Users/rmm/sb/swarm/.stack-work/docker/_sandbox/default:/Users/rmm/sb/swarm/.stack-work/docker -v /Users/rmm/sb/swarm/.stack-work/docker/_home:/Users/rmm/sb/swarm/.stack-work/docker/default -v /Users/rmm/.stack:/Users/rmm/sb/swarm/.stack-work/docker/default/.stack -v /Users/rmm/sb/swarm/.stack-work/docker/default/.ghc:/Users/rmm/sb/swarm/.stack-work/docker/default/.ghc -v /Users/rmm/sb/swarm/.stack-work/docker/default/.cabal:/Users/rmm/sb/swarm/.stack-work/docker/default/.cabal -v /Users/rmm/sb/swarm/.stack-work/docker/default/.ghcjs:/Users/rmm/sb/swarm/.stack-work/docker/default/.ghcjs -t -i sb/swarm-build:latest stack --internal-re-exec -v solver
d6e56bfea54d42bc6ef1cd8dce73d2e6d15b3a3182c3f5dd58068956a6f18751
> /usr/local/bin/docker start -a --interactive=false d6e56bfea54d42bc6ef1cd8dce73d2e6d15b3a3182c3f5dd58068956a6f18751
*** Running gosu stack stack --internal-re-exec -v solver...
Version 0.1.3.1, Git revision 908b04205e6f436d4a5f420b1c6c646ed2b804d7
2015-08-14 17:31:56.27672: [debug] Checking for project config at: /Users/rmm/sb/swarm/stack.yaml @(stack-0.1.3.1:Stack.Config src/Stack/Config.hs:459:9)
2015-08-14 17:31:56.277136: [debug] Loading project config file stack.yaml @(stack-0.1.3.1:Stack.Config src/Stack/Config.hs:482:13)
2015-08-14 17:31:56.290369: [debug] Run process: ghc --info @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:31:56.342953: [debug] Run process: ghc --numeric-version @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:31:56.367249: [debug] Run process: ghc-pkg --no-user-package-db field --simple-output Cabal id @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:31:56.422647: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:31:56.47134: [info] This command is not guaranteed to give you a perfect build plan @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:152:5)
2015-08-14 17:31:56.471541: [info] It's possible that even with the changes generated below, you will still need to do some manual tweaking @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:153:5)
2015-08-14 17:32:00.119112: [debug] Run process: ghc --numeric-version @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:32:00.19316: [info] Asking cabal to calculate a build plan, please wait @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:72:5)
2015-08-14 17:32:00.193307: [debug] Run process: cabal --config-file=/tmp/cabal-solver26/cabal.config install -v --dry-run --only-dependencies --reorder-goals --max-backjumps=-1 --package-db=clear --package-db=global /Users/rmm/sb/swarm/swarm-api-server/ /Users/rmm/sb/swarm/swarm-data-server/ /Users/rmm/sb/swarm/swarm-session-server/ /Users/rmm/sb/swarm/swarm-shared/ /Users/rmm/sb/swarm/thrift/lib/hs/ @(stack-0.1.3.1:System.Process.Read src/System/Process/Read.hs:255:3)
2015-08-14 17:32:13.627338: [info] extra-deps: @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:184:19)
2015-08-14 17:32:13.628789: [info] - ekg-0.4.0.8 @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:184:19)
2015-08-14 17:32:13.62886: [info] - ekg-json-0.1.0.0 @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:184:19)
2015-08-14 17:32:13.628944: [info] - etcd-1.0.5 @(stack-0.1.3.1:Stack.Solver src/Stack/Solver.hs:184:19)
*** gosu exited with status 0.
*** Killing all processes...
>

I noted in the Stack.Docker module that you can get stack to stop making the container interactive if stack thinks it's in jenkins or bamboo or not using a terminal, however the --no-terminal flag to stack doesn't apparently affect whether the -i flag gets put in. I think it's because this check:

     isStdoutTerminal <- asks getTerminal
     (isStdinTerminal,isStderrTerminal) <-
       liftIO ((,) <$> hIsTerminalDevice stdin
                   <*> hIsTerminalDevice stderr)

isn't consulting the global option, but that's a bit beyond my understanding of Stack's internals.

[borsboom]

I think I get what you're doing and I once tried something similar with some success. Nevertheless, I'm going to close this issue since the way you're using trying to use stack w/ boot2docker is definitely not supported for now.

The logic for determining when to pass docker run --internal is annoyingly complex, and is the result of many tweaks to try to get it to behave the best way for most cases. In this case, we usually still want to allow interaction with the process even when not on a terminal, for example in an Emacs shell, thus why --no-terminal doesn't disable it.

You should be able to force it off by using something like stack --docker-run-args="--interactive=false".

[Dridus]

the --docker-run-args="--interactive=false" workaround is quite helpful, thanks.