--only-dependencies instead of --only-snapshot

[gregwebs]

As a stack user, I don't know what the terminology "snapshot" means. Also, I am not sure if it means only the resolver or if extra-deps are included. The behavior that I would like is for extra-deps to be included (well at least the ones that are from hackage tip, maybe not a local directory) so it is more similar to cabal install --only-dependencies.

[gregwebs]

--only-snapshot ignores extra-deps that are just packages from hackage instead of the resolver. For my continuous integration build I need a --only-dependencies flag that install all dependencies. So I am changing the ticket name.

[snoyberg]

See the discussion in #310 to see how we got to where we are now.

[gregwebs]

#310 seems to say that --only-dep is used for ci builds. This ticket is to take the concept to its logical conclusion of installing extra deps

[snoyberg]

Sorry, I just realized that some of that discussion happened outside of the issue. The basic idea is: there are weird corner cases that come into play if we try to install only dependencies, specifically because of cycles between dependencies and local packages. Consider working on the wai package: some of the dependencies (like wai-logger) depend on non-dependencies (wai). Do we install wai and wai-logger, or not? The obvious cut-off was to just install whatever would go into the snapshot database and drop anything local/custom.

That's not to say that there isn't value in doing something better, I'm just explaining how we got to the current place. If someone wants to put in the time to figure out a better solution, that would be good. But it doesn't seem like this is high-value enough to be worth it (unless you have use cases I'm not anticipating).

[gregwebs]

For wai-logger, sure you cannot install that one ahead of time. But that seems like a rare corner case (that should be handled properly). The usual case is just wanting packages from the head of hackage instead of the LTS. And for some projects, that could mean a lot or packages.

For us it is a big deal because even though we only have a few packages, it slows down every CI build. We need a step to cache as much as possible, and obviously we don't want to cache the actual build.

However, I think I can work around this for now by just manually installing the extra-deps.

[gregwebs]

Here is my current workaround to install everything in extra-deps:

# install extra-deps from stack.yaml
# This is hacky, there is probably a better way
# look for the leading '-' in the yaml file and the ending '-' with a version number
stack install $(cat stack.yaml | grep -e '^\s*-.*-[0-9]' | awk '{print $2}' | xargs)

[zimbatm]

In my scenario the project source files aren't available during the build step. Basically I want to build all the dependencies except the top-level ones.

stack build --only-snapshot
# Install extra-deps
grep -e '^\s*-.*-[0-9]' stack.yaml | awk '{print $2}' | xargs stack build
# Install github dependencies
grep 'git: [email protected]:' stack.yaml | sed -e 's/.*:.*\/\(.*\).git.*/\1/g' | xargs stack build

Instead of --only-snapshot having a --excluded <list-of-packages> would also work for me.

[gregwebs]

Another related issue for me is that the extra dependencies are installed into the .stack-work directory: this makes them difficult to cache between CI builds. Perhaps more "nix-like" package management can solve this issue, although I could also solve it with a custom snapshot.

[gregwebs]

We are solving this issue by creating a custom snapshot. That is a good enough solution for me. Perhaps this workflow should just be documented when the next release is made.

[snoyberg]

See plan laid out at: #651 (comment)

[snoyberg]

Implemented for #651, please review PR #753.

[gregwebs]

This is working great for us! We are also going to create a custom snapshot now and watch our build time plummet :)