Using environment variables in stack.yaml

[wdanilo]

Hi! Would you be so nice and dd ability to use environment variables within stack.yam configuration files? This would be a life saver in a situation when a big team develops a set of libraries and every person keeps these libraries in different places, so the environment variable would point where are the sources that should override packages from Hackage.

Another small topic is to add extra dependencies recursively - all packages found in folder X. In such use case we can easily override packages for development time without changing stack yams configuration and this (n combination with the above idea) is a very useful utility, portable across developer environments.

Thank you! :)

[3noch]

I'm sure you've thought of this, but you could have a script generate the stack.yaml file. Your team would need to run this script on a fresh clone/checkout or when something significant changed.

[borsboom]

Hi! Would you be so nice and dd ability to use environment variables within stack.yam configuration files? This would be a life saver in a situation when a big team develops a set of libraries and every person keeps these libraries in different places, so the environment variable would point where are the sources that should override packages from Hackage.

Yes, I can see this being a good reason to support environment variables. Aside: this is also an excellent use case for Stack's Docker integration (or other ways to achieve similar objectives, like Vagrant).

Another small topic is to add extra dependencies recursively - all packages found in folder X. In such use case we can easily override packages for development time without changing stack yams configuration and this (n combination with the above idea) is a very useful utility, portable across developer environments.

This seems like it's a recipe for losing reproducibility, unless done with great care. And it's the temptation to do this sort of thing that causes me to have misgivings about letting anything external (like environment variables) effect how stack.yaml is interpreted.

Legit uses for this outweigh the misgivings, though, but it should be restricted to simple string substitution (avoid the temptation to essentially make a templating solution for stack.yaml). Perhaps something like the environment substitution supported by Yesod's settings.yml.

[3noch]

Perhaps instead you could provide an environment variable that tells stack to run a script to get its stack.yaml file. For example, export STACK_YAML_SCRIPT=$PWD/generate-stack.yaml.sh arg1. This would allow stack to virtually punt on the problem but let users who really need the functionality to get it without needing to wrap stack in some other script to get the same behavior.

[mgsloan]

Sharing info between stack.yaml files just came up in #haskell-stack (pinging @bitemyapp @MichaelXavier). This is another example of nice-to-have features for the configuration in general. I'm a bit concerned that going down this road could be a pandora's box of complexity for both stack code and for users' mental model.

I like @3noch's idea, though it would be nice to support arbitrary stack.yaml generation in a way that users wouldn't need to know about it. This could look like having a generate field, like so:

generate: yaml-generator arg1 arg2

resolver: lts-3.13
packages:
  - yaml-generator

The generate field would specify "Run this program to generate the stack.yaml on stdout". The rest of the configuration would be used to install that generator program, if necessary.

[borsboom]

@3noch/@mgsloan: I hope I'm not being too obstinate, but I really think if you need this level of sophistication you should wrap stack in your own build system. Stack does not aim to replace make/shake/, and this feels like its encroaching on that territory. Some support for simple cases (like basic variable substitution) is reasonable, though.

[borsboom]

A PR adding simple environment variable substitution would be welcome (but if you want to do more than that, we should discuss further first).

[3noch]

@borsboom I'm with you. I'd actually prefer not to have any special handling for this case at all, since it greatly complicates things. EDIT: I only suggested that idea in the hopes of moving sophistication out of stack. Basic environment variable substitution seems like a nice compromise.

[bitemyapp]

Tend to agree with @borsboom - most of my more developed projects wrap Stack with Makefiles and I'm pretty grateful (so far) that stack.yaml has stayed dumb even if template inheritance would save me a little boilerplate. It wouldn't save me much because usually my stack.yaml variants are for different versions of GHC or compilers entirely and I can't really share library versions whole-sale between them anyway.

[mgsloan]

Yeah, I agree with that! Making this the domain of your own build system / scripts is quite reasonable. I like that the stack.yaml will necessarily be in the filesystem (whereas with my generate field above, it isn't clear that would happen).

It might be good to figure out some recommended approaches for this templating / inheritance, though. This way, we're more likely to end up with a bunch of projects which do something similar, so it's easier to start developing on them. This can wait a while, let's see what folks come up with.

[MichaelXavier]

I'll just say +1 on keeping things simple. I hit this very case today where for ease of development it made sense that my default stack.yaml would not use docker, but to produce consistent bins for testing and deployment, I wanted the docker functionality. It was 3 or 4 lines in my shake build to add a rule for building stack_production.yaml. I still prefer using shake to do this sort of thing over futzing with m4 or anything like this. It may be worth noting in the documentaton that these are some ways to solve it so at least people know it has been considered.

[acfoltzer]

The initial environment variable substitution proposal would be extremely useful for SMACCMPilot. It looks like the Ruby world uses embedded ruby syntax to splice in environment variables, e.g.:

foo: <%= ENV['FOO'] %>`

It would be ... overkill to import hint and evaluate Haskell inside of splices, but I'm left wondering where the appropriate contexts would be to evaluate simpler env-var-only splices like

location: $SOME_REPO/some-package

It seems like we could

1. perform the substitution before parsing the stack.yaml file
2. perform the substitution during parsing, requiring an upstream change to yaml but saving an extra pass over the file
3. perform the substitution only in particular contexts once the file has been parsed

I'm not very familiar with the code either for Stack or yaml, but my intuition is that 1. would be the most straightforward option. 2. is probably too application-specific for a general-purpose package like yaml, and 3. would require careful attention to make sure all the relevant fields of the config are substituted before they're used.

[sjakobi]

I'm bumping the priority here because more issues have cropped up that would profit from a solution of this.

[CMCDragonkai]

This would be useful for the global stack configuration.

[DanBurton]

(My attention was drawn here from Haskell Weekly.)

If what is desired is something like this:

location: $SOME_REPO/some-package

What about using something like an "extra-deps" folder that is gitignored and contains symlinks?

location: extra-deps/some-package

Where extra-deps/some-package is a symlink on each dev's machine to the desired source. Or extra-deps itself could be a symlink to each dev's preferred parent location for these sources. I'm not sure I see a compelling case for adding env var interpolation to stack.yaml.

[seagreen]

Depending on non-versionable settings like environment variables would make Stack less reproducible, which is my favorite thing about it. For something we all have to use like stack files the simpler, the better.

I'd vote to close this, which would also bring down the number of open issues to a nice round 695:grin:

[mgsloan]

Sure, closing for now! I am starting to like the idea of allowing flags and interpolation by default in stack yaml, I have seen some good uses for it, particularly conditionals on OS. But I definitely like keeping it as simple as possible. Perhaps the future solution will be to have something like stack.yaml.mustache (just an example). This way, you'd be able to see what is going on in the generated stack.yaml, while still having a degree of programatic templating