Multiple include blocks

[yorinasub17]

This is the implementation of the feature described in #1803

This is backward compatible for the most part, except for the expose feature, where you must now index into the label - any prior references to exposed include must be updated to include[""]. E.g., if the user had:

include {
  path = find_in_parent_folders()
  expose = true
}

inputs = {
  region = include.locals.region
}

the include.locals.region reference must be updated to include[""].locals.region.

include {
  path = find_in_parent_folders()
  expose = true
}

inputs = {
  region = include[""].locals.region
}

[brikis98]

Nice work Yori 👍

[brikis98]

I mentioned this in the other PR, but we probably want to encourage putting things that you typically change on an env-by-env basis in the env itself so it works better with CI / CD; and putting things that you would change globally (albeit rarely) into an _env folder. So for example, env name, instance type, number of instances, ref (version) are all things you may want to change in one env, but not another, and you may want CI / CD to be able to detect it based on the file you changed, so it makes more sense to define those params directly in stage/app/terragrunt.hcl for stage and prod/app/terragrunt.hcl for prod. For everything else, you can put it in _env, under the assumption that it changes very rarely, and you can somehow manually trigger CI / CD as necessary for that change.

That does make me think of an alternative approach, which might be nonsense, but I figured it's worth thinking through: would it make sense to version the terragrunt.hcl files in _env? Ignoring implementation details, imagine for a second that all the _env stuff is defined in infrastructure-modules; inside the terragrunt.hcl files in _env, it uses relative paths within infrastructure-modules; and the whole repo is versioned as usual. Then, in infrastructure-live, you include not from a path but from a versioned source parameter. That allows you to use different versions of _env in each actual env, and promote changes from one env to another. In that case, you could define environment-specific values in _env as well (e.g., instance type, instance count, etc), as you'd bump the version number in stage or prod to pick up the new configs, and CI / CD would pick that up.

So, in effect, infra-live would become VERY minimal: just a bunch of terragrunt.hcl marker files that use include to pull in the actual configs from versioned URLs.

[yorinasub17]

That does make me think of an alternative approach, which might be nonsense, but I figured it's worth thinking through: would it make sense to version the terragrunt.hcl files in _env? Ignoring implementation details, imagine for a second that all the _env stuff is defined in infrastructure-modules; inside the terragrunt.hcl files in _env, it uses relative paths within infrastructure-modules; and the whole repo is versioned as usual. Then, in infrastructure-live, you include not from a path but from a versioned source parameter. That allows you to use different versions of _env in each actual env, and promote changes from one env to another. In that case, you could define environment-specific values in _env as well (e.g., instance type, instance count, etc), as you'd bump the version number in stage or prod to pick up the new configs, and CI / CD would pick that up.

Yes I had this exact thought! This actually has another advantage of better handling variable updates across module versions. E.g., in the current setup, there is a limitation where the shared inputs are in a static file shared across dev, stage, and prod. What if a version bump removes a variable, or there is a new required variable, or an old variable type changed? Ideally, you would make the change in the shared variables in _env, but then that would break stage and prod.

[yorinasub17]

I mentioned this in the other PR, but we probably want to encourage putting things that you typically change on an env-by-env basis in the env itself so it works better with CI / CD; and putting things that you would change globally (albeit rarely) into an _env folder. So for example, env name, instance type, number of instances, ref (version) are all things you may want to change in one env, but not another, and you may want CI / CD to be able to detect it based on the file you changed, so it makes more sense to define those params directly in stage/app/terragrunt.hcl for stage and prod/app/terragrunt.hcl for prod. For everything else, you can put it in _env, under the assumption that it changes very rarely, and you can somehow manually trigger CI / CD as necessary for that change.

Good point. I updated the docs to use expose instead of read_terragrunt_config for the module version, and updated read_terragrunt_config docs to be something more generic: c034ca9

[brikis98]

That does make me think of an alternative approach, which might be nonsense, but I figured it's worth thinking through: would it make sense to version the terragrunt.hcl files in _env? Ignoring implementation details, imagine for a second that all the _env stuff is defined in infrastructure-modules; inside the terragrunt.hcl files in _env, it uses relative paths within infrastructure-modules; and the whole repo is versioned as usual. Then, in infrastructure-live, you include not from a path but from a versioned source parameter. That allows you to use different versions of _env in each actual env, and promote changes from one env to another. In that case, you could define environment-specific values in _env as well (e.g., instance type, instance count, etc), as you'd bump the version number in stage or prod to pick up the new configs, and CI / CD would pick that up.

Yes I had this exact thought! This actually has another advantage of better handling variable updates across module versions. E.g., in the current setup, there is a limitation where the shared inputs are in a static file shared across dev, stage, and prod. What if a version bump removes a variable, or there is a new required variable, or an old variable type changed? Ideally, you would make the change in the shared variables in _env, but then that would break stage and prod.

We should def experiment with this as the way to make the Ref Arch super DRY, versioned, etc.

[brikis98]

This is great work Yori 👍

[brikis98]

Neat 👍

[brikis98]

Ah, nice, this is a better example 👍

[brikis98]

Passing in the name is a great solution 👍

[yorinasub17]

Thanks for review! Will merge this in, cut a new release as backward compatible (given how much code change there is to the include machinery), and then open a follow up PR to update that function docs.