RDD: Introduce import blocks to support multiple includes

[yorinasub17]

This is a README Driven Development PR, and only contains the docs. Note that some of the design decisions are based on an in-progress implementation (see section down below)

This PR is the README updates for the upcoming changes to support including multiple terragrunt configurations in a child. Note that this is different from nested include, which is more complicated to implement due to the implications to the relevant include functions like get_parent_terragrunt_dir and path_relative_to_include.

IMPORTANT NOTE

If we wish to maintain backward compatibility with the existing include functionality, this implementation requires defining a new import block to replace include. The main reason for this is due to a design choice in the HCL parser where it does not support optional labels on blocks. That is, you can not parse the following two blocks into the same go struct:

include {}
include "root" {}

Once you require a label on the internal struct, the corresponding hcl block MUST have one label - otherwise, the hcl parser errors out saying as such. I found this out in the WIP implementation being done in the branch https://github.com/gruntwork-io/terragrunt/tree/yori-multiple-include-blocks.

[yorinasub17]

NOTE: These are purposely spaced out by 5 so that it is easier to inject additional pages inbetween later.

[brikis98]

Overall, this looks pretty good.

For completeness, I'm going to think out loud and ask a few questions, in part for clarity, in part because it might be worth updating the docs to capture the answers:

1. Should we remove include from most of the docs and recommend only import? Having multiple ways to do something is confusing, so I'd much rather have one, clear, supported, path. We could leave include in the reference docs and mark it as deprecated, so if you stumble across it in your code, you know what it is, but otherwise, I'd be tempted to remove it from all the quick start guides and the rest.

2. I think I asked this way back when we did the imports RFC, but am struggling to find the answer, so apologies if this is a repeat. Typically, when I think of import, I think of pulling in a namespace, and then explicitly pulling the things I want out of it. Here, this is more like a multi-include, where calling import implicitly changes lots of my configs. Is that intentional? Is import the right name then?

3. As a sanity check, is it worth trying to come up with a skeleton Ref Arch example that uses import to show what the result looks like? I did that for my earlier DRY Ref Arch and you were able to use that to find some important issues.

4. NIT: the read_terragrunt_config centralizes the versions of the module used in different envs to a single file. Does that break the way we do CI / CD today?

[yorinasub17]

Typically, when I think of import, I think of pulling in a namespace, and then explicitly pulling the things I want out of it. Here, this is more like a multi-include, where calling import implicitly changes lots of my configs. Is that intentional? Is import the right name then?

That's a good point and I don't think you asked this on the RFC. I agree with you here and struggled with the naming. Ideally we can use include as is, but the hcl parser really limits us here...

That said, after sleeping on it, I am now thinking if there might be a way to implement the optional label by taking a two-pass parsing approach with different structs. Let me see if that works and report back. If that works, then we solve 1 & 2 by being able to use include, and also being able to keep include {} as a shorthand for include "" {}, which feels less like a "separate way".

[yorinasub17]

As a sanity check, is it worth trying to come up with a skeleton Ref Arch example that uses import to show what the result looks like? I did that for my earlier DRY Ref Arch and you were able to use that to find some important issues.

Definitely. My plan was to try out implementing the DRY Ref Arch with the prototype implementation once the direction is finalized from the RDD. I'm actually mostly done with the implementation (just need a few tests to verify functionality), so it's not too far off.

[yorinasub17]

UPDATE: switched back to include instead of import. The implementation turned out to be not too bad once I realized I can use the inline editing functionality of hclwrite, like aws-provider-patch (see https://github.com/gruntwork-io/terragrunt/blob/yori-multiple-include-blocks/config/include.go#L549-L703)

[yorinasub17]

UPDATE: Implemented this version here - #1804

[brikis98]

As a sanity check, is it worth trying to come up with a skeleton Ref Arch example that uses import to show what the result looks like? I did that for my earlier DRY Ref Arch and you were able to use that to find some important issues.

Definitely. My plan was to try out implementing the DRY Ref Arch with the prototype implementation once the direction is finalized from the RDD. I'm actually mostly done with the implementation (just need a few tests to verify functionality), so it's not too far off.

Really looking forward to seeing this. From a read of the docs and RFC, the approach here looks good, but seeing it used in a real Ref Arch, even a totally dummy one for testing, makes it way more effective to think about the pros and cons of this design.

UPDATE: Implemented this version here - #1804

Hah, nice, I'll take a look!

[yorinasub17]

NOTE: I will be closing this as the README has now been ported over to the implementation PR. We can continue any discussions there.

[yorinasub17]

Ok got the example DRY setups done:

• Toy example: https://github.com/gruntwork-io/terragrunt-infrastructure-live-example/tree/yori-dry-experiment
• A more comprehensive one: https://github.com/gruntwork-io/terragrunt-infrastructure-live-example/tree/yori-dry-experiment-comprehensive

There isn't much difference between the two approaches (the one that you set up using include + read_terragrunt_config, and the one using multiple includes) at a glance, but the new version using multiple includes has a few tiny advantages:

• Can leverage exposed includes to get rid of extraneous locals references.
• Can compose multiple configurations to DRY common frontend and backend. (NOTE: I didn't implement this yet, but was planning to)
• Don't need to repeat the remote_state and generate blocks using read_terragrunt_config. Because of that, it feels slightly more intuitive.

[brikis98]

Ok got the example DRY setups done:

• Toy example: https://github.com/gruntwork-io/terragrunt-infrastructure-live-example/tree/yori-dry-experiment
• A more comprehensive one: https://github.com/gruntwork-io/terragrunt-infrastructure-live-example/tree/yori-dry-experiment-comprehensive

There isn't much difference between the two approaches (the one that you set up using include + read_terragrunt_config, and the one using multiple includes) at a glance, but the new version using multiple includes has a few tiny advantages:

• Can leverage exposed includes to get rid of extraneous locals references.
• Can compose multiple configurations to DRY common frontend and backend. (NOTE: I didn't implement this yet, but was planning to)
• Don't need to repeat the remote_state and generate blocks using read_terragrunt_config. Because of that, it feels slightly more intuitive.

That looks really nice!

If we update Ref Arch 2.0 to use this, it would be a huge improvement. However, is it worth trying one more experiment first, where the include can read a versioned source URL, so _env lives in infra-modules? I have no sense of how much work that is, both in terms of Terragrunt and Ref Arch 2.0 improvements, so totally up to you. We should probably update the Terragrunt example repos to use whichever approach we pick too.