Support automatic migration depending on module ref vs module state ref

[nitrocode]

Great tool by the way.

Please let me know if this is feasible.

Context

Previously applied module code which creates module.s3.aws_s3_bucket.that resource.

module "s3" {
  source = "some-module-source?ref=tags/0.1.0"

Now the source has been bumped to 0.2.0 which changes the name that to this

The new version contains a directory migrate/0.1.0_to_0.2.0.hcl file which can be downloaded and run via tfmigrate

Proposal

User updates the source to a new version of a module

terraform init -upgrade
terraform plan

User notices creations and destroys of similar resources

Running this or similar

tfmigrate plan --auto-detect
tfmigrate plan --target module.s3
tfmigrate plan
tfmigrate apply

1. Detect previously applied module versions
2. Detect if new module version will be applied
3. Check for a standardized module migration directory migrate/ which contains the previous version and new version like x.y.z_to_a.b.c.hcl
   • could also use multiple directories to autodetect like tfmigrate, tfupgrade, upgrade, etc
4. if plan, do not prompt user
5. if apply, prompt user to migrate

[minamijoyo]

Interesting. It's a good idea that a module author provides how to migrate an old state.

I think there are some problems:

• The module author doesn't know actual resource address and can provide only a migration template file which cannot directly be applied by the module user.
• The tfmigrate was designed to be suitable for GitOps, so fully automatic will be unclear what happens and cannot confirm by the module user and other reviewers. I prefer more declarative way that the module user to generate a migration file from the template.
• Previously applied version is not recorded in the state. It can only be detected from source code differences. It's not impossible, but it will be hard to implement.
• Ideally, the migration file should be generated at the same time as editing the terraform configuration file, which knows the full context, but there are many issues to be solved in implementation.
• If previously applied version is an outdated patch version and the module has multiple migration templates, which file to be applied can be detected by the semantic versioning. It's also not impossible, but it needs many rules to work correctly.
• If we support the automatic migration, then the module user probably expects to resolve all diffs with no changes, but it's a wrong expectation. Not all breaking changes for the module can be migrated with state operations., for example, adding a new resource or attribute. This leads a bad UX. The module author should clarify when and how to migrate state in the CHANGELOG or README.

By the way, I guess we may be able to generate a migration file only from a tfplan file with some heuristics in a simple case such as renaming a resource in module. If we go forward this direction, we can use it for wider range of cases, not only module upgrade, but also fixing state drifts caused by changing resource outside of Terraform. Of course, it's not perfect, it's best-effort feature, but at least the module user probably doesn't expect everything will be fixed magically.

[barryib]

I think it'll be better to let module developers to generate migration file from template. That was my first thought for terraform-aws-eks. But after I started, I noticed that I had a lot of code duplication.

Because template generation can be a pain, I was wondering if we can add annotation like in ORM Schema Migration directly in code. Something like this (thinking loud):

#tfmigrate:mv:aws_autoscaling_group.foo:your-comment
resource "aws_autoscaling_group" "workers_launch_template" {
  count = var.create_eks ? local.worker_group_launch_template_count : 0
  name_prefix = join(
    "-",
    compact(
      [
        coalescelist(aws_eks_cluster.this[*].name, [""])[0],
        lookup(var.worker_groups_launch_template[count.index], "name", count.index)
      ]
    )
  )
}

Where aws_autoscaling_group.foo was the old resource name.

With that, we can:

• tfmigrate generate to generate migration file from config and state
• tfmigrate plan
• tfmigrate apply

In all cases, this is a really wanted feature for all module developers.

[minamijoyo]

@barryib Thank you for your idea!

Aside from the difficulty of implementing parsing comments, I think it would probably work for a simple case, but if a resource in a parent module will be moved into a submodule and a root module has two instances of the parent module, how can we know their old address to generate a migration file? The tfmigrate doesn't have a context how to map two old addresses to new ones.

[nitrocode]

if a resource in a parent module will be moved into a submodule and a root module has two instances of the parent module, how can we know their old address to generate a migration file? The tfmigrate doesn't have a context how to map two old addresses to new ones.

Perhaps the tfmigrate generate can be semi-automated

$ tfmigrate generate <module-name> <version-old> <version-new>

initializing modules...
analyzing resources of <module-name> <version-old>...
analyzing resources of <module-name> <version-new>...
analyzing differences....

Found 5 differences

#1: module.<name>.security_group exists in <version-old> but not in <version-new>
#1: module.<name>.module.security_group is new
#1: module.<name>.module.security_group.security_group is new
#1: is deleted module.<name>.security_group the same as module.security_group.security_group ? [Ny]  y

#2: #...

...

Generated `tfmigrate_<module-name>_<version-old>_<version-new>.hcl!

[minamijoyo]

FYI: I noticed that the Terraform core team seems to be working on a native support for "moved" block. Looks promising.
hashicorp/terraform#29126

[apparentlymart]

Hello! Thanks for linking over to that docs PR I was working on, because I hadn't previously noticed this discussion over here!

From what I understood of the discussion above it does seem like we're aiming at a similar set of use-cases as what this issue is discussing. Our development work on this is still pretty early and we don't have a functioning implementation in the main branch yet, but if anyone here would be interested we'd love to have some folks try out the functionality in an alpha build and let us know how it works out for some real-world modules. We're certainly not trying to solve every possible problem in the first release, because of course then we'd likely never actually ship it, but our initial area of focus is on this idea of renaming objects (resources/modules) or moving them into and out of modules and so I expect you all will have some interesting real examples to try it out against!

Once we have things ready for feedback in at least one alpha release someone on our team will post a note over in our community forum about it, though I'll do my best to remember to come back and note it here too.

[minamijoyo]

Hi @nitrocode, @barryib, and all who are interested in this feature,

An alpha build of Terraform v1.1 is now available which includes the "moved" block feature:
https://discuss.hashicorp.com/t/request-for-feedback-config-driven-refactoring/30730

Try it and send your feedback to Terraform core team. Thanks!

[apparentlymart]

Ahh, thanks @minamijoyo! I guess after all I did forget to come here and post a note about that community forum post, so thanks for doing it for me. 🙇

We'd love to hear any feedback you have over in that forum topic.

[minamijoyo]

@nitrocode Can we close this issue by a new moved block introduced in Terraform v1.1?
https://www.hashicorp.com/blog/terraform-1-1-improves-refactoring-and-the-cloud-cli-experience