Track resource renaming through configuration

[BogdanSorlea]

Hey,

Although this idea might get refuted, I think it could ease the pain of refactoring quite a lot. Example...

Let's say I have a terraform-based setup with two instances (does not really matter how many) and some SGs created, all located in the same folder. Thus, let's say that my SG, whose "name" (terraform resource name) is integration, SG which is attached to both instances. Therefore, when specifying the SG ID to associate to the instances, we reference it by ${aws_security_group.integration.id}.

Now let's say we move the SG declaration to a separate module, security-groups, which is located outside the working directory and is referenced accordingly. Inside this module, we define the output

output "integration_sg_id" {
  value = "${aws_security_group.integration.id}"
}

Because of this change, the SG ID associated to the instances needs to change to ${module.security-groups.integration_sg_id}. Now, assuming that before the refactoring we have actually spawned these resources, just applying the new manifest will result in errors, requiring to destroy and re-create the stack.

For "large-scale" deployments, destroying and recreating is not always an option, so I am suggesting to somehow allow the defining of some resource equivalence maps, i.e. a file where you could say something like

resource "equivalence" "integration-refactoring-20160926" {
  this = "aws.security-groups.integration"
  that = "module.security-groups.integration_sg_id"
}

Of course, there would be something related to scope that would have to be figured out - and, of course, it is very much a bit of a hairy problem, but I can assure you that at least for our setup it is very crucial.

If applied, this mechanism should probably be envisioned as a database incremental migration routine: the state file should keep track of all the equivalence definitions that have already been applied, add a new equivalence resource to the state file after it has been successfully applied and update the structure/values of/in the state file to reflect the post-apply state.

Please let me know if there's anything that is not very clear.

[apparentlymart]

Hi @BogdanSorlea! Thanks for this feature request.

Do you think you could achieve some of what you're asking here using the terraform state mv command? This doesn't allow the resource to appear under two different names, but it is intended to allow you to easily rename a resource or move it into a module without recreating it.

The workflow in this case would be:

• Move the security group config into the security-groups module and define `output "integration_sg_id" there.
• Add the module "security-groups" definition to the existing project and change interpolations to "${module.security-groups.integration_sg_id}".
• Run terraform state mv aws_security_group.integration module.security-groups.aws_security_group.integration to move the state about the existing resource into the module.
• Run terraform plan and see that no changes are required, because the config and the state are already matching.

The main difference between this and what you are describing, I think, is that using terraform state mv requires that you complete the refactoring all in a single operation, whereas your approach could potentially allow moving the resource but deferring the update of all of the referencing interpolations until some later time.

[BogdanSorlea]

Hey, thanks, stupid of me, I wasn't aware of the state move command, I will try it out and let you know.

[afeld]

state mv works well, but only if you're running apply manually; there's not a good way to work it into deployment pipelines. Does feel like there could be some way to help with migrations of resources between modules, and being able to specify an equivalence/alias would be a way to do that.

[jamesarosen]

Run terraform state mv aws_security_group.integration module.security-groups.aws_security_group.integration to move the state about the existing resource into the module.
Run terraform plan and see that no changes are required, because the config and the state are already matching.

The problem with this flow is that I would have to run the mv before opening a pull request on my infrastructure repo. That means changing global state (the terraform remote's state) before code review.

For a flow that involves review by a second engineer, I think it would be very useful to have something like

terraform plan -with=name.before:name.after

[mcintyre321]

I think this is a really important feature. Refactoring the terraform resource names is a pretty common things, and it really needs to work with CI and source control.

Other approaches might be:

resource "something" "newname" {
    previous_names: ["oldname"]
}

Terraform should then upgrade any oldnames to the newname.

Another option could be for terraform to attempt to detect renames, by having the developer mark an output variable as a 'resource identifier'. e.g. suppose you have a resource that creates an s3 bucket with a specific name and path. During plan time, terraform could look for any deleted resources with that arn, and if they match that of one being created, it can treat that as a renamed resource

[jlsjonas]

Just found this ticket after a 2nd search through open issues, it looks like this one even predates #18347 where it was said that state mv shouldn't be used, which contradicts what was suggested as a workaround here.

Can we get consensus between the contributors or in some other way make a battle plan so that the community can contribute to a renaming feature?

[afeld]

Thought I had posted it here before, but I started tfmv a couple years ago to try and address this problem. Haven't gotten back to it, but the problem just bit me again. Would love some help on it if there's interest. [email protected]

[philomory]

I really, really wish Terraform had the second feature @mcintyre321 suggests, where specific attributes of a resource could be flagged as a resource identifier, similar in some ways to a Puppet "namevar", but potentially specified on a per-resource basis by the user. This would not only help with refactoring, it would also potentially allow for automatic importing of existing resources without resorting to terraform import.

[jlsjonas]

it would also potentially allow for automatic importing of existing resources without resorting to terraform import.

While I'd really like the feature, and allowing to define an identifier would certainly make this more manageable for moving.
For importing however, I'd argue that this would have to be an explicit action (an extra "these resources were already identified , do you want to import them?" question before/after confirming applying, CLI flag when non-interactive) to avoid importing a naming collision, with the resource ending up in 2 TF states

[philomory]

@jlsjonas For sure, I do agree there should be a confirmation before auto-importing; I only mean "auto" in the sense that you don't have to type potentially hundreds of terraform import statements manually.

[apparentlymart]

Hi all,

Making it so that all side-effects are made using the standard terraform plan and terraform apply workflow, rather than various secondary commands, is a long-term design goal for Terraform, but unfortunately it's also something that we've identified as being too big a piece of work to happen prior to stabilizing Terraform as 1.0, which is the goal all of our current work is heading towards.

With that said, I can acknowledge that this is something we intend to do eventually but I also want to be clear with you all that it is not going to happen in the near future. I can't predict far into the future with any certainty, but I strongly suspect that it will be at least a year before we'd be able to begin design and implementation work for this huge shift in Terraform's current model of plans, because work to stabilize Terraform's current featureset is taking priority. We don't intend to accept community contributions for the feature because the changes required would be too disruptive to Terraform's internals to happen concurrently with the stabilization work.

I understand that this is annoying, and I want to have this new workflow at least as much as you all do, but I also want to be honest about the current situation in case it inspires any of you to seek third-party external solutions in the meantime.

[mwarkentin]

I've come across https://github.com/minamijoyo/tfmigrate

Haven't used it personally but maybe helpful for some people in this thread.

[schollii]

IIUC, this means that instead of manually renaming and moving resource definitions, we would write a file that represents desired state. The state being, in this case, of the root module. Let's look at a simple use case: I have a root module that is getting large and I want to refactor some of its code into a separate sub-module.

To do this manually I use terraform state list and note all the resources that will get moved; I create a subfolder in root, move the resources there, and create some outputs that get consumed by the root module (from resources and outputs); then, I use terraform state mv to "rename" the resources (in this case, prepending the module name like module.sub_module.), and finally I can do terraform plan to confirm that there are no changes necessary to the provisioned resources

To do this declaratively in a file I would create a file with a "desired state" that lists the resources in a new file in a new subfolder, and the new outputs. A command like terraform module-config plan would compare the current module state to the described one, and show where the different resource descriptions will be moved to, and how they will be renamed. Then apply would do it.

Example module-config state file that represents current state:

module_config "root" {
    file "main.tf" {
      data aws_s3_bucket.name0 id0
      resource aws_iam_role.name1 id1
      resource aws_s3_bucket.name2 id2
    }
    
    file "variables.tf" {
      variable name3 id3
      variable name4 id4
    }
    
    file "outputs.tf" {
      output name5 id5
      output name6 id6
    }
}

where the ID is whatever is required to refer to the actual object in terraform state, such as ARN for resources or some unique ID that terraform generates for variables, outputs, data, modules etc. This is to support renaming.

The above config state says that the current module consists of 3 files with 2 resources, a data, 2 variables, and 2 outputs, no submodules. There are likely other entities that should appear there, like tfvars file, providers/versions file, backend, etc.

Then if you modified that file by moving variable/output/data/resources between files, and into submodules, terraform module-config plan/apply would know what to do. Eg if the above file gets modified to this:

module_config "root" {
    file "main.tf" {
      data aws_s3_bucket.new_name0 id0
      resource aws_iam_role.name1 id1
      module new_module
    }
    
    file "variables.tf" {
      variable new_name3 id3
    }
    
    file "outputs.tf" {
      output new_name5 id5
    }
}

module_config "new_module" {
    dir "folder"

    file "main.tf" {
      data aws_s3_bucket.new_name0
      resource aws_s3_bucket.name2 id2
    }
    
    file "variables.tf" {
      variable name4
    }
    
    file "outputs.tf" {
      output name6
    }
}

Terraform knows exactly how to refactor everything: what folder to create, what files to put in there, what objects to move, and can rename all references to variables and resources.

Interestingly, this could also be used to import resources, or to capture the configuration of resources:

file "main.tf" {
  import resource aws_iam_role.name1 id1
  import resource aws_s3_bucket.name2 id2
}

which would tell terraform to ensure that main.tf has the said resources and that the state file manages them.

In any case such module-config declaration of state should only be necessary when modifying existing state, so the "current" state should be exportable, eg terraform module-config export would create the state file, then you could edit it per above.

It would also be straightforward to have chains of migrations: eg if you have migration1.tf, migration2.tf, migration3.tf. It is sufficient for 3 to mark 2 as predecessor, and 2 to mark 1 as predecessor, and 1 to have nil predecessor (mean the initial module config is its predecessor), and for the state file to track what is the latest migration that successfully completed. Then it can tell whether a module-config apply is an upgrade, downgrade, no-op, or not-allowed (eg after 2 applied, you can only apply 1 or 3, 2 being a no-op, and 4 or nil are not-allowed).

Not sure yet how to handle partial failure of a module-config apply. This will have moved some resource definitions and renamed some resources in state file, but others will be pending. And the problem might require mods to the migration file. Therefore perhaps module-config apply command can create a temp file similar to the lock file, and keep track of every mod that has succeeded, and deletes it if all succeeded. If partway failure occurs, you can do module-config apply on the previous migration and the apply command can see that it is a revert on an incomplete migration.

[aristosvo]

Would it not be great if we could use a workflow similar to terraform state mv, something like:

resource "example" "test" {
  name      = "Test149"
  feature_x = "enabled"

  lifecycle {
    state_mv {
      previous_state = example.previous_test
    }
  }
}

A similar approach would be amazing to have for terraform import:

resource "example" "test" {
  name      = "Test149"
  feature_x = "enabled"

  lifecycle {
    import {
      id = "Cloudx/Test149"
    }
  }
}

This would save us a lot of scripts and/or manual work if naming is refactored..

[mwarkentin]

https://discuss.hashicorp.com/t/request-for-feedback-config-driven-refactoring/30730

[apparentlymart]

Thanks for pointing out this issue too, @mwarkentin! 😀

I didn't notice until now that we had this earlier request that #19354 later duplicated. I just replied over there and closed that one and what I said over there applies here too, in particular that if you have feedback on the new feature please create new issues for each bug report or feedback item rather than adding more comments here, because we'll be using issues to track work during the v1.1 beta period.

Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.