Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to move resource to new module #21346

Closed
sudoforge opened this issue May 17, 2019 · 16 comments · Fixed by #22299
Closed

Unable to move resource to new module #21346

sudoforge opened this issue May 17, 2019 · 16 comments · Fixed by #22299

Comments

@sudoforge
Copy link

sudoforge commented May 17, 2019

Terraform Version

Terraform v0.12.0-rc1
+ provider.github v2.1.0

Terraform Configuration Files

provider "github" {
  organization = "ORG_NAME_HERE"
  version      = "~> 2.1.0"
}

resource "github_team" "foo" {
  name    = "Some Team Name"
  privacy = "closed"
}

module "foo" {
  source      = "PATH_TO_MODULE"
  name        = "Some Team Name"
}

A reference module can be as simple as:

variable "name" {
  type = string
  description = "The name of the team"
}

resource "github_team" "this" {
  name = var.name
}

Debug Output

2019/05/16 18:54:44 [INFO] Terraform version: 0.12.0 rc1 
2019/05/16 18:54:44 [INFO] Go runtime version: go1.12.1
2019/05/16 18:54:44 [INFO] CLI args: []string{"/Users/username/.local/bin/tf12rc1", "state", "mv", "github_team.foo", "module.foo.github_team.this"}
2019/05/16 18:54:44 [DEBUG] Attempting to open CLI config file: /Users/username/.terraformrc
2019/05/16 18:54:44 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/05/16 18:54:44 [INFO] CLI command args: []string{"state", "mv", "github_team.foo", "module.foo.github_team.this"}
2019/05/16 18:54:44 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2019/05/16 18:54:44 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2019/05/16 18:54:44 [DEBUG] New state was assigned lineage "2f78a26c-330c-43e7-9f71-d82486f48d23"
2019/05/16 18:54:44 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2019/05/16 18:54:44 [TRACE] Meta.Backend: instantiated backend of type <nil>
2019/05/16 18:54:44 [DEBUG] checking for provider in "."
2019/05/16 18:54:44 [DEBUG] checking for provider in "/Users/username/.local/bin"
2019/05/16 18:54:44 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"
2019/05/16 18:54:44 [DEBUG] found provider "terraform-provider-github_v2.1.0_x4"
2019/05/16 18:54:44 [DEBUG] found valid plugin: "github", "2.1.0", "/private/tmp/foo/.terraform/plugins/darwin_amd64/terraform-provider-github_v2.1.0_x4"
2019/05/16 18:54:44 [DEBUG] checking for provisioner in "."
2019/05/16 18:54:44 [DEBUG] checking for provisioner in "/Users/username/.local/bin"
2019/05/16 18:54:44 [DEBUG] checking for provisioner in ".terraform/plugins/darwin_amd64"
2019/05/16 18:54:44 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2019/05/16 18:54:44 [TRACE] backend/local: CLI option -backup is overriding state backup path to -
2019/05/16 18:54:44 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at 
2019/05/16 18:54:44 [DEBUG] checking for provider in "."
2019/05/16 18:54:44 [DEBUG] checking for provider in "/Users/username/.local/bin"
2019/05/16 18:54:44 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"
2019/05/16 18:54:44 [DEBUG] found provider "terraform-provider-github_v2.1.0_x4"
2019/05/16 18:54:44 [DEBUG] found valid plugin: "github", "2.1.0", "/private/tmp/foo/.terraform/plugins/darwin_amd64/terraform-provider-github_v2.1.0_x4"
2019/05/16 18:54:44 [DEBUG] checking for provisioner in "."
2019/05/16 18:54:44 [DEBUG] checking for provisioner in "/Users/username/.local/bin"
2019/05/16 18:54:44 [DEBUG] checking for provisioner in ".terraform/plugins/darwin_amd64"
2019/05/16 18:54:44 [TRACE] backend/local: CLI option -backup is overriding state backup path to -
2019/05/16 18:54:44 [TRACE] statemgr.Filesystem: preparing to manage state snapshots at terraform.tfstate
2019/05/16 18:54:44 [TRACE] statemgr.Filesystem: existing snapshot has lineage "a022c743-4cae-c580-101f-35d46d150be5" serial 1
2019/05/16 18:54:44 [TRACE] statemgr.Filesystem: locking terraform.tfstate using fcntl flock
2019/05/16 18:54:44 [TRACE] statemgr.Filesystem: writing lock metadata to .terraform.tfstate.lock.info
2019/05/16 18:54:44 [TRACE] statemgr.Filesystem: reading latest snapshot from terraform.tfstate
2019/05/16 18:54:44 [TRACE] statemgr.Filesystem: read snapshot with lineage "a022c743-4cae-c580-101f-35d46d150be5" serial 1
2019/05/16 18:54:44 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info

2019/05/16 18:54:44 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
Error: Invalid target address

Cannot move to module.foo.github_team.this: module.foo does not exist in the
current state.

Expected Behavior

Terraform 0.11.13 (and below) would move the resource to the module, assuming you provided the full path to the resource contained within the module, e.g. module.foo.whatever_resource.resource_name.

Actual Behavior

Terraform 0.12.0-rc1 errors out, stating that module.foo doesn't exist in the current state. And that's true, it's not currently being tracked.

Steps to Reproduce

  1. terraform-12-rc1 init
  2. terraform-12-rc1 import github_team.foo SOME_TEAM_ID
  3. terraform-12-rc1 state mv github_team.foo module.foo.github_team.this

Additional Context

N/A

References

N/A (I searched and didn't find anything; feel free to edit if reference issues are found)

@sudoforge
Copy link
Author

sudoforge commented May 17, 2019

It should be noted that current documentation for terraform state mv suggests that this should be possible:

$ terraform-12-rc1 state mv github_team.foo module.foo
Error: Invalid target address

Cannot move github_team.all_developers to <invalid>..: the target must also be
a whole resource.

@hwstovall
Copy link

hwstovall commented Jun 4, 2019

We've had some success working around this with the following.

Let's say we have an existing resource identified by ABC123 that was created by Terraform as a resource called module.old.resource_type.foo.

# Import the component created by the old module into the new module.
# This create entries for the new module in the state.
terraform import module.new.resource_type.foo ABC123

# Delete the old module's state entry for that component.
terraform state rm module.old.resource_type.foo

From here on out, the new module will exist in state and terraform state mv should work as expected.

While this has worked for us, I want to make sure it is clear that this could potentially lead to state inconsistencies if not done correctly. Use this approach with caution.

@sudoforge
Copy link
Author

sudoforge commented Jun 4, 2019

To be clear, @hwstovall's comment is not a method for moving modules and resources, but a full deletion of the old and import of the new. This is the obvious way to accomplish the end goal, but requires importing the resource(s) that are being "moved", which means you need to know something about the resource -- the format of the import string, and any details such as the resource name, ID, or other such property.

This issue was created to highlight that the new version of the command line client was unable to move A to B, which does not require any knowledge about the resource, and does not require re-importing the resource.

@pselle pselle modified the milestones: v0.12.1, TBD Jun 4, 2019
@pauldraper
Copy link

pauldraper commented Jun 7, 2019

Also happens with final 0.12 versions. (I'm using 0.12.1.)

Can't figure out a good way around this; @hwstovall's (incomplete) workaround is the best I can find.

And I have to deal with "resource aws_cloudwatch_log_stream doesn't support import"

@sudoforge sudoforge changed the title terraform-v0.12.0-rc1 not able to move resource to new module Unable to move resource to new module Jun 9, 2019
@sudoforge
Copy link
Author

@pauldraper You could manually modify the state file, but this isn't recommended. If you have already migrated your plans to 0.12, however, that remains your only option for resources which do not support imports.

@bohdanyurov-gl
Copy link

bohdanyurov-gl commented Jun 10, 2019

Moving tf resource:
module.project.module.service-dbm-ecr.aws_ecr_repository.main ->
module.project.module.service-dbm-ecr.aws_ecr_repository.this[0]

Error: Invalid target address

Cannot move module.project.module.service-dbm-ecr.aws_ecr_repository.main to
<invalid>..: the target must also be a whole resource.

Terraform version 0.12.1

@sudoforge
Copy link
Author

@bohdanyurov-gl Thanks for reporting that you're experiencing the same issue!

In the future, a simple 👍 reaction on the original comment is a great way to "me too" an issue.

@pauldraper
Copy link

@sudoforge thanks for the tip; modifying the state file manually was indeed the solution I used.

@pauldraper
Copy link

pauldraper commented Jun 12, 2019

Alright, here is can move a resource to a new module in 0.12, with minimal risk: use a dummy resource (e.g. null_resource) in the new module, and -target its creation before moving your actual resources.

Version 1 (initial)

main.tf

module "example" {
  number = 1
  source = "./example"
}

example/main.tf

variable "number" {}

resource "local_file" "file" {
  count = var.number
  content = "content"
  filename = "${path.module}/file-${count.index}.txt"
}
# terraform init
# terraform apply -auto-approve
module.example.local_file.file[0]: Creating...
module.example.local_file.file[0]: Creation complete after 0s [id=040f06fd774092478d450774f5ba30c5da78acc8]

Version 2 (update, and failed mv)

main.tf

module "example" {
  number = 0
  source = "./example"
}

module "example2" {
  number = 1
  source = "./example"
}

example/main.tf

variable "number" {}

resource "local_file" "file" {
  count = var.number
  content = "content"
  filename = "${path.module}/file-${count.index}.txt"
}
# terraform state mv module.example.local_file.file module.example2.local_file.file

Error: Invalid target address

Cannot move to module.example2.local_file.file: module.example2 does not exist
in the current state.

Version 3 (create dummy, and successfully mv)

main.tf

module "example" {
  number = 0
  source = "./example"
}

module "example2" {
  number = 1
  source = "./example"
}

example/main.tf

variable "number" {}

resource "local_file" "file" {
  count = var.number
  content = "content"
  filename = "${path.module}/file-${count.index}.txt"
}

resource "null_resource" "null" {}
# terraform init

# terraform apply -auto-approve -target=module.example2.null_resource.null
module.example2.null_resource.null: Creating...
module.example2.null_resource.null: Creation complete after 0s [id=885875772923890418]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

# terraform state mv module.example.local_file.file module.example2.local_file.file
Move "module.example.local_file.file" to "module.example2.local_file.file"
Successfully moved 1 object(s).

Version 4 (remove dummy)

main.tf

module "example" {
  number = 0
  source = "./example"
}

module "example2" {
  number = 1
  source = "./example"
}

example/main.tf

variable "number" {}

resource "local_file" "file" {
  count = var.number
  content = "content"
  filename = "${path.module}/file-${count.index}.txt"
}
# terraform apply -auto-approve
module.example2.null_resource.null: Refreshing state... [id=2880189106572971671]
module.example2.local_file.file[0]: Refreshing state... [id=040f06fd774092478d450774f5ba30c5da78acc8]
module.example.null_resource.null: Creating...
module.example.null_resource.null: Creation complete after 0s [id=6086028694931568030]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

(I did run into #21529 when I was trying this with more complex configurations, but I didn't record how I got around that.)

@navaati
Copy link

navaati commented Jul 9, 2019

Hi.

Nice workaround with the dummy in the module, although it's impractical when you're getting the module from git or something like that.

So here is a way to do the exact same thing without modifying your module source, hacking the statefile instead. Of course the usual warning apply, that could eat your statefile:

terraform state pull |jq --arg module_path module.dali --arg null_id $(uuidgen) '.resources+=[{module:$module_path,mode:"managed",type:"null_resource",name:"dummy",provider:"provider.null",instances:[{schema_version:0,attributes:{id:$null_id,triggers:null},private:"bnVsbA=="}]}]|.serial+=1' |terraform state push -

You can use it by modifying the "module.mymodule" to the path of the actual module you want to move resources to. You need the jq and uuidgen commands.

Then do your move:

terraform state mv resource_type.my_resource module.mymodule.resource_type.my_resource # Now that works
terraform init # to download the null provider, that's needed at this point
terraform apply # That will delete the null resource, since it's not present in the code

Now, that is still a horrible hack :).

Dear Hashicorp, this bug is breaking basically all module-related refactoring possibilities, I really think it needs love fast !

@apottere
Copy link

apottere commented Aug 6, 2019

If you use the workaround provided by @navaati, you can use the following command to remove the dummy null_resource after moving your resources into the module. In my case I wasn't able to plan or apply because I didn't have a provider for the dummy null_resource.

terraform state rm module.<name>.null_resource.dummy

@jcarlson
Copy link

jcarlson commented Aug 9, 2019

@jbardin Why was this issue closed? I am still seeing this behavior in Terraform 0.12.5. Either this defect still exists, or the documentation should be updated to indicate that it is not possible.

In my use case, I am trying to refactor a large module into sub-modules. I am trying to do something like:

terraform state mv \
  module.monolith.random_string.password \
  module.monolith.module.credentials.random_string.password

I get the following error:

Error: Invalid target address

Cannot move to
module.monolith.module.credentials.random_string.password:
module.monolith.module.credentials does not exist in the current
state.

From the documentation:
https://www.terraform.io/docs/commands/state/mv.html#example-move-a-resource-into-a-module

The module will be created if it doesn't exist.

@teamterraform
Copy link
Contributor

Hi @jcarlson, the Close message above has a link to the PR that closed the issue, which was merged after 0.12.6. It will be included in the next release.

@jcarlson
Copy link

jcarlson commented Aug 9, 2019

Apologies, I missed that PR link. Thank you!

@sandangel
Copy link

sandangel commented Aug 22, 2019

hi I'm facing the same issue with terraform 0.12.6

tf state mv aws_iam_role.ecs_cloud_watch_read_role module.monitoring.module.iam.aws_iam_role.ecs_cloud_watch_read_role

Error: Invalid target address

Cannot move to
module.monitoring.module.iam.aws_iam_role.ecs_cloud_watch_read_role:
module.monitoring.module.iam does not exist in the current state.
# the old state here
tf state show aws_iam_role.ecs_cloud_watch_read_role
# aws_iam_role.ecs_cloud_watch_read_role:
resource "aws_iam_role" "ecs_cloud_watch_read_role" {
    arn                   = "arn:aws:iam::229482903727:role/ecsCloudWatchReadRole"
    assume_role_policy    = jsonencode(
        {
            Statement = [
                {
                    Action    = "sts:AssumeRole"
                    Effect    = "Allow"
                    Principal = {
                        Service = "ecs-tasks.amazonaws.com"
                    }
                },
            ]
            Version   = "2012-10-17"
        }
    )
    create_date           = "2019-08-15T08:09:15Z"
    force_detach_policies = false
    id                    = "ecsCloudWatchReadRole"
    max_session_duration  = 3600
    name                  = "ecsCloudWatchReadRole"
    path                  = "/"
    tags                  = {}
    unique_id             = "AROATK3R7HCX65ATUVF7C"
}
tf show tfplan
  - aws_iam_role.ecs_cloud_watch_read_role

  - aws_iam_role_policy.ecs_cloud_watch_read_policy

-/+ module.monitoring.aws_ecs_task_definition.z_grafana_td (new resource required)

  ~ module.monitoring.module.ecs_cluster.aws_launch_template.ecs_cluster_lt

  # I created the new resource here
  + module.monitoring.module.iam.aws_iam_role.ecs_cloud_watch_read_role

  + module.monitoring.module.iam.aws_iam_role.ecs_efs_instance_role

  + module.monitoring.module.iam.aws_iam_role_policy.ecs_cloud_watch_read_policy

  + module.monitoring.module.iam.aws_iam_role_policy.ecs_rexray_efs_policy

  + module.monitoring.module.iam.aws_iam_role_policy.ecs_rexray_volume_policy

  + module.monitoring.module.iam.aws_iam_role_policy_attachment.ecs_for_ec2_role

  + module.monitoring.module.iam.aws_iam_role_policy_attachment.efs_read_only_access
tf --version
Terraform v0.12.6
+ provider.aws v2.24.0
+ provider.cloudflare v1.17.1

@ghost
Copy link

ghost commented Sep 2, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Sep 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.