terraform 0.12 Crash running apply after upgrade

[bhawk123]

crash.log.zip

[apparentlymart]

Sorry for this crash @bhawk123, and thanks for reporting it.

Stack trace for this crash:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1b9c255]

goroutine 65 [running]:
github.com/hashicorp/terraform/backend/local.(*Local).renderPlan(0xc00064b5f0, 0xc000101810, 0xc0001a6558, 0xc000117200)
	/private/tmp/terraform-20190523-61862-1w2ypr9/terraform-0.12.0/src/github.com/hashicorp/terraform/backend/local/backend_plan.go:265 +0x755
github.com/hashicorp/terraform/backend/local.(*Local).opPlan(0xc00064b5f0, 0x2be63e0, 0xc0003d6d40, 0x2be63e0, 0xc0003d6d80, 0xc0003da540, 0xc0003d6d00)
	/private/tmp/terraform-20190523-61862-1w2ypr9/terraform-0.12.0/src/github.com/hashicorp/terraform/backend/local/backend_plan.go:165 +0xb9c
github.com/hashicorp/terraform/backend/local.(*Local).Operation.func1(0xc0005ac640, 0xc0005ac650, 0xc0005ac660, 0xc0003da540, 0xc00064b5f0, 0xc0003d6d00, 0xc0005ac630, 0x2be63e0, 0xc0003d6d40, 0x2be63e0, ...)
	/private/tmp/terraform-20190523-61862-1w2ypr9/terraform-0.12.0/src/github.com/hashicorp/terraform/backend/local/backend.go:356 +0x135
created by github.com/hashicorp/terraform/backend/local.(*Local).Operation
	/private/tmp/terraform-20190523-61862-1w2ypr9/terraform-0.12.0/src/github.com/hashicorp/terraform/backend/local/backend.go:340 +0x36b

This seems to be in the plan rendering codepath, checking if a resource instance object is tainted:

terraform/backend/local/backend_plan.go

Lines 260 to 267 in c7c69a5

	// check if the change is due to a tainted resource
	tainted := false
	if !state.Empty() {
	rs := state.ResourceInstance(rcs.Addr)
	if rs != nil {
	tainted = rs.Current.Status == states.ObjectTainted
	}
	}

My guess, given this context, is that the existing state contains a resource instance that currently has only deposed objects, and not a "current" object. This logic is assuming (incorrectly) that any instance that appears in the plan will have a current object.

I think there's another related inaccuracy here, thinking through what this code is attempting to do: the resource instance change we're rendering might itself have a DeposedKey, in which case we ought to be checking the tainted status of the corresponding deposed object, not the "current" object.

I think something like this would get the desired effect here (just off the top of my head, not tested at all yet):

// check if the change is due to a tainted resource
tainted := false
if !state.Empty() {
    if is := state.ResourceInstance(rcs.Addr); is != nil {
        if obj := is.GetGeneration(rcs.DeposedKey.Generation()); os != nil {
            tainted = obj.Status == states.ObjectTainted
        }
    }
}

[naps62]

Also getting this issue. Weirdly enough, I have 4 instances of the same code (each with different parameters, but overall the same structure should be in all of them) and only one of them crashes

[apparentlymart]

I think the trigger for this crash would be something in the latest Terraform state snapshot, rather than something in the configuration itself: if a previous Terraform version left a "deposed" instance in the state (can result in case of a catastrophic failure during a create_before_destroy operation, where the create step failed hard) then it may trigger this bug.

We had expected that running terraform apply one last time on Terraform 0.11.14 would clean up any such leftovers by destroying the leftover deposed objects and creating new objects to replace them, but it seems like there is some situation where that isn't true. Unfortunately because Terraform state persists between versions, the results of bugs and errors in older versions of Terraform can sometimes inadvertently persist across upgrades and cause issues for later releases, and possibly that is what is happening here.

We do intend to fix this, probably by doing something similar to what I mentioned in my earlier comment (since the existing logic is not totally correct anyway) but in the mean time it may be possible to work around this by making sure terraform apply can complete successfully on all of your configurations/workspaces on Terraform 0.11.14 before upgrading, if this is a problem that Terraform 0.11 knows how to correct in the state.

[naps62]

@apparentlymart I already downgraded the problematic folder back to 0.11, and applied the changes I had pending.
later today I can try the upgrade to see if it works now. Thanks

[nauxliu]

@apparentlymart the workaround doesn't work for me.

[nauxliu]

@apparentlymart
I have fixed my state and the crash is gone, here is how I did it

1. Clone from master and delete these line https://github.com/hashicorp/terraform/blob/master/backend/local/backend_plan.go#L263-L266
2. Run make bin to build my own build.
3. Use my own build to run terraform plan, then I saw a deposed resource that is not shown in 0.11.4 plan which should have been removed a long time ago.
4. Use terraform state rm to remove that state
5. Official 0.12.0 build did not cause a crash anymore.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.