Upgrade of tfstate version on first remote fetch of 0.7.0 leads to CacheRefreshConflict

[glasser]

Terraform Version

This bug occurs when upgrading from a version of Terraform such as 0.6.15 to
current master (eg, 592162c).

Affected Resource(s)

Please list the resources as a list, for example:

• core (state file management and cache)

Terraform Configuration Files

No configuration file is necessary.

Steps to Reproduce

In an empty directory, with old and new versions of Terraform available:

• Configure remote state with the old version of Terraform. This writes a
  "version 1" state file to S3.
• Delete the local state
• Configure remote state with the new version of Terraform. This downloads the
  "version 1" state file but rewrites it to say "version 2" without changing the
  serial, which causes an error.

glasser@glasser-lyrid 0 ~/Scratch/tf-upgrade $ /usr/local/bin/terraform --version
Terraform v0.6.15

glasser@glasser-lyrid 0 ~/Scratch/tf-upgrade $ AWS_PROFILE=dev /usr/local/bin/terraform remote config -backend=s3 -backend-config=bucket="dev-meteor-terraform-state" -backend-config="key=glasser-tfbug-terraform.tfstate" -backend-config=region=us-east-1
Initialized blank state with remote state enabled!
Remote state configured and pulled.
glasser@glasser-lyrid 0 ~/Scratch/tf-upgrade $ cat .terraform/terraform.tfstate 
{
    "version": 1,
    "serial": 0,
    "remote": {
        "type": "s3",
        "config": {
            "bucket": "dev-meteor-terraform-state",
            "key": "glasser-tfbug-terraform.tfstate",
            "region": "us-east-1"
        }
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {}
        }
    ]
}
glasser@glasser-lyrid 0 ~/Scratch/tf-upgrade $ aws --profile=dev s3 cp s3://dev-meteor-terraform-state/glasser-tfbug-terraform.tfstate /dev/stdout
{
    "version": 1,
    "serial": 0,
    "remote": {
        "type": "s3",
        "config": {
            "bucket": "dev-meteor-terraform-state",
            "key": "glasser-tfbug-terraform.tfstate",
            "region": "us-east-1"
        }
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {}
        }
    ]
}
download: s3://dev-meteor-terraform-state/glasser-tfbug-terraform.tfstate to ../../../../dev/stdout
glasser@glasser-lyrid 0 ~/Scratch/tf-upgrade $ ~/go/bin/terraform -version
Terraform v0.7.0-dev

glasser@glasser-lyrid 0 ~/Scratch/tf-upgrade $ rm -rf .terraform
glasser@glasser-lyrid 0 ~/Scratch/tf-upgrade $ TF_LOG=TRACE AWS_PROFILE=dev ~/go/bin/terraform remote config -backend=s3 -backend-config=bucket="dev-meteor-terraform-state" -backend-config="key=glasser-tfbug-terraform.tfstate" -backend-config=region=us-east-1
2016/05/17 17:11:18 [INFO] Terraform version: 0.7.0 dev 
2016/05/17 17:11:18 [DEBUG] Detected home directory from env var: /Users/glasser
2016/05/17 17:11:18 [DEBUG] Detected home directory from env var: /Users/glasser
2016/05/17 17:11:18 [DEBUG] Attempting to open CLI config file: /Users/glasser/.terraformrc
2016/05/17 17:11:18 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2016/05/17 17:11:18 [DEBUG] Detected home directory from env var: /Users/glasser
2016/05/17 17:11:18 [WARN] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
Initialized blank state with remote state enabled!
2016/05/17 17:11:18 [WARN] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
Error while performing the initial pull. The error message is shown
2016/05/17 17:11:19 [DEBUG] plugin: waiting for all plugin processes to complete...
below. Note that remote state was properly configured, so you don't
need to reconfigure. You can now use `push` and `pull` directly.

Unknown refresh result: Local and remote state conflict, manual resolution required
glasser@glasser-lyrid 1 ~/Scratch/tf-upgrade $ cat .terraform/terraform.tfstate 
{
    "version": 2,
    "serial": 0,
    "remote": {
        "type": "s3",
        "config": {
            "bucket": "dev-meteor-terraform-state",
            "key": "glasser-tfbug-terraform.tfstate",
            "region": "us-east-1"
        }
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {}
        }
    ]
}

Expected Behavior

Running terraform remote config in an empty directory (no local tfstate file)
with a "version 1" tfstate file in S3 should not lead to an error.

Actual Behavior

It does lead to an error, because the version is converted to 2 on download
without changing the serial, which leads to a CacheRefreshConflict.

Debug Output

Debug output included above because it is short.

Workaround

-pull=false makes this command succeed and it does not appear to prevent
further commands from working.

[glasser]

I think the fix might be to this line in terraform/state.go:

// WriteState writes a state somewhere in a binary format.
func WriteState(d *State, dst io.Writer) error {
    // Make sure it is sorted
    d.sort()

    // Ensure the version is set
    d.Version = StateVersion

Perhaps only assign to Version if d.Version == 0? Though in that case, it's unclear to me that it would ever get upgraded to 2.

[bigkraig]

Yep, we use S3 buckets for our state files and always clear out the local .terraform directory before we run TF, this is a problem for us to go to 0.7.

[bigkraig]

Thoughts on this? It should only increment the serial number when we are upgrading a state.

diff --git a/terraform/state.go b/terraform/state.go
index 12fb901..ce6f909 100644
--- a/terraform/state.go
+++ b/terraform/state.go
@@ -1557,6 +1557,11 @@ func WriteState(d *State, dst io.Writer) error {
        // Make sure it is sorted
        d.sort()

+       // Update the serial number whenever we upgrade the state
+       if d.Serial != 0 && d.Version != StateVersion {
+               d.Serial = d.Serial + 1
+       }
+
        // Ensure the version is set
        d.Version = StateVersion

[glasser]

That makes sense to me though I don't know why the Serial != 0 check, and I'm not a Terraform developer.

[bigkraig]

Yeah we may not need that, my assumption was that if its a serial 0, its a new state file and wouldn't need to be bumped. I haven't tested the condition.

[bigkraig]

@jen20 this is a problem for anyone using remote states and upgrading to 0.7 AFAIK

[phinze]

Hi @bigkraig and @glasser - we'll make sure this is handled before the next RC 👍

[bigkraig]

It looks like this was addressed in #7402

[phinze]

Indeed it was! Feel free to follow up if you're seeing this issue on RC3 or later.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.