Specified terraform version (using required_version) not used when required_providers is specified

[zlangbert]

I have a project that I just upgraded to terraform 0.13, and that on it's own worked fine. I then attempted to use the new syntax for specifying a provider version, and that seems to cause atlantis to be unable to parse the required_version specified in the terraform. For example, with this configuration plans are successful:

terraform {
  required_version = "0.13.0"

  backend "s3" {
    region         = "us-west-2"
    bucket         = "<redacted>"
    key            = "<redacted>"
    dynamodb_table = "<redacted>"
  }
}

But with this configuration:

terraform {
  required_version = "0.13.0"

  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 3"
    }
  }

  backend "s3" {
    region         = "us-west-2"
    bucket         = "<redacted>"
    key            = "<redacted>"
    dynamodb_table = "<redacted>"
  }
}

The plan fails with

Warning: Provider source not supported in Terraform v0.12

  on main.tf line 5, in terraform:
   5:     aws = {
   6:       source = "hashicorp/aws"
   7:       version = "~> 3"
   8:     }

A source was declared for provider aws. Terraform v0.12 does not support the
provider source attribute. It will be ignored.


Error: Unsupported Terraform Core version

This configuration does not support Terraform version 0.12.27. To proceed,
either choose another supported Terraform version or update the root module's
version constraint. Version constraints are normally set for good reason, so
updating the constraint may lead to other errors or unexpected behavior.

It seems specifying the provider info breaks required_version = "0.13.0".

[joerx]

Seems to be related to the "source" argument in the provider block

[joerx]

I'm seeing this error message in the logs:

2020/08/17 03:42:43+0000 [EROR] project_command_builder.go:450 [censored]#79: Trying to detect required version: Unsuitable value type: Unsuitable value: string required

[lkysow]

PRs welcome! Can someone dig into this?

[joerx]

Ok, so I did a bit more digging and I think I got to the root of it. Here's what I found:

The error we're seeing in the logs seems to originate here:

atlantis/server/events/project_command_builder.go

Line 476 in 1d59cf6

module, diags := tfconfig.LoadModule(absProjDir)

2020/08/20 02:48:37+0000 [EROR] project_command_builder.go:478 joerx/tf-sandbox#1: Trying to detect required version: Unsuitable value type: Unsuitable value: string required

Looks like tfconfig is complaining about something in our Terraform config. Adding some additional output to find out what exactly it's complaining about:

2020/08/20 02:48:37+0000 [DBUG] project_command_builder.go:480 joerx/tf-sandbox#1: Unsuitable value: string required in /home/atlantis/.atlantis/repos/joerx/tf-sandbox/1/default/custom-provider-poc/backend.tf:5

Which corresponds to the required provider block in my test config (this PR: joerx/tf-sandbox#1):

terraform {
  required_version = "0.13.0"

  required_providers {
    cheesecake = {
      source  = "joerx/cheesecake"
      version = "0.2.3"
    }
  }
}

Apparently the tfconfig library expects a string where now is a provider config object. Seems the corresponding dependency is a few revisions behind. I'll try to bump it and see if that helps.

[llamahunter]

Is there a workaround, since #1171 hasn't been released yet?

[llamahunter]

Hmm... I tried using the top-of-trunk atlantis, but it didn't work for me either. It seems to be ignoring the terraform required_version specification, and is not overriding the default terraform version 0.12.29 set via environment variables. By adding an atlantis.yaml via a repo config, I was able to set the right version 0.13.4, and things worked. But the docs seem to say that the specified required_version in the terraform config should be sufficient.
https://www.runatlantis.io/docs/terraform-versions.html#via-terraform-config

How are you supposed to do a transition from one major version of terraform to another? We have our atlantis instance defaulting to 0.12.29 so that all the old terraform 12 stuff will still work, by default. But when trying to move to 0.13 in a specific repo, it seems to not be obeying the directives in the HCL files.

[llamahunter]

Does the terraform required_version specification only work if specifying an exact version, not a version range, even tho that is valid there?

[llamahunter]

Looked at the source code and answered my own question. It does not, for reasons that are unclear to me. Filed #1217.

[joerx]

For what it's worth, I wrote a small wrapper around Terraform that will automatically download the correct TF version based on a semver constraint. It's here: https://github.com/joerx/tfx - we're using it in a custom workflow for Atlantis and it works reasonably well so far.

[llamahunter]

will automatically download the correct TF version based on a semver constraint.

You're scraping the HTML with a regex, right?

[joerx]

Yeah. haven’t found a better way to do it yet

…

Sent from my iPhone

On 12 Oct 2020, at 12:28 PM, Richard Lee ***@***.***> wrote:  will automatically download the correct TF version based on a semver constraint. You're scraping the HTML with a regex, right? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[scruplelesswizard]

There is a similar project called terraform-switcher that uses the official Terraform releases list and selects the correct release based on the required_version constraint

[scruplelesswizard]

Looked at the source code and answered my own question. It does not, for reasons that are unclear to me. Filed #1217.

@llamahunter if you have a moment can you please make note of where in the source you found the constraint to make it easier for someone to pick up?

[jseiser]

I wanted to report that this still doesnt work in the latest helm chart.

terraform {
  required_version = "0.13.5"
  backend "s3" {
    bucket         = "terraform-tfstate-us-east-1"
    encrypt        = "true"
    key            = "ops/restricted" # Change as needed
    dynamodb_table = "terraform_locks_us-east-1"
    region         = "us-east-1"
  }
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.19.0"
      region  = "us-east-1"
    }
  }
}

Results in

Error: Unsupported Terraform Core version

  on providers.tf line 2, in terraform:
   2:   required_version = "0.13.5"

This configuration does not support Terraform version 0.13.3. To proceed,
either choose another supported Terraform version or update this version
constraint. Version constraints are normally set for good reason, so updating
the constraint may lead to other errors or unexpected behavior.

We are specifying defaultTFVersion: 0.13.3 chart, and then attempting to upgrade section by section to 0.13.5, then we would bump up the default version.

[stafot]

Same here @jseiser looks as blocker to proceed with custom workflows with terraform > 0.12.27 in our case is 0.13.5.

[scruplelesswizard]

#1217 is in place to fix this, but it hasn't been merged yet.

[nitrocode]

By the way, for any people who find this issue, you can now set the required version and atlantis will download it for you.

https://www.runatlantis.io/docs/terraform-versions.html#via-terraform-config