The s3 backend does not respect the NO_PROXY variable

[mnival]

Terraform Version

Terraform v1.6.0
on linux_amd64

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket="tf-state-test"
    region="eu-central-1"
    key="test/test.tfstate"
  }
}

Debug Output

https://gist.github.com/mnival/cb0857de9b72e2020bdc46b3ae00fbba

Expected Behavior

For requests to http://169.254.169.254/, Terraform should not use the proxy because I have the following NO_PROXY value:
NO_PROXY=169.254.169.254
Below is the result with Curl which uses the HTTP_PROXY, HTTPS_PROXY and NO_PROXY variables:

$ curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" -v -o /dev/null
* Uses proxy env variable NO_PROXY == '169.254.169.254'
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/8.0.1
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 21600
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Length: 56
< Content-Type: text/plain
< Date: Thu, 05 Oct 2023 09:40:58 GMT
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Connection: close
< Server: EC2ws
<
{ [56 bytes data]
* Closing connection 0

Actual Behavior

Terraform only uses HTTP_PROXY and HTTPS_PROXY variables but not NO_PROXY. So some requests use a proxy even though they are only available when local and therefore without a proxy

Steps to Reproduce

terraform init

Additional Context

export HTTP_PROXY="http://proxy:3128"
export HTTP_PROXY="http://proxy:3128"
export NO_PROXY="169.254.169.254"

References

• backend/s3: add http_proxy and related arguments #33748

[DerPauli]

We encounter this problem as well, but the other way around. Our atlantis server uses Terraform v1.6.0 and can't refresh the state from S3 anymore through our corporate proxy.
The atlantis output for terraform init looks like:

│ Error: No valid credential sources found
│ 
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, failed to retrieve
│ credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded
│ maximum number of attempts, 3, https response error StatusCode: 0,
│ RequestID: , request send failed, Post
│ "https://sts.eu-central-1.amazonaws.com/": proxyconnect tcp: dial tcp :0:
│ connect: connection refused
│ 

[StevenKGER]

We encounter this problem as well.

It seems like it's due to #33765 and there due to hashicorp/aws-sdk-go-base#81. Why did it work before? Because Golang uses the environmental proxy settings in the standard HTTPTransport, which has been used before.

This function returns always the same url - therefore ignoring NO_PROXY.

An issue should be opened in aws-sdk-go-base to support NO_PROXY settings and after it has been changed there, we can add NO_PROXY here again. Otherwise reverting the changes which have been added with the minor update 1.6.0 can recover the state which used the work before.

[wernerwws]

Is there any workaround?

[joaquin386]

@wernerwws 1.5.2 will work 1.6.X will not

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.