[Bug]: aws_iam_role role_last_used has changed outside of Terraform

[jburgess777]

Terraform Core Version

1.1.4

AWS Provider Version

4.64.0

Affected Resource(s)

• aws_iam_role

Expected Behavior

We expect role_last_used to change frequently. The value is not controlled by Terraform so I don't think it should be reported as a change to the resource when generating the plan.

Actual Behavior

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # aws_iam_role.state_machine_terminate has changed
  ~ resource "aws_iam_role" "state_machine_terminate" {
        id                    = "state_machine_terminate-jburgess3"
        name                  = "state_machine_terminate-jburgess3"
      ~ role_last_used        = [
          ~ {
              ~ last_used_date = "2023-04-21T10:24:09Z" -> "2023-04-21T10:24:11Z"
              ~ region         = "us-east-1" -> "us-west-2"
            },
        ]
        tags                  = {}
        # (10 unchanged attributes hidden)

        # (5 unchanged blocks hidden)
    }

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

It looks like this would appear for any IAM role created by Terraform

Steps to Reproduce

Apply terraform plan which creates an AWS IAM role
Use the role
Might need to wait some time for AWS to update the role usage metadata
Run plan again

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

role_last_used is a new feature in the 4.64 release: #30750

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[kristofmartens]

This is a super annoying bug as this will constantly update the terraform state when that role is being used which is constantly... Please give this the proper priority if you can

[anthonybouton]

Hi there, I am currently working on a fix, PR should be ready to review soon

[anthonybouton]

@jburgess777
I'm having issues reproducing this
My environment

OS Ubuntu 23.04
Terraform Version : 1.4.5
Provider version: 4.64.0

What I have done

• Create a new terraform project and create a Role assumable by Lambda togheter with a dummy lambda function

• Trigger the lambda a few times manually in the Console

• Wait for AWS to update the last_used field and validate the result from aws iam get-role --role-name xx

• terraform plan

• no changes

• modified the name of the role to trigger a change plan
  
  only one change is noticed by terraform, seems legit?

• I abort the change, and I change the name back to the original value

• terraform plan
  

• I also tried with a modification that does not require a replacement ( such as altering the description ) but still no luck as the change set looks good
  

• As an extra attempt, I modified my local state file and altered the last_used datestamps in there, but still no luck

Am I missing something? Is this OS related?
I noticed you using an older terraform core version, did updating help?

[mycrEEpy]

I noticed you using an older terraform core version, did updating help?

I can confirm it's happening in one of our projects with Terraform Core 1.1.9.
The same codebase in another project with Terraform Core 1.3.9 does not have this problem.

[bit-nibble-byte]

Currently experiencing this with terraform version 1.4.6 and provider hashicorp/aws v4.64.0

I don't see these changes when running terraform plan like the original reporter but I DO see changes related to "role_last_used" every time I run terraform plan -refresh-only. "role_last_used" seems guaranteed to change every lambda execution and is making the refresh plan super inflated

[ZerxXxes]

Hello, I also have this problem when running terraform in Github Actions using terraform 1.3.3 and provider hashicorp/aws v4.65.0

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

# module.environment.aws_iam_role.ecs_instance_role has changed
  ~ resource "aws_iam_role" "ecs_instance_role" {
        id                    = "ecs-instance-role"
        name                  = "ecs-instance-role"
      ~ role_last_used        = [
          ~ {
              ~ last_used_date = "2023-05-04T07:49:36Z" -> "2023-05-04T08:49:03Z"
                # (1 unchanged element hidden)
            },
        ]
        tags                  = {}
        # (9 unchanged attributes hidden)
    }

[rito-sixt]

Hi. I have this issue as well while using https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest where the IAM role created constantly shows up as "changed" because of the last used timestamp.

Super annoying, and would be great if can be fixed.

[Flydiverny]

Adding another variant:
This always shows as a change when using terraform plan --refresh=false, resulting in a role_last_used = (known after apply), while a terraform plan shows no changes. Very noisy and frustrating 😄

[gabrielrinaldi]

Any updates on the PR to fix this? The issue for me is that it creates a false drift detection in Terraform, it is hard to know if there is an actual problem or just normal usage of the platform. From my understanding this has been fixed in the past and recently came back.

[anthonybouton]

I haven't been able to reproduce this on the latest version(s) of either terraform and the AWS provider.
I've volunteered to pickup this issue but I'm not sure what the strategy is for backwards compatibility across older terraform versions and how far back they should go.

[gabrielrinaldi]

@anthonybouton since it is a field that we cannot update (at least per the documentation) should it be saved in the state? Every time someone uses a role AWS will update that field causing the issue. But I am not very familiar on how providers work, just how to use them :)

[anthonybouton]

@anthonybouton since it is a field that we cannot update (at least per the documentation) should it be saved in the state? Every time someone uses a role AWS will update that field causing the issue. But I am not very familiar on how providers work, just how to use them :)

That is what I am attempted to believe, however I am not able to reproduce this, neither on my small repo, or on other client's projects where terraform with the AWS provider is being used heavily.

I've taken a look at the code and the required IGNORE attributes seem to be present for this particular attribute.

[gabrielrinaldi]

@anthonybouton do you think it could be a configuration issue? Like should I add a lifecycle entry for it? I am on version 4.67.0 and it happens everyday. Also this is happening on the oidc dynamic roles for GitHub and Terraform, not on other roles.

[anthonybouton]

@anthonybouton do you think it could be a configuration issue? Like should I add a lifecycle entry for it? I am on version 4.67.0 and it happens everyday. Also this is happening on the oidc dynamic roles for GitHub and Terraform, not on other roles.

Maybe, we have a similar setup ( using BitBucket instead of GitHub ) and not seeing the issue neither.
What version of terraform are you running? And if it's not the latest, did upgrading help?

[sammccants]

@gabrielrinaldi adding a lifecycle entry did not solve it for me, it just added this sort of ironic warning:

Adding an attribute name to ignore_changes tells Terraform to ignore
future changes to the argument in configuration after the object has
been created, retaining the value originally configured.

The attribute role_last_used is decided by the provider alone and
therefore there can be no configured value to compare with. Including
this attribute in ignore_changes has no effect. Remove the attribute
from ignore_changes to quiet this warning.

For info I'm using the most recent version of Terraform v1.4.6 and I believe the most recent version of the AWS provider (I have its version set to ~> 4.0 which I believe will have it use the latest version of 4.67.0)

[gabrielrinaldi]

@anthonybouton I am using the latest v1.4.6

@sammccants does it happen for all roles for you?

[Flydiverny]

Regarding -refresh=false planning:

Spent some time digging into our various terraform targets, their current plans and when the provider was upgraded, and whether or not they've been applied since.

My theory was that some targets had not been applied since the provider was upgraded (our CICD will skip apply if plan has no changes), and applying them once fixes the -refresh=false plan.

With this in mind, I checked a couple of more deployment targets, and targets where the state file had not been modified since the provider update and inspected the state file seeing that there is no role_last_used in there, vs state files that had been modified after upgrade had the role_last_used fields in the state file.

Hoping this can help some others! I don't know if there's any relation to the issue when not using -refresh=false.

what I was doing and writing before coming up with ☝🏻

In our/my case it is not limited to any specific set of roles as far as I can tell. I have roles used in IAM Roles for Service Accounts, roles used by EC2 instances and datadog integration roles etc.

We reuse the same terraform with different variables for multiple environments.

Using terraform 1.4.6, and various AWS provider versions .65, .66, .67

"target"	terrafrom plan -refresh=false	terraform plan	Comment
A stg	'No changes'	'No changes'	As expected
A prod	'No changes'	'No changes'	As expected
B stg	role_last_used = (known after apply)	'No changes'	Unexpected
B prod	'No changes'	'No changes'	As expected

For B stg, running -refresh=false after a plan with no changes still gives known after apply. Running terraform apply (no changes) then running with -refresh=false gives the expected No changes

Regarding terraform plan -refresh-only:

It does indeed show as changes made outside of Terraform each time, which is noisy. But I suppose this is expected. Similar behavior exists in other resources like aws_db_instance with latest_restorable_time.

[eduardoasantana]

Would someone be able to share their lifecycle policies used to ignore changes to the IAM role role_last_used and last_used_date values in state?

As others have mentioned, this causes a lot of false alarms and noise in our Terraform drift detection alerts.

[ollytheninja]

The below does not appear to work @eduardoasantana, I'd love to know how to ignore it too!

lifecycle {
  ignore_changes = [
    role_last_used.0.last_used_date,
  ]
}

I've also tried just role_last_used but that also doesn't work. Not sure if I'm missing something or if the ignore flag isn't working properly?

[bryantbiggs]

@dpowley / @jar-b - can we revert #30750 - it doesn't seem like it was a highly requested feature and its causing more harm than good currently. I'm not sure of the value of this attribute in Terraform either

[jar-b]

Hi everyone - Thanks for your reports, and apologies for any frustration or confusion this addition has caused.

To restate my understanding of the issue, the roles_last_used attribute is causing pre-v1.2.0 versions of Terraform (more on this later) to display a note indicating changes were made outside of Terraform. While no changes are actually planned, the note, and its resemblance to a plan/apply output when changes actually are necessary, is causing confusion and logging noise.

Under the hood, Terraform core is handling what changes are displayed at plan time. A core change introduced in v1.2.0 (specifically hashicorp/terraform#30486) should help this case:

When showing a plan, Terraform CLI will now only show "Changes outside of Terraform" if they relate to resources and resource attributes that contributed to the changes Terraform is proposing to make.

A previous comment in this thread documents how changes to this computed attribute are correctly hidden in later versions of the Terraform CLI. A similar reproduction on a different computed attribute can be found in this issue comment, which explicitly shows the differences between plan outputs across Terraform CLI versions.

In summary, upgrading to Terraform CLI >=v1.2.0 should prevent excess noise caused by changes to this computed attribute.

---

Edit: For anyone experiencing plan differences (not just a note about changes outside Terraform on a plan with no changes), or still observing displayed differences on Terraform versions >v1.2.0, please open a new issue linking back to this one. Refresh-only plans may be a separate case that we need to handle regardless of Terraform CLI version, so a separate issue report with more details would be greatly appreciated!

[bryantbiggs]

@jar-b - @yurrriq reported this issue with Terraform 1.4.0 running in Terraform Enterprise (showing up in TFE drift detection), so I don't think its just an issue with < 1.2.0

[jar-b]

Could you please share the relevant Terraform configuration and plan output?

[bryantbiggs]

see above, there are several references

[dpowley]

@jar-b - @yurrriq reported this issue with Terraform 1.4.0 running in Terraform Enterprise (showing up in TFE drift detection), so I don't think its just an issue with < 1.2.0

Drift detection is a new feature from HashiCorp and all computed values that change show up as drift as the feature is currently implemented. This is something that we've brought up with the team in the past but is currently expected behavior. Have we seen it show up in a later version than 1.2 in a TF plan?

[bryantbiggs]

and all computed values that change show up as drift as the feature is currently implemented

Which I somewhat understand, thats fundamentally how Terraform works - this is why I raised the question of reverting the changes made in #30750. It doesn't appear that it was a highly requested feature addition and it doesn't fit within the Terraform model. If we know that a computed value, that is not controlled by Terraform, is always going to change and keep changing, why is that a valuable addition to the provider?

[dpowley]

and all computed values that change show up as drift as the feature is currently implemented

Which I somewhat understand, thats fundamentally how Terraform works - this is why I raised the question of reverting the changes made in #30750. It doesn't appear that it was a highly requested feature addition and it doesn't fit within the Terraform model. If we know that a computed value, that is not controlled by Terraform, is always going to change and keep changing, why is that a valuable addition to the provider?

That's a great question. First off, I added it to the provider to use it in the new continuous validation feature from Terraform so we can get notified on unused roles.

As far as computed values go, these are all over the place in TF. Oftentimes they don't change (like the AWS generated AROA unique ID for an AWS IAM role as an example), but there are some computed attributes that change frequently (the accounts attribute on an aws_organizations_organization resource or the resource_count attribute on the tfe_workspace resource or the etag attribute on the google_project_iam_binding resource are three examples). I do think these are valuable to have in TF as they are resource attributes that are sometimes useful to reference. But I think drift detection should largely ignore computed values as they are expected to change outside of TF as you said.

[bryantbiggs]

But I think drift detection should largely ignore computed values as they are expected to change outside of TF as you said.

I think thats a bit heavy handed to state - computed values that DO change are most likely changing for a reason and that change will trigger cascading effect to its dependencies (i.e. - if computed value foo changes, and resource bar is dependent on that attribute, bar will need to be re-evaluated/updated based on those changes). So we can't simply just ignore them.

If a value is changing, Terraform will detect that and make it known in the plan/apply - that is the premise of Terraform. I believe you are trying to mix declarative with dynamic/imperative (i.e. - if this happens, do this) and that is not what Terraform was designed for (@apparentlymart please correct me if I am wrong)

[dpowley]

But I think drift detection should largely ignore computed values as they are expected to change outside of TF as you said.

I think thats a bit heavy handed to state - computed values that DO change are most likely changing for a reason and that change will trigger cascading effect to its dependencies (i.e. - if computed value foo changes, and resource bar is dependent on that attribute, bar will need to be re-evaluated/updated based on those changes). So we can't simply just ignore them.

If a value is changing, Terraform will detect that and make it known in the plan/apply - that is the premise of Terraform. I believe you are trying to mix declarative with dynamic/imperative (i.e. - if this happens, do this) and that is not what Terraform was designed for (@apparentlymart please correct me if I am wrong)

100% agree. In the case that you outline, bar will show as having drift since a non-computed value is changing for resource bar. As far as what is TF designed for, I'll leave the official take to HashiCorp product managers and engineers that work directly on the product!

[pspot2]

When I first saw this behavior, I obviously thought of it as a feature similar to the detection of external changes in many other resources/attributes. E.g. those who don't want to see it, should be able to ignore that part. However, as @ollytheninja mentioned above, this doesn't work by design for read-only attributes:

│ Warning: Redundant ignore_changes element
│ 
│   on ../../../../modules/my_module/my_module.tf line 58, in resource "aws_iam_role" "user_role":
│   58: resource "aws_iam_role" "user_role" {
│ 
│ Adding an attribute name to ignore_changes tells Terraform to ignore future
│ changes to the argument in configuration after the object has been created,
│ retaining the value originally configured.
│ 
│ The attribute role_last_used is decided by the provider alone and therefore
│ there can be no configured value to compare with. Including this attribute
│ in ignore_changes has no effect. Remove the attribute from ignore_changes
│ to quiet this warning.

After stumbling upon this discussion, I'm more inclined to consider this a "bug" as well for the following reasons:

• Cannot be silenced/ignored through configuration (!)
• Even if it was possible to ignore it, the lifecycle block would need to be added to all existing (and future) aws_iam_role (and other) resources. "Opt-out" doesn't look like a good way to manage this.
• I do agree that for many other resources/attributes, drift detection (changes outside of Terraform) is a neat feature to have a quick overview whether external changes are legit or not. But this one is completely useless (if somebody needs to track last used times of roles, Terraform plan is one of the worst places to do that. A proper reporting solution would be more appropriate for this use-case.)

[jar-b]

After consideration of community feedback, we've decided to remove the role_last_used attribute from the aws_iam_role resource, but leave it in the corresponding data source. While technically representing a breaking change, we're tentatively planning to release this in the next minor version, v5.1.0. We believe a breaking change is acceptable in this context for the following reasons:

1. The attribute, while functioning correctly, is causing a negative user experience, evidenced by the number of upvotes and comments on this issue in a relatively short time since its release.
2. The attribute is relatively new (release with v4.64.0, from April 2023), read-only, and unlikely to be used as an input to any other resource.
3. The corresponding data source provides an alternative source for this information.

Please let us know if you have any questions or concerns with this proposed change. Thanks!

[github-actions]

This functionality has been released in v5.1.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.