[Bug]: aws_chatbot_slack_channel_configuration with more than one SNS topic causes "Error: Provider produced inconsistent result after apply"

[chrisdenton-ct]

Terraform Core Version

1.9.2

AWS Provider Version

5.64.0

Affected Resource(s)

aws_chatbot_slack_channel_configuration

Expected Behavior

terraform apply creates AWS Chatbot Slack Channel configuration with multiple associated SNS topics and completes successfully.

Actual Behavior

terraform apply creates AWS Chatbot Slack Channel configuration with multiple associated SNS topics but then throws errors.

Relevant Error/Panic Output Snippet

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to aws_chatbot_slack_channel_configuration.NAME, provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an
│ unexpected new value: .sns_topic_arns[0]: was cty.StringVal("arn:aws:sns:AWSREGION:AWSACCOUNTID:TOPICARN1"), but now
│ cty.StringVal("arn:aws:sns:AWSREGION:AWSACCOUNTID:TOPICARN3").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to aws_chatbot_slack_channel_configuration.NAME, provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an
│ unexpected new value: .sns_topic_arns[1]: was
│ cty.StringVal("arn:aws:sns:AWSREGION:AWSACCOUNTID:TOPICARN2"), but now
│ cty.StringVal("arn:aws:sns:AWSREGION:AWSACCOUNTID:TOPICNARN1").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to aws_chatbot_slack_channel_configuration.NAME, provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an
│ unexpected new value: .sns_topic_arns[2]: was cty.StringVal("AWSREGION:AWSACCOUNTID:TOPICARN3"), but now
│ cty.StringVal("AWSREGION:AWSACCOUNTID:TOPICARN2").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform Configuration Files

resource "aws_chatbot_slack_channel_configuration" "NAME" {
  configuration_name    = "CONFIGNAME-${var.env}"
  iam_role_arn          = aws_iam_role.chatbot_role.arn
  slack_channel_id      = var.slack_channel_id
  slack_team_id         = "SLACKTEAMID"
  guardrail_policy_arns = ["arn:aws:iam::aws:policy/ReadOnlyAccess"]
  sns_topic_arns        = [data.terraform_remote_state.REPO1.outputs.TOPICARN1, data.terraform_remote_state.REPO2.outputs.TOPICARN2, data.terraform_remote_state.REPO3.outputs.TOPICARN3]
  tags                  = local.tags
}

Steps to Reproduce

Create a valid aws_chatbot_slack_channel_configuration resource where sns_topic_arns is a list of more than 1 arns. I have hit this issue by creating a list of 3 arns.

Debug Output

none

Panic Output

none

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @chrisdenton-ct 👋 Thank you for taking the time to raise this! I'm going to leave this open for the team/community to address more permanently. That said, looking at the error, it seems the ordering between plan and apply is throwing things off. While it's usually not necessary, the tolist() function should normalize the list and make sure that the order is consistent across the plan/apply. I haven't yet validated this, but it may be worth testing as a workaround until this issue is prioritized.

[joshaw]

Using the tolist() function to normalise the list wasn't sufficient for me. I've ended up using a lifecycle {} block to ignore the changes for the time being:

resource "aws_chatbot_slack_channel_configuration" "NAME" {
  ...
  lifecycle {
    ignore_changes = [sns_topic_arns]
  }
}

[Sacrimantius]

I encountered this issue today and determined sort() worked for my use case:

resource "aws_chatbot_slack_channel_configuration" "this" {
    ...
    sns_topic_arns     = sort([aws_sns_topic.low.arn, aws_sns_topic.high.arn])
}

I tested tolist() but it didn't appear to match due to provider returning alphabetically (at least from small sample size)

[chrisdenton-ct]

I've tried all the suggested workarounds but unfortunately I can't make any of them work for me.

So, still holding out hope for a code fix...

[Jeff-SearchPilot]

Same here, it looks like its related to the ordering of the SNS topics given back by the AWS API. If the ordering matches the input list (or only has 1 topic) then its fine but otherwise this error is thrown. sorting and reverseing lists can help but once you have more than 2 or 3 topics in the list you're probably stuck with having to add an ignore_changes lifecycle block (which will cause other issues).

I did have a quick look at @madhavvishnubhatta 's original PR (!38124) which recently added this resource. I assume its possible to just sort both lists internally to ensure consistency, but its a bit beyond my golang knowledge.

Probably duplicated in #39370 and #38956 too

[jpetre0]

Any update on this please? 🙏 We are also being impacted by this

[i5o]

I'm facing a similar issue but with MS Teams.

resource "aws_chatbot_teams_channel_configuration" "shared_devops_chatbot" {
  count = local.teams_channels[var.env_name] != null ? 1 : 0

  configuration_name    = "Shared-${upper(var.env_name)}-ChatOps"
  team_id               = local.team_id
  channel_id            = local.teams_channels[var.env_name]
  channel_name          = "AWS Chatbot - ${var.env_name}"
  sns_topic_arns        = split(",", data.external.chatbot_sns_topics.result["sns_topic_arns"])
  tenant_id             = local.tenant_id
  iam_role_arn          = aws_iam_role.chatbot_readonly_role.arn
  guardrail_policy_arns = ["arn:aws:iam::aws:policy/ReadOnlyAccess"]
}

Whenever I hit apply I get the following:

Acquiring state lock. This may take a few moments...
aws_chatbot_teams_channel_configuration.shared_devops_chatbot[0]: Modifying...
aws_chatbot_teams_channel_configuration.shared_devops_chatbot[0]: Still modifying... [10s elapsed]
aws_chatbot_teams_channel_configuration.shared_devops_chatbot[0]: Still modifying... [20s elapsed]
aws_chatbot_teams_channel_configuration.shared_devops_chatbot[0]: Still modifying... [30s elapsed]
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to aws_chatbot_teams_channel_configuration.shared_devops_chatbot[0], provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value: .sns_topic_arns[0]: was cty.StringVal("arn:aws:sns:us-east-1:xxxxx:bastion_host_patch_alerts"), but now cty.StringVal("arn:aws:sns:us-east-1:xxxxx:lexid-cw-alarms-topic").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to aws_chatbot_teams_channel_configuration.shared_devops_chatbot[0], provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value: .sns_topic_arns[2]: was cty.StringVal("arn:aws:sns:us-east-1:xxxxx:xxxxxx-topic"), but now cty.StringVal("arn:aws:sns:us-east-1:xxxxxx:xxxxx").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

[i5o]

Well, to add to my case, I figured out that the provider returns the following:
Topics sorted, but in reverse order.

So doing this solved my issue:

sns_topic_arns        = reverse(sort(split(",", data.external.chatbot_sns_topics.result["sns_topic_arns"])))

[JvRok]

None of the solutions provided here solved it consistently for me, and I was creating many integrations across many accounts and got a weird variety of responses.

I believe the error occurs due to

1. The provider cares about the order of the topics due to using a list rather than a set.
2. AWS chatbot seemingly decides a random but consistent order no matter how much you adjust the list manually or in IAC. So your provider has to match the order, which is seemingly random.

For 2 above - I actually manually adjusted the list in the AWS console to try to change the order of the sns topics, and after navigating away and back to the page, the list was back to the same order.

This order was consistent within thee account, but not between accounts. E.g. There did not seem to be a pattern to follow that worked across accounts.

E.g. Sometimes it would order as:

foo -> bar -> baz

but then on another account
bar -> baz -> foo

And consistently, stubbornly remained that way no matter what I did.

I could only fix the error in the provider through adjusting (manually) the order per account into what that particular account had decided the ordering should be.

I did manage to fix the issue by editing the provider's sns_topic type to set, rather than a list, but I have never touched golang before so I've raised it as a draft PR for someone with better knowledge of the provider/bug to take a look and advise on whether this is the way to go.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.78.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.