Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.32 tf refresh fails with AuthorizationFailed errors #3921

Closed
Justin-W opened this issue Jul 24, 2019 · 16 comments
Closed

1.32 tf refresh fails with AuthorizationFailed errors #3921

Justin-W opened this issue Jul 24, 2019 · 16 comments

Comments

@Justin-W
Copy link
Contributor

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.11.10
AzureRM v1.32.0

Affected Resource(s)

The errors occur when running the terraform refresh CLI command. They don't seem to be resource-specific.

  • azurerm_XXXXX

Terraform Configuration Files

# not applicable

Debug Output

Panic Output

Expected Behavior

Upgrading from AzureRM v1.31.0 to v1.32.0 (with no other changes to HCL, Azure accounts, etc.) should have been a no-op, and all terraform CLI commands should have behaved the same with either AzureRM provider version.

Actual Behavior

The terraform refresh command fails (every time) with errors like the following:

provider.azurerm.customer: Error ensuring Resource Providers are registered: Cannot register provider Microsoft.Maps with Azure Resource Manager: resources.ProvidersClient#Register: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=“AuthorizationFailed” Message=“The client ‘REDACTED’ with object id ‘REDACTED’ does not have authorization to perform action ‘Microsoft.Maps/register/action’ over scope ‘/subscriptions/REDACTED’ or the scope is invalid. If access was recently granted, please refresh your credentials.“.

Steps to Reproduce

  1. terraform refresh

Important Factoids

References

@katbyte katbyte added this to the v1.32.1 milestone Jul 25, 2019
@katbyte
Copy link
Collaborator

katbyte commented Jul 25, 2019

Hi @Justin-W,

We added a new client this release (azure maps) so the SP/authorization methold you are using to access Azure needs permission to register new clients. it looks like for some reason you do not have such permissions? How are you authenticating to azure and what roles/permission does that grant?

Outside of changing permissions there you could potentially register the client in the portal.

If your authentication method does not grant such permission you can skip provider registration by using the skip_provider_registration flag

@katbyte katbyte added question and removed bug labels Jul 25, 2019
@lambodar
Copy link

Hi Guys,

I too have same issue, I believe it should have made optional, skip provider registration is not the right thing to do also I'm not using Microsoft.Maps Resource providers explicitly so why my SP need to have permission to register it.

@watsonlu
Copy link

Hi,

Our setup is broken because of this and we are not using the map resources. The provider respects access restrictions on all other resources, why is the maps resource an exception to this?

We are going to hardcode our provider version to resolve this for now, but this is def breaking a bug

@lambodar
Copy link

Just a workaround we made it to point below which is working fine

version = "= 1.31" insted of version = "~> 1.31"

More here: https://www.terraform.io/docs/configuration/providers.html

@katbyte
Copy link
Collaborator

katbyte commented Jul 25, 2019

@watsonlu, @lambodar,

It sounds like you both have access restrictions on resource registrations for providers other then Microsoft.Maps?

The provider attempts to register all the resource providers that we use and it does not treat Microsoft.Maps any differently than the others. So if this is an exception it sounds like there might be something on the Azure side that is different?

@Justin-W
Copy link
Contributor Author

Hi @Justin-W,

We added a new client this release (azure maps) so the SP/authorization methold you are using to access Azure needs permission to register new clients.

Well, that qualifies as a potentially-breaking change to existing configurations, doesn't it? Shouldn't such a backwards-incompatible change have been at least documented somewhere? E.g. In the release notes or change log?

it looks like for some reason you do not have such permissions? How are you authenticating to azure and what roles/permission does that grant?

Correct. The authentication credentials I am using were specifically created with limited permissions, so that they are only able to do what they "should" be able to do for the purposes they are used for.

The newly-required permissions are not something that they are "supposed" to be able do, and it feels somewhat like a hack if we would need to update our service account permissions just because the AzureRM provider added a new capability, even though we aren't using it at all.

Is the set of "minimum required" permissions documented somewhere? And is it kept up to date whenever changes like this one in v1.32.0 occur?

Outside of changing permissions there you could potentially register the client in the portal.

This also feels lack a hack solution.

If your authentication method does not grant such permission you can skip provider registration by using the skip_provider_registration flag

I'm aware of that option, but the docs for that feature toggle don't seem to be detailed enough for me to understand the full implications of disabling auto-registration, and whether that would be a good and/or feasible solution. What else might break if we do that? In what circumstances "should" that option be true, and in which should it be false?

IMO, the whole point of using permissions-limited credentials was to ensure that if anything changed the behavior of the terraform CLI operations, then regardless of the source and cause of such a change, the limited credentials would limit the scope of the effects of such changes (especially unintended and/or otherwise-invisible changes).

So basically, the limited-permissions account worked exactly as we intended. It prevented the AzureRM provider from doing something "new", when the "change" that failed was something that: 1. we didn't previously need; 2. we still don't need (AFAIK); and 3. we don't really want.

However, I'd like to better understand your views on the "correct" way for us to handle this whole issue. Thanks.

@terminatingcode
Copy link

terminatingcode commented Jul 26, 2019

Hi there,

Me and my team use Service Providers to interact with the Azure Provider and ran into this issue.

We tried to use the hack:

variable "skip_provider_registration" {
  type    = bool
  default = true
}

provider "azurerm" {
  subscription_id            = "${var.subscription_id}"
  client_id                  = "${var.appId}"
  client_secret              = "${var.password}"
  tenant_id                  = "${var.tenant}"
  skip_provider_registration = "${var.skip_provider_registration}"
}

Unfortunately, it did not solve our problem and we encountered another error:

azurerm_storage_account.storage_account: Creating...

Error: Error updating Azure Storage Account enable_advanced_threat_protection "backupandrestoresdk": security.AdvancedThreatProtectionClient#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '<CLIENT>' with object id '<OBJECT>' does not have authorization to perform action 'Microsoft.Security/advancedThreatProtectionSettings/read' over scope '/subscriptions/<SUBSCRIPTION>/resourceGroups/bbr/providers/Microsoft.Storage/storageAccounts/<BLAH>/providers/Microsoft.Security/advancedThreatProtectionSettings/current' or the scope is invalid. If access was recently granted, please refresh your credentials."

@katbyte could you provide more information as to what you mean by registering the client via the portal? Register which client with what?
Azure is one of many cloud IaaS's I work with everyday so the terminology specific to Azure is not super fresh.

cc @BooleanCat

@katbyte
Copy link
Collaborator

katbyte commented Jul 26, 2019

@Justin-W,

I’m sorry that the latest release has broken your workflow, but the provider is not doing anything new in v1.32. It has always attempted to register every Resource Provider (RP) we support so end users do not have to go through the hassle of registering each one in each region. Whilst we could not do this by default it's a tradeoff between the provider not working by default and users having to turn off the registrations not wanted versus the manual work of turning them on. We believe that the vast majority of users will want them automatically registered compared to those who would manage RP registration themselves.

What the skip_provider_registration flag does is turn off those automatic registrations, nothing more. As you do not want to grant this SP provider registration access (and I assume only register RPs you use) you would set the skip_provider_registration flag to true and then manage those registrations yourself. Please see https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-supported-services for more information and detail about how to do that.

What did happen in 1.32 was we added support for a new RP (Microsoft.Maps) and the provider is now attempting to register that for you. This is something we do on average every 2-3 months as we support more services and not a breaking change. It is regrettable that it was not mentioned in the changelog (historically we haven’t included new RPs) but you make a good point and I think it is a good idea to include them going forward

I presume the reason this was not an issue for you before was all the RPs were already registered (most likely by running tf with the required access) and only upon upgrading to 1.32 was the provider attempting to register a new one causing the error.

I hope this clarifies what happened and how you can manage the client registrations going forward.

@katbyte
Copy link
Collaborator

katbyte commented Jul 26, 2019

@terminatingcode,

That is an unrelated issue that is being tracked here #3920. I've opened #3947 to resolve it in a hotfix hopefully to go out early next week.

@lambodar
Copy link

The label should be "bug"instead of "question".

@katbyte katbyte removed this from the v1.32.1 milestone Jul 29, 2019
@terminatingcode
Copy link

@terminatingcode,

That is an unrelated issue that is being tracked here #3920. I've opened #3947 to resolve it in a hotfix hopefully to go out early next week.

Hi @katbyte,
We bumped our provider to the fixed version in the referenced bug fix and still encounter this error. For context, Advanced Threat Protection is available in our region (West Europe) and we do not have it enabled. Is this a different issue?

cc @alamages

@katbyte
Copy link
Collaborator

katbyte commented Aug 5, 2019

@terminatingcode,

Apologies, it looks like that is a different threat protection error entirely, and most likely because the SP/auth method of choice does not have the correct permissions to enable advanced threat protection (different then this issue which is about permission required to register a Resource Provider) I presume you'll need to create a role with the advancedThreatProtectionSettings permission and add your SP to it. If this continues to be a problem would you mind opening a new issue as it is unrelated this this one?

@terminatingcode
Copy link

terminatingcode commented Aug 5, 2019

Hi @katbyte
We are not trying to enable advanced threat protection nor do we use it anywhere. The error seems to be failing on a read.
does not have authorization to perform action 'Microsoft.Security/advancedThreatProtectionSettings/read'

Perhaps the provider is trying to detect whether advanced threat protection is enabled or not?

I'll post this in a new issue if you feel it warrants it.

@tombuildsstuff
Copy link
Contributor

@terminatingcode

Perhaps the provider is trying to detect whether advanced threat protection is enabled or not?

Exactly that - as the Azure Provider now supports configuring Advanced Threat Protection, we now need to look this up in the Read function to confirm if it's configured or not (to be able to determine if it should be added/removed); as such Terraform needs permissions to read these settings. In this case the error being returned from the Azure API here states that you don't have permission to read the Advanced Threat Protection settings - as such adding this (read-only) permission should allow this to work.

@tombuildsstuff
Copy link
Contributor

Closing this in favour of #3950

@ghost
Copy link

ghost commented Sep 15, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

@ghost ghost locked and limited conversation to collaborators Sep 15, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants