403, substatus 5301 #3110
Comments
|
Do you have the full exception.ToString()? Looking for the diagnostics from the SDK to better understand what the client was doing. |
|
Just the call stack. But that exception itself isn't really the issue. It's throwing this error because the token it has says it does not have the proper permissions. The issue is why does the token say it doesn't have the proper permissions when it in fact does. Microsoft.Azure.Cosmos.CosmosException: |
|
One scenario where the service might be having an issue is reported on #3092, are you by any chance deleting and recreating the container/database? |
No, we are not. It's something to do with the AAD token.
I also don't understand why if AAD doesn't initially return the proper role assignments, why the cosmos refresh of the token doesn't either. This instance of the cosmos client will forever throw the 403 error until I restart the executable or create a new cosmos client. |
|
Please upgrade to the latest SDK. The following bug was fixed and exists in 3.24.0. I don't think this will fix the root issue it, but it should fix the need to recreate the client. |
|
Ok, I will try it. i'll close this for now because it'll take awhile to verify that it works. |
Describe the bug
We are using RBAC for data plane access. We have microservices running inside service fabric on a vm scaleset. The scaleset has a system managed identity. We have granted this managed identity the 00000000-0000-0000-0000-000000000002 role definition id. We create a single cosmos client for each instance of a service. Sometimes a service will fail on all calls to cosmos with the following error:
Response status code does not indicate success: Forbidden (403); Substatus: 5301; ActivityId: XXXX; Reason: (Request blocked by Auth XXXX : Request is blocked because principal [XXXX] does not have required RBAC permissions to perform action [Microsoft.DocumentDB/databaseAccounts/readMetadata] on resource [3TxEAO5CNTo=]. Learn more: https://aka.ms/cosmos-native-rbac.
ActivityId: XXXX, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.17763 cosmos-netstandard-sdk/3.24.1);
But other services running on the same machine in the same scaleset work just fine. If we shut down the service with the problem and start it back up it usually fixes the problem.
So,
To Reproduce
This is very hard to reproduce, I can't publish a repo where you can reproduce this.
Expected behavior
We expect cosmos to get a token from AAD (either initially or on refresh) with the proper permissions.
Actual behavior
Sometime AAD does not initially return the proper permissions and when cosmos does a token refresh behind the scenes that it gets a proper token. I'm not sure if the refresh is only updating the expireson or if it gets a new token with the proper permissions.
Environment summary
SDK Version: 3.24.1
OS Version (e.g. Windows, Linux, MacOSX) Windows Server Core 2019
Additional context
Add any other context about the problem here (for example, complete stack traces or logs).
The text was updated successfully, but these errors were encountered: