[C++] Implement all important authentication methods for Azure filesystem

[Tom-Newton]

Describe the enhancement requested

So far the Azure filesystem implementation only supports account key authentication. This is ok initially during development but for normal operation role-based access control (RBAC) is recommended.

Azure authentication that I think are important to supported (roughly ordered with most important first):

• [C++][FS][Azure] Default credential authentication #39262
• [C++][FS][Azure] Workload Identity Authentication #39318
• [C++][FS][Azure] Managed Identity authentication #39320
• [C++][FS][Azure] Client secret authentication #39343
• [C++][FS][Azure] Azure CLI authentation #39344
• [C++][FS][Azure] Environment credential authentication #39345
• [C++][FS][Azure] SAS token authentication #44308
• None, for un-secured accounts. Implemented by GH-39449: [C++] Use default Azure credentials implicitly and support anonymous credentials explicitly #39450

Related Issues:

• [C++] Filesystem implementation for Azure Blob Storage #18014 (is a child of)

Component(s)

C++

[kou]

Can we split this issue to each authentication method?

[pitrou]

Can we split this issue to each authentication method?

Seconded. What we did in S3 is simply start with the simplest methods (anonymous, secret key) and then let contributors add whichever auth method they needed.

[Tom-Newton]

Ok 👍. I'll probably make each individual auth method a sub ticket of this ticket.

I plan to start working on this quite soon.

[sahil1105]

Hi, I was wondering if there were any examples of using SAS tokens with the AzureFileSystem?

SAS token. I believe SAS token just needs to be appended to the URL in the right places.

My understanding is that the SAS tokens need to be appended to the API requests and therefore we may still need to provide a way at the FileSystem level to specify these tokens so that they can be used when making the API calls. Please correct me if I'm mistaken. When we tried to include the SAS token in the path to "OpenOutputStream" (when trying to write a parquet file), we saw errors like:

Check for Hierarchical Namespace support on 'https://<STORAGE_ACCOUNT_NAME>.blob.core.windows.net/<CONTAINER_NAME>' failed. Azure Error: [AuthenticationFailed] 403 Server failed to authenticate

I'd be interested to pick this up if I can get some guidance on it.

[kou]

You're right.

We need to design API for SAS token support as the first step. Could you investigate APIs of other related libraries?

[Tom-Newton]

It looks like there is AzureSasCredential https://github.com/Azure/azure-sdk-for-cpp/blob/101f20f2bbf3dd5f6438565cd9f709a231317f77/sdk/tables/azure-data-tables/inc/azure/data/tables/credentials/azure_sas_credential.hpp#L16C9-L16C27 in the Azure C++ SDK so I think we can just do it exactly the same as all the other authentication options.

Checking the API on a couple of similar libraries:
adlfs just has a sas_token string argument when initialising the filesystem https://github.com/fsspec/adlfs/blob/7f06dbdd410224047df09d0ef9e5e9913c64bf7d/adlfs/spec.py#L247
object_store has a with_sas_authorization method, but interestingly it wants the URL syntax of the SAS token broken down into query_pairs: impl Into<Vec<(String, String)>>.

[Tom-Newton]

All the sub issues are now complete, so I'm going to mark this as completed.