auth: Support AzureDevOps Pipeline OIDC auth (similar to Github OIDC auth)

[magodo]

This PR adds support for Azure DevOps Pipeline OIDC authentication, similar to the existing Github OIDC authentication.

Currently users are able to use OIDC token under ADO pipeline to exchange for an ARM access token, assuming the OIDC token is from the AzureCLI task or other tasks that support OIDC. The problem of this auth way is when the arm access token expires, there is no way to refresh and get a new one OIDC token.
This PR makes the provider get a OIDC token by itself, via the OIDC request token + request URL + ADO service connection (properly configured).

With this, the users will be able to use 3 flavors of OIDC auths:

• Direct OIDC token
• Github OIDC flow
• ADO OIDC flow (new)

Test

I used the following pipeline definition to run the acctest:

trigger: 
 - none

pool:
   vmImage: 'ubuntu-latest'

resources:
  repositories:
    - repository: go-azure-sdk
      type: github
      endpoint: magodo-pat-read-public-repo
      name: magodo/go-azure-sdk
      ref: ado_pipeline_oidc

steps: 
- task: GoTool@0
  inputs:
    version: '1.23.3'

- checkout: go-azure-sdk

- task: AzureCLI@2
  inputs:
    azureSubscription: $(CONNECTION_ID)
    scriptType: bash
    scriptLocation: "inlineScript"
    inlineScript: |
      set -e
      cd sdk/auth
      go test -v -run="TestAccADOPipelineOIDCAuthorizer" .
  env:
    ACCTEST: 1
    ARM_TENANT_ID: $(AZURE_TENANT_ID)
    ARM_CLIENT_ID: $(AZURE_CLIENT_ID) 
    ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID: $(CONNECTION_ID)
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    SYSTEM_OIDCREQUESTURI: $(System.OidcRequestUri)

Following is the screenshot of the result:

References

• https://devblogs.microsoft.com/azure-sdk/improve-security-posture-in-azure-service-connections-with-azurepipelinescredential/
• https://devblogs.microsoft.com/devops/workload-identity-federation-for-azure-deployments-is-now-generally-available/
• https://devblogs.microsoft.com/devops/introduction-to-azure-devops-workload-identity-federation-oidc-with-terraform/

Update [Jan.10]

I've added more tests at the client (ACC) and auth (UT) packages. Meanwhile, I've updated the auth/README.md file. Besides, I've adjusted the order of ADO and Github OIDC in the NewAuthorizerFromCredentials function, which enables users who intend to use ADO OIDC while enbaled both won't end up using Github OIDC auth and fail. This can work since ADO OIDC has one more condition (i.e. the service connection ID) than Github OIDC.

Tests

Github Action

name: go-azure-sdk OIDC test
on: [workflow_dispatch]

permissions:
  id-token: write
  contents: read
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: 'Checkout go-azure-sdk repo'
        uses: actions/checkout@v4
        with:
          repository: 'magodo/go-azure-sdk'
          ref: 'ado_pipeline_oidc'
      - name: 'Setup Go'
        uses: actions/setup-go@v5
        with:
          go-version: '1.23'
      - name: 'Go Test'
        run: |
          cd sdk
          export ACCTEST=1
          export ARM_TENANT_ID=${{ secrets.AZURE_TENANT_ID }}
          export ARM_CLIENT_ID=${{ secrets.AZURE_CLIENT_ID }}
          go test -v -run="TestAccGitHubOIDCAuthorizer" ./auth
          go test -v -run="TestAccClient" ./client

ADO

trigger: 
 - none

pool:
   vmImage: 'ubuntu-latest'

resources:
  repositories:
    - repository: go-azure-sdk
      type: github
      endpoint: magodo-pat-read-public-repo
      name: magodo/go-azure-sdk
      ref: ado_pipeline_oidc

steps: 
- task: GoTool@0
  inputs:
    version: '1.23.3'

- checkout: go-azure-sdk

- task: AzureCLI@2
  inputs:
    azureSubscription: $(CONNECTION_ID)
    scriptType: bash
    scriptLocation: "inlineScript"
    inlineScript: |
      set -e
      cd sdk
      go test -v -run="TestAccADOPipelineOIDCAuthorizer" ./auth
      go test -v -run="TestAccClient" ./client
  env:
    ACCTEST: 1
    ARM_TENANT_ID: $(AZURE_TENANT_ID)
    ARM_CLIENT_ID: $(AZURE_CLIENT_ID) 
    ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID: $(CONNECTION_ID)
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    SYSTEM_OIDCREQUESTURI: $(System.OidcRequestUri)

[jaredfholgate]

This is looking really good. I appreciate the environment variable names will be defined in the provider, but thinking ahead, I suggest they match what has been implemented for azapi, so that from a usability perspective customers only need to set one env var.

Docs here: https://registry.terraform.io/providers/Azure/azapi/latest/docs/guides/service_principal_oidc#configuring-the-service-principal-in-terraform

This is what the azapi env vars look like:

 - task: AzureCLI@2
    displayName: Acc Tests with OIDC Azure Pipeline
    inputs:
      azureSubscription: 'azapi-oidc-test' // Azure Service Connection ID
      scriptType: 'pscore'
      scriptLocation: 'inlineScript'
      inlineScript: |
        $env:ARM_TENANT_ID = $env:tenantId
        $env:ARM_CLIENT_ID = $env:servicePrincipalId
        $env:ARM_OIDC_REQUEST_TOKEN = "$(System.AccessToken)"
        $env:ARM_OIDC_AZURE_SERVICE_CONNECTION_ID = "azapi-oidc-test"
        $env:ARM_USE_OIDC = 'true'
        terraform plan
      addSpnToEnvironment: true

You'll also notice that the TOKEN REQUEST URI is not specified here. That is because the azure go sdk handles finding that. I think we could do the same here?

• https://github.com/Azure/terraform-provider-azapi/blob/df7ff137be43ac025ff95a88eb49b0bc31915d4f/internal/provider/provider.go#L949
• https://github.com/Azure/azure-sdk-for-go/blob/45c9053e200d7cdf488af284806a3f1fca1d32bc/sdk/azidentity/azure_pipelines_credential.go#L79

[magodo]

@jaredfholgate Good spots! The namings of the env vars will be defined in the provider, instead of the go-azure-sdk, which is slightly different than how Azure track2 SDK does. Since you can already see that the github oidc implementation in this SDK didn't auto-discover those env vars, but prefer an explicit definition by the users:

	// IdTokenRequestUrl is the URL for the OIDC provider from which to request an ID token.
	// Usually exposed via the ACTIONS_ID_TOKEN_REQUEST_URL environment variable when running in GitHub Actions
	IdTokenRequestUrl string

Followings are only defined in the test files:

	GitHubToken                   = os.Getenv("ACTIONS_ID_TOKEN_REQUEST_TOKEN")
	GitHubTokenURL                = os.Getenv("ACTIONS_ID_TOKEN_REQUEST_URL")
	ADOPipelineToken              = os.Getenv("SYSTEM_ACCESSTOKEN")
	ADOPipelineTokenURL           = os.Getenv("SYSTEM_OIDCREQUESTURI")
	ADOServiceConnectionId        = os.Getenv("ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID")

When this PR get merged and we integrate this feature into the azuread, azurerm and azure backend, we'll keep the default namings for the env vars.

[jaredfholgate]

@jaredfholgate Good spots! The namings of the env vars will be defined in the provider, instead of the go-azure-sdk, which is slightly different than how Azure track2 SDK does. Since you can already see that the github oidc implementation in this SDK didn't auto-discover those env vars, but prefer an explicit definition by the users:

	// IdTokenRequestUrl is the URL for the OIDC provider from which to request an ID token.
	// Usually exposed via the ACTIONS_ID_TOKEN_REQUEST_URL environment variable when running in GitHub Actions
	IdTokenRequestUrl string

Followings are only defined in the test files:

	GitHubToken                   = os.Getenv("ACTIONS_ID_TOKEN_REQUEST_TOKEN")
	GitHubTokenURL                = os.Getenv("ACTIONS_ID_TOKEN_REQUEST_URL")
	ADOPipelineToken              = os.Getenv("SYSTEM_ACCESSTOKEN")
	ADOPipelineTokenURL           = os.Getenv("SYSTEM_OIDCREQUESTURI")
	ADOServiceConnectionId        = os.Getenv("ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID")

When this PR get merged and we integrate this feature into the azuread, azurerm and azure backend, we'll keep the default namings for the env vars.

Just to be clear, will the provider look up the SYSTEM_OIDCREQUESTURI, so the user doesn't need to supply it like azapi? Wouldn't it be better to implement that in the library like the azure go sdk does?

Or are you expecting the user to explicitly pass that url? It seems unnecessary to add that requirement given it is not required for azapi.

Apologies if I am missing something.

[magodo]

@jaredfholgate The library won't auto-detect those environment variables by naming convention, as it has been like this for the github case, so I won't break this convention by recognizing ADO env vars.

On the other hand, the provider will detect those env vars, including SYSTEM_OIDCREQUESTURI for the ADO case, and specify it during constructing the auth structure when using this library. So it provides a same experience as azapi, from user's perspective.

[jaredfholgate]

@jaredfholgate The library won't auto-detect those environment variables by naming convention, as it has been like this for the github case, so I won't break this convention by recognizing ADO env vars.

On the other hand, the provider will detect those env vars, including SYSTEM_OIDCREQUESTURI for the ADO case, and specify it during constructing the auth structure when using this library. So it provides a same experience as azapi, from user's perspective.

Got it, thanks. Just wanted to ensure the experience for each provider is the same. I'm really pleased this is getting implemented it will make a huge difference for our customers.

[jackofallops]

Thanks @magodo - This looks good to me. Just one comment to be mindful of as we carry this forward into the provider(s).

As mentioned offline, due to the nature of this change and some logistical factors the team has, I'm going to have to hold off merging this for the time being, but we'll get it in so we can progress it and make it available asap.

Thanks again.

[jackofallops]

Assuming we take a similar approach when plugging this into the provider, we're going to need to be careful not to make assumptions about which value the user intends to be consumed of these two if both exist on the host.

[magodo]

In case either of these have both env vars available,do you think it makes sense to just error out?

Cc @ms-henglu for azapi.

[jackofallops]

I suspect that we should document the priority of selection in the provider configuration documentation and leave it to the user? There may be cases where users need/want to have both in their environment?

[magodo]

Sure, we'll need to document the environment variable selection order for the users and let them make the call. Whilst, I don't know if there is a case that both environment variables will be available.