[WIP] Add the ability to support authentication to AWS ECR

[bzon]

This fixes #539. In order for this to work, the fluxd container must have the credentials
to access AWS API via API Keys or IAM roles.

Sidecar integration approach:

• Run the ecr-sidecar container alongside the fluxd container (flux Pod)..

• The ecr-sidecar must use registry-ids and region flag values which are required when invoking the get token request.

• The ecr-sidecar container fetches the docker login token using AWS ECR SDK with a configurable interval using fetch-interval flag and save it in memory.

• The ecr-sidecar container exposes a /awsauth endpoint which returns the authentication in JSON format.

  {
    "auths": {
      "https://12345.dkr.ecr-use-east-1.amazonaws.com": {
        "auth": "QVdTOmVjcnBhc3N3b3JkCg=="
      },
      "https://602401143452.dkr.ecr.us-east-1.amazonaws.com": {
        "auth": "QVdTOnh4eHh4Cg=="
      }
    }
  }

• WIP: Now, when fluxd invoke the credsFor function we can add a new condition for AWS related docker repos:

  if strings.HasSuffix(host, "amazonaws.com") {
    sidecar := newAWSSidecar()
    if cred, err := sidecar.GetAWSLoginCreds(host); err == nil {
      return cred
    }
  }

  This is not tested yet..

Considerations:

Initially, I thought it would be nicer if GetAWSLoginCreds function can be executed as a new goroutine during the fluxd runtime. But it was taking so much effort for me to understand the end to end flow of this demon. In my opinion, adding a sidecar made it much simpler for me to integrate this feature. We may be able to do the same for Azure authentication in the future.

References:

• https://docs.aws.amazon.com/cli/latest/reference/ecr/get-authorization-token.html
• https://docs.aws.amazon.com/sdk-for-go/api/service/ecr/#ECR.GetAuthorizationTokenRequest
• https://github.com/bzon/ecr-k8s-secret-creator

[squaremo]

Thanks for the clear explanation of your thinking in the post. I like the sidecar approach, all in all. It means less special-case stuff in the "core" code.

Can we segregate the AWS-specific bits even more? I am thinking

• move registry/aws.go to its own package
• make the opt-in explicit in an argument to fluxd, so you have to provide e.g., the sidecar address, or at least switch it on, and that registers the special case for ".amazonaws.com".

One complication this brings up is that it adds another image to be built, etc.. I don't expect it to change all that often, but it does mean more release machinery. We could consider either putting the sidecar in its own repo with releases on their own cadence. (Then I guess we'd need to vendor it to get the client shim. Blargh)

[squaremo]

We could consider either putting the sidecar in its own repo

Actually I am inclined to just version this along with the flux image. That means that occasionally, a change here will have to wait for a new flux release; and, there will be a bunch of releases of the sidecar image that don't actually change anything. But .. 🤷‍♂️

[bzon]

@squaremo Should we have a sidecar package? So we can have aws, azure, etc here in the future. I also like the switch idea via flags maybe?

Btw, I need help on understanding how to test if flux uses the credentials fetched from the sidecar. I was able to verify that flux actually calls the ecr sidecar endpoint but it wasn't using it.

[bzon]

Actually I am inclined to just version this along with the flux image. That means that occasionally, a change here will have to wait for a new flux release; and, there will be a bunch of releases of the sidecar image that don't actually change anything. But .. 🤷‍♂️

I think it's better to put this in the same repo for now.

[bzon]

Closing this PR for now. I feel like my approach is not the best way to implement flux authentication to ECR. We should probably first observe how the kubelet deals with ECR authentication here https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/aws/aws_credentials.go and see if we can reuse some of package methods built in to flux instead of creating a sidecar.