K8s authenticator fails to authenticate using Deployment name for recent K8s versions

[diverdane]

Summary

The Kubernetes authenticator plugin is failing to authenticate and results in 401 errors when:

• A deployment name is being used as a Kubernetes authn identity
• Kubernetes version that the app is running on is recent (v1.16 or newer)
• (This may not be an absolute requirement) Follower is running externally to the cluster that the app is on

This failure was reported by a DAP customer.

When follower debugs are enabled with the configuration described above, the following error is seen in the follower logs:

<15>1 2020-03-16T14:58:27.000+00:00 dap-access-node-66f8df9946-qg2k4 conjur-possum 1277 - [meta sequenceId="12"] [origin=192.168.247.106] [request_id=27e5a551-2584-406d-bb70-ad1fae84cee6] [tid=1477] Authentication Error: #<Authentication::AuthnK8s::K8sObjectLookup::K8sForbiddenError: forbidden: User "system:serviceaccount:dap:conjur-cluster" cannot get path "/oapi/v1">

Full log is here: https://cyberark.lightning.force.com/lightning/r/ContentDocument/0692J000008icAbQAI/view

What's curious is that the issue was reported in a non-OpenShift environment, but the API endpoint being queried is an OpenShift specific endpoint.

Looking at the code where this error is happening, you can see some kubernetes clients being created in the K8s authenticator:

conjur/app/domain/authentication/authn_k8s/k8s_object_lookup.rb

Lines 150 to 163 in 1278b66

	# If more API versions appear, add them here.
	# List them in the order that you want them to be searched for methods.
	def k8s_clients
	@clients ||= [
	kubectl_client,
	KubeClientFactory.client(api: 'apis/apps', version: 'v1beta2', host_url: api_url, options: options),
	KubeClientFactory.client(api: 'apis/apps', version: 'v1beta1', host_url: api_url, options: options),
	KubeClientFactory.client(api: 'apis/extensions', version: 'v1beta1', host_url: api_url, options: options),
	# OpenShift 3.3 DeploymentConfig
	KubeClientFactory.client(api: 'oapi', version: 'v1', host_url: api_url, options: options),
	# OpenShift 3.7 DeploymentConfig
	KubeClientFactory.client(api: 'apis/apps.openshift.io', version: 'v1', host_url: api_url, options: options)
	]
	end

       # If more API versions appear, add them here.
      # List them in the order that you want them to be searched for methods.
      def k8s_clients
        @clients ||= [
          kubectl_client,
          KubeClientFactory.client(api: 'apis/apps', version: 'v1beta2', host_url: api_url, options: options),
          KubeClientFactory.client(api: 'apis/apps', version: 'v1beta1', host_url: api_url, options: options),
          KubeClientFactory.client(api: 'apis/extensions', version: 'v1beta1', host_url: api_url, options: options),
          # OpenShift 3.3 DeploymentConfig
          KubeClientFactory.client(api: 'oapi', version: 'v1', host_url: api_url, options: options),
          # OpenShift 3.7 DeploymentConfig
          KubeClientFactory.client(api: 'apis/apps.openshift.io', version: 'v1', host_url: api_url, options: options)
        ]
      end

When an authentication request comes in, this list is scanned for the first client that is capable of handling a get of the kubernetes resource of interest (in this case, a deployment). For a deployment resource, it's expected that the kubectl_client baseline client, or the v1beta1 client should be found as capable of handling a get of deployment, so the search for appropriate client should not reach a check of the oapi client version.

It should be noted that in Kubernetes v1.16, support of deployment resources using the deprecated v1beta1 API version was dropped in the server code.

So it's very likely that a more explicit v1 version with some appropriate api endpoints needs to be added to support Kubernetes clusters that are recent (v1.16 or newer, when deployment API was deprecated for v1beta1).

Steps to Reproduce

See description above.

Expected Results

Authenication should happen. No 401 errors.

Actual Results (including error logs, if applicable)

A clear and concise description of what actually did happen.

Additional Information

Add any other context about the problem here.