Conjur OSS API server fails to GET binary values

[saprette]

Summary

Conjur OSS API server fails to GET binary values

Steps to Reproduce

Set a variable value to a pkcs12

root@conjur-admin-configure-geftdtqi38-c9cdc:/tmp# openssl pkcs12 -in truststore.p12 -nodes | openssl x509 -noout -enddate
Enter Import Password:
notAfter=Oct 27 14:53:44 2042 GMT
root@conjur-admin-configure-geftdtqi38-c9cdc:/tmp# conjur variable values add redacted/redacted/pki_truststore < truststore.p12 
Value added
root@conjur-admin-configure-geftdtqi38-c9cdc:/tmp# conjur variable value redacted/redacted/pki_truststore > test.p12
root@conjur-admin-configure-geftdtqi38-c9cdc:/tmp# openssl pkcs12 -in test.p12 -nodes | openssl x509 -noout -enddate
Enter Import Password:
notAfter=Oct 27 14:53:44 2042 GMT

Note that the CLI is able to add the variable to the pkcs12 value, and is also able to read it without breaking the file.

Use cyberark/secrets-provider-for-k8s:1.1.0 to update this secret

kind: Secret
apiVersion: v1
metadata:
  namespace: {{ .Values.cycle }}-{{ .Values.project }}
  name: pki-truststore
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/sync-wave: "-15"
stringData:
  conjur-map: |-
    pki-truststore: redacted/redacted/pki_truststore
type: Opaque

The job doing the update

apiVersion: batch/v1
kind: Job
metadata:
  namespace: {{ .Values.cycle }}-{{ .Values.project }}
  name: conjur-secret-provider
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    argocd.argoproj.io/sync-wave: "-10"
spec:
  activeDeadlineSeconds: 180
  template:
    spec:
      serviceAccountName: kubernetes-authenticator-client
      containers:
        - image: {{ .Values.conjur.secretsproviderfork8s.image | quote }}
          imagePullPolicy: IfNotPresent
          name: kubernetes-authenticator-client
          volumeMounts:
            - name: trusted-ca-bundle
              mountPath: /tmp/trusted-ca-bundle
          env:
            - name: DEBUG
              value: {{ .Values.conjur.debug | quote }}
            - name: CONTAINER_MODE
              value: init
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: CONJUR_VERSION
              valueFrom:
                configMapKeyRef:
                  name: config.conjur
                  key: version
            - name: CONJUR_APPLIANCE_URL
              valueFrom:
                configMapKeyRef:
                  name: config.conjur
                  key: applianceUrl
            - name: CONJUR_AUTHN_URL
              valueFrom:
                configMapKeyRef:
                  name: config.conjur
                  key: authenticationUrl
            - name: CONJUR_ACCOUNT
              valueFrom:
                configMapKeyRef:
                  name: config.conjur
                  key: account
            - name: CONJUR_AUTHN_LOGIN
              value: 'host/conjur/authn-k8s/authenticator/identities/redacted'
            - name: CONJUR_CERT_FILE
              value: '/tmp/trusted-ca-bundle/ca-bundle.crt'
            - name: K8S_SECRETS
              value: 'pki-truststore'
            - name: SECRETS_DESTINATION
              value: k8s_secrets
      volumes:
        - name: trusted-ca-bundle
          configMap:
            name: trusted-ca-bundle
      restartPolicy: Never

Logs of the secrets-provider-for-k8s job

INFO: 2020/11/28 13:08:36 k8s_secrets_client.go:19: CSPFK005I Retrieving Kubernetes secret 'pki-truststore' from namespace 'redacted'
DEBUG: 2020/11/28 13:08:36 provide_conjur_secrets.go:127: CSPFK009D Processing 'conjur-map' data entry value of Kubernetes Secret 'pki-truststore'
INFO: 2020/11/28 13:08:36 k8s_secrets_client.go:54: CSPFK004I Creating Kubernetes client
INFO: 2020/11/28 13:08:36 conjur_secrets_retriever.go:11: CSPFK003I Retrieving following secrets from DAP/Conjur: [redacted/redacted/pki_truststore]
INFO: 2020/11/28 13:08:36 conjur_client.go:21: CSPFK002I Creating DAP/Conjur client
ERROR: 2020/11/28 13:08:36 provide_conjur_secrets.go:77: CSPFK034E Failed to retrieve DAP/Conjur secrets. Reason: 500 Internal Server Error. 
ERROR: 2020/11/28 13:08:36 main.go:94: CSPFK016E Failed to provide DAP/Conjur secrets

Matching logs in conjur-oss

[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32] default:host:conjur/authn-k8s/authenticator/identities/redacted fetched default:variable:redacted/redacted/pki_truststore
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32] Completed 500 Internal Server Error in 18ms
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32]   
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32] Encoding::UndefinedConversionError ("\x82" from ASCII-8BIT to UTF-8):
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32]   
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32] app/controllers/secrets_controller.rb:73:in `batch'
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32] app/controllers/application_controller.rb:74:in `block in run_with_transaction'
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32] app/controllers/application_controller.rb:73:in `run_with_transaction'
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32] lib/rack/remove_request_parameters.rb:26:in `call'
[origin=10.128.2.70] [request_id=cd294a8e-8a69-47ea-a45a-3b192a24d077] [tid=32] lib/rack/default_content_type.rb:68:in `call'

Expected Results

conjur-oss does not have a UndefinedConversionError and secrets-provider-for-k8s is able to update my secret with the pkcs12 value

Reproducible

Always

Version/Tag number

repository: cyberark/conjur
tag: 'v1.11.0'

repository: nginx
tag: '1.15'

repository: postgres
tag: '10.12'

Environment setup

OpenShift cluster version 4.5.19.

Additional Information

Note that we use such setup to fetch OK many different kinds of secrets, including json documents and SSL private keys

[doodlesbykumbi]

Hi @saprette. Thank you for creating this issue. We really appreciate you taking the time to provide details on how to reproduce, it makes looking into the issue a lot easier.

I took a look and think I was able to narrow down the issue.

1. I noted that secrets-provider-for-k8s uses conjur-api-go to fetch secrets. Specifically it uses the batch retrieval endpoint, see https://github.com/cyberark/secrets-provider-for-k8s/blob/master/pkg/secrets/clients/conjur/conjur_secrets_retriever.go#L19
2. I tried to replicate the issue by writing a binary secret to Conjur OSS (openssl rand -hex 10 | xxd -r -p) and retrieve it using conjur-api-go. I noticed that RetrieveSecret works, but RetrieveBatchSecrets results in a 500 and the same error you shared Encoding::UndefinedConversionError.
3. I made a comparison of the endpoints, show (RetrieveSecret) and batch (RetrieveBatchSecrets). It seems that the show endpoint respects the resource mime_type and uses the Rails send_data method, while the batch endpoint puts the secrets into JSON. I think it is here where things fall apart. See https://github.com/cyberark/conjur/blob/master/app/controllers/secrets_controller.rb#L47-L73.

Here's what I'll do

• Schedule updates to the Conjur OSS README, and the README of any dependencies that consume the batch retrieval endpoint to inform readers of the limitations.
• Bring this up internally so we can talk about getting it scheduled.

If you would like to contribute the resolution of this bug, please allow me to point you to our contribution guide