Provide a easy way to add a custom CA #7572

klausenbusk · 2021-10-29T08:40:20Z

Summary

Argo supports adding a custom TLS certificate per repository, but it isn't very scalable if you ex: want to MITM all traffic.

Motivation

In our case we want to MITM all traffic with a proxy and we don't necessarily know all server names up front. Adding a TLS certificate per server name is cumbersome and the only alternative seems to BYOI (add the certificate and run update-ca-certificates).

Proposal

We could perhaps use the existing infrastructure and support a "default certificate" in:

argo-cd/util/cert/cert.go

Lines 321 to 334 in b37eee1

    
           // Gets the full path for a certificate bundle configured from a ConfigMap 
        
           // mount. This function makes sure that the path returned actually contain 
        
           // at least one valid certificate, and no invalid data. 
        
           func GetCertBundlePathForRepository(serverName string) (string, error) { 
        
           	certPath := fmt.Sprintf("%s/%s", GetTLSCertificateDataPath(), ServerNameWithoutPort(serverName)) 
        
           	certs, err := GetCertificateForConnect(serverName) 
        
           	if err != nil { 
        
           		return "", nil 
        
           	} 
        
           	if len(certs) == 0 { 
        
           		return "", nil 
        
           	} 
        
           	return certPath, nil 
        
           }

The text was updated successfully, but these errors were encountered:

jgallucci32 · 2022-03-02T21:01:22Z

This can be achieved by creating a Kubernetes secret and adding it as volume mounts to the argo-cd pods. Since ArgoCD is written in Go it will automatically look for the custom CA certs placed in /etc/ssl/certs/. Here are the steps:

Create a Kubernetes secret called custom-ca-certificates using your custom CA bundle:

kubectl create secret generic custom-ca-certificates --from-file=custom-ca-certificates.crt=/path/to/custom/bundle.crt

Add volume and volumeMount parameters to the Helm values.yaml answer file:

  controller:
    volumeMounts:
      - name: custom-ca-certificates
        mountPath: /etc/ssl/certs/custom-ca-certificates.crt
        subPath: custom-ca-certificates.crt
    volumes:
      - name: custom-ca-certificates
        secret:
          defaultMode: 420
          secretName: custom-ca-certificates
  repoServer:
    volumeMounts:
      - name: custom-ca-certificates
        mountPath: /etc/ssl/certs/custom-ca-certificates.crt
        subPath: custom-ca-certificates.crt
    volumes:
      - name: custom-ca-certificates
        secret:
          defaultMode: 420
          secretName: custom-ca-certificates
  server:
    volumeMounts:
      - name: custom-ca-certificates
        mountPath: /etc/ssl/certs/custom-ca-certificates.crt
        subPath: custom-ca-certificates.crt
    volumes:
      - name: custom-ca-certificates
        secret:
          defaultMode: 420
          secretName: custom-ca-certificates

Upgrade Helm chart and it now can use custom certificates from your SSL proxy or other sites with custom signed certificates.

Morriz · 2022-04-27T08:15:42Z

The operator has no such fields. Any insights?

WesselAtWork · 2023-02-27T02:27:54Z

I had multiple certs (root and intermediaries) so I mounted the certs on /etc/ssl/certs.
Sadly that overrides EVERYTHING on that.
Everything was fine till I needed to do helm charts. Our helm charts uses some upstream charts and while connecting to our main repo worked fine, connecting to the upstream failed.
Fixed it with:

initContainers:
  name: generate-certs
  image: docker.io/alpine/curl:latest
  command:
    - "/usr/sbin/update-ca-certificates"
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    runAsNonRoot: false
    allowPrivilegeEscalation: true
    readOnlyRootFilesystem: false
    capabilities:
      add:
      - ALL
  volumeMounts:
   - name: ssl-certs
      mountPath: /etc/ssl/certs
   - name: ssl-local
      mountPath: /usr/local/share/ca-certificates
  volumes:
   - name: ssl-certs
     emptyDir: {}
   - name: ssl-local
     emptyDir: {}

I put this init job on every application that required the certs:

Controller
ApplicationSet Controller
Server
Repo Server
Notification Server (unsure if this one is actually required)

I don't think you need the insanely privileged securityContext.
I was struggling getting it to work because I accidentally mounted /usr/share/ca-certificates which overode the /mozilla which was were all the default certs were and update-ca-certificates --fresh wasn't working the way I thought it was.
If you can do it with less privileges (or specific privileges) I'd love to see them.

lknite · 2023-10-03T17:16:49Z

I've been using the work around to use a mount to overwrite /etc/ssl/certs but this is not a full solution. Overwriting /etc/ssl/certs with my onprem certs means that some certs normally verified with the default /etc/ssl/certs were are no longer verified, such as dev.azure.com in my situation. I can add them manually but eventually they expire.

In the past I have copied the ca-certificates bundle which comes with linux by default, appended my onprem certs, then added this to a configmap or secret, but it turned out the file was too big so this work-around does not always work.

We need a way to specify onprem certs that will then be added using whatever method the source argocd image uses such as 'update-ca-trust' or 'update-ca-certificates' after adding to an import directory.

lknite · 2023-10-04T17:52:52Z

I wrote up a way to implement this at another site, maybe we could use the same technique:
bitnami/charts#19744

global:
  # specify by providing the name of a secret and use all data as .crt files
  ca-bundle-secret: ca-bundle
  # or specify directly
  ca-bundle: |
    -----BEGIN CERTIFICATE-----
    MIIFrDCCBJSgAwIBAgIQCfluwpVVXyR0nq8eXc7UnTANBgkqhkiG9w0BAQwFADBh
    MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIFrDCCBJSgAwIBAgIQCfluwpVVXyR0nq8eXc7UnTANBgkqhkiG9w0BAQwFADBh
    MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
    -----END CERTIFICATE-----

lknite · 2023-10-04T21:24:18Z

Summary

Here's a pvc-based workaround inspired by @WesselAtWork (needed to get around size limitations of a configmap / secret, otherwise we could just mount one of those):

mount one or more certs from a ca-bundle secret to the /tmp/source folder
loop through certs and split into multiple cert files as needed, save to /usr/local/share/ca-certificates
run update-ca-certificates to import the additional certs
copy the /etc/ssl/certs folder contents to /tmp/target (which is the pvc)
mount the pvc to /etc/ssl/certs in the pod

Requirements

RWX required OR:
- One pod mounts pvc as RWO and performs the initialization
  - All other pods mount a pvc as ROX (ReadOnlyMany) with 1st pvc as datasource
- Or each pod can have its own rwo pvc
HA deployments may need additional considerations

Recommend

Update ca-bundle secret using something like external secrets, this way if the bundle is updated, pods will be updated at next start (or when skater restarts them).

Additional

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ca-bundle-server
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ca-bundle-reposerver
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Mi

initContainers

  server:
    initContainers:
    - name: get-ca-bundle
      image: docker.io/alpine/curl:latest
      command:
      - "/bin/sh"
      - "-c"
      args:
      - |
        for FILE in /tmp/source/*; do awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "/usr/local/share/ca-certificates/cert." c ".crt"}' < $FILE; done
        /usr/sbin/update-ca-certificates
        rm -rf /tmp/target/*
        cp -r /etc/ssl/certs/* /tmp/target/
      volumeMounts:
      - name: ca-bundle
        mountPath: /tmp/source
      - name: pvc-ca-bundle
        mountPath: /tmp/target
    volumes:
    - name: ca-bundle
      secret:
        secretName: ca-bundle
    - name: pvc-ca-bundle
      persistentVolumeClaim:
        claimName: ca-bundle-server

    volumeMounts:
    - name: pvc-ca-bundle
      mountPath: "/etc/ssl/certs"

  repoServer:
    initContainers:
    - name: get-ca-bundle
      image: docker.io/alpine/curl:latest
      command:
      - "/bin/sh"
      - "-c"
      args:
      - |
        for FILE in /tmp/source/*; do awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "/usr/local/share/ca-certificates/cert." c ".crt"}' < $FILE; done
        /usr/sbin/update-ca-certificates
        rm -rf /tmp/target/*
        cp -r /etc/ssl/certs/* /tmp/target/
      volumeMounts:
      - name: ca-bundle
        mountPath: /tmp/source
      - name: pvc-ca-bundle
        mountPath: /tmp/target
    volumes:
    - name: ca-bundle
      secret:
        secretName: ca-bundle
    - name: pvc-ca-bundle
      persistentVolumeClaim:
        claimName: ca-bundle-reposerver

    volumeMounts:
    - name: pvc-ca-bundle
      mountPath: "/etc/ssl/certs"

lknite · 2023-10-04T23:17:09Z

The idea to just mount a configmap /secret should not be considered a solution. This works in most cases but requires a person to start googling until they find out the right location to put the file vs just searching for 'bundle' or 'cert' in the values.yaml and being done. Why not just let the user specify a secret to use and put it where it needs to go ... the consumer shouldn't have to know what language or image file is being used to get the onprem certs installed.

hamelg · 2023-10-05T16:58:47Z

Another workaround with the operator by specifying volumes and volumeMounts in the crd argocds.argoproj.io, ca certificats are in the configmap ca-certs :

  repo:
    volumeMounts:
      - mountPath: /etc/ssl/certs/ca-bundle-private.crt
        name: ca-certs
        subPath: ca-bundle.crt
    volumes:
      - configMap:
          name: ca-certs
        name: ca-certs

jmanuelortizn · 2023-10-20T15:04:16Z

I am facing the same issue when pulling docker images from a custom private registry with self signed cert.

I tried the proposed solution with mounted volumes but the same issue is still happening: (I am able to see the cusom certs in /etc/ssl/certs/* and also run update-ca-certificates)

Failed to pull image "docker.pvt-registry.net/image-services:2.0.0.76-develop": 
rpc error: code = Unknown desc = failed to pull and unpack image "docker.pvt-registry.net/image-services:2.0.0.76-develop": 
failed to resolve reference "docker.pvt-registry.net/image-services:2.0.0.76-develop": 
failed to do request: Head "https://docker.pvt-registry.net/v2/image-service/manifests/2.0.0.76-develop": 
x509: certificate signed by unknown authority

Is there other work around that worked for anyone?

jle-pass · 2024-01-24T16:14:08Z

+1

We add the special key `_default_ca_fallback_` in argocd-tls-certs-cm that acts as a fallback. So when connecting to `server.example.com` we will look for the entry `server.example.com` and fall back to the entry `_default_ca_fallback_` if the former is missing. Closes: argoproj#7572 Signed-off-by: Michele Baldessari <[email protected]>

klausenbusk added the enhancement New feature or request label Oct 29, 2021

ShimiTaNaka mentioned this issue Jul 19, 2023

fix(argo-cd): Use ingress crt for dex when enabled argoproj/argo-helm#2172

Closed

6 tasks

lknite mentioned this issue Oct 4, 2023

global ca-bundle bitnami/charts#19744

Open

mbaldessari linked a pull request Mar 13, 2024 that will close this issue

feat: Implement default CA cert fallback in argocd-tls-certs-cm #17503

Open

14 tasks

rassie mentioned this issue Jun 12, 2024

Create a companion container for CA certificate handling adoptium/containers#573

Open

ngealy mentioned this issue Oct 18, 2024

Keycloak integration fails with the error "x509: certificate signed by unknown authority" argoproj/argo-helm#1374

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide a easy way to add a custom CA #7572

Provide a easy way to add a custom CA #7572

klausenbusk commented Oct 29, 2021

jgallucci32 commented Mar 2, 2022

Morriz commented Apr 27, 2022

WesselAtWork commented Feb 27, 2023

lknite commented Oct 3, 2023 •

edited

Loading

lknite commented Oct 4, 2023 •

edited

Loading

lknite commented Oct 4, 2023 •

edited

Loading

lknite commented Oct 4, 2023

hamelg commented Oct 5, 2023

jmanuelortizn commented Oct 20, 2023 •

edited

Loading

jle-pass commented Jan 24, 2024

Provide a easy way to add a custom CA #7572

Provide a easy way to add a custom CA #7572

Comments

klausenbusk commented Oct 29, 2021

Summary

Motivation

Proposal

jgallucci32 commented Mar 2, 2022

Morriz commented Apr 27, 2022

WesselAtWork commented Feb 27, 2023

lknite commented Oct 3, 2023 • edited Loading

lknite commented Oct 4, 2023 • edited Loading

lknite commented Oct 4, 2023 • edited Loading

Summary

Requirements

Recommend

Additional

initContainers

lknite commented Oct 4, 2023

hamelg commented Oct 5, 2023

jmanuelortizn commented Oct 20, 2023 • edited Loading

jle-pass commented Jan 24, 2024

lknite commented Oct 3, 2023 •

edited

Loading

lknite commented Oct 4, 2023 •

edited

Loading

lknite commented Oct 4, 2023 •

edited

Loading

jmanuelortizn commented Oct 20, 2023 •

edited

Loading