Is it possible to use a cert-issuer for all certs, without ECK controlling ca?

[jonathon2nd]

Hello!

To preface I have read the following, which does not seem to cover what I want:
#5798
https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-custom-http-certificate.html#k8s_custom_self_signed_certificate_using_cert_manager
https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-tls-certificates.html#k8s-setting-up-your-own-certificate

What I would like to do is use cert-manager to provision all certs.

I was looking in the operator for some way to set the following, so that it can create certs using cert-manager

  issuerRef:
    group: certmanager.step.sm
    kind: StepClusterIssuer
    name: step-issuer

We have an internal PKI built out using smallstep, and what I would like to do is have the operator use the StepClusterIssuer that we have setup to request certs, and use the the ca.crt it gets from there. We have an internal issuing-ca that all of our other internal apps are using. I do not want it to have it's own ca that it uses to create certs, I would like it to request all the certs needed using the issuer.

Thanks in advance!

[jonathon2nd]

Reviewing this: https://www.elastic.co/guide/en/cloud-on-k8s/2.16/k8s-transport-settings.html
And looking at the certs created by default, I am seeing some issues.

The transport cert secret is multi-key, containing a key for each node. I looked here: https://www.elastic.co/guide/en/cloud-on-k8s/2.16/k8s-api-elasticsearch-k8s-elastic-co-v1.html#k8s-api-jackfan.us.kg-elastic-cloud-on-k8s-v2-pkg-apis-elasticsearch-v1-transporttlsoptions, to see if it was possible to make some changes to accommodate a separate cert and secret for each node, but not seeing anything.

[jonathon2nd]

Ok great after reviewing the docs more and a bunch of debugging I was able to get a setup fully working
The operator still creates extra things that say they are ca's and whatnot, but none of them are actually mounted and used. I have validated all endpoints for kibana, elastic, and agents to ensure they are all using the correct certs.

Posting the full yaml I am using here if it is every helpful. Going to leave this open, as it would be very useful if the operator was designed to use cert-manager for creating certs instead of this approach.

### This is the CA from the issuing CA server for us.
---
apiVersion: v1
kind: Secret
metadata:
  name: step-root-ca
  namespace: trust-manager
type: kubernetes.io/tls
data:
  tls.crt: >-
    EEE
  tls.key: ''

---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: elasticsearch-es-transport-ca-internal
  namespace: elastic
spec:
  sources:
    - secret:
        name: step-root-ca
        key: tls.crt
        namespace: elastic
  target:
    configMap:
      name: elasticsearch-es-transport-ca-internal
      key: ca.crt

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: elasticsearch-es-http-certs-internal
  namespace: elastic
spec:
  secretName: elasticsearch-es-http-certs-internal
  issuerRef:
    group: certmanager.step.sm
    kind: StepClusterIssuer
    name: step-issuer
  commonName: "elasticsearch-es-http.elastic.es.local"
  dnsNames:
    - "elasticsearch-es-http"
    - "elasticsearch-es-http.elastic"
    - "elasticsearch-es-http.elastic.svc"
    - "elasticsearch-es-http.elastic.es.local"
    - "elasticsearch-es-http.elastic.svc.cluster.local"
    - "*.elasticsearch-es-default.elastic.svc"
    - "elasticsearch-es-internal-http.elastic"
    - "elasticsearch-es-internal-http.elastic.svc"
  privateKey:
    algorithm: RSA
    size: 2048

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kibana-kb-http-certs-internal
  namespace: elastic
spec:
  secretName: kibana-kb-http-certs-internal
  issuerRef:
    group: certmanager.step.sm
    kind: StepClusterIssuer
    name: step-issuer
  commonName: "kibana-kb-http.elastic.svc"
  dnsNames:
    - "kibana-kb-http.elastic"
    - "kibana-kb-http.elastic.svc"
    - "kibana-kb-http.elastic.svc.cluster.local"
  privateKey:
    algorithm: RSA
    size: 2048

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: fleet-server-agent-http-certs-internal
  namespace: elastic
spec:
  secretName: fleet-server-agent-http-certs-internal
  issuerRef:
    group: certmanager.step.sm
    kind: StepClusterIssuer
    name: step-issuer
  commonName: "fleet-server-agent-http.elastic.svc"
  dnsNames:
    - "fleet-server-agent-http.elastic"
    - "fleet-server-agent-http.elastic.svc"
    - "fleet-server-agent-http.elastic.svc.cluster.local"
  privateKey:
    algorithm: RSA
    size: 2048

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elastic
spec:
  version: 8.15.0
  count: 1
  elasticsearchRef:
    name: elasticsearch
  http:
    tls:
      selfSignedCertificate:
        disabled: true
      certificate:
        secretName: kibana-kb-http-certs-internal
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.elastic.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.fpelastic.svc:8220"]
    xpack.fleet.packages:
    - name: system
      version: latest
    - name: elastic_agent
      version: latest
    - name: fleet_server
      version: latest
    - name: apm
      version: latest
    xpack.fleet.agentPolicies:
    - name: Fleet Server on ECK policy
      id: eck-fleet-server
      is_default_fleet_server: true
      namespace: elastic
      monitoring_enabled:
      - logs
      - metrics
      unenroll_timeout: 900
      package_policies:
      - name: fleet_server-1
        id: fleet_server-1
        package:
          name: fleet_server
    - name: Elastic Agent on ECK policy
      id: eck-agent
      namespace: elastic
      monitoring_enabled:
      - logs
      - metrics
      unenroll_timeout: 900
      is_default: true
      package_policies:
      - name: system-1
        id: system-1
        package:
          name: system
      - package:
          name: apm
        name: apm-1
        inputs:
        - type: apm
          enabled: true
          vars:
          - name: host
            value: 0.0.0.0:8200
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic
spec:
  version: 8.15.0
  transport:
    tls:
      selfSignedCertificates:
        disabled: true
      certificateAuthorities:
        configMapName: elasticsearch-es-transport-ca-internal
  http:
    tls:
      selfSignedCertificate:
        disabled: true
      certificate:
        secretName: elasticsearch-es-http-certs-internal
  secureSettings:
  - secretName: s3-secret
  nodeSets:
  - name: default
    config:
      node.roles: ["master","data","ingest","ml"]
      node.store.allow_mmap: false
      # Tell Elasticsearch where to find the transport key/cert (mounted by CSI):
      xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/cert-manager-csi/transport/tls.key
      xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/cert-manager-csi/transport/tls.crt
      xpack.security.transport.ssl.certificate_authorities: ["/usr/share/elasticsearch/config/cert-manager-csi/transport/ca.crt"]

    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          volumeMounts:
            - name: transport-certs
              mountPath: /usr/share/elasticsearch/config/cert-manager-csi/transport
          # specify resource limits and requests
          resources:
            requests:
              memory: 4Gi
              cpu: 1
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms2g -Xmx2g"
        volumes:
          - name: transport-certs
            csi:
              driver: csi.cert-manager.io
              readOnly: true
              volumeAttributes:
                csi.cert-manager.io/issuer-name: step-issuer
                csi.cert-manager.io/issuer-group: certmanager.step.sm
                csi.cert-manager.io/issuer-kind: StepClusterIssuer
                # Each Pod gets a unique transport cert based on its own Pod DNS
                csi.cert-manager.io/common-name: "${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local"
    count: 3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 500Gi
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: elastic
spec:
  version: 8.15.0
  kibanaRef:
    name: kibana
  elasticsearchRefs:
  - name: elasticsearch
  mode: fleet
  fleetServerEnabled: true
  http:
    tls:
      selfSignedCertificate:
        disabled: true
      certificate:
        secretName: fleet-server-agent-http-certs-internal
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: fleet-server
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
        - name: agent
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata: 
  name: elastic-agent
  namespace: elastic
spec:
  version: 8.15.0
  kibanaRef:
    name: kibana
  fleetServerRef: 
    name: fleet-server
  mode: fleet
  deployment:
    replicas: 1
    podTemplate:
      spec:
        securityContext:
          runAsUser: 0
        containers:
        - name: agent
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: apm
  namespace: elastic
spec:
  selector:
    agent.k8s.elastic.co/name: elastic-agent
  ports:
  - protocol: TCP
    port: 8200
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fleet-server
rules:
- apiGroups: [""]
  resources:
  - pods
  - namespaces
  - nodes
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fleet-server
  namespace: elastic
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fleet-server
subjects:
- kind: ServiceAccount
  name: fleet-server
  namespace: elastic
roleRef:
  kind: ClusterRole
  name: fleet-server
  apiGroup: rbac.authorization.k8s.io