Add Kubernetes Secret as a store format

[wmedlar]

First-party Kubernetes support would be a wonderful addition to sops, the ability to read from and write to secrets manifests. It shouldn't be too dissimilar to the YAML or JSON stores, with decrypted values base64'd and wrapped in a k8s.io/api/core/v1.Secret.

I wouldn't mind taking a stab at this, implementation seems straightforward enough, and it would be a good precursor to writing a custom sops-backed CRD a la Bitnami's SealedSecrets.

[autrilla]

Agreed, go for it! This is definitely something we want.

[wmedlar]

Fitting it into a store wasn't quite as trivial as I expected, so I'm looking into going full-on CRD route with this. @autrilla would you or other sops maintainers be willing to give feedback on a proposal for this feature before I dive too deep in?

[autrilla]

Sure, I’m willing to give feedback. What gave you trouble with the store approach?

On Thu 20. Dec 2018 at 21:46, Will Medlar ***@***.***> wrote: Fitting it into a store wasn't quite as trivial as I expected, so I'm looking into going full-on CRD route with this. @autrilla <https://github.com/autrilla> would you or other sops maintainers be willing to give feedback on a proposal for this feature before I dive too deep in? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#401 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJ-V6tCg0dpnvlPJ8RpY-YgO7E1KVHNks5u6_c8gaJpZM4Y6tEE> .

-- Thanks, Adrian Utrilla

[ajvb]

@wmedlar fyi this is coming, but in a different form, with sops publish (#473) - The idea is that you will be able to publish secrets from sops to k8s secrets (as well as other services like Vault or AWS Secrets Manager).

[mr-karan]

The sops publish looks good but it doesn't really conform to Gitops way of doing things.

All cluster config must be in declarative state including the secrets. If people use tools like fluxctl which internally does kubectl apply, we still cannot use sops with it, because the decryption key must be present inside the cluster and K8s needs to know that it's a "special secret" not a regular.

I guess CRD could still make sense along with sops publish too, if the maintainers think it's useful I can probably take a look at this.

[jvehent]

The idea is to have sops publish as part of the provisioning process to send secrets to k8s secrets. We still respect the git way of doing things because sops documents are stored in git, decrypted during provisioning and sent to k8s.

Of course, that's just one way of doing things. YMMV.

[hobti01]

What about an Admission Controller that publishes the declarative CRD to a Secret owned by the CRD?

Publish rules per namespace or cluster could be a separate CRD.

[jlongtine]

@hobti01 I've put some thought into this approach. I don't think it'd be too difficult to create a controller that takes something like:

apiVersion: "mozilla.org/v1"
kind: SOPSSecret
metadata:
  name: test_secret
spec:
  API_KEY: ENC[<encrypted_API_KEY>]
  PASSWORD: ENC[<encrypted_PASSWORD>]
  sops:
    kms:
    - <sops encryption info>
    lastmodified: '2019-04-22T19:52:23Z'
    mac: 
    unencrypted_suffix: _unencrypted
    version: 3.3.0

And created this:

apiVersion: v1
kind: Secret
metadata:
  name: test_secret
type: Opaque
data:
  API_KEY: <decrypted+base64ed_API_KEY>
  PASSWORD: <decrypted+base64ed_PASSWORD>

But, I haven't had time to work on implementing it yet. Anyone done anything around this?

[wmedlar]

@jlongtine I played around with the idea a few months back and got a proof-of-concept controller working with GCP KMS. The hard part is integrating a k8s controller in a way that doesn't diminish the user experience of sops.

Unfortunately I haven't had time to work on it since, but I can share the knowledge I gained if you'd like to pursue it.

[cyphermaster]

FYI
There is a way to use sops in order to manage secrets : with kustomize-sops

early stage, but looks promising

[skang0601]

I've had a lot of success just using helm-secrets: https://github.com/futuresimple/helm-secrets

I don't completely get what the controller architecture would achieve/solve at the moment however. Unless you want the target cluster to be the only entity that can decrypt the payload, but that seems to contradict the current workflow of SOPS.

The base64 encoding of the secret values isn't necessary as well since the k8 Secrets object accepts StringData when writing.

[gitirabassi]

I don't see why a CRD is usefull. More abstractions aren't needed here I think. I don't even undestand why a publish to K8S would be helpful, that could be subsituted by a sops -d <enc_file> | kubectl apply -f - in whichever CI/CD tool you're using. I feel that the only thing needed here is that the values from keys apiVersion, kind,type and metadata to not be encrypted. and that would need unencrypted_suffix to support []string instead string. Am I right?

[hobti01]

A CRD is useful because of all the reasons for SealedSecrets

• do not execute sops in the pipeline
• do not give pipeline access to decrypt all secrets, only the cluster controller can decrypt
• declarative configuration

For us, a nice workflow with SOPS is:

• person uses sops -crd MySecretData.yaml to
  • create/edit secret values
  • encrypt with specified keys, including cluster key(s)
  • store as CRD
• pipeline/gitops deploys CRD to cluster
• admission controller or operator decrypts CRD using cluster key and creates Secret

Unfortunately we can't spare the cycles to contribute to this effort at the moment, but we really like sops and will keep watching :)

[devstein]

Related project for those that end up here: https://github.com/viaduct-ai/kustomize-sops . A good CRD-less approach

[ryedin]

Also related to this thread, fluxcd does now natively support sops content: fluxcd/flux#2580

[ghost]

Just came to this topic due to actual need as well.

Any proposal done here seems to be focused on actually decrypting data and then storing them as secrets. However, that (from my perspective) contradicts the actual security goals as we all know that the secrets are not stored encrypted inside. Although that can be changed implementation-wise, it provides a false sense of security for beginners.

So, what I was looking forward to is some kind of interceptor/hook that allows me to mutate a Secret or ConfigMap while being mounted to a container (or just before container entrypoint/command execution), so that the decrypted data is mounted, but not stored anywhere and executed in the context of the Pod/Service.

That way, not even the k8s cluster itself needs to have access to decryption keys, only the Pods that want to mount the encrypted data need to have an appropriate role or otherwise access to key data.

Maybe there already is something like that but if so, I missed it.

[ryedin]

@enote-kane all very valid concerns. the system that you're looking for that does all those things is Hashicorp's Vault. That said, what I mean by "those things" is the general idea of securely storing and distributing secrets at runtime (not specifically doing anything with k8s secrets resources).

However, there are also a couple things here that might be misconceptions...

That way, not even the k8s cluster itself needs to have access to decryption keys, only the Pods that want to mount the encrypted data need to have an appropriate role or otherwise access to key data.

The way sops works, (e.g. integration with a KMS provider) means no one ever gets any decryption keys. They are never distributed. The KMS service does all the decryption and never leaks the keys. When you create the service principal or IAM user that has encrypt/decrypt privileges against that KMS bucket, sops authenticates with the provider as that user, sends the encrypted file to the service over https, and receives the decrypted payload. No keys are ever leaked. So that's one thing.

The other thing is just about how k8s secrets work in general.

Firstly, they are provided to your pods in a safe manner using tmpfs (ramdisk):
Secrets are only provided to nodes that have a scheduled pod that requires it, and the Secret is stored in tmpfs, not written to disk. When the last pod on a node that requires a Secret is deleted, the Secret is deleted from the node's tmpfs. Secrets are stored within a given namespace and can only be accessed by pods within the same namespace.

Secondly, depending on which k8s service you're using, it's highly likely these days that your cluster has encryption-at-rest enabled by default (unless you've rolled your own cluster), so the cluster-level storage of the secrets is very likely fine. For example if you're in Azure's AKS, the etcd disks are always encrypted at rest with no option to create a cluster that has that feature disabled.

Finally:
it provides a false sense of security for beginners - absolutely none of this is for beginners xD.

[ryedin]

Oh, and give flux a good look. It works very nicely and allows you to safely store your encrypted k8s secrets files in your repo. It's really the perfect balance between safety and convenience IMO. Once you need more robustness you pretty much have one place to go next (Hashicorp Vault) and that comes with all kinds of additional infrastructure and required knowledge to manage (it's great but it has a very real cost to it)

[ghost]

@ryedin Thank you a lot for clearing things up for me here.

I'll have a deeper look into the existing solutions then to get a better understanding where exactly they hook in.

[ghost]

After taking a deep dive on this matter again, I am using the following approach now, which doesn't require to trust the secrets service or other parts of the system provided by the cloud provider, except for the memory of the node system:

apiVersion: v1
kind: Pod
metadata:
  ...
spec:
  securityContext:
    runAsUser: 12345
    runAsGroup: 12345
    fsGroup: 12345
    allowPrivilegeEscalation: false

  initContainers:
  - name: secrets-preprocessor
    image: ...
    env:
    - name: AWS_REGION
      value: eu-central-1
    - name: AWS_ROLE_ARN
      value: ...arn...
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    command: sops
    args: --decrypt --output /decrypted/config.json /encrypted/config.json
    volumeMounts:
    - name: service-config
      mountPath: /encrypted
      readOnly: true
    - name: secrets-data
      mountPath: /decrypted
      readOnly: false
    - name: app-kms-token
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readOnly: true
    - name: aws-iam-token
      mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      readOnly: true

  containers:
  - name: ...
    image: ...
    command: ...
    args: ...
    volumeMounts:
    - name: secrets-data
      mountPath: /app/etc
      readOnly: true

  volumes:
  - name: service-config
    secret:
      secretName: my-sops-encrypted-config
  - name: secrets-data
    emptyDir:
      medium: Memory
      sizeLimit: 10Mi
  - name: app-kms-token
    secret:
      defaultMode: 420
      secretName: app-kms-token
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token

This way, we can also eliminate some of the risks as outlined in the official documentation, mainly:

• A user who can create a Pod that uses a secret can also see the value of that secret. Even if the API server policy does not allow that user to read the Secret, the user could run a Pod which exposes the secret.
• Currently, anyone with root permission on any node can read any secret from the API server, by impersonating the kubelet. ...

Of course, protecting access to the key (or service accounts in my case) may impose another issue.

However, the biggest benefits for me on this approach are:

• no additional tooling required
• no modifications to/integrations with the CD required
• CD does not need permission to decrypt
• no additional services/CRD's/controllers inside Kubernetes
• fully transparent support of what sops offers
• the additional init-container and service accounts can be injected using standard kustomize patches to implement them as a cross-cutting concern at the CD-level

[IronCore864]

Can we revisit this after almost 6 years? I think this could be one of SOPS's best applications. You can encrypt K8s secrets YAMLs and share them across teams, everybody can see the structure of the data, the metadata, the keys, but not the values. At the moment, if SOPS is used in a situation like this, it will also encrypt apiVersion, kind, and metadata, making it impossible to read.

[devstein]

Hey @IronCore864 have you tried customizing your .sops.yaml to not encrypted those field? This is how we recommend users support K8s secrets with KSOPS

# .sops.yaml
creation_rules:
  - unencrypted_regex: "^(apiVersion|metadata|kind|type)$"