KSOPS Issue With Strategic Merge Patch In ArgoCD #134
Comments
|
Hi @evercast-chris thanks for making an issue. Can you paste the output |
|
|
sorry its a long manifest, but thats what I have after patching to repo-server @devstein |
|
@evercast-chris The patch is missing # 4. Set the XDG_CONFIG_HOME env variable to allow kustomize to detect the plugin
env:
- name: XDG_CONFIG_HOME
value: /.configLet me know if you run into any more errors. Just comment the error here. |
|
@devstein I think we're on the right track. I added the env variable but am still getting an error from argocd when creating an application. Here is the information below. Here's what the ArgoCD error looks like:
Also, here is what it looks like looks like in the argo-repo-server Here is what the env variable values are: |
|
@evercast-chris Based on those environmnet variables you shared, they aren't getting set properly in Argo CD. argocd@argocd-repo-server-86f84b7775-6kr58:/$ echo $XDG_CONFIG_HOME
/.configI recommend re-reviewing the repo server patch in the README. |
|
@devstein I changed the env variable but still getting the same result. I would like to try to re-patch but im struggling to find a command that actually successfully patches this to the deployment file. The closest command I can find is the follow --> ``` |
|
@devstein as you can see the patch has no changes but its a different patch i am trying why am I getting no changes? This has happend often in the process. The patch I am currently trying again to patch is the following... |
|
@evercast-chris The patch is incompatible with # argo-cd-repo-server-ksops-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
# 1. Define an emptyDir volume which will hold the custom binaries
volumes:
- name: custom-tools
emptyDir: {}
# 2. Use an init container to download/copy custom binaries into the emptyDir
initContainers:
- name: install-ksops
image: viaductoss/ksops:v3.0.1
command: ["/bin/sh", "-c"]
args:
- echo "Installing KSOPS...";
mv ksops /custom-tools/;
mv $GOPATH/bin/kustomize /custom-tools/;
echo "Done.";
volumeMounts:
- mountPath: /custom-tools
name: custom-tools
# 3. Volume mount the custom binary to the bin directory (overriding the existing version)
containers:
- name: argocd-repo-server
volumeMounts:
- mountPath: /usr/local/bin/kustomize
name: custom-tools
subPath: kustomize
# Verify this matches a XDG_CONFIG_HOME=/.config env variable
- mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
name: custom-tools
subPath: ksops
# 4. Set the XDG_CONFIG_HOME env variable to allow kustomize to detect the plugin
env:
- name: XDG_CONFIG_HOME
value: /.config
## If you use AWS or GCP KMS, don't forget to include the necessary credentials to decrypt the secrets!
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: argo
key: aws_access_key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: argo
key: aws_access_secret_keyNotice the difference in |
|
Thank you for taking a look @devstein im going to try this on my primary argocd environment so I have a clean deployment to work with. I want to confirm if I should be using the command |
Yes
I think it should work, but I always use |
|
@devstein Ok cool I think when it comes to kustomize to patch this file, I don't have the ArgoCD manifests in a repo anywhere its just in the cluster under the ArgoCD namespace. Is the kustomize command for patching using the strategicmergepatch in a kustomization.yaml or is there actually an actual command for patch, such as there is for the build command? |
|
Got it, if this is a one-off operation and you want to sanity check the results with
kubectl -n argocd get deployment/argocd-repo-server -o yaml > base.yaml
resources:
- ./base.yaml
patches:
- ./argo-cd-repo-server-ksops-patch.yaml
kustomize build --enable-alpha-plugins ./ | kubectl apply -f - |
|
Hey @devstein thanks for the explanation its very helpful. Im getting an error trying to clarify what the patches part of the this kustomization.yaml file is doing here. It looks like im putting the output for argo-repo-server into base.yaml locally. So, im trying to confirm that I am patching the new patch and not the same patch over base.yaml, should I be patching the argo-cd-repo-server-ksops-patch.yaml instead of is that correct? |
|
@evercast-chris Right! |
|
@devstein Awesome it built, I just needed to tweak the kustomization.yaml a bit with However, it is not allowing me to create the file. Im getting the following error: I tried using the validate=false command as suggested but also received this message: Researching, this might seem like a syntax error somewhere but not sure. What would you recommend at this point? |
|
Hey @evercast-chris you are right this is a syntax error in the |
|
@devstein ok the the base.yaml checks out as valid yaml. The only error I have seen with it is the resources here is the section of the yaml. It states in VS Code no resource limits specified for this container - this could starve other process. |
|
@devstein here is the base.yaml file I am using... |
|
@devstein any suggestions on this? I think I've tried just about everything to get this init container to patch correctly. Thanks. |
|
Hey @evercast-chris. I assume you have already, but if you haven't tried from a fresh, non-running deployment of Argo CD I would recommend starting there. This is what the patch in the README is intended for. The only alternative would be to pursue the custom docker image approach instead. |
|
@devstein thanks dev, i did attempt to patch this to an unedited repo-server file but I keep getting the no changes patched command in the terminal. I did try the Docker image once but wasn't able to successfully build the image, are there more detailed steps you can provide other than just the image? I wish there was more information out there on ksops, like a demo, some documentation with instructions on how to use the image correctly with argocd, etc. |
|
Sorry to hear @evercast-chris -- for the image, once it's successfully built you can use it in the Adding more links to examples, or including an example in the repo is a good idea. Here are two links to examples that I know of that could be helpful |
|
@devstein Thanks Dev, im sure im missing something on my end. I think im going to give building the image another try. Awesome thank you for the additional resources this is helpful. I appreciate the time! |
|
@devstein you were right the patch did work on a brand new argocd environment, which is definitely progress thanks! I am running ksops locally and it seems to work fine with the secret.yaml I am testing. However, I am getting an error from ArgoCD, im not exactly sure what the error is saying, but I believe I may have the secret-generator in the wrong folder, I am following the references you sent me and locally everything is good. Any clue on how what this error might be saying?
|
|
Glad you made progress @evercast-chris! That error is hard to parse, but looking this message Does Argo CD have access to the private keys to decrypt this secret? If you are using a PGP key, I recommend looking at this previous issue #24 for tips |
|
@devstein actually this one doesn't since its a new argocd cluster, thank you for the reminder! I'll keep you posted on the results it might just be the missing piece needed. |
|
@devstein still troubleshooting I did find this error in the running pod for argo-repo-server and it's definitely not able to find the gpg-keys. I don't have this mounted path. Im wondering if you have seen this issue before? I tried to place the gpg keys in argocd but its kinda not that simple since it requires an ASCII-armored. Anyways, wondering im wondering if there is an efficient way to correct this? |
|
Hey @evercast-chris I have not; however, if you haven't I would try referencing #24 and https://github.com/james-callahan/example-gitops/tree/master/argocd for potential solutions |
|
@devstein I have to be getting pretty close but im wondering if you have seen this error before? If so do you happen to know what it might mean? It is longer than this but don't want to overwhelm. Thank you! |
|
@evercast-chris I have not, but it looks like it's not specific to KSOPS. Want to share your |
|
@devstein yes this does look more like a sops error than a ksops issue. Do I need include a secret-generator and a secret ksops yaml file as well? This is the kustomization file I have in my overlay. Secret.enc.yml is my encrypted file. |
|
Yes you do. Try this apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: evercast-qa
bases:
- ../../../base
patches:
- ingress.yml
- image.yml
generators:
- secret-generator.yamlwhere apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: secret-generator
files:
- ./secret.enc.yml |
|
@devstein ok cool, I also have this secret generator setup in my base kustomization as well. One secret-generator.yaml in base and one in the overlay of choice? I also think my aws kms permission in my .sops yaml file might be making things more complicated than needed to be. |
|
@evercast-chris Oh, in that case what is the goal of referencing resources:
- secret.enc.ymlin the overlay? |
|
@devstein so when im creating an application on argo, as long as I reference the secret.enc.yaml from my base folder in kustomization.yaml it will recognize that there is an encrypted secret. However it will wont fully sync the argocd application. I thought perhaps it is because the secret I am referencing in my base folder is not in my cluster. Then, I did a kubectl apply -f of the secret.enc.yaml in my cluster argocd is deployed on, but still cannot get the secret to fully sync in the argocd application. The only time I could get a fully sync on a secret manifest is when it was unencrypted. Sooo, my guess was to apply the secret in the overlay and maybe argo will fully recognize it, but unfortunately it gives a long error on argocd and it also does not kustomize build locally from the overlay file. |
|
@evercast-chris You never want to directly reference a SOPS encrypted secrets file in In general, secrets are often overlay specific, so it typically makes sense to use KSOPS in the overlay. |
|
@devstein this is great to know, I was actually following the documentation at https://dev.to/stack-labs/gitops-demo-with-argo-cd-and-ksops-on-gke-2a0l where the ksops secret and generator are in the base folder. Is there where it should be stored or only in the overlay folder which I am trying to create the argocd application for, or both? Sorry for all the questions, I appreciate the patience. |
Hello, I have been trying to connect KSOPS with ArgoCD for a while now. I seem to not be having luck with the
kustomize.buildOptions: "--enable-alpha-plugins"command in the argo-cd configmap via the strategic merge patch with init containers. KSOPS works fine locally and the patch on the argo-cd-configmap is done correctly as well but still nothing.I recently also tried using the ksops 2.5 image version and changing the flag to
--enable_alpha_plugins. image:viaductoss/ksops:v3.0.1 to-->image: viaductoss/ksops:v2.5.0inside the repo sever patch. Still it does not sync with ArgoCD.Here are the error am receiving via ArgoCD. -->
Unable to create application: application spec is invalid: InvalidSpecError: Unable to generate manifests in base rpc error: code = Unknown desc = 'kustomize build' (my_git_repo) --enable_alpha_plugins' failed exit status 1: Error: unknown flag: --enable-alpha-pluginsAnother error from ArgoCD -->
unable to find root -tried ("; homed in $KUSTOMIZE_PLUGIN_HOME), ("kustomize/plugin'; homed in $XDG_CONFIG_HOME), ('/home/argocd/.config/kustomize/plugin'; homed in default value of $XDG_CONFIG_HOME)That is the patch command I am using...
Please let me know what I can do to finally get KSOPS working with Argo.
Thank you!
The text was updated successfully, but these errors were encountered: