kamus-cli - unclear error when trying to encrypt a secret for SA in default namespace

[itsvikasgupta]

I installed kamus using these steps:

$ key=$(openssl rand -base64 32 | tr -d '\n')
$ helm upgrade --install kamus soluto/kamus --set keyManager.AES.key=$key

Here are my pods:

NameSpace     POD_NAME                                     CONTAINER_NAME
kube-system   etcd-docker-for-desktop                      etcd
kube-system   kamus-decryptor-65fb5845b-qmzcj              decryptor-api
kube-system   kamus-decryptor-65fb5845b-xrcrq              decryptor-api
kube-system   kamus-encryptor-5fd59d766d-2qzqd             encryptor-api
kube-system   kamus-encryptor-5fd59d766d-brzht             encryptor-api
kube-system   kube-apiserver-docker-for-desktop            kube-apiserver
kube-system   kube-controller-manager-docker-for-desktop   kube-controller-manager
kube-system   kube-dns-86f4d74b45-fwgc4                    kubedns,dnsmasq,sidecar
kube-system   kube-proxy-zqhl8                             kube-proxy
kube-system   kube-scheduler-docker-for-desktop            kube-scheduler
kube-system   kubernetes-dashboard-669f9bbd46-65lhk        kubernetes-dashboard
kube-system   tiller-deploy-78c6868dd6-bkscs               tiller

Continuing the steps:

$ export POD_NAME=$(kubectl get pods --namespace kube-system -l "app=kamus,release=kamus,component=encryptor" -o jsonpath="{.items[0].metadata.name}")

$ kubectl port-forward $POD_NAME 8080:9999 &

$ kamus-cli encrypt --secret mysecret --service-account default --namespace kube-system --kamus-url http://localhost:8080 --allow-insecure-url

[info  kamus-cli]: Encryption started...
[info  kamus-cli]: service account: default
[info  kamus-cli]: namespace: kube-system
[warn  kamus-cli]: Auth options were not provided, will try to encrypt without authentication to kamus
Handling connection for 8080
[error kamus-cli]: Error while trying to encrypt with kamus: Encrypt request failed due to unexpected error. Status code: 400

Here's the relevant log from the first encryptor-api container:

{"Timestamp":"2019-03-27T18:54:08.5567116+00:00","Level":"Information","MessageTemplate":"Route matched with {RouteData}. Executing action {ActionName}","Properties":{"RouteData":"{action = \"Encrypt\", controller = \"Encrypt\"}","ActionName":"Kamus.Controllers.EncryptController.Encrypt (encrypt-api)","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"be18cad6-7833-48b3-83e2-a4e57653760c"}}
{"Timestamp":"2019-03-27T18:54:08.5575067+00:00","Level":"Information","MessageTemplate":"Executing action method {ActionName} with arguments ({Arguments}) - Validation state: {ValidationState}","Properties":{"ActionName":"Kamus.Controllers.EncryptController.Encrypt (encrypt-api)","Arguments":["Kamus.Models.EncryptRequest"],"ValidationState":"Valid","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"be18cad6-7833-48b3-83e2-a4e57653760c"}}
{"Timestamp":"2019-03-27T18:54:08.5577029+00:00","Level":"Information","MessageTemplate":"Encryption request started, SourceIP: {sourceIp}, ServiceAccount: {sa}, Namespace: {namespace}","Properties":{"sourceIp":"::ffff:127.0.0.1","sa":"default","namespace":"kube-system","log_type":"audit","SourceContext":"Kamus.Controllers.EncryptController","ActionId":"be18cad6-7833-48b3-83e2-a4e57653760c","ActionName":"Kamus.Controllers.EncryptController.Encrypt (encrypt-api)"}}
{"Timestamp":"2019-03-27T18:54:08.5584394+00:00","Level":"Information","MessageTemplate":"Executed action method {ActionName}, returned result {ActionResult} in {ElapsedMilliseconds}ms.","Properties":{"ActionName":"Kamus.Controllers.EncryptController.Encrypt (encrypt-api)","ActionResult":"Microsoft.AspNetCore.Mvc.BadRequestObjectResult","ElapsedMilliseconds":0.1087,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"be18cad6-7833-48b3-83e2-a4e57653760c"}}
{"Timestamp":"2019-03-27T18:54:08.5585630+00:00","Level":"Information","MessageTemplate":"Executing ObjectResult, writing value of type '{Type}'.","Properties":{"Type":"System.String","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor","ActionId":"be18cad6-7833-48b3-83e2-a4e57653760c","ActionName":"Kamus.Controllers.EncryptController.Encrypt (encrypt-api)"}}
{"Timestamp":"2019-03-27T18:54:08.5604943+00:00","Level":"Information","MessageTemplate":"Executed action {ActionName} in {ElapsedMilliseconds}ms","Properties":{"ActionName":"Kamus.Controllers.EncryptController.Encrypt (encrypt-api)","ElapsedMilliseconds":3.5635000000000003,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"be18cad6-7833-48b3-83e2-a4e57653760c"}}

{"Timestamp":"2019-03-27T18:54:09.9472619+00:00","Level":"Information","MessageTemplate":"Route matched with {RouteData}. Executing action {ActionName}","Properties":{"RouteData":"{action = \"IsAlive\", controller = \"Monitoring\"}","ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d"}}
{"Timestamp":"2019-03-27T18:54:09.9473877+00:00","Level":"Information","MessageTemplate":"Executing action method {ActionName} - Validation state: {ValidationState}","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ValidationState":"Valid","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d"}}
{"Timestamp":"2019-03-27T18:54:09.9474734+00:00","Level":"Information","MessageTemplate":"Executed action method {ActionName}, returned result {ActionResult} in {ElapsedMilliseconds}ms.","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ActionResult":"Microsoft.AspNetCore.Mvc.ObjectResult","ElapsedMilliseconds":0.0109,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d"}}
{"Timestamp":"2019-03-27T18:54:09.9475511+00:00","Level":"Information","MessageTemplate":"Executing ObjectResult, writing value of type '{Type}'.","Properties":{"Type":"System.Boolean","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d","ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)"}}
{"Timestamp":"2019-03-27T18:54:09.9482698+00:00","Level":"Information","MessageTemplate":"Executed action {ActionName} in {ElapsedMilliseconds}ms","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ElapsedMilliseconds":0.90570000000000006,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d"}}
{"Timestamp":"2019-03-27T18:54:11.0972374+00:00","Level":"Information","MessageTemplate":"Route matched with {RouteData}. Executing action {ActionName}","Properties":{"RouteData":"{action = \"IsAlive\", controller = \"Monitoring\"}","ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d"}}
{"Timestamp":"2019-03-27T18:54:11.0984310+00:00","Level":"Information","MessageTemplate":"Executing action method {ActionName} - Validation state: {ValidationState}","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ValidationState":"Valid","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d"}}
{"Timestamp":"2019-03-27T18:54:11.0994953+00:00","Level":"Information","MessageTemplate":"Executed action method {ActionName}, returned result {ActionResult} in {ElapsedMilliseconds}ms.","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ActionResult":"Microsoft.AspNetCore.Mvc.ObjectResult","ElapsedMilliseconds":0.0085,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d"}}
{"Timestamp":"2019-03-27T18:54:11.0996033+00:00","Level":"Information","MessageTemplate":"Executing ObjectResult, writing value of type '{Type}'.","Properties":{"Type":"System.Boolean","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d","ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)"}}
{"Timestamp":"2019-03-27T18:54:11.0999477+00:00","Level":"Information","MessageTemplate":"Executed action {ActionName} in {ElapsedMilliseconds}ms","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ElapsedMilliseconds":2.1046,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"5b11275d-a927-4614-8ee8-de05e6c4a28d"}}

Here's the relevant log from the second encryptor-api container:

{"Timestamp":"2019-03-27T18:56:33.7197061+00:00","Level":"Information","MessageTemplate":"Route matched with {RouteData}. Executing action {ActionName}","Properties":{"RouteData":"{action = \"IsAlive\", controller = \"Monitoring\"}","ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510"}}
{"Timestamp":"2019-03-27T18:56:33.7203973+00:00","Level":"Information","MessageTemplate":"Executing action method {ActionName} - Validation state: {ValidationState}","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ValidationState":"Valid","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510"}}
{"Timestamp":"2019-03-27T18:56:33.7215596+00:00","Level":"Information","MessageTemplate":"Executed action method {ActionName}, returned result {ActionResult} in {ElapsedMilliseconds}ms.","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ActionResult":"Microsoft.AspNetCore.Mvc.ObjectResult","ElapsedMilliseconds":0.0829,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510"}}
{"Timestamp":"2019-03-27T18:56:33.7232645+00:00","Level":"Information","MessageTemplate":"Executing ObjectResult, writing value of type '{Type}'.","Properties":{"Type":"System.Boolean","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510","ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)"}}
{"Timestamp":"2019-03-27T18:56:33.7238806+00:00","Level":"Information","MessageTemplate":"Executed action {ActionName} in {ElapsedMilliseconds}ms","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ElapsedMilliseconds":3.5866000000000002,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510"}}

{"Timestamp":"2019-03-27T18:56:36.6676545+00:00","Level":"Information","MessageTemplate":"Route matched with {RouteData}. Executing action {ActionName}","Properties":{"RouteData":"{action = \"IsAlive\", controller = \"Monitoring\"}","ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510"}}
{"Timestamp":"2019-03-27T18:56:36.6685187+00:00","Level":"Information","MessageTemplate":"Executing action method {ActionName} - Validation state: {ValidationState}","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ValidationState":"Valid","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510"}}
{"Timestamp":"2019-03-27T18:56:36.6699962+00:00","Level":"Information","MessageTemplate":"Executed action method {ActionName}, returned result {ActionResult} in {ElapsedMilliseconds}ms.","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ActionResult":"Microsoft.AspNetCore.Mvc.ObjectResult","ElapsedMilliseconds":0.8342,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510"}}
{"Timestamp":"2019-03-27T18:56:36.6712641+00:00","Level":"Information","MessageTemplate":"Executing ObjectResult, writing value of type '{Type}'.","Properties":{"Type":"System.Boolean","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510","ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)"}}
{"Timestamp":"2019-03-27T18:56:36.7264841+00:00","Level":"Information","MessageTemplate":"Executed action {ActionName} in {ElapsedMilliseconds}ms","Properties":{"ActionName":"Kamus.Controllers.MonitoringController.IsAlive (encrypt-api)","ElapsedMilliseconds":58.156000000000006,"EventId":{"Id":2},"SourceContext":"Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker","ActionId":"d04286dc-501a-41f9-beb7-c502320b3510"}}

Does anyone know why I am getting 400 during the encryption step or if I am missing some step?

Thank you.

[omerlh]

Currently encrypting for the default service account is not supported. Kamus try to encourage you to create a service account per application.

[itsvikasgupta]

You are right. When I create a service account and use that in the command, the encrypted data is successfully returned.

However, if I encrypt twice, the encrypted data is different every time. Shouldn't the encrypted data be same each time?

$ kubectl create sa dummy

$ kamus-cli encrypt --secret mysecret --service-account dummy --namespace this-namespace-does-not-exist --kamus-url http://localhost:8080 --log-level verbose --allow-insecure-url
[info  kamus-cli]: Encryption started...
...
[info  kamus-cli]: Successfully encrypted data to dummy service account in this-namespace-does-not-exist namespace
[info  kamus-cli]:
Encrypted data:
02Gd0QdhT84FHG1rI/KitQ==:R/sL5YNCorv6pvNCVnbeAg==
 
$ kamus-cli encrypt --secret mysecret --service-account dummy --namespace this-namespace-does-not-exist --kamus-url http://localhost:8080 --log-level verbose --allow-insecure-url
[info  kamus-cli]: Encryption started...
...
[info  kamus-cli]: Successfully encrypted data to dummy service account in this-namespace-does-not-exist namespace
[info  kamus-cli]:
Encrypted data:
rwaqqmTh53dNqBIzgFZi5w==:a1lEw3Hy2o3CLwn9W8WL7A== <---- DOESN'T MATCH the previously encrypted data above

[omerlh]

Nope, the IV used for the symmetric encryption is generated per request (see this answer), this is why you're getting different responses for the same input. This is expected :)

[itsvikasgupta]

Can this be marked as a documentation bug?

$ kamus-cli encrypt --secret mysecret --service-account default --namespace kube-system --kamus-url http://localhost:8080 --allow-insecure-url

[info  kamus-cli]: Encryption started...
[info  kamus-cli]: service account: default
[info  kamus-cli]: namespace: kube-system
[warn  kamus-cli]: Auth options were not provided, will try to encrypt without authentication to kamus
Handling connection for 8080
[error kamus-cli]: Error while trying to encrypt with kamus: Encrypt request failed due to unexpected error. Status code: 400

should give a better error.

[omerlh]

Yes, added. The behavior will change when will add the CRD (#13) - the CRD will use the default SA in the namespace, so we'll have to allow users to encrypt for the default SA.

[shaikatz]

It was probably fixed by the latest versions, please reopen if relevant.