Error 500: unknown token type "\"\"" when working with AppRole

[arusso]

Describe the bug

After upgrading to Vault 1.2.1, we can no longer login (or even read the configuration of) one of our AppRole roles. Other roles appear unaffected. We receive the following error:

Error reading auth/approle/role/my-role: Error making API request.

URL: GET https://vault.example.com:8200/v1/auth/approle/role/my-role
Code: 500. Errors:

* 1 error occurred:
        * unknown token type "\"\""

We've tried to re-write the AppRole with a valid configuration, but get:

Error writing data to auth/approle/role/my-role: Error making API request.

URL: PUT https://vault.example.com:8200/v1/auth/approle/role/my-role
Code: 500. Errors:

* 1 error occurred:
        * internal error

To Reproduce

Steps to reproduce the behavior:

1. Run vault read auth/approle/role/my-role
2. See error

Expected behavior

Shows configuration for that approle / allows login via approle.

Environment:

• Vault Server Version (retrieve with vault status): 1.2.1
• Vault CLI Version (retrieve with vault version): 1.2.1
• Server Operating System/Architecture: RHEL 7.7 x86_64

Vault server configuration file(s):

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "secrets/"
}

ui = true
cluster_name = "cluster-name"
pid_file = "/var/run/vault.pid"
disable_clustering = true

listener "tcp" {
  address     = "1.2.3.5:8200"
  tls_cert_file = "/etc/pki/tls/certs/vault.crt"
  tls_key_file = "/etc/pki/tls/private/vault.key"
  tls_min_version = "tls12"
  proxy_protocol_behavior = "allow_authorized"
  proxy_protocol_authorized_addrs = "1.2.3.4,2.3.4.5,3.4.5.6"
  tls_disable_client_certs = "true"
}

api_addr = "https://vault-02.example.com:8200"

Additional context
We upgraded vault from 1.1.3 to 1.2.0 and then 1.2.1. We're not sure if this issue was present as well in 1.2.0 because of the other issues preventing AppRoles from working.

We have a number of other AppRoles, and this one seems to be the only one affected. Nothing really special about it's configuration:

{
  "bind_secret_id": true,
  "bound_cidr_list": null,
  "local_secret_ids": false,
  "period": 0,
  "policies": [
    "a-policy",
    "b-policy"
  ],
  "secret_id_bound_cidrs": null,
  "secret_id_num_uses": 0,
  "secret_id_ttl": 0,
  "token_bound_cidrs": null,
  "token_max_ttl": 600,
  "token_num_uses": 0,
  "token_ttl": 600
}

[briankassouf]

Hi @arusso, thanks for the thorough report! Is there anything in your vault server logs when you get the internal error response back?

[fortman]

Releated to #7231
?

[arusso]

Here's what I get when I run vault read vault read auth/approle/role/my-role:

{"time":"2019-08-07T21:30:54.060517223Z","type":"request","auth":{"client_token":"REDACTED","accessor":"VgKnyaGPkcacEdyIJfSDgAqW","display_name":"ldap-arusso","policies":["default","svc.vault.admin"],"token_policies":["default","svc.vault.admin"],"metadata":{"username":"arusso"},"entity_id":"783c936d-7f24-fa7d-bdc6-a17ae4afc7ea","token_type":"service"},"request":{"id":"9f350bd5-4aaf-8247-c89c-7364541c0ab1","operation":"read","client_token":"REDACTED","client_token_accessor":"VgKnyaGPkcacEdyIJfSDgAqW","namespace":{"id":"root"},"path":"auth/approle/role/my-role","remote_address":"5.4.3.2"}}
{"time":"2019-08-07T21:30:54.06065671Z","type":"response","auth":{"client_token":"REDACTED","accessor":"VgKnyaGPkcacEdyIJfSDgAqW","display_name":"ldap-arusso","policies":["default","svc.vault.admin"],"token_policies":["default","svc.vault.admin"],"metadata":{"username":"arusso"},"entity_id":"783c936d-7f24-fa7d-bdc6-a17ae4afc7ea","token_type":"service"},"request":{"id":"9f350bd5-4aaf-8247-c89c-7364541c0ab1","operation":"read","client_token":"REDACTED","client_token_accessor":"VgKnyaGPkcacEdyIJfSDgAqW","namespace":{"id":"root"},"path":"auth/approle/role/my-role","remote_address":"5.4.3.2"},"response":{},"error":"1 error occurred:\n\t* unknown token type \"\\\"\\\"\"\n\n"}

And here's one when I try to re-write the config with cat my-role.json | vault write auth/approle/role/my-role - (minus the local_secret_ids key since that can only be set during creation):

{"time":"2019-08-07T21:34:54.799936902Z","type":"request","auth":{"token_type":"default"},"request":{"id":"98f6e96a-9287-dc85-ae87-55c53f85518e","operation":"update","client_token":"REDACTED","client_token_accessor":"VgKnyaGPkcacEdyIJfSDgAqW","namespace":{"id":"root"},"path":"auth/approle/role/my-role","data":{"bind_secret_id":true,"bound_cidr_list":null,"period":"hmac-sha256:80349c9d73d3122dfebd2e49e911ed905d44044d5c17a3d9dd9ca2eb636c1adb","policies":["hmac-sha256:0a9d27f1201c72c13eda8525f81a20b50f4bd40584d654fcb6b2ff9d30653ab5","hmac-sha256:c2c61848b037c8d612669506337fe2daf7d6a9b8785b500d86ffc9b0093afcda"],"secret_id_bound_cidrs":null,"secret_id_num_uses":"hmac-sha256:80349c9d73d3122dfebd2e49e911ed905d44044d5c17a3d9dd9ca2eb636c1adb","secret_id_ttl":"hmac-sha256:80349c9d73d3122dfebd2e49e911ed905d44044d5c17a3d9dd9ca2eb636c1adb","token_bound_cidrs":null,"token_max_ttl":"hmac-sha256:e6295af594e68d175bb490b9d288854edb41ae3817b947527b9108f7dfb9d1af","token_num_uses":"hmac-sha256:80349c9d73d3122dfebd2e49e911ed905d44044d5c17a3d9dd9ca2eb636c1adb","token_ttl":"hmac-sha256:e6295af594e68d175bb490b9d288854edb41ae3817b947527b9108f7dfb9d1af"},"remote_address":"5.4.3.2"},"error":"internal error"}
{"time":"2019-08-07T21:34:54.800057995Z","type":"response","auth":{"token_type":"default"},"request":{"id":"98f6e96a-9287-dc85-ae87-55c53f85518e","operation":"update","client_token":"REDACTED","client_token_accessor":"VgKnyaGPkcacEdyIJfSDgAqW","namespace":{"id":"root"},"path":"auth/approle/role/my-role","data":{"bind_secret_id":true,"bound_cidr_list":null,"period":"hmac-sha256:80349c9d73d3122dfebd2e49e911ed905d44044d5c17a3d9dd9ca2eb636c1adb","policies":["hmac-sha256:0a9d27f1201c72c13eda8525f81a20b50f4bd40584d654fcb6b2ff9d30653ab5","hmac-sha256:c2c61848b037c8d612669506337fe2daf7d6a9b8785b500d86ffc9b0093afcda"],"secret_id_bound_cidrs":null,"secret_id_num_uses":"hmac-sha256:80349c9d73d3122dfebd2e49e911ed905d44044d5c17a3d9dd9ca2eb636c1adb","secret_id_ttl":"hmac-sha256:80349c9d73d3122dfebd2e49e911ed905d44044d5c17a3d9dd9ca2eb636c1adb","token_bound_cidrs":null,"token_max_ttl":"hmac-sha256:e6295af594e68d175bb490b9d288854edb41ae3817b947527b9108f7dfb9d1af","token_num_uses":"hmac-sha256:80349c9d73d3122dfebd2e49e911ed905d44044d5c17a3d9dd9ca2eb636c1adb","token_ttl":"hmac-sha256:e6295af594e68d175bb490b9d288854edb41ae3817b947527b9108f7dfb9d1af"},"remote_address":"5.4.3.2"},"response":{},"error":"1 error occurred:\n\t* internal error\n\n"}

[calvn]

Hi @arusso I was able to reproduce this by slightly modifying one of the existing test:

=== RUN   TestAppRole_TokenutilUpgrade
--- FAIL: TestAppRole_TokenutilUpgrade (0.00s)
    path_role_test.go:1874: unknown token type "\"\""

I'm currently doing a bit more investigation and working on a fix for this.

[sivanag1974]

Even in vault 1.2.3, I am getting the error

./vault read -tls-skip-verify /auth/approle/role/abc_automation
Error reading auth/approle/role/adds_automation: Error making API request.

URL: GET https://<<>>/v1/auth/approle/role/abc_automation
Code: 500. Errors:

1 error occurred:

• unknown token type """"
  [root@5a2ee033ssb8 tmp]# ./vault -v
  Vault v1.2.3