Nomad 0.8.4 panics when talking to vault 0.11.0

[groggemans]

Nomad version

Nomad v0.8.4 (dbee1d7)

Operating system and Environment details

Raspbian linux

Issue

Nomad panics when trying to access the vault api

Reproduction steps

Run nomad agent in server mode and connect with a vault 0.11.0 cluster

Nomad Server logs (if appropriate)

WARNING: keyring exists but -encrypt given, using keyring
==> Loaded configuration from /etc/nomad.d/base.hcl, /etc/nomad.d/client.hcl, /etc/nomad.d/server.hcl
==> Starting Nomad agent...
panic: interface conversion: interface {} is nil, not []interface {}

goroutine 67 [running]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/vault/api.(*Sys).Capabilities(0x13bd8dc8, 0x13b20181, 0x24, 0x13bfd17, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/vault/api/sys_capabilities.go:38 +0x4c4
github.com/hashicorp/nomad/vendor/github.com/hashicorp/vault/api.(*Sys).CapabilitiesSelf(0x13bd8dc8, 0x13bfd17, 0x15, 0x0, 0x13784780, 0x76d6a6c8, 0x0, 0x0)
        /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/vault/api/sys_capabilities.go:6 +0x3c
github.com/hashicorp/nomad/nomad.(*vaultClient).hasCapability(0x13b821b0, 0x13bfd17, 0x15, 0x1f0d668, 0x2, 0x2, 0x22ac4, 0x13ac0320, 0x10, 0x10, ...)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:753 +0x38
github.com/hashicorp/nomad/nomad.(*vaultClient).validateCapabilities(0x13b821b0, 0x13b1c8a1, 0xd, 0x139c4100, 0x0, 0x0)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:696 +0x68
github.com/hashicorp/nomad/nomad.(*vaultClient).parseSelfToken(0x13b821b0, 0x1, 0x0)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:668 +0x1a0
github.com/hashicorp/nomad/nomad.(*vaultClient).establishConnection(0x13b821b0)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:428 +0x11c
github.com/hashicorp/nomad/nomad.(*vaultClient).(github.com/hashicorp/nomad/nomad.establishConnection)-fm()
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:247 +0x1c
github.com/hashicorp/nomad/nomad.wrapNilError.func1(0x0, 0x1399d040)
        /opt/gopath/src/github.com/hashicorp/nomad/nomad/vault.go:1186 +0x1c
github.com/hashicorp/nomad/vendor/gopkg.in/tomb%2ev2.(*Tomb).run(0x139a8e40, 0x1370f718)
        /opt/gopath/src/github.com/hashicorp/nomad/vendor/gopkg.in/tomb.v2/tomb.go:153 +0x1c
created by github.com/hashicorp/nomad/vendor/gopkg.in/tomb%2ev2.(*Tomb).Go
        /opt/gopath/src/github.com/hashicorp/nomad/vendor/gopkg.in/tomb.v2/tomb.go:149 +0xa4

I think this needs a fix in the vault code, but posting here for completeness and follow up.

[jefferai]

Nomad needs to pull in updated API libs

[jefferai]

This doesn't seem to be a 0.11 bug. It appears to be this: hashicorp/vault#4553 which was fixed in Vault 0.10.2. This, however, appears to be after Nomad last pulled Vault API libs.

We'll put in a workaround for 0.11.1 but Nomad should pull in updated API libs.

[kak-tus]

May be problem is not only in hashicorp/vault#4553?

I dumped request/response in this case.

Request:

(*http.Request)(0xc420c36900)({
 Method: (string) (len=4) "POST",
 URL: (*url.URL)(0xc4211d0280)(http://vault.query.consul:8200/v1/sys/capabilities-self),
 Proto: (string) (len=8) "HTTP/1.1",
 ProtoMajor: (int) 1,
 ProtoMinor: (int) 1,
 Header: (http.Header) (len=1) {
  (string) (len=13) "X-Vault-Token": ([]string) (len=1 cap=1) {
   (string) (len=36) "mytoken"
  }
 },
 Body: (ioutil.nopCloser) {
  Reader: (*bytes.Buffer)(0xc420414930)({"path":"sys/capabilities-self","token":"mytoken"}
)

In curl form

curl -H 'X-Vault-Token:mytoken' -d '{"path":"sys/capabilities-self","token":"mytoken"}' 'http://vault.query.consul:8200/v1/sys/capabilities-self'

Response on this request is

{"request_id":"f6610f0f-1818-0188-8273-0290172cda05","lease_id":"","renewable":false,"lease_duration":0,"data":{"capabilities":["root"],"sys/capabilities-self":["root"]},"wrap_info":null,"warnings":null,"auth":null}

So, as you can see - there is exists "capabilities" key in response - in "data" subkey. And fix in https://github.com/hashicorp/vault/pull/4553/files do

capabilitiesRaw, ok := result["capabilities"].([]interface{})

And this code not fully fixed the problem - it try to get key from wrong response path.

May be this is misunderstanding with Vault team and response format?

[adamlc]

I'm also having this issue! I'm a little worried that if my nomad servers restart for whatever reason then they aren't going to come back. So it sounds like updating the vault libs alone doesn't fix the issue?

[kak-tus]

@adamlc I fixed it in file sys_capabilities.go and use nomad version with this patch

[adamlc]

@kak-tus thanks I'll give it a try!

[adamlc]

I can confirm the above fix is working perfectly :)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.