Upgrade from 1.1.0 to 1.1.1 started causing error in vault logs

[raoofm]

Describe the bug

Using eth0 for VAULT_CLUSTER_ADDR: https://100.xxx.x.x:8201
telemetry.disable_hostname has been set to false. Recommended setting is true for Prometheus to avoid poorly named metrics.
==> Vault server configuration:
               HA Storage: dynamodb
             Api Address: https://vault.dev.net
                     Cgo: disabled
         Cluster Address: https://100.xxx.x.x:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled", x_forwarded_for_authorized_addrs: "[100.xx.x.x/11]", x_forwarded_for_hop_skips: "0", x_forwarded_for_reject_not_authorized: "true", x_forwarded_for_reject_not_present: "false")
              Listener 2: tcp (addr: "0.0.0.0:8300", cluster address: "0.0.0.0:8301", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: info
                   Mlock: supported: true, enabled: true
                 Storage: s3
                 Version: Vault v1.1.1
             Version Sha: a3dcd63451cf6da1d04928b601bbe9748d53842e
 ==> Vault server started! Log data will stream in below:
 2019-04-15T18:56:18.278Z [INFO]  core: vault is unsealed
2019-04-15T18:56:18.278Z [INFO]  core.cluster-listener: starting listener: listener_address=0.0.0.0:8201
2019-04-15T18:56:18.279Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2019-04-15T18:56:18.279Z [INFO]  core.cluster-listener: starting listener: listener_address=0.0.0.0:8301
2019-04-15T18:56:18.279Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8301
2019-04-15T18:56:18.279Z [INFO]  core: entering standby mode
2019-04-15T18:56:20.288Z [INFO]  core: acquired lock, enabling active operation
2019-04-15T18:56:20.558Z [INFO]  core: post-unseal setup starting
2019-04-15T18:56:20.595Z [INFO]  core: loaded wrapping token key
2019-04-15T18:56:20.596Z [INFO]  core: successfully setup plugin catalog: plugin-directory=
2019-04-15T18:56:20.627Z [INFO]  core: successfully mounted backend: type=system path=sys/
2019-04-15T18:56:20.628Z [INFO]  core: successfully mounted backend: type=identity path=identity/
2019-04-15T18:56:20.724Z [ERROR] core: failed to create mount entry: path=secret/ error="upgrade done but error checking/creating policy: cannot write to storage during setup"
2019-04-15T18:56:20.724Z [INFO]  core: pre-seal teardown starting
2019-04-15T18:56:20.725Z [INFO]  core: pre-seal teardown complete
2019-04-15T18:56:20.725Z [ERROR] core: post-unseal setup failed: error="failed to setup mount table"
2019-04-15T18:56:21.748Z [INFO]  core: acquired lock, enabling active operation
2019-04-15T18:56:21.946Z [INFO]  core: post-unseal setup starting
2019-04-15T18:56:21.962Z [INFO]  core: loaded wrapping token key
2019-04-15T18:56:21.962Z [INFO]  core: successfully setup plugin catalog: plugin-directory=
2019-04-15T18:56:21.983Z [INFO]  core: successfully mounted backend: type=system path=sys/
2019-04-15T18:56:21.984Z [INFO]  core: successfully mounted backend: type=identity path=identity/
2019-04-15T18:56:22.003Z [ERROR] core: failed to create mount entry: path=secret/ error="upgrade done but error checking/creating policy: cannot write to storage during setup"

To Reproduce
Steps to reproduce the behavior:

1. Upgraded vault from v1.1.0 to v1.1.1

Additional context
Stateful set with 3 pods in kubernetes. Uses s3 as backend and dynamodb for HA.

[raoofm]

@briankassouf was there a breaking change?

[jefferai]

We identified a regression in certain situations and are working on a fix.

[raoofm]

thanks @jefferai

[jefferai]

This is fixed in hashicorp/vault-plugin-secrets-kv#31

[vickysy84]

Hi @raoofm. I am new to Vault and am looking at the audit logs but I don't see the update log similar to what you pasted above. Where can I find this? Hoping you can help me in this inquiry, thanks!