Goroutine leak when mongodb-database-plugin can't find target replicaset

[berniecamus]

Describe the bug
When using vault (current 1.5.4) to broker connections to mongodb, if the mongodb replicaset configured in the plugin is unavailable, goroutines begins to climb. These get higher and higher (along with open file handles) until eventually vault becomes unresponsive.

To Reproduce
Steps to reproduce the behavior:

1. Configure vault mongodb database plugin to provide credentials to a mongodb replicaset
2. Break or disable mongodb on that replicaset (but leave the servers themselves available, just with no mongo running)
3. Watch open goroutines climb indefinitely

Expected behavior
Vault database plugin closes goroutines after being unable to establish a connection to mongodb

Environment:

• Vault Server Version (retrieve with vault status): 1.5.4 (also observed in 1.0.3, which we upgraded from in an attempt to resolve this issue, as we're aware a similar fault was fixed in 1.3.2)
• Vault CLI Version (retrieve with vault version): v1.5.4
• Server Operating System/Architecture: Ubuntu 16.04

Vault server configuration file(s):

api_addr = "xxxxxxxxxxxxxxxxxxx:8200"

cluster_addr = "https://xxxxxxxxxxxxxxxxxx:8201"

log_level = "info"

storage "mysql" {
  address  = "xxxxxxxxxxxxxxxxxxxxx:3306"
  username = "vault"
  password = "xxxxxxxxxxxxxxxxxxxxx"
  tls_ca_file = "/etc/vault/xxxxxxxxxxxx.pem"
}

ha_storage "dynamodb" {
  ha_enabled     = "true"
  region         = "eu-west-2"
  table          = "xxxxxxxxxxxx"
  read_capacity  = 20
  write_capacity = 10
}

ui = true

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/etc/ssl/vault/xxxxxxx.crt"
  tls_key_file = "/etc/ssl/vault/xxxxxxx.key"
  tls_min_version = "tls12"
  tls_cipher_suites = "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA"
}

telemetry {
  statsd_address = "127.0.0.1:8125"
}

# Might need to re-enabled
# https://www.vaultproject.io/docs/configuration/index.html#disable_mlock
disable_mlock = true

seal "awskms" {
  kms_key_id = "alias/xxxxxxxxxxxxxxxxxxx"
  region     = "xxxxxx"
}

Additional context
We are aware disable_mlock probably should be "false" and have tried that too, it doesn't affect this issue. We also have a lot of traffic hitting the replicaset requesting creds. Perhaps ten creds a minute, but there is no resource contention of CPU, mem etc

[berniecamus]

Note: this is behaviour that I think should have been fixed by: #8140, which was then rolled out in 1.3.2, but the problem still seems to persist

[ncabatoff]

My hunch is that the goroutines you were seeing were related to lease revocation. I believe the rework of expiration in 1.7 would have addressed this. Could you try to reproduce with a current version? And if you succeed, can you attach the output from running vault debug against the active node when this happen?

[heatherezell]

Since it's been a few months since there was last activity on this issue, I'll go ahead and close it now. If you have further bug reports, or more activity on this issue, please feel free to re-open it. Don't forget you can get help on our discuss forum, as well! https://discuss.hashicorp.com/c/vault/30

[berniecamus]

Apologies enough time passed that I left the firm I was working for when I opened this. Thanks for the info though!