Vault crashes with too large requests

[fbsb]

Describe the bug
Vault crashes when requests are too large

Nov 17 08:20:53 vault[10257]: panic: runtime error: slice bounds out of range [:3466612] with capacity 1572684 [recovered]
Nov 17 08:20:53 vault[10257]:         panic: runtime error: slice bounds out of range [:3466612] with capacity 1572684
Nov 17 08:20:53 vault[10257]: goroutine 397883 [running]:
Nov 17 08:20:53 vault[10257]: github.com/samuel/go-zookeeper/zk.encodePacket.func1(0xc002ed3de8)
Nov 17 08:20:53 vault[10257]:         /builds/go/pkg/mod/github.com/samuel/[email protected]/zk/structs.go:489 +0xf8
Nov 17 08:20:53 vault[10257]: panic(0x35f1c20, 0xc003980060)
Nov 17 08:20:53 vault[10257]:         /usr/local/go/src/runtime/panic.go:969 +0x166
Nov 17 08:20:53 vault[10257]: github.com/samuel/go-zookeeper/zk.encodePacketValue(0xc0010800b4, 0x17ff4c, 0x17ff4c, 0x2f4a300, 0xc0345f8010, 0x197, 0x197, 0x0, 0x0)
Nov 17 08:20:53 vault[10257]:         /builds/go/pkg/mod/github.com/samuel/[email protected]/zk/structs.go:565 +0x8e3
Nov 17 08:20:53 vault[10257]: github.com/samuel/go-zookeeper/zk.encodePacketValue(0xc00108000c, 0x17fff4, 0x17fff4, 0x2ed7980, 0xc0345f8000, 0x16, 0x8, 0x0, 0x0)
Nov 17 08:20:53 vault[10257]:         /builds/go/pkg/mod/github.com/samuel/[email protected]/zk/structs.go:519 +0x3b1
Nov 17 08:20:53 vault[10257]: github.com/samuel/go-zookeeper/zk.encodePacket(0xc00108000c, 0x17fff4, 0x17fff4, 0x2ed7980, 0xc0345f8000, 0x0, 0x0, 0x0)
Nov 17 08:20:53 vault[10257]:         /builds/go/pkg/mod/github.com/samuel/[email protected]/zk/structs.go:498 +0x17a
Nov 17 08:20:53 vault[10257]: github.com/samuel/go-zookeeper/zk.(*Conn).sendData(0xc0000ecc60, 0xc02ba26000, 0x3, 0x0)
Nov 17 08:20:53 vault[10257]:         /builds/go/pkg/mod/github.com/samuel/[email protected]/zk/conn.go:754 +0x145
Nov 17 08:20:53 vault[10257]: github.com/samuel/go-zookeeper/zk.(*Conn).sendLoop(0xc0000ecc60, 0x0, 0x0)
Nov 17 08:20:53 vault[10257]:         /builds/go/pkg/mod/github.com/samuel/[email protected]/zk/conn.go:794 +0x402
Nov 17 08:20:53 vault[10257]: github.com/samuel/go-zookeeper/zk.(*Conn).loop.func1(0xc001ef7ce0, 0xc0000ecc60, 0xc00cedd1b0)
Nov 17 08:20:53 vault[10257]:         /builds/go/pkg/mod/github.com/samuel/[email protected]/zk/conn.go:509 +0x53
Nov 17 08:20:53 vault[10257]: created by github.com/samuel/go-zookeeper/zk.(*Conn).loop
Nov 17 08:20:53 vault[10257]:         /builds/go/pkg/mod/github.com/samuel/[email protected]/zk/conn.go:504 +0x3fc
Nov 17 08:20:53 systemd[1]: vault.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 17 08:20:53 systemd[1]: vault.service: Failed with result 'exit-code'.

To Reproduce
Steps to reproduce the behavior:

1. Have a large secret (e.g. java truststore)
2. Have zookeeper max object size set to 2 MiB
3. Run vault kv put test_mount/large_secret_3m file=@large_secret_3m
4. Vault crashes and is sealed after restart

Expected behavior
The request should fail but the vault server should not crash.

Environment:

• Vault Server Version v1.5.4:
• Vault CLI Version v1.5.4:
• Server Operating System/Architecture: debian/buster amd64

Vault server configuration file(s):

backend "zookeeper" {
    address = "localhost:2181"
    advertise_addr = "<redacted>"
    scheme = "https"
    max_parallel = 128
    znode_owner = "ip:127.0.0.1"
}

listener "tcp" {
    address = "<redacted>:8200"
    tls_key_file = "/etc/vault/vault.key"
    tls_cert_file = "/etc/vault/vault.pem"
    tls_min_version = "tls12"
}

listener "tcp" {
    address = "<redacted>:8200"
    tls_key_file = "/etc/vault/vault.key"
    tls_cert_file = "/etc/vault/vault.pem"
    tls_min_version = "tls12"
}

disable_cache = false 
plugin_directory = "/var/lib/vault/plugins"
disable_mlock = false 
default_lease_ttl = "720h" 
max_lease_ttl = "720h"

Additional context
We have mitigated this issue by lowering the max_request_size to the value set in zookeeper config.
While this works for us I think the server should still not crash and restart after a too big request has been received.

[juliantaylor]

likely the crash would be fixed with an up to date go-zookeeper module, e.g. this change changes the panic to a shortbuffer error:
samuel/go-zookeeper#216

which was updated in #10147 included in vault 1.6.0

[swayne275]

Closing this issue as it's been 30+ days. Please re-open if Julian's comment didn't resolve!