vault panics due to concurrent map writes

[gobins]

Vault version: 0.7.2

We have an automation tool that lists and compares policies in Vault. The tool makes a lot of read calls for reading all existing policies in Vault. Vault panics and dies every time we run the step to read all policies in Vault.
We have been able to reproduce the error multiple times.

Jun 5 13:36:36 vault-test vault: fatal error: concurrent map writes Jun 5 13:36:36 vault-test vault: goroutine 5971 [running]: Jun 5 13:36:36 vault-test vault: runtime.throw(0x1a804ac, 0x15) Jun 5 13:36:36 vault-test vault: /goroot/src/runtime/panic.go:596 +0x95 fp=0xc4218ab048 sp=0xc4218ab028 Jun 5 13:36:36 vault-test vault: runtime.mapassign(0x17a66a0, 0xc421da4690, 0xc4218ab230, 0x601ff) Jun 5 13:36:36 vault-test vault: /goroot/src/runtime/hashmap.go:499 +0x667 fp=0xc4218ab0e8 sp=0xc4218ab048 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/vault.NewACL(0xc4221f8b80, 0xc, 0x10, 0xc4212caa00, 0x0, 0x0) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/vault/acl.go:114 +0xab6 fp=0xc4218ab320 sp=0xc4218ab0e8 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/vault.(*PolicyStore).ACL(0xc4206ff940, 0xc421b465b0, 0xc, 0xd, 0x0, 0x0, 0x3796c464) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/vault/policy_store.go:356 +0x2ca fp=0xc4218ab400 sp=0xc4218ab320 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/vault.(*Core).fetchACLandTokenEntry(0xc420156a80, 0xc421442620, 0x0, 0x0, 0x0, 0x0) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/vault/core.go:603 +0x33c fp=0xc4218ab4b8 sp=0xc4218ab400 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/vault.(*Core).checkToken(0xc420156a80, 0xc421442620, 0x0, 0x0, 0x0, 0x0) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/vault/core.go:615 +0x145 fp=0xc4218ab580 sp=0xc4218ab4b8 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/vault.(*Core).handleRequest(0xc420156a80, 0xc421442620, 0x0, 0x0, 0x0, 0x0) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/vault/request_handling.go:119 +0x15f fp=0xc4218ab898 sp=0xc4218ab580 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/vault.(*Core).HandleRequest(0xc420156a80, 0xc421442620, 0x0, 0x0, 0x0) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/vault/request_handling.go:45 +0xc77 fp=0xc4218ab9a8 sp=0xc4218ab898 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/http.request(0xc420156a80, 0x2630b60, 0xc42019b1e8, 0xc421400e00, 0xc421442620, 0x0, 0x0) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/http/handler.go:209 +0x3c fp=0xc4218aba08 sp=0xc4218ab9a8 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/http.handleLogical.func1(0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/http/logical.go:121 +0xfb fp=0xc4218aba78 sp=0xc4218aba08 Jun 5 13:36:36 vault-test vault: net/http.HandlerFunc.ServeHTTP(0xc4204ea0c0, 0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /goroot/src/net/http/server.go:1942 +0x44 fp=0xc4218abaa0 sp=0xc4218aba78 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/http.handleRequestForwarding.func1(0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/http/handler.go:168 +0x761 fp=0xc4218abc08 sp=0xc4218abaa0 Jun 5 13:36:36 vault-test vault: net/http.HandlerFunc.ServeHTTP(0xc4204ea0e0, 0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /goroot/src/net/http/server.go:1942 +0x44 fp=0xc4218abc30 sp=0xc4218abc08 Jun 5 13:36:36 vault-test vault: net/http.(*ServeMux).ServeHTTP(0xc4204061e0, 0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /goroot/src/net/http/server.go:2238 +0x130 fp=0xc4218abc70 sp=0xc4218abc30 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/http.wrapHelpHandler.func1(0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/http/help.go:22 +0x17f fp=0xc4218abcc8 sp=0xc4218abc70 Jun 5 13:36:36 vault-test vault: net/http.HandlerFunc.ServeHTTP(0xc4204ea120, 0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /goroot/src/net/http/server.go:1942 +0x44 fp=0xc4218abcf0 sp=0xc4218abcc8 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/http.wrapGenericHandler.func1(0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/http/handler.go:86 +0xb1 fp=0xc4218abd38 sp=0xc4218abcf0 Jun 5 13:36:36 vault-test vault: net/http.HandlerFunc.ServeHTTP(0xc4204ea140, 0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /goroot/src/net/http/server.go:1942 +0x44 fp=0xc4218abd60 sp=0xc4218abd38 Jun 5 13:36:36 vault-test vault: net/http.serverHandler.ServeHTTP(0xc4204e0630, 0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /goroot/src/net/http/server.go:2568 +0x92 fp=0xc4218abda8 sp=0xc4218abd60 Jun 5 13:36:36 vault-test vault: net/http.initNPNRequest.ServeHTTP(0xc421d9bc00, 0xc4204e0630, 0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /goroot/src/net/http/server.go:3088 +0x93 fp=0xc4218abf10 sp=0xc4218abda8 Jun 5 13:36:36 vault-test vault: net/http.(*initNPNRequest).ServeHTTP(0xc420dc70a0, 0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: <autogenerated>:312 +0x74 fp=0xc4218abf50 sp=0xc4218abf10 Jun 5 13:36:36 vault-test vault: net/http.(Handler).ServeHTTP-fm(0x2630b60, 0xc42019b1e8, 0xc421400e00) Jun 5 13:36:36 vault-test vault: /goroot/src/net/http/h2_bundle.go:4319 +0x4d fp=0xc4218abf80 sp=0xc4218abf50 Jun 5 13:36:36 vault-test vault: github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc420e6dc00, 0xc42019b1e8, 0xc421400e00, 0xc420f85440) Jun 5 13:36:36 vault-test vault: /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:1964 +0x89 fp=0xc4218abfc0 sp=0xc4218abf80 Jun 5 13:36:36 vault-test vault: runtime.goexit()

[jefferai]

I see what the issue is; any chance you can provide your test tool so I can easily reproduce?

[jefferai]

If you are able to test the issue-2804 branch that would be great. I wasn't able to reproduce this error in a test, unfortunately, but the cause was straightforward.