Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Concurrency-related crashes from Prometheus client #4132

Closed
ghost opened this issue Aug 9, 2017 · 2 comments · Fixed by #5706
Closed

Concurrency-related crashes from Prometheus client #4132

ghost opened this issue Aug 9, 2017 · 2 comments · Fixed by #5706
Assignees
@Stebalien
Labels
kind/bug A bug in existing code (including security flaws) regression
Milestone
go-ipfs 0.4.14

Comments

@ghost
Copy link

ghost commented Aug 9, 2017

Version information: go-ipfs/0.4.10/4679f80

Type: bug

Severity: critical

Description:

Since I deployed v0.4.11-pre to the gateways, I see regular crashes (~1/h) out of the Prometheus client lib (client_golang). Note that these crashes might have occured earlier too, just less frequently since the gateways way running on a lot less potent machines until yesterday.

These are supposedly fixed in client_golang's master, but updating it is non-trivial since it has seen structural refactorings and is now multiple packages that all need to be gxed.

fatal error: concurrent map iteration and map write

goroutine 140025520 [running]:
runtime.throw(0x561c52fef158, 0x26)
  /usr/lib/go/src/runtime/panic.go:596 +0x97 fp=0xc43e745e20 sp=0xc43e745e00
runtime.mapiternext(0xc43e745f58)
  /usr/lib/go/src/runtime/hashmap.go:737 +0x7f0 fp=0xc43e745ed0 sp=0xc43e745e20
runtime.mapiterinit(0x561c536bb180, 0xc431686840, 0xc43e745f58)
  /usr/lib/go/src/runtime/hashmap.go:727 +0x2b5 fp=0xc43e745f28 sp=0xc43e745ed0
gx/ipfs/QmX3QZ5jHEPidwUrymXV1iSCSUhdGxj15sm2gP4jKMef7B/client_golang/prometheus.computeApproximateRequestSize(0xc44a077a00, 0xc4256773e0, 0x13)
  /go/src/gx/ipfs/QmX3QZ5jHEPidwUrymXV1iSCSUhdGxj15sm2gP4jKMef7B/client_golang/prometheus/http.go:303 +0x8d fp=0xc43e745fc8 sp=0xc43e745f28
runtime.goexit()
  /usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc43e745fd0 sp=0xc43e745fc8
created by gx/ipfs/QmX3QZ5jHEPidwUrymXV1iSCSUhdGxj15sm2gP4jKMef7B/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1
  /go/src/gx/ipfs/QmX3QZ5jHEPidwUrymXV1iSCSUhdGxj15sm2gP4jKMef7B/client_golang/prometheus/http.go:275 +0x14f
@ghost ghost added kind/bug A bug in existing code (including security flaws) regression labels Aug 9, 2017
@ghost ghost self-assigned this Aug 9, 2017
@ghost ghost added this to the Ipfs 0.4.11 milestone Aug 9, 2017
@magik6k magik6k mentioned this issue Aug 20, 2017
Merged
@ghost
Copy link
Author

ghost commented Sep 16, 2017

Possibly useful info: prometheus/client_golang#189 && prometheus/client_golang#311

@magik6k magik6k modified the milestones: Ipfs 0.4.11, Ipfs 0.4.12 Nov 2, 2017
@Kubuxu Kubuxu modified the milestones: Ipfs 0.4.12, go-ipfs 0.4.13 Nov 6, 2017
@Stebalien Stebalien mentioned this issue Jul 23, 2018
Merged
@Stebalien
Copy link
Member

@lgierth I've updated prometheus. Can you check to see if this has been fixed?

Stebalien added a commit that referenced this issue Oct 30, 2018
@Stebalien
fix prometheus concurrent map write bug
4674f19 
fixes #4132

License: MIT
Signed-off-by: Steven Allen <[email protected]>
@Stebalien Stebalien mentioned this issue Oct 30, 2018
Merged
@ghost ghost assigned Stebalien Oct 30, 2018
@ghost ghost added the status/in-progress In progress label Oct 30, 2018
@ghost ghost removed the status/in-progress In progress label Oct 31, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Stebalien Stebalien

Labels
kind/bug A bug in existing code (including security flaws) regression
Projects
None yet
Milestone
go-ipfs 0.4.14
Development

Successfully merging a pull request may close this issue.

fix prometheus concurrent map write bug ipfs/kubo
3 participants
@Stebalien @Kubuxu @magik6k