Kafka Scaler - fatal error: concurrent map read and map write

[mariusgrigoriu]

Keda sometimes crashes with concurrent map read and write. Our test involves 1000 ScaledObjects all using the Kafka scaler.

Steps to Reproduce the Problem

Create many SO with the Kafka scaler and wait. We experienced a crash after about 10 minutes from startup with 1000 SO, but YMMV.

Logs from KEDA operator

2020-12-16T22:33:32.088Z	INFO	controllers.ScaledObject	Initializing Scaling logic according to ScaledObject Specification	{"ScaledObject.Namespace": "default", "ScaledObject.Name": "scaled-object-51"}
2020-12-16T22:33:32.103Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "default", "ScaledObject.Name": "scaled-object-176"}
2020-12-16T22:33:32.385Z	INFO	kafka_scaler	invalid offset found for topic exp-kargokedatest-avro in group keda-kafka-loadtest-51 and partition 0, probably no offset is committed yet
2020-12-16T22:33:32.441Z	INFO	controllers.ScaledObject	Initializing Scaling logic according to ScaledObject Specification	{"ScaledObject.Namespace": "default", "ScaledObject.Name": "scaled-object-176"}
2020-12-16T22:33:32.442Z	INFO	kafka_scaler	invalid offset found for topic exp-kargokedatest-avro in group keda-kafka-loadtest-172 and partition 0, probably no offset is committed yet
fatal error: concurrent map read and map write

goroutine 16484 [running]:
runtime.throw(0x216d5a9, 0x21)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc035b07870 sp=0xc035b07840 pc=0x437692
runtime.mapaccess1_faststr(0x1e40740, 0xc0436ce720, 0x2138690, 0x5, 0xc03eb1f1a8)
	/usr/local/go/src/runtime/map_faststr.go:21 +0x465 fp=0xc035b078e0 sp=0xc035b07870 pc=0x413d45
github.com/kedacore/keda/pkg/scalers.parseKafkaMetadata(0xc035b07e38, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/workspace/pkg/scalers/kafka_scaler.go:110 +0x245 fp=0xc035b07a38 sp=0xc035b078e0 pc=0x1bf53e5
github.com/kedacore/keda/pkg/scalers.NewKafkaScaler(0xc035b07e38, 0x5, 0x2138127, 0x5, 0x0)
	/workspace/pkg/scalers/kafka_scaler.go:75 +0x45 fp=0xc035b07c20 sp=0xc035b07a38 pc=0x1bf4ea5
github.com/kedacore/keda/pkg/scaling.buildScaler(0xc02c12430b, 0x5, 0xc035b07e38, 0xc045cdca80, 0xc02a6657e0, 0xc045c752f8, 0xc02c1242a0)
	/workspace/pkg/scaling/scale_handler.go:437 +0x29d fp=0xc035b07c88 sp=0xc035b07c20 pc=0x1c596fd
github.com/kedacore/keda/pkg/scaling.(*scaleHandler).buildScalers(0xc00090e240, 0xc045d38140, 0xc045c75200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3, 0x3)
	/workspace/pkg/scaling/scale_handler.go:354 +0x37d fp=0xc035b07e90 sp=0xc035b07c88 pc=0x1c580bd
github.com/kedacore/keda/pkg/scaling.(*scaleHandler).GetScalers(0xc00090e240, 0x20efd00, 0xc04b6d5860, 0x6, 0x251cde0, 0xc0469300c0, 0xc0437a3c50, 0x221faf8)
	/workspace/pkg/scaling/scale_handler.go:68 +0xee fp=0xc035b07ef8 sp=0xc035b07e90 pc=0x1c5570e
github.com/kedacore/keda/pkg/scaling.(*scaleHandler).startPushScalers(0xc00090e240, 0x250dcc0, 0xc0517d4fc0, 0xc02c128140, 0x20efd00, 0xc04b6d5860, 0x24dd700, 0xc02c124398)
	/workspace/pkg/scaling/scale_handler.go:145 +0x1e5 fp=0xc035b07fa0 sp=0xc035b07ef8 pc=0x1c566c5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc035b07fa8 sp=0xc035b07fa0 pc=0x46d521
created by github.com/kedacore/keda/pkg/scaling.(*scaleHandler).HandleScalableObject
	/workspace/pkg/scaling/scale_handler.go:94 +0x30a

Specifications

• KEDA Version: 2.0.0
• Kubernetes Version: 1.17.13
• Scaler(s): Kafka

[nrmitchi]

Want to add in here that think issue is not Kafka specific. We ran into it recently with a significantly smaller number of ScaledObjects (< 10), and using only CPU and Prometheus scalers.

[coderanger]

Might have been fixed by #1812.

[aslom]

I have similiar problem - no crashing but also no scaling happens - instead getting long stream of

2021-07-19T04:03:24.568Z	INFO	kafka_scaler	invalid offset found for topic topic10 in group knative-group10 and partition 0, probably no offset is committed yet
2021-07-19T04:03:34.709Z	INFO	kafka_scaler	invalid offset found for topic topic10 in group knative-group10 and partition 0, probably no offset is committed yet
2021-07-19T04:03:44.820Z	INFO	kafka_scaler	invalid offset found for topic topic10 in group knative-group10 and partition 0, probably no offset is committed yet
...

That was with v2.1.0 and I am now testing with latest from git.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.