Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic due to type casting between etcdserverpb.Request and RequestV2 #12454

Closed
kjin97 opened this issue Nov 5, 2020 · 3 comments
Closed

Panic due to type casting between etcdserverpb.Request and RequestV2 #12454

kjin97 opened this issue Nov 5, 2020 · 3 comments

Comments

@kjin97
Copy link

kjin97 commented Nov 5, 2020

Issue: etcdserver hits a panic in a test case
Environment: ubuntu14 docker container with go1.14.2.linux-amd64.tar.gz
Potentially relevant lines in go.mod:

require (
	github.com/coreos/bbolt v1.3.5 // indirect
	github.com/coreos/etcd v3.3.25+incompatible
	github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf // indirect
	github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
	github.com/gogo/protobuf v1.3.1 // indirect
	github.com/golang/protobuf v1.4.3
	google.golang.org/grpc v1.33.1
	google.golang.org/protobuf v1.25.0 // indirect
)
replace github.com/coreos/bbolt => go.etcd.io/bbolt v1.3.5
replace google.golang.org/grpc v1.33.1 => google.golang.org/grpc v1.26.0

go 1.14

To reproduce: start a single-node etcd cluster, and connect to it with the ETCD v2 client. Then, spawn 100 goroutines, each of which attempts to acquire a lock at /distributed_lock by trying to write its own id to that key. Each goroutine holds the lock for a while (can be simulated with a time.Sleep). Note that I am not using the etcd concurrency package. However, I don't think the issue is within my code because the stacktrace is entirely within etcd's library code. This could be a lesser-hit codepath because it requires writing a request expensive enough to warrant the call to etcdserver.warnOfExpensiveReques, and a type casting between Request and RequestV2.

For certain reasons, it is not very easy for me to upgrade to the v3 client. How possible is it for you guys to take a look at this panic and push a fix?

[2020-10-29 05:39:31.971264+0000]  2020-10-29 05:39:26.903535 I | etcdserver: name = name-default-39338
[2020-10-29 05:39:31.971278+0000]  2020-10-29 05:39:26.903601 I | etcdserver: data dir = $ETCD_DIR/test-etcd-name-default-39338418782820
[2020-10-29 05:39:31.971286+0000]  2020-10-29 05:39:26.903637 I | etcdserver: member dir = $ETCD_DIR/test-etcd-name-default-39338418782820/member
[2020-10-29 05:39:31.971295+0000]  2020-10-29 05:39:26.903678 I | etcdserver: heartbeat = 10ms
[2020-10-29 05:39:31.971303+0000]  2020-10-29 05:39:26.903713 I | etcdserver: election = 100ms
[2020-10-29 05:39:31.971311+0000]  2020-10-29 05:39:26.903755 I | etcdserver: snapshot count = 100000
[2020-10-29 05:39:31.971319+0000]  2020-10-29 05:39:26.903800 I | etcdserver: advertise client URLs = http://[localhost]:39338
[2020-10-29 05:39:31.971347+0000]  2020-10-29 05:39:26.903832 I | etcdserver: initial advertise peer URLs = http://[localhost]:46563
[2020-10-29 05:39:31.971355+0000]  2020-10-29 05:39:26.903869 I | etcdserver: initial cluster = name-default-39338=http://[localhost]:46563
[2020-10-29 05:39:31.971364+0000]  2020-10-29 05:39:26.914089 I | etcdserver: starting member a8cd15281488d33c in cluster 8ca5d47f17b554f
[2020-10-29 05:39:31.971404+0000]  2020-10-29 05:39:26.917714 I | etcdserver: starting server... [version: 3.3.25, cluster version: to_be_decided]
[2020-10-29 05:39:31.971412+0000]  2020-10-29 05:39:26.917960 I | etcdserver: a8cd15281488d33c as single-node; fast-forwarding 9 ticks (election ticks 10)
[2020-10-29 05:39:31.971421+0000]  2020-10-29 05:39:26.919689 I | etcdserver/membership: added member a8cd15281488d33c [http://[localhost]:46563] to cluster 8ca5d47f17b554f
[2020-10-29 05:39:31.971481+0000]  2020-10-29 05:39:26.996207 I | etcdserver: published {Name:name-default-39338 ClientURLs:[http://[localhost]:39338]} to cluster 8ca5d47f17b554f
[2020-10-29 05:39:31.971497+0000]  2020-10-29 05:39:27.007533 I | etcdserver: setting up the initial cluster version to 3.3
[2020-10-29 05:39:31.971506+0000]  2020-10-29 05:39:27.007898 N | etcdserver/membership: set the initial cluster version to 3.3
...
Write some expensive requests to etcd
...
[2020-10-29 05:39:31.976002+0000]  panic: field etcdserverpb.Request.ID has invalid type: got uint64, want pointer
[2020-10-29 05:39:31.976010+0000]  
[2020-10-29 05:39:31.976018+0000]  goroutine 498 [running]:
[2020-10-29 05:39:31.976027+0000]  google.golang.org/protobuf/internal/impl.fieldInfoForScalar(0x17723a0, 0xc00042d000, 0x1364e13, 0x2, 0x0, 0x0, 0x17724c0, 0x13a3aa0, 0x1364e17, 0x29, ...)
[2020-10-29 05:39:31.976035+0000]  	$PROJECT_DIRECTORY/vendor/google.golang.org/protobuf/internal/impl/message_reflect_field.go:228 +0xa22
[2020-10-29 05:39:31.976043+0000]  google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc(0xc0001dea00, 0xb0, 0xffffffffffffffff, 0x98, 0xffffffffffffffff, 0xc000ef81e0, 0xc000ef8210, 0xc000ef8240, 0xc000ef8270)
[2020-10-29 05:39:31.976054+0000]  	$PROJECT_DIRECTORY/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go:67 +0xc45
[2020-10-29 05:39:31.976062+0000]  google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs(0xc0001dea00, 0x17724c0, 0x152f680, 0xb0, 0xffffffffffffffff, 0x98, 0xffffffffffffffff, 0xc000ef81e0, 0xc000ef8210, 0xc000ef8240, ...)
[2020-10-29 05:39:31.976074+0000]  	$PROJECT_DIRECTORY/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go:36 +0xc8
[2020-10-29 05:39:31.976083+0000]  google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce(0xc0001dea00)
[2020-10-29 05:39:31.976091+0000]  	$PROJECT_DIRECTORY/vendor/google.golang.org/protobuf/internal/impl/message.go:90 +0x222
[2020-10-29 05:39:31.976099+0000]  google.golang.org/protobuf/internal/impl.(*MessageInfo).init(0xc0001dea00)
[2020-10-29 05:39:31.976107+0000]  	$PROJECT_DIRECTORY/vendor/google.golang.org/protobuf/internal/impl/message.go:72 +0x63
[2020-10-29 05:39:31.976115+0000]  google.golang.org/protobuf/internal/impl.(*messageReflectWrapper).Has(0xc0013c70b0, 0x17723a0, 0xc00042d000, 0xc00042d000)
[2020-10-29 05:39:31.976123+0000]  	$PROJECT_DIRECTORY/vendor/google.golang.org/protobuf/internal/impl/message_reflect_gen.go:185 +0x5a
[2020-10-29 05:39:31.976131+0000]  github.com/golang/protobuf/proto.(*textWriter).writeMessage(0xc000ef8150, 0x1769ac0, 0xc0013c70b0, 0x0, 0x0)
[2020-10-29 05:39:31.976140+0000]  	$PROJECT_DIRECTORY/vendor/github.com/golang/protobuf/proto/text_encode.go:278 +0xb41
[2020-10-29 05:39:31.976148+0000]  github.com/golang/protobuf/proto.(*TextMarshaler).marshal(0x1f7e3a4, 0x174b580, 0xc0017f7080, 0x740a8da3, 0x0, 0xbfdeb343ed225481, 0x6d68615d, 0x1ff3040)
[2020-10-29 05:39:31.976156+0000]  	$PROJECT_DIRECTORY/vendor/github.com/golang/protobuf/proto/text_encode.go:86 +0x2cf
[2020-10-29 05:39:31.976164+0000]  github.com/golang/protobuf/proto.(*TextMarshaler).Text(...)
[2020-10-29 05:39:31.976172+0000]  	$PROJECT_DIRECTORY/vendor/github.com/golang/protobuf/proto/text_encode.go:44
[2020-10-29 05:39:31.976181+0000]  github.com/golang/protobuf/proto.CompactTextString(...)
[2020-10-29 05:39:31.976198+0000]  	$PROJECT_DIRECTORY/vendor/github.com/golang/protobuf/proto/text_encode.go:106
[2020-10-29 05:39:31.976207+0000]  github.com/coreos/etcd/etcdserver/etcdserverpb.(*Request).String(...)
[2020-10-29 05:39:31.976216+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/etcdserver.pb.go:51
[2020-10-29 05:39:31.976224+0000]  github.com/coreos/etcd/etcdserver.(*RequestV2).String(0xc0017f7680, 0x6d68615d, 0x1ff3040)
[2020-10-29 05:39:31.976232+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/v2_server.go:164 +0xd4
[2020-10-29 05:39:31.976241+0000]  github.com/coreos/etcd/etcdserver.warnOfExpensiveGenericRequest(0xbfdeb343ed225481, 0x6d68615d, 0x1ff3040, 0x17338a0, 0xc0017f7680, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
[2020-10-29 05:39:31.976249+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/util.go:157 +0x13d
[2020-10-29 05:39:31.976257+0000]  github.com/coreos/etcd/etcdserver.warnOfExpensiveRequest(0xbfdeb343ed225481, 0x6d68615d, 0x1ff3040, 0x17338a0, 0xc0017f7680, 0x0, 0x0, 0x0, 0x0)
[2020-10-29 05:39:31.976265+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/util.go:109 +0xca
[2020-10-29 05:39:31.976279+0000]  github.com/coreos/etcd/etcdserver.(*EtcdServer).applyV2Request(0xc000429400, 0xc0017f7680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x17337a0, 0xc000ef8120)
[2020-10-29 05:39:31.976287+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/apply_v2.go:116 +0x26f
[2020-10-29 05:39:31.976295+0000]  github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntryNormal(0xc000429400, 0xc0008ed270)
[2020-10-29 05:39:31.976304+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/server.go:1440 +0x42b
[2020-10-29 05:39:31.976312+0000]  github.com/coreos/etcd/etcdserver.(*EtcdServer).apply(0xc000429400, 0xc00127ea80, 0x1f, 0x21, 0xc000161960, 0xc0001619c0, 0x187, 0xc001788850)
[2020-10-29 05:39:31.976320+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/server.go:1387 +0x53f
[2020-10-29 05:39:31.976333+0000]  github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntries(0xc000429400, 0xc000161960, 0xc00164f180)
[2020-10-29 05:39:31.976341+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/server.go:1024 +0x169
[2020-10-29 05:39:31.976349+0000]  github.com/coreos/etcd/etcdserver.(*EtcdServer).applyAll(0xc000429400, 0xc000161960, 0xc00164f180)
[2020-10-29 05:39:31.976358+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/server.go:888 +0x99
[2020-10-29 05:39:31.976366+0000]  github.com/coreos/etcd/etcdserver.(*EtcdServer).run.func6(0x1752ac0, 0xc0003a5340)
[2020-10-29 05:39:31.976374+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/etcdserver/server.go:840 +0x54
[2020-10-29 05:39:31.976382+0000]  github.com/coreos/etcd/pkg/schedule.(*fifo).run(0xc0001a1680)
[2020-10-29 05:39:31.976391+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:157 +0x11f
[2020-10-29 05:39:31.976399+0000]  created by github.com/coreos/etcd/pkg/schedule.NewFIFOScheduler
[2020-10-29 05:39:31.976407+0000]  	$PROJECT_DIRECTORY/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:70 +0x2b2
@ptabor
Copy link
Contributor

ptabor commented Nov 5, 2020

In general etcd is not compatible with github.com/golang/protobuf >=v1.4.0.

The reasons is that protos were generated by gogo/protobuf 1.3.x (and it seems there will be no 1.4.x compatible release:
gogo/protobuf#691).

Highest versions of dependencies you can use are:

google.golang.org/grpc v1.29.1
github.com/golang/protobuf v1.3.5

@jingyih
Copy link
Contributor

jingyih commented Nov 5, 2020

Is this a dup of #12197?

@kjin97
Copy link
Author

kjin97 commented Nov 5, 2020

Looks like it, wasn't able to find that issue when googling. Thanks!

@kjin97 kjin97 closed this as completed Nov 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ptabor @jingyih @kjin97