-
Notifications
You must be signed in to change notification settings - Fork 9.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
clientv3, etcdmain, proxy: support authed RPCs with grpcproxy #8289
Conversation
proxy/grpcproxy/cluster.go
Outdated
@@ -175,3 +181,33 @@ func (cp *clusterProxy) MemberList(ctx context.Context, r *pb.MemberListRequest) | |||
resp := (pb.MemberListResponse)(*mresp) | |||
return &resp, err | |||
} | |||
|
|||
func (ap *authedClusterProxy) MemberAdd(ctx context.Context, r *pb.MemberAddRequest) (*pb.MemberAddResponse, error) { | |||
token := getAuthTokenFromClient(ctx) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these are very invasive; would it be possible to use a unary interceptor instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, I'll try the interceptor mechanism.
f84dc07
to
0fdbaef
Compare
@heyitsanthony I followed the interceptor way, it is quite convenient! I'm still considering how to handle the watch & auth combination. Possibly OCC styled checking like 1. check permission with range 2. watch until something happen 3. check the permission again, would work but I'm not sure this is the best solution yet. |
@mitake as a first attempt, the watch permission checking could simply check at time of watch creation like the server does now |
@heyitsanthony I thought the checking during watch creation doesn't work well because the result from the watch can be forwarded to other clients. And the clients would be owned by different users. Because of the same reason, caching range results for serializable clients in the grpcproxy wouldn't be able to coexist with auth. Probably an option for turning off the caching would be required? |
Or letting jwt tokens have permission information and check it in proxy would be reasonable? https://github.com/coreos/etcd/blob/master/auth/jwt.go#L66 Clearly it makes tokens bloated, but is useful for handling watch in proxy. simple token isn't for serious use and CN based auth cannot work with proxy because clients shouldn't share a single CN. |
@mitake I was thinking at watch creation time the proxy could get user / role information from etcd to determine if the user is authorized for reading the keys it's requesting before attaching to any watch that's already open. Caching could be made safe by having a separate cache per user. |
@heyitsanthony even a user has a permission at the first time, the permission can be dropped after issuing a watch RPC. So I thought the OCC styled double checking would be required. Per user caching seems to be nice. But this idea would have a similar problem like the above. A user can be deleted and a new user with the name of the deleted user can be created. Of course they are corner cases :) |
Perhaps there's some way to associate it with auth tokens? If the auth token changes, check to see if the user still has authorization. Serving cached data after token expiration should be OK since all the data in the cache was acquired using that expired token in the first place.
The etcd server has this same issue, right? A user can create a watch and the permission can be dropped but it'll keep streaming keys. I'm not too worried about this for the time being; binding the permission at the time of acquiring a resource is common, although not ideal. Or am I missing something? A stream interceptor could handle cutting off the watch. At the moment, I can't think of a simple way to make it consistent with auth that wouldn't involve issuing auth rpcs for every watch response. |
@heyitsanthony on the second thought, watch RPCs don't involve Raft so the very strict checking would be overkill. As you say, the stream interceptor approach would be reasonable. I'll try it. Thanks for your comments. |
BTW, I'm still thinking about grpcproxy's caching mechanism (it is not related to watch). If a Range request is serializable, a proxy tries to return its cached result. But this cached result wouldn't be invalidated by Put requests that bypass the proxy (e.g. Put requests served by etcd directly or served by other proxies). This semantics is weaker than serializable. Is this ok? |
It's too weak for many purposes. #7624 would get serializability for real. |
I see, thanks for the pointer. |
@heyitsanthony I supported watch RPC in the latest version. It couldn't be implemented in a non intrusive manner like other RPCs. This is because:
|
@heyitsanthony I'll add test cases for covering the new changes. Can we have an agreement about the intrusive change required for supporting watch? |
@mitake this approach looks OK. Try removing the |
@heyitsanthony ah I see. I revived the tests with the latest commit. |
@heyitsanthony it seems that the previous CI failures were caused by slowness of the server. I rebased and updated this PR. Could you take a look if the latest CI goes well? |
hmm... the CIs are causing unseen errors. I'm checking them. |
@heyitsanthony I found a silly mistake in handling snapshot RPC and solved it in the new commit (9e6e21c). It would solve the CI failure problem. Could you take a look after CI passing? |
Something is still broken in integration tests... I'm looking at it. |
@heyitsanthony I found that the clientv3's integration pass caused a failure because of its too short timeout value. I changed it with the latest commit: 22abc7a . I want to watch what happen on the CI servers with the change. |
@heyitsanthony I checked the latest CI failures (which produce very lengthy logs!) and found that the bloated error logs came from the failure of The failure of |
75c163e
to
c16785c
Compare
0019a99
to
267b9f5
Compare
@heyitsanthony the CIs are still failing but the latest failures wouldn't be related to this PR:
Possibly 2nd and 3rd problems need issues? Also I resolved all failed tests related to this PR. The main reason of the failures was breaking watch API's semantics by newly added permission checking in grpcproxy. It took very long because of two test cases so I describe the summary below:
|
@heyitsanthony I looked at the latest CI failures for a while but they seem to be not related with this PR (not 100% sure, though). Could you take a look when you have a time? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor nits
proxy/grpcproxy/watch.go
Outdated
Serializable: true, | ||
Key: key, | ||
RangeEnd: rangeEnd, | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CountOnly: 1, Limit: 1
so this doesn't fetch a lot of data
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I'll update for the comments soon.
proxy/grpcproxy/watch_broadcast.go
Outdated
@@ -59,6 +59,12 @@ func newWatchBroadcast(wp *watchProxy, w *watcher, update func(*watchBroadcast)) | |||
clientv3.WithCreatedNotify(), | |||
} | |||
|
|||
// Forward a token from client to server. | |||
token := getAuthTokenFromClient(w.wps.stream.Context()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this code happens twice, should probably have a function like:
cctx = withClientAuthToken(cctx, w.wps.stream.Context())
proxy/grpcproxy/util.go
Outdated
@@ -0,0 +1,64 @@ | |||
// Copyright 2016 The etcd Authors |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2017
proxy/grpcproxy/util.go
Outdated
package grpcproxy | ||
|
||
import ( | ||
"golang.org/x/net/context" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"context"
This commit lets grpcproxy support authed RPCs. Auth tokens supplied by clients are now forwarded to etcdserver by grpcproxy.
This commit lets grpcproxy handle authed watch. The main changes are: 1. forwrading a token of a new broadcast client 2. checking permission of a new client that participates to an existing broadcast
Like the previous commit 10f783e, this commit lets grpcproxy forward an auth token supplied by its client in an explicit manner. snapshot is a stream RPC so this process is required like watch.
@heyitsanthony updated for the latest comments, could you take a look? It seems that CIs are failing because of environmental specific issues.
Probably 1 and 2 were caused by resource shortage? I couldn't reproduce the errors on my env, of course. |
lgtm. CI failure is not related. Merging. |
Actually I reran the semaphoreci again. It is still failing. So it is probably a legit issue. |
|
@gyuho @xiang90 I'm not sure the latest error in semaphore is same to #8581 . In this PR, the error is caused by |
On the master branch, I could reproduce a little bit similar (but different) e2e failure like below:
A single test (in this case, |
@mitake I can confirm it's not related to this change. It's from our recent client balancer patch. |
ok. let us merge this in. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm.
This commit lets grpcproxy support authed RPCs. Auth tokens supplied
by clients are now forwarded to etcdserver by grpcproxy.
Fix #8247
/cc @zhuqunzhou @heyitsanthony
This PR is still WIP because I'm considering how to protect proxy based watch. It isn't simple like other RPCs because of its streaming and broadcast nature.
@zhuqunzhou could you try it for your purpose? Authed KV requests with proxy should work now.