Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gracefully exit the program when the lease expired #2655

Merged
merged 10 commits into from
Jan 19, 2022

Conversation

px3303
Copy link
Contributor

@px3303 px3303 commented Jan 12, 2022

This PR can let trillian proactively Listen "LeaseKeepAliveResponse" channel returned by KeepAlive in ETCD client. When automatic renewal interruption is detected, Exit the program by canceling the context.

Fixes #2654,#2249

Checklist

Listen "LeaseKeepAliveResponse" channel returned by keepalive in ETCD client. When automatic renewal interruption is detected, the caller will be informed through Context, and then the caller can cancel the global context to exit the progrd.
@px3303 px3303 requested a review from a team as a code owner January 12, 2022 12:17
@px3303 px3303 requested a review from mhutchinson January 12, 2022 12:17
@mhutchinson
Copy link
Contributor

Sorry for the slow reply, I've been at work trying to understand this change. The pattern you're using here feels a bit opaque:

  1. Code calls a sub-method, which returns a child context
  2. Code then awaits cancellation of child context
  3. On cancellation of the child context, the parent context is cancelled

I've created a change that I think is equivalent with respect to propagating etcd cancellation, but does it as:

  1. Code passes the cancellation function into the sub-method
  2. If the lease is cancelled, the cancel function is called

This change is on my fork at https://github.com/google/trillian/compare/master...mhutchinson:etcd_keepalive?expand=1. If you like this, then please do integrate this into the PR. If this change breaks something, then let me know!

@px3303 px3303 closed this Jan 15, 2022
@px3303 px3303 reopened this Jan 15, 2022
@px3303
Copy link
Contributor Author

px3303 commented Jan 15, 2022

@mhutchinson

Sorry for my bad english. Thanks a lot for your suggestion, I decided to go with it. But you removed the AwaitContext function, stop the rpc server blocking depending on it, so i added it again.

@mhutchinson
Copy link
Contributor

/gcbrun


// listenKeepAliveRsp listens to `keepAliveRspCh` channel, and calls the cancel function
// to notify the lease expired.
func listenKeepAliveRsp(ctx context.Context, keepAliveRspCh <-chan *clientv3.LeaseKeepAliveResponse, cancel func()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it'd be slightly easier to grok if this method blocked and the caller is responsible for doing go listenKeepAliveRsp(...)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

util/process.go Outdated
@@ -42,3 +42,21 @@ func AwaitSignal(ctx context.Context, doneFn func()) {
glog.Infof("AwaitSignal canceled: %v", ctx.Err())
}
}

// AwaitContext waits for context done, then runs the given function.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be tempted to keep this as an unexported function in the serverutil package it's called from...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@px3303 px3303 requested a review from AlCutter January 18, 2022 06:10
@AlCutter
Copy link
Member

/gcbrun

Copy link
Member

@AlCutter AlCutter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mhutchinson for a 2nd pair of eyes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

KeepAlive in etcd client cannot ensure the permanent validity of lease
3 participants