Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop watching leases for custodian controller #688

Merged
merged 4 commits into from
Oct 3, 2023

Conversation

shreyas-s-rao
Copy link
Contributor

@shreyas-s-rao shreyas-s-rao commented Sep 29, 2023

How to categorize this PR?

/area quality
/kind bug

What this PR does / why we need it:
Custodian controller no longer watches lease objects, but rather does an initial reconciliation on any etcd object that has finished reconciliation, and requeues the event for after SyncPeriod interval, as was being done previously as well.

This PR ensures that custodian controller does not update the etcd status too frequently, thus saving hundreds of calls to kube-apiserver every second per seed cluster, not to mention the reduction in number of failed etcd status update calls from etcd controller to update observedGeneration and lastError information.

Additionally, the predicate EtcdReconciliationFinished has been modified to check equality of status.ObservedGeneration and metadata.Generation, rather than status.Ready. Reason for this is that reconciliation is technically finished only when etcd reconciler marks the observedGeneration with the generation it "observes", and has nothing to do with whether the etcd cluster is ready or not.

Which issue(s) this PR fixes:
Fixes #683

Special notes for your reviewer:
/invite @unmarshall @seshachalam-yv
/assign @unmarshall @seshachalam-yv

Release note:

Custodian controller no longer watches leases owned by the etcd resources, thus reducing frequency of etcd status updates and now honouring `custodian-sync-period` value.
`custodian-sync-period` value is set to `15s` in the Helm chart for etcd-druid.

@shreyas-s-rao shreyas-s-rao added this to the v0.20.0 milestone Sep 29, 2023
@shreyas-s-rao shreyas-s-rao requested a review from a team as a code owner September 29, 2023 17:07
@gardener-robot gardener-robot added area/quality Output qualification (tests, checks, scans, automation in general, etc.) related kind/bug Bug needs/review Needs review size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) labels Sep 29, 2023
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Sep 29, 2023
@gardener-robot-ci-1 gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Sep 30, 2023
@gardener-robot
Copy link

@unmarshall, @seshachalam-yv You have pull request review open invite, please check

controllers/custodian/config.go Outdated Show resolved Hide resolved
controllers/custodian/config.go Outdated Show resolved Hide resolved
pkg/predicate/predicate.go Show resolved Hide resolved
pkg/predicate/predicate.go Show resolved Hide resolved
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 3, 2023
@gardener-robot-ci-3 gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 3, 2023
@shreyas-s-rao shreyas-s-rao force-pushed the fix/custodian-requeue branch from b1958b9 to f225e10 Compare October 3, 2023 01:24
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 3, 2023
@gardener-robot-ci-3 gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 3, 2023
@shreyas-s-rao
Copy link
Contributor Author

/test pull-etcd-druid-e2e-kind-alpha-features

@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Oct 3, 2023
@shreyas-s-rao
Copy link
Contributor Author

@unmarshall thanks for your comments. I have addressed them. PTAL

@shreyas-s-rao
Copy link
Contributor Author

/retest

@shreyas-s-rao
Copy link
Contributor Author

/test pull-etcd-druid-e2e-kind

@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Oct 3, 2023
Copy link
Contributor

@unmarshall unmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@gardener-robot gardener-robot added reviewed/lgtm Has approval for merging and removed needs/review Needs review labels Oct 3, 2023
@gardener-robot-ci-3 gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 3, 2023
Copy link
Contributor

@seshachalam-yv seshachalam-yv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @shreyas-s-rao Nice Fix
Saving a lot KAPI calls 🚀
/lgtm

@seshachalam-yv seshachalam-yv merged commit 138e1b9 into gardener:master Oct 3, 2023
@gardener-robot gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Oct 3, 2023
@shreyas-s-rao shreyas-s-rao deleted the fix/custodian-requeue branch October 3, 2023 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/quality Output qualification (tests, checks, scans, automation in general, etc.) related kind/bug Bug needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) reviewed/lgtm Has approval for merging reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) status/closed Issue is closed (either delivered or triaged)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Custodian controller does not honour SyncPeriod and updates etcd status too frequently
6 participants