Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "Parallelize creating cluster snapshot" #7695

Closed
wants to merge 1 commit into from
Closed

Revert "Parallelize creating cluster snapshot" #7695

wants to merge 1 commit into from

Conversation

x13n
Copy link
Member

@x13n x13n commented Jan 15, 2025

Reverts #7630

There's a race condition when clearing the cache, which sometimes breaks unit tests:

WARNING: DATA RACE
Write at 0x00c001446ae8 by goroutine 49556:
  k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot/store.(*internalDeltaSnapshotData).clearPodCaches()
      /cluster-autoscaler/simulator/clustersnapshot/store/delta.go:185 +0x124
  k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot/store.(*internalDeltaSnapshotData).clearCaches()
      /cluster-autoscaler/simulator/clustersnapshot/store/delta.go:178 +0xb6
  k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot/store.(*internalDeltaSnapshotData).addPodsToNode()
      /cluster-autoscaler/simulator/clustersnapshot/store/delta.go:265 +0x82
  k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot/store.(*DeltaSnapshotStore).setClusterStatePodsParallelized.func1()
      /cluster-autoscaler/simulator/clustersnapshot/store/delta.go:485 +0x11b
  k8s.io/client-go/util/workqueue.ParallelizeUntil.func1()
      /gopath/pkg/mod/k8s.io/[email protected]/util/workqueue/parallelizer.go:90 +0x1c1

Previous write at 0x00c001446ae8 by goroutine 49567:
  k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot/store.(*internalDeltaSnapshotData).clearPodCaches()
      /cluster-autoscaler/simulator/clustersnapshot/store/delta.go:185 +0x124
  k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot/store.(*internalDeltaSnapshotData).clearCaches()
      /cluster-autoscaler/simulator/clustersnapshot/store/delta.go:178 +0xb6
  k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot/store.(*internalDeltaSnapshotData).addPodsToNode()
      /cluster-autoscaler/simulator/clustersnapshot/store/delta.go:265 +0x82
  k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot/store.(*DeltaSnapshotStore).setClusterStatePodsParallelized.func1()
      /cluster-autoscaler/simulator/clustersnapshot/store/delta.go:485 +0x11b
  k8s.io/client-go/util/workqueue.ParallelizeUntil.func1()
      /gopath/pkg/mod/k8s.io/[email protected]/util/workqueue/parallelizer.go:90 +0x1c1

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: x13n

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 15, 2025
@k8s-ci-robot k8s-ci-robot added area/cluster-autoscaler approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 15, 2025
@towca
Copy link
Collaborator

towca commented Jan 15, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 15, 2025
macsko
macsko reviewed Jan 15, 2025
View reviewed changes
cluster-autoscaler/simulator/clustersnapshot/store/delta.go
}

// Maybe consider deleting from the list in the future. Maybe not.
data.clearCaches()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is revert the best option? Looks like this data.clearCaches() is redundant when setting cluster state and its removal will fix the race.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can quickly fix it then we can skip the revert I think

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But in general it is safer to revert first, and submit fixed version later, so that tests are not broken at HEAD.

@macsko
Copy link
Member

macsko commented Jan 15, 2025

/hold (#7695 (comment))

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 15, 2025
@macsko macsko mentioned this pull request Jan 15, 2025
Merged
@BigDarkClown
Copy link
Contributor

/lgtm

@x13n
Copy link
Member Author

x13n commented Jan 15, 2025

Fixed in #7696, closing this one.

@x13n x13n closed this Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@macsko macsko macsko left review comments

@aleksandra-malinowska aleksandra-malinowska Awaiting requested review from aleksandra-malinowska

@feiskyer feiskyer Awaiting requested review from feiskyer

Assignees

@towca towca

@BigDarkClown BigDarkClown

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/cluster-autoscaler cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@x13n @k8s-ci-robot @towca @macsko @BigDarkClown