Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

Throttle vacuum workers based on chunk age #1761

Merged
merged 1 commit into from
Nov 28, 2022
Merged

Throttle vacuum workers based on chunk age #1761

merged 1 commit into from
Nov 28, 2022

Conversation

jgpruitt
Copy link
Contributor

@jgpruitt jgpruitt commented Nov 18, 2022

Description

Throttle vacuum workers based on chunk age. Figure out how many workers to used based on the oldest transaction id age of the chunks in our batch. The age should be between vacuumFreezeMinAge and autovacuumFreezeMaxAge. The closer the maxChunkAge is to the autovacuumFreezeMaxAge, the more workers we assign. If maxChunkAge is == vacuumFreezeMinAge, then we just use 1 worker.

	       vacuumFreezeMinAge
	        |              maxChunkAge
	        |               |       autovacuumFreezeMaxAge
	0-------|---------------X--------|------
	        1000            6789     11000

In this example, maxChunkAge is 57.89% of the way between vacuumFreezeMinAge and autovacuumFreezeMaxAge. If maxParallelism is 5, then we will use 3 workers.

If the autovacuum engine beats us to a chunk, skip that chunk.

Merge requirements

Please take into account the following non-code changes that you may need to make with your PR:

  • CHANGELOG entry for user-facing changes
  • Updated the relevant documentation

@jgpruitt jgpruitt self-assigned this Nov 18, 2022
@jgpruitt jgpruitt linked an issue Nov 18, 2022 that may be closed by this pull request
@jgpruitt jgpruitt marked this pull request as ready for review November 18, 2022 21:27
@jgpruitt jgpruitt requested a review from a team as a code owner November 18, 2022 21:27
Copy link
Contributor

@cevian cevian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, but would like some cleanup esp wrt using current_setting

pkg/vacuum/vacuum.go Outdated Show resolved Hide resolved
pkg/vacuum/vacuum.go Show resolved Hide resolved
pkg/vacuum/vacuum.go Outdated Show resolved Hide resolved
pkg/vacuum/vacuum.go Outdated Show resolved Hide resolved
pkg/vacuum/vacuum.go Outdated Show resolved Hide resolved
pkg/vacuum/vacuum.go Outdated Show resolved Hide resolved
pkg/vacuum/vacuum.go Outdated Show resolved Hide resolved
pkg/vacuum/vacuum.go Show resolved Hide resolved
pkg/vacuum/vacuum.go Show resolved Hide resolved
Copy link
Contributor

@cevian cevian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but I'd like some of Arun's comments addressed. Especially the ones relating to metric exposition

@jgpruitt jgpruitt force-pushed the john/vacuum branch 5 times, most recently from fe723ce to 086a8c6 Compare November 28, 2022 18:40
Throttle vacuum workers based on chunk age. Figure out how many workers
to used based on the oldest transaction id age of the chunks in our
batch. The age should be between vacuumFreezeMinAge and
autovacuumFreezeMaxAge. The closer the maxChunkAge is to the
autovacuumFreezeMaxAge, the more workers we assign. If maxChunkAge is
== vacuumFreezeMinAge, then we just use 1 worker.

If the autovacuum engine beats us to a chunk, skip that chunk.
@jgpruitt jgpruitt requested review from arajkumar and cevian November 28, 2022 18:44
@jgpruitt jgpruitt merged commit ac1cf99 into master Nov 28, 2022
@jgpruitt jgpruitt deleted the john/vacuum branch November 28, 2022 20:44
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Modify the vacuum engine to be more adaptive and use less CPU
4 participants