Create guide doc for partition compaction #6512
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CharlieTLe
reviewed
Jan 22, 2025
* Purge expired postings cache items due inactivity Signed-off-by: alanprot <[email protected]> * Fix comments Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]>
* Update thanos to 4ba0ba403896 Signed-off-by: Daniel Sabsay <[email protected]> * run go mod vendor Signed-off-by: Daniel Sabsay <[email protected]> --------- Signed-off-by: Daniel Sabsay <[email protected]> Co-authored-by: Daniel Sabsay <[email protected]> Signed-off-by: Alex Le <[email protected]>
…cortexproject#6505) Bumps the actions-dependencies group with 2 updates in the / directory: [actions/upload-artifact](https://github.com/actions/upload-artifact) and [github/codeql-action](https://github.com/github/codeql-action). Updates `actions/upload-artifact` from 4.5.0 to 4.6.0 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@6f51ac0...65c4c4a) Updates `github/codeql-action` from 3.28.0 to 3.28.1 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@48ab28a...b6a472f) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-dependencies - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Alex Le <[email protected]>
Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]>
* Implement partition compaction grouper Signed-off-by: Alex Le <[email protected]> * fix comment Signed-off-by: Alex Le <[email protected]> * replace level 1 compaction limits with ingestion replication factor Signed-off-by: Alex Le <[email protected]> * fix doc Signed-off-by: Alex Le <[email protected]> * update compaction_visit_marker_timeout default value Signed-off-by: Alex Le <[email protected]> * update default value for compactor_partition_index_size_limit_in_bytes Signed-off-by: Alex Le <[email protected]> * refactor code Signed-off-by: Alex Le <[email protected]> * address comments and refactor Signed-off-by: Alex Le <[email protected]> * address comment Signed-off-by: Alex Le <[email protected]> * address comment Signed-off-by: Alex Le <[email protected]> * update config name Signed-off-by: Alex Le <[email protected]> * Implement partition compaction planner Signed-off-by: Alex Le <[email protected]> * fix after rebase Signed-off-by: Alex Le <[email protected]> * addressed comments Signed-off-by: Alex Le <[email protected]> * updated doc and refactored metric Signed-off-by: Alex Le <[email protected]> * fix test Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]>
Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]>
…les (cortexproject#6507) * Add cleaner logic to clean partition compaction blocks and related files Signed-off-by: Alex Le <[email protected]> * refactored metrics Signed-off-by: Alex Le <[email protected]> * refactor Signed-off-by: Alex Le <[email protected]> * update logs Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]>
Maintainers would like an additional week to get the partition compactor changes in before the first release candidate for 1.19. Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]>
…rtexproject#6514) Signed-off-by: yeya24 <[email protected]> Signed-off-by: Alex Le <[email protected]>
…ortexproject#6518) Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]>
cortexproject#6519) Signed-off-by: Alex Le <[email protected]>
…ject#6522) Signed-off-by: Alex Le <[email protected]>
…oject#6523) * expose grpc client connect timeout config Signed-off-by: yeya24 <[email protected]> * changelog Signed-off-by: yeya24 <[email protected]> --------- Signed-off-by: yeya24 <[email protected]> Signed-off-by: Alex Le <[email protected]>
…#6510) * Implemented partition compaction end to end with custom compaction lifecycle Signed-off-by: Alex Le <[email protected]> * removed unused variable Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * refactor according to comments Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * check context error inside sharded posting Signed-off-by: Alex Le <[email protected]> * fix lint Signed-off-by: Alex Le <[email protected]> * fix integration test for memberlist Signed-off-by: Alex Le <[email protected]> * make compactor initial wait cancellable Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]>
* Test for nil on expire expanded postings Signed-off-by: alanprot <[email protected]> * stopping ingester Signed-off-by: alanprot <[email protected]> * refactor the test to not timeout Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]>
* log when a request starts running in querier Signed-off-by: Ahmed Hassan <[email protected]> * log when a request starts running in querier for frontend processor Signed-off-by: Ahmed Hassan <[email protected]> --------- Signed-off-by: Ahmed Hassan <[email protected]> Signed-off-by: Alex Le <[email protected]>
…6508) Signed-off-by: Friedrich Gonzalez <[email protected]> Signed-off-by: Alex Le <[email protected]>
Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]>
Signed-off-by: Daniel Deluiggi <[email protected]> Signed-off-by: Alex Le <[email protected]>
https://github.com/cortexproject/cortex/blob/master/RELEASE.md#show-that-a-release-is-in-progress Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]>
Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]>
Signed-off-by: Alex Le <[email protected]>
…roup (cortexproject#6547) * Fixed blocksGroupWithPartition unable to reuse functions from blocksGroup Signed-off-by: Alex Le <[email protected]> * update tests Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]>
Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]>
Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]>
* Create partial_data Signed-off-by: Justin Jung <[email protected]> * Fix lazyquery so that warning message is returned Signed-off-by: Justin Jung <[email protected]> * Add QueryPartialData limit Signed-off-by: Justin Jung <[email protected]> * Fix broken mock Signed-off-by: Justin Jung <[email protected]> * Make response with warnings to be not cached Signed-off-by: Justin Jung <[email protected]> * Updated streamingSelect in distributor_queryable Signed-off-by: Justin Jung <[email protected]> * Update query.go Signed-off-by: Justin Jung <[email protected]> * Update replication_set Signed-off-by: Justin Jung <[email protected]> * Lint Signed-off-by: Justin Jung <[email protected]> * Lint again Signed-off-by: Justin Jung <[email protected]> * Generated doc Signed-off-by: Justin Jung <[email protected]> * Changelog Signed-off-by: Justin Jung <[email protected]> * Update config description Signed-off-by: Justin Jung <[email protected]> * Do not remove warnings from seriesSet Signed-off-by: Justin Jung <[email protected]> * Avoid cache only if the warning message contains partial data error Signed-off-by: Justin Jung <[email protected]> * Remove context usage for partial data Signed-off-by: Justin Jung <[email protected]> * Refactor how partial data info is passed + apply to series and label methods as well Signed-off-by: Justin Jung <[email protected]> * Lint + fix tests Signed-off-by: Justin Jung <[email protected]> * Fix build Signed-off-by: Justin Jung <[email protected]> * Create separate config for ruler partial data Signed-off-by: Justin Jung <[email protected]> * Genereta doc Signed-off-by: Justin Jung <[email protected]> * Add more tests Signed-off-by: Justin Jung <[email protected]> * Change error Signed-off-by: Justin Jung <[email protected]> * Fix test Signed-off-by: Justin Jung <[email protected]> * Update changelog Signed-off-by: Justin Jung <[email protected]> * Update changelog Signed-off-by: Justin Jung <[email protected]> * Nit Signed-off-by: Justin Jung <[email protected]> * Nit Signed-off-by: Justin Jung <[email protected]> --------- Signed-off-by: Justin Jung <[email protected]> Signed-off-by: Alex Le <[email protected]>
* add dynamodb kv with timeout enforced Signed-off-by: yeya24 <[email protected]> * add tests Signed-off-by: yeya24 <[email protected]> * docs Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: yeya24 <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Alex Le <[email protected]>
…cortexproject#6564) Bumps the actions-dependencies group with 2 updates in the / directory: [github/codeql-action](https://github.com/github/codeql-action) and [actions/setup-go](https://github.com/actions/setup-go). Updates `github/codeql-action` from 3.28.1 to 3.28.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@b6a472f...6e54559) Updates `actions/setup-go` from 5.2.0 to 5.3.0 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@3041bf5...f111f33) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-dependencies - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Alex Le <[email protected]>
…in the future" (cortexproject#6562) * improve fuzz test for expanded postings cache Signed-off-by: alanprot <[email protected]> * create more tests on the expanded postings cache Signed-off-by: alanprot <[email protected]> * adding get series call on the test Signed-off-by: alanprot <[email protected]> * no use CachedBlockChunkQuerier when query time range is completely after the last sample added in the head Signed-off-by: alanprot <[email protected]> * adding comments Signed-off-by: alanprot <[email protected]> * increase the number of fuzz test from 100 to 300 Signed-off-by: alanprot <[email protected]> * add get series fuzzy testing Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]>
* Filter readOnly ingesters when sharding Signed-off-by: Daniel Deluiggi <[email protected]> * Extend shard on READONLY Signed-off-by: Daniel Deluiggi <[email protected]> * Remove old code Signed-off-by: Daniel Deluiggi <[email protected]> * Fix test Signed-off-by: Daniel Deluiggi <[email protected]> * update changelog Signed-off-by: Daniel Deluiggi <[email protected]> --------- Signed-off-by: Daniel Deluiggi <[email protected]> Signed-off-by: Alex Le <[email protected]>
Signed-off-by: Alex Le <[email protected]>
Co-authored-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> Signed-off-by: Alex Le <[email protected]>
Signed-off-by: Alex Le <[email protected]>
alexqyle
force-pushed
the
partition-compaction-doc
branch
from
65285a1 to
a7466d1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Create guide doc for partition compaction and update CHANGELOG to include partition compaction
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]