Use partitions ring in write path and ingesters consumption #7376
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pracucci
commented
Feb 14, 2024
| // It can be nil, in which case a simple `go f()` will be used. | ||
| // See Config.ReusableIngesterPushWorkers on how to configure this. | ||
| ingesterDoBatchPushWorkers func(func()) | ||
| doBatchPushWorkers func(func()) |
There was a problem hiding this comment.
Note to reviewers: renamed because it's also used for partitions.
dimitarvdimitrov
approved these changes
Feb 14, 2024
Comment on lines
+62
to
+64
| func NewPartitionReaderForPusher(kafkaCfg KafkaConfig, partitionID int32, consumerGroup string, pusher Pusher, logger log.Logger, reg prometheus.Registerer) (*PartitionReader, error) { | ||
| consumer := newPusherConsumer(pusher, reg, logger) | ||
| return newPartitionReader(kafkaCfg, partitionID, consumer, logger, reg) | ||
| return newPartitionReader(kafkaCfg, partitionID, consumerGroup, consumer, logger, reg) |
There was a problem hiding this comment.
do you want to fold the ConsumerGroup into KafkaConfig in the interest of fewer parameters? (I'd even say partitionID should have been there in the first place, but maybe later on)
There was a problem hiding this comment.
Right now, KafkaConfig is what you user can configure while input args is internally config. I would keep this distinction for now. I find it easier to reason about.
Signed-off-by: Marco Pracucci <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> Co-authored-by: Peter Štibraný <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
pracucci
force-pushed
the
write-and-consume-partitions
branch
from
6568441 to
784ad79
Compare
Signed-off-by: Marco Pracucci <[email protected]>
dimitarvdimitrov
approved these changes
Feb 14, 2024
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR is another piece towards the partitions ring implementation, as part of the experimental ingest storage based on Kafka. In particular, in this PR I'm changing distributors (write path only) and ingesters to use partitions from the partitions ring.
How it works:
ingester-zone-a-0owns the partition0)ingestion_partitions_tenant_shard_sizeWhat's excluded from this PR:
This PR is based on grafana/dskit#484
Which issue(s) this PR fixes or relates to
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.