Reset offsets supervisor API #14772
Merged
Reset offsets supervisor API #14772
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add a new endpoint /druid/indexer/v1/supervisor/<supervisorId>/resetOffsets which accepts DataSourceMetadata as a body parameter. - Update logs, unit tests and docs.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
...exing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaDataSourceMetadataTest.java
Fixed
Show resolved
Hide resolved
...g-service/src/test/java/org/apache/druid/indexing/kinesis/KinesisDataSourceMetadataTest.java
Fixed
Show fixed
Hide fixed
...a/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisorStateTest.java
Fixed
Show fixed
Hide fixed
...nce/src/main/java/org/apache/druid/indexing/materializedview/MaterializedViewSupervisor.java
Outdated
Show resolved
Hide resolved
abhishekrb19
force-pushed
the
supervisor_reset_offsets_api
branch
from
d3500ef to
9212182
Compare
abhishekagarwal87
added
the
Needs web console change
jon-wei
reviewed
Aug 9, 2023
.../main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
Outdated
Show resolved
Hide resolved
abhishekrb19
force-pushed
the
supervisor_reset_offsets_api
branch
2 times, most recently
from
fe09257 to
e9b4dec
Compare
abhishekrb19
force-pushed
the
supervisor_reset_offsets_api
branch
from
e9b4dec to
03b9324
Compare
abhishekrb19
force-pushed
the
supervisor_reset_offsets_api
branch
from
9d97584 to
8bcb1e5
Compare
|
The CodeQL alert about log entries depending on a user-provided value appears to be flagging existing logging code that's not part of this PR. I think it's a false positive alert. |
vogievetsky
reviewed
Aug 15, 2023
abhishekrb19
force-pushed
the
supervisor_reset_offsets_api
branch
3 times, most recently
from
94f0b4f to
11671f6
Compare
abhishekrb19
force-pushed
the
supervisor_reset_offsets_api
branch
3 times, most recently
from
c57d0fc to
3293590
Compare
abhishekrb19
force-pushed
the
supervisor_reset_offsets_api
branch
from
3293590 to
eaa1d26
Compare
abhishekrb19
force-pushed
the
supervisor_reset_offsets_api
branch
from
c6c23ed to
9e04ad5
Compare
AmatyaAvadhanula
approved these changes
Aug 17, 2023
|
Ignoring the CodeQL issues per the comment above |
vogievetsky
removed
the
Needs web console change
This was referenced Aug 24, 2023
pagrawal10
pushed a commit
to confluentinc/druid
that referenced
this pull request
Jun 12, 2024
* Add supervisor /resetOffsets API. - Add a new endpoint /druid/indexer/v1/supervisor/<supervisorId>/resetOffsets which accepts DataSourceMetadata as a body parameter. - Update logs, unit tests and docs. * Add a new interface method for backwards compatibility. * Rename * Adjust tests and javadocs. * Use CoreInjectorBuilder instead of deprecated makeInjectorWithModules * UT fix * Doc updates. * remove extraneous debugging logs. * Remove the boolean setting; only ResetHandle() and resetInternal() * Relax constraints and add a new ResetOffsetsNotice; cleanup old logic. * A separate ResetOffsetsNotice and some cleanup. * Minor cleanup * Add a check & test to verify that sequence numbers are only of type SeekableStreamEndSequenceNumbers * Add unit tests for the no op implementations for test coverage * CodeQL fix * checkstyle from merge conflict * Doc changes * DOCUSAURUS code tabs fix. Thanks, Brian!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new API:
/druid/indexer/v1/supervisor/<supervisorId>/resetOffsets. This API accepts a reset datasource metadata in the body and only resets the specified partition offsets in the metadata store. Unlike the/resetAPI, this new API retains any previously checkpointed partition offsets not specified in the payload, so tasks will resume from where they left off. Further, only tasks serving the reset partitions will be restarted.Description
A few use cases:
/resetthat will clear the entire datasource metadata and start from the earliest or latest offset, users can now specify specific partition offsets.Why not re-purpose the existing
resetAPI endpoint and the associated interface method?resetAPI, so accepting a new body in the existing API will cause some clients to break.The API flow is as follows:
/resetOffsetsendpointResetOffsetsNoticeis createdRelease note
Users can reset specific partition offsets for a supervisor using the
resetOffsetsAPI.Key changed/added classes in this PR
SupervisorResource.javaSupervisor.javaSeekableStreamSupervisor.javaSeekableStreamSupervisorStateTest.javaTestSeekableStreamDataSourceMetadata.javaThis PR has: