chore: merge the partition implementation of s3stream into new trunk branch of automq kafka #879

daniel-y · 2024-03-05T04:46:40Z

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.

Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

1. add commit wal object test Signed-off-by: TheR1sing3un <[email protected]>

1. still commit valid stream in wal object Signed-off-by: TheR1sing3un <[email protected]>

1. replay WalObjectRecord to advance range's end offset Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): advance range's end offset by WalObjectRecord

1. support new version stream open/close operations Signed-off-by: TheR1sing3un <[email protected]>

1. pass checkstyle Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): support new version stream open/close operations

1. support get streams' offset range 2. remove useless CompactObject related request/response 3. refactor WalObjectRequest to contain compacted objects and stream objects 4. increase all stream related request/response apiKey Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): support get streams' offset range

1. complete wal object commit process in controller Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): complete wal object commit process in controller

1. replay new added state in StreamImage Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): replay new added state in StreamImage

1. boostrap with controller-metadata-manager Signed-off-by: TheR1sing3un <[email protected]>

* fix: fix checkstyle and add ci flow for checkstyle and spotbugs (#106) ci(s3): add checkstyle and spotbugs in ci 1. add checkstyle and spotbugs in ci 2. fix to pass checkstyle Signed-off-by: TheR1sing3un <[email protected]> * fix: fix fetching problem brought by thread pool separating; fix style problems; add more thread for fetching and appending thread pool (#107) * fix: fix fetching problem brought by thread pool separating; fix style problems; add more thread for fetching and appending thread pool Signed-off-by: Curtis Wan <[email protected]> * remove 'SLOW_FETCH_TIMEOUT_MILLIS - 1' case in quickFetch test and change 2 -> 'SLOW_FETCH_TIMEOUT_MILLIS / 2' in slowFetch test Signed-off-by: Curtis Wan <[email protected]> * refactor: add more comments to make logic more clear Signed-off-by: Curtis Wan <[email protected]> --------- Signed-off-by: Curtis Wan <[email protected]> * refactor: close #105; add more threads for partition opening or closing (#109) Signed-off-by: Curtis Wan <[email protected]> * feat(es): client factory SPI (#112) Signed-off-by: Robin Han <[email protected]> * fix: close #108; make sure topics are all cleaned at the end of the test (#115) Signed-off-by: Curtis Wan <[email protected]> * fix: close #110; shutdown additional thread pools right now when broker is in shutdown (#111) * fix: close #110; shutdown additional thread pools right now when broker is in shutdown Signed-off-by: Curtis Wan <[email protected]> * fix: move thread pools into AlwaysSuccessClient Signed-off-by: Curtis Wan <[email protected]> * fix: add more logs; fix partition leading or following Signed-off-by: Curtis Wan <[email protected]> --------- Signed-off-by: Curtis Wan <[email protected]> * fix: close #117; fix return too early; add stack for log closing (#118) Signed-off-by: Curtis Wan <[email protected]> * fix: close #114; use position rather than offset as nextOffset for indexes; close metastream when log is closing (#116) * wip: add more logs Signed-off-by: Curtis Wan <[email protected]> * fix: add ExceptionUtil; use position rather than offset as nextOffset for indexes Signed-off-by: Curtis Wan <[email protected]> * fix: fix style check Signed-off-by: Curtis Wan <[email protected]> --------- Signed-off-by: Curtis Wan <[email protected]> * fix: close #119; skip deleting segments when metaStream is closed (#120) Signed-off-by: Curtis Wan <[email protected]> * feat: close #123; Handle no-retryable exceptions thrown by Elastic Stream SDK (#124) * feat: close #123; Handle no-retryable exceptions thrown by Elastic Stream SDK. The corresponding partitions will be offline; Add context info to pass unit tests for cases that fetching just follows appending. Signed-off-by: Curtis Wan <[email protected]> * feat: enhancement codes refered in discussions Signed-off-by: Curtis Wan <[email protected]> * fix: style problem Signed-off-by: Curtis Wan <[email protected]> * test: wait for more time for quick/slow fetch tests Signed-off-by: Curtis Wan <[email protected]> * test: start fetching after appending finished Signed-off-by: Curtis Wan <[email protected]> --------- Signed-off-by: Curtis Wan <[email protected]> * test: add slowFetchDelay (#135) Signed-off-by: Curtis Wan <[email protected]> * feat(core): Add implementation of AutoBalancer components and unit tests (#134) * feat(core): Add customized MetricsReporter and partition-level metrics - add partition level metrics for BytesInPerSec and BytesOutPerSec - implement CruiseControlMetrics to monitor and report interested Yammer metrics Closes #78 Signed-off-by: sc.nieh <[email protected]> * feat(core): AutoBalancerMetricsReporter optimization - pre-aggregate for broker and partition level metrics - fill with empty value for probably missing metrics #78 Signed-off-by: sc.nieh <[email protected]> * feat(core): Implement load retriever for auto balancer #77 Signed-off-by: sc.nieh <[email protected]> * feat(core): Implement AutoBalancerManager and AnomalyDetector #77 Signed-off-by: sc.nieh <[email protected]> * fix(core): Fix inconsistent in-flight request check Closes #121 Signed-off-by: sc.nieh <[email protected]> * feat(core): Create metrics reporter topic on controller Signed-off-by: sc.nieh <[email protected]> * fix(core): fix execute interval of ExecutionManager Signed-off-by: sc.nieh <[email protected]> * fix: add esUnit tag to unit tests of autobalancer Signed-off-by: sc.nieh <[email protected]> --------- Signed-off-by: sc.nieh <[email protected]> --------- Signed-off-by: TheR1sing3un <[email protected]> Signed-off-by: Curtis Wan <[email protected]> Signed-off-by: Robin Han <[email protected]> Signed-off-by: sc.nieh <[email protected]> Co-authored-by: TheR1sing3un <[email protected]> Co-authored-by: Robin Han <[email protected]> Co-authored-by: Shichao Nie <[email protected]>

1. support controller-kv client 2. change some logs' level from info to trace Signed-off-by: TheR1sing3un <[email protected]>

1. replace object-id with order-id for WAL Signed-off-by: TheR1sing3un <[email protected]>

1. return first object id in preparedObjectResponse Signed-off-by: TheR1sing3un <[email protected]>

…ager` (#64) * refactor(s3): remove inflight wal objects 1. remove inflight wal objects 2. delete redundant classes Signed-off-by: TheR1sing3un <[email protected]> * feat(s3): support blocking `getObjects` in `StreamMetadataManager` 1. support blocking `getObjects` in `StreamMetadataManager` 2. refactor `getObjects` 3. change the unit of `s3.cache.size` from `MB` to `B` Signed-off-by: TheR1sing3un <[email protected]> * feat(s3): more suitable log level 1. more suitable log level Signed-off-by: TheR1sing3un <[email protected]> * fix(s3): add more concurrent protection 1. add more concurrent protection Signed-off-by: TheR1sing3un <[email protected]> --------- Signed-off-by: TheR1sing3un <[email protected]>

1. handle stream objects commit in controller Signed-off-by: TheR1sing3un <[email protected]>

1. handle invalid commit request Signed-off-by: TheR1sing3un <[email protected]>

* feat: rename GetStreamsOffset to GetOpeningStreams Signed-off-by: Robin Han <[email protected]> * feat: get broker opening streams Signed-off-by: Robin Han <[email protected]> --------- Signed-off-by: Robin Han <[email protected]>

Signed-off-by: Robin Han <[email protected]>

* feat(s3): unify S3 object metadata 1. unify S3 object metadata Signed-off-by: TheR1sing3un <[email protected]> * fix(s3): fix after rebasing 1. fix after rebasing Signed-off-by: TheR1sing3un <[email protected]> * feat(s3): log stream-objects' info when commit wal object 1. log stream-objects' info when commit wal object Signed-off-by: TheR1sing3un <[email protected]> --------- Signed-off-by: TheR1sing3un <[email protected]>

1. support retry to-controller request Signed-off-by: TheR1sing3un <[email protected]>

* feat: get broker opening streams Signed-off-by: Robin Han <[email protected]> * feat: recover from crash (WAL buggy version) Signed-off-by: Robin Han <[email protected]> --------- Signed-off-by: Robin Han <[email protected]>

Signed-off-by: Robin Han <[email protected]>

* fix(s3): fixed bugs on uploading WAL object during compaction - eliminate race condition on uploading WAL object by using sequential writing - prevent blocking on multipart upload when part size is less than MIN_PART_SIZE Signed-off-by: Shichao Nie <[email protected]> * fix(s3): used pooled buffer in DataBlockWriter; add newFixedThreadPool to Threads Signed-off-by: Shichao Nie <[email protected]> --------- Signed-off-by: Shichao Nie <[email protected]>

…er (#843) Signed-off-by: Shichao Nie <[email protected]>

Signed-off-by: Shichao Nie <[email protected]>

Signed-off-by: Robin Han <[email protected]>

Signed-off-by: Shichao Nie <[email protected]>

* feat(metrics): metrics on fetch limiters Signed-off-by: Ning Yu <[email protected]> * feat(metrics): metrics on fetch executors' queue size Signed-off-by: Ning Yu <[email protected]> * style: fix check style Signed-off-by: Ning Yu <[email protected]> --------- Signed-off-by: Ning Yu <[email protected]>

Signed-off-by: Shichao Nie <[email protected]>

- deduplication for consumer lag calculation on partition reassignment - add get/put object avg latency panels Signed-off-by: Shichao Nie <[email protected]>

* fix(telemetry): add missing percentile metrics Signed-off-by: Shichao Nie <[email protected]> * fix(telemetry): increase metric expiration time for collector Signed-off-by: Shichao Nie <[email protected]> --------- Signed-off-by: Shichao Nie <[email protected]>

Signed-off-by: Shichao Nie <[email protected]>

Signed-off-by: Robin Han <[email protected]>

Signed-off-by: Shichao Nie <[email protected]>

…eporter interval (#875) Signed-off-by: Shichao Nie <[email protected]>

Signed-off-by: Shichao Nie <[email protected]>

CLAassistant · 2024-03-05T04:46:49Z

All committers have signed the CLA.

…branch of automq kafka

…t exist (#879) Signed-off-by: SSpirits <[email protected]>

TheR1sing3un and others added 30 commits August 29, 2023 11:05

test(s3): add commit wal object test

5dff89c

1. add commit wal object test Signed-off-by: TheR1sing3un <[email protected]>

fix(s3): still commit valid stream in wal object

51d471a

1. still commit valid stream in wal object Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): replay WalObjectRecord to advance range's end offset

7a4e2e5

1. replay WalObjectRecord to advance range's end offset Signed-off-by: TheR1sing3un <[email protected]>

Merge pull request #34 from AutoMQ/refactor_range_end_offset

bb18f95

feat(s3): advance range's end offset by WalObjectRecord

feat(s3): support new version stream open/close operations

a65ca17

1. support new version stream open/close operations Signed-off-by: TheR1sing3un <[email protected]>

style(s3): pass checkstyle

e4a4bf2

1. pass checkstyle Signed-off-by: TheR1sing3un <[email protected]>

Merge pull request #37 from AutoMQ/feat_streammanager

accf654

feat(s3): support new version stream open/close operations

Merge pull request #38 from AutoMQ/support_get_stream_offset

4cb74be

feat(s3): support get streams' offset range

feat(s3): complete wal object commit process in controller

195b865

1. complete wal object commit process in controller Signed-off-by: TheR1sing3un <[email protected]>

Merge pull request #39 from AutoMQ/feat_new_wal_handler

068b3c4

feat(s3): complete wal object commit process in controller

feat(s3): replay new added state in StreamImage

7597551

1. replay new added state in StreamImage Signed-off-by: TheR1sing3un <[email protected]>

Merge pull request #42 from AutoMQ/feat_wal_image

a054d1f

feat(s3): replay new added state in StreamImage

feat(s3): boostrap Kafka on S3 (#49)

844f2f8

1. boostrap with controller-metadata-manager Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): support controller-kv client (#55)

a9f637b

1. support controller-kv client 2. change some logs' level from info to trace Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): replace object-id with order-id for WAL (#56)

295ba7c

1. replace object-id with order-id for WAL Signed-off-by: TheR1sing3un <[email protected]>

refactor(s3): return first object id in preparedObjectResponse (#60)

4e87747

1. return first object id in preparedObjectResponse Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): handle stream objects commit in controller (#71)

09fd581

1. handle stream objects commit in controller Signed-off-by: TheR1sing3un <[email protected]>

feat(s3): handle invalid commit request (#75)

543e422

1. handle invalid commit request Signed-off-by: TheR1sing3un <[email protected]>

feat: get broker opening streams (#85)

bd13e80

* feat: rename GetStreamsOffset to GetOpeningStreams Signed-off-by: Robin Han <[email protected]> * feat: get broker opening streams Signed-off-by: Robin Han <[email protected]> --------- Signed-off-by: Robin Han <[email protected]>

fix: convert brokerId to int32 (#90)

1dbd3a9

Signed-off-by: Robin Han <[email protected]>

feat(s3): support retry to-controller request (#87)

142d55e

1. support retry to-controller request Signed-off-by: TheR1sing3un <[email protected]>

feat: recover from crash (#107)

46ba6f0

* feat: get broker opening streams Signed-off-by: Robin Han <[email protected]> * feat: recover from crash (WAL buggy version) Signed-off-by: Robin Han <[email protected]> --------- Signed-off-by: Robin Han <[email protected]>

fix: fix WAL iterator & data position (#113)

8487cab

Signed-off-by: Robin Han <[email protected]>

feat: directly split one stream WAL (#122)

56f6c04

Signed-off-by: Robin Han <[email protected]>

fix: fix controller check prepare object NPE (#124)

e2bed96

Signed-off-by: Robin Han <[email protected]>

SCNieh and others added 21 commits February 26, 2024 18:12

feat(core): redirect JUL log to sl4j and remove unused logging export…

05842b1

…er (#843) Signed-off-by: Shichao Nie <[email protected]>

fix(core): fix node id regex in broker dashboard (#841)

d971f8e

Signed-off-by: Shichao Nie <[email protected]>

feat: record pooled record memory usage (#846)

6b7f329

Signed-off-by: Robin Han <[email protected]>

fix(metrics): add label 'version' to kafka.request.count (#847)

ea2ed9e

Signed-off-by: Shichao Nie <[email protected]>

feat(telemetry): add host name to OTel resource (#849)

8bf8103

Signed-off-by: Shichao Nie <[email protected]>

feat(metrics): add buffer and thread metrics (#851)

143eea9

feat(telemetry): add direct memory panels (#853)

8baa89c

Signed-off-by: Shichao Nie <[email protected]>

fix(telemetry): fix read ahead throughput panel unit (#854)

9450e52

Signed-off-by: Shichao Nie <[email protected]>

feat(metrics): rename DirectByteBufAlloc to ByteBufAlloc (#855)

9b428ea

fix(telemetry): fix memory allocation metrics name (#856)

553c95e

Signed-off-by: Shichao Nie <[email protected]>

feat(metrics): enable buffer pools metrics (#857)

e172762

fix(telemetry): fix jvm metrics (#859)

1989644

Signed-off-by: Shichao Nie <[email protected]>

fix(telemetry): refine grafana dashboard (#860)

33a22e4

- deduplication for consumer lag calculation on partition reassignment - add get/put object avg latency panels Signed-off-by: Shichao Nie <[email protected]>

fix(telemetry): fix topic dashboard name (#868)

51fa8ba

Signed-off-by: Shichao Nie <[email protected]>

feat(issues871): log request info for stream object compaction (#872)

f95ca67

Signed-off-by: Robin Han <[email protected]>

feat(core): merge multiple reassignments for same partition (#873)

272a28c

Signed-off-by: Shichao Nie <[email protected]>

feat(dashboard): add node level network metrics (#874)

13e5099

Signed-off-by: Shichao Nie <[email protected]>

feat(dashboard): set query interval to 30s to match default metrics r…

881bc78

…eporter interval (#875) Signed-off-by: Shichao Nie <[email protected]>

fix(core): support alter auto balancer topic partitions (#876)

4a52cc8

Signed-off-by: Shichao Nie <[email protected]>

chore: merge the partition implementation of s3stream into new trunk …

ba76df0

…branch of automq kafka

daniel-y force-pushed the s3stream-to-trunk branch from d044671 to ba76df0 Compare March 5, 2024 04:50

superhx approved these changes Mar 5, 2024

View reviewed changes

superhx merged commit bcfd816 into merge_trunk Mar 5, 2024
1 check passed

superhx deleted the s3stream-to-trunk branch March 5, 2024 04:51

daniel-y restored the s3stream-to-trunk branch March 14, 2024 08:50

daniel-y pushed a commit that referenced this pull request Mar 14, 2024

fix(stream): use getBucketACL instead of headBucket to check if bucke…

a38ec06

…t exist (#879) Signed-off-by: SSpirits <[email protected]>

ShadowySpirits added a commit that referenced this pull request Mar 14, 2024

fix(stream): use getBucketACL instead of headBucket to check if bucke…

872a41d

…t exist (#879) Signed-off-by: SSpirits <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: merge the partition implementation of s3stream into new trunk branch of automq kafka #879

chore: merge the partition implementation of s3stream into new trunk branch of automq kafka #879

daniel-y commented Mar 5, 2024

CLAassistant commented Mar 5, 2024 •

edited

Loading

chore: merge the partition implementation of s3stream into new trunk branch of automq kafka #879

chore: merge the partition implementation of s3stream into new trunk branch of automq kafka #879

Conversation

daniel-y commented Mar 5, 2024

Committer Checklist (excluded from commit message)

CLAassistant commented Mar 5, 2024 • edited Loading

CLAassistant commented Mar 5, 2024 •

edited

Loading