Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(s3): return first object id in preparedObjectResponse #60

Merged
merged 1 commit into from
Sep 5, 2023

Conversation

TheR1sing3un
Copy link
Contributor

  1. return first object id in preparedObjectResponse

1. return first object id in preparedObjectResponse

Signed-off-by: TheR1sing3un <[email protected]>
@TheR1sing3un TheR1sing3un requested a review from superhx September 5, 2023 09:57
@TheR1sing3un TheR1sing3un merged commit ab111f3 into develop Sep 5, 2023
@TheR1sing3un TheR1sing3un deleted the feat_wal_compact_metadata branch September 5, 2023 10:19
daniel-y pushed a commit that referenced this pull request Mar 5, 2024
1. return first object id in preparedObjectResponse

Signed-off-by: TheR1sing3un <[email protected]>
superhx added a commit that referenced this pull request Mar 5, 2024
…branch of automq kafka (#879)

* test(s3): add commit wal object test

1. add commit wal object test

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): still commit valid stream in wal object

1. still commit valid stream in wal object

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): replay WalObjectRecord to advance range's end offset

1. replay WalObjectRecord to advance range's end offset

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): support new version stream open/close operations

1. support new version stream open/close operations

Signed-off-by: TheR1sing3un <[email protected]>

* style(s3): pass checkstyle

1. pass checkstyle

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): support get streams' offset range

1. support get streams' offset range
2. remove useless
CompactObject
related request/response
3. refactor WalObjectRequest to
contain
compacted objects and stream objects
4. increase all stream
related request/response apiKey

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): complete wal object commit process in controller

1. complete wal object commit process in controller

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): replay new added state in StreamImage

1. replay new added state in StreamImage

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): boostrap Kafka on S3 (#49)

1. boostrap with controller-metadata-manager

Signed-off-by: TheR1sing3un <[email protected]>

* feat: merge ESK (#54)

* fix: fix checkstyle and add ci flow for checkstyle and spotbugs (#106)

ci(s3): add checkstyle and spotbugs in ci

1. add checkstyle and spotbugs in ci
2. fix to pass checkstyle

Signed-off-by: TheR1sing3un <[email protected]>

* fix: fix fetching problem brought by thread pool separating; fix style problems; add more thread for fetching and appending thread pool (#107)

* fix: fix fetching problem brought by thread pool separating; fix style problems; add more thread for fetching and appending thread pool

Signed-off-by: Curtis Wan <[email protected]>

* remove 'SLOW_FETCH_TIMEOUT_MILLIS - 1' case in quickFetch test and change 2 -> 'SLOW_FETCH_TIMEOUT_MILLIS / 2' in slowFetch test

Signed-off-by: Curtis Wan <[email protected]>

* refactor: add more comments to make logic more clear

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* refactor: close #105; add more threads for partition opening or closing (#109)

Signed-off-by: Curtis Wan <[email protected]>

* feat(es): client factory SPI (#112)

Signed-off-by: Robin Han <[email protected]>

* fix: close #108; make sure topics are all cleaned at the end of the test (#115)

Signed-off-by: Curtis Wan <[email protected]>

* fix: close #110; shutdown additional thread pools right now when broker is in shutdown (#111)

* fix: close #110; shutdown additional thread pools right now when broker is in shutdown

Signed-off-by: Curtis Wan <[email protected]>

* fix: move thread pools into AlwaysSuccessClient

Signed-off-by: Curtis Wan <[email protected]>

* fix: add more logs; fix partition leading or following

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* fix: close #117; fix return too early; add stack for log closing (#118)

Signed-off-by: Curtis Wan <[email protected]>

* fix: close #114; use position rather than offset as nextOffset for indexes; close metastream when log is closing (#116)

* wip: add more logs

Signed-off-by: Curtis Wan <[email protected]>

* fix: add ExceptionUtil; use position rather than offset as nextOffset for indexes

Signed-off-by: Curtis Wan <[email protected]>

* fix: fix style check

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* fix: close #119; skip deleting segments when metaStream is closed (#120)

Signed-off-by: Curtis Wan <[email protected]>

* feat: close #123; Handle no-retryable exceptions thrown by Elastic Stream SDK (#124)

* feat: close #123; Handle no-retryable exceptions thrown by Elastic Stream SDK. The corresponding partitions will be offline; Add context info to pass unit tests for
cases that fetching just follows appending.

Signed-off-by: Curtis Wan <[email protected]>

* feat: enhancement codes refered in discussions

Signed-off-by: Curtis Wan <[email protected]>

* fix: style problem

Signed-off-by: Curtis Wan <[email protected]>

* test: wait for more time for quick/slow fetch tests

Signed-off-by: Curtis Wan <[email protected]>

* test: start fetching after appending finished

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* test: add slowFetchDelay (#135)

Signed-off-by: Curtis Wan <[email protected]>

* feat(core): Add implementation of AutoBalancer components and unit tests  (#134)

* feat(core): Add customized MetricsReporter and partition-level metrics

- add partition level metrics for BytesInPerSec and BytesOutPerSec
- implement CruiseControlMetrics to monitor and report interested Yammer
metrics

Closes #78

Signed-off-by: sc.nieh <[email protected]>

* feat(core): AutoBalancerMetricsReporter optimization

- pre-aggregate for broker and partition level metrics
- fill with empty value for probably missing metrics

#78

Signed-off-by: sc.nieh <[email protected]>

* feat(core): Implement load retriever for auto balancer

#77

Signed-off-by: sc.nieh <[email protected]>

* feat(core): Implement AutoBalancerManager and AnomalyDetector

#77

Signed-off-by: sc.nieh <[email protected]>

* fix(core): Fix inconsistent in-flight request check

Closes #121

Signed-off-by: sc.nieh <[email protected]>

* feat(core): Create metrics reporter topic on controller

Signed-off-by: sc.nieh <[email protected]>

* fix(core): fix execute interval of ExecutionManager

Signed-off-by: sc.nieh <[email protected]>

* fix: add esUnit tag to unit tests of autobalancer

Signed-off-by: sc.nieh <[email protected]>

---------

Signed-off-by: sc.nieh <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>
Signed-off-by: Curtis Wan <[email protected]>
Signed-off-by: Robin Han <[email protected]>
Signed-off-by: sc.nieh <[email protected]>
Co-authored-by: TheR1sing3un <[email protected]>
Co-authored-by: Robin Han <[email protected]>
Co-authored-by: Shichao Nie <[email protected]>

* feat(s3): support controller-kv client (#55)

1. support controller-kv client
2. change some logs' level from info to
trace

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): replace object-id with order-id for WAL (#56)

1. replace object-id with order-id for WAL

Signed-off-by: TheR1sing3un <[email protected]>

* refactor(s3): return first object id in preparedObjectResponse (#60)

1. return first object id in preparedObjectResponse

Signed-off-by: TheR1sing3un <[email protected]>

* feat(stream-client): optimize getting objects from `StreamMetadataManager` (#64)

* refactor(s3): remove inflight wal objects

1. remove inflight wal objects
2. delete redundant classes

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): support blocking `getObjects` in `StreamMetadataManager`

1. support blocking `getObjects` in `StreamMetadataManager`
2. refactor
`getObjects`
3. change the unit of `s3.cache.size` from `MB`
to `B`

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): more suitable log level

1. more suitable log level

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): add more concurrent protection

1. add more concurrent protection

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): handle stream objects commit in controller (#71)

1. handle stream objects commit in controller

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): handle invalid commit request (#75)

1. handle invalid commit request

Signed-off-by: TheR1sing3un <[email protected]>

* feat: get broker opening streams (#85)

* feat: rename GetStreamsOffset to GetOpeningStreams

Signed-off-by: Robin Han <[email protected]>

* feat: get broker opening streams

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* fix: convert brokerId to int32 (#90)

Signed-off-by: Robin Han <[email protected]>

* feat(s3): unify S3 object metadata (#86)

* feat(s3): unify S3 object metadata

1. unify S3 object metadata

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): fix after rebasing

1. fix after rebasing

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): log stream-objects' info when commit wal object

1. log stream-objects' info when commit wal object

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): support retry to-controller request (#87)

1. support retry to-controller request

Signed-off-by: TheR1sing3un <[email protected]>

* feat: recover from crash (#107)

* feat: get broker opening streams

Signed-off-by: Robin Han <[email protected]>

* feat: recover from crash (WAL buggy version)

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* fix: fix WAL iterator & data position (#113)

Signed-off-by: Robin Han <[email protected]>

* feat: directly split one stream WAL (#122)

Signed-off-by: Robin Han <[email protected]>

* fix: fix controller check prepare object NPE (#124)

Signed-off-by: Robin Han <[email protected]>

* fix(s3): fixed bugs on uploading WAL object during compaction (#130)

* fix(s3): fixed bugs on uploading WAL object during compaction

- eliminate race condition on uploading WAL object by using sequential
writing
- prevent blocking on multipart upload when part size is less
than MIN_PART_SIZE

Signed-off-by: Shichao Nie <[email protected]>

* fix(s3): used pooled buffer in DataBlockWriter; add newFixedThreadPool to Threads

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat(s3): support trim-stream operation (#137)

* feat(s3): add trim-stream operation protocol

1. add trim-stream operation protocol

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): support trim-stream operation

1. support trim-stream operation

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): minor fix

1. minor fix

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): delete destroyed object in S3 (#161)

* feat(s3): delete destroyed object in S3

1. delete destroyed object in S3

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): configuration about mock S3 operator

1. configuration about mock S3 operator

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): support trim beyond end offset (#162)

1. support trim beyond end offset

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): support continuity check when commit wal object (#166)

* feat(s3): support continuity check when commit wal object

1. support continuity check when commit wal object
2. when trim offset
larger than current end offset, still keep current range's end offset

Signed-off-by: TheR1sing3un <[email protected]>

* style(s3): fix suppress warning

1. fix suppress warning

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>

* fix: fix stream open (#176)

Signed-off-by: Robin Han <[email protected]>

* feat(s3): support broker epoch (#179)

* feat(s3): add broker epoch field in stream related request protocol

1. add `broker-epoch` field in stream related request protocol
2.
replace `OpenStreamMetadata` with `StreamMetadata` and remove
`OpenStreamMetadata.class`

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): send stream-related request with `broker-epoch`

1. send stream-related request with `broker-epoch`

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): check broker epoch

1. check broker epoch
2. fix wal object record replaying bug

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): handle response with broker epoch error

1. handle response with broker epoch error

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>

* refactor: extract stream part0 rename package (#186)

Signed-off-by: Robin Han <[email protected]>

* refactor: extract s3stream to s3stream module (#192)

* refactor: extract s3stream to s3stream module

Signed-off-by: Robin Han <[email protected]>

* refactor: fix netty conflict

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat(s3): support delete stream (#191)

* feat(s3): support delete stream

1. support delete stream
2. simple fix unclosed stream when close
partition

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): fix checkstyle

1. fix checkstyle

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>

* style(wal): Code beautification and formatting (#196)

* refactor(wal): Make `capacity` a required config

Signed-off-by: Ning Yu <[email protected]>

* style(wal): Remove `Throwable` in try-catch

Signed-off-by: Ning Yu <[email protected]>

* style(wal): Standardize logs and error messages

Signed-off-by: Ning Yu <[email protected]>

* refactor(wal): Make `nextOffset` align to `BLOCK_SIZE`

Signed-off-by: Ning Yu <[email protected]>

* docs(wal): Replace all Chinese comments to English

Signed-off-by: Ning Yu <[email protected]>

* fix spotbugs

Signed-off-by: Ning Yu <[email protected]>

---------

Signed-off-by: Ning Yu <[email protected]>

* refactor: s3stream remove dependency to metadata (#198)

* refactor: s3stream remove dependency to metadata

Signed-off-by: Robin Han <[email protected]>

* fix: fix ut

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat: await partition close (#230)

Signed-off-by: Robin Han <[email protected]>

* fix(s3Stream): make sure to read continuous data blocks when data is trimmed on compaction (#231)

Signed-off-by: Shichao Nie <[email protected]>

* feat(s3): support batch request (#233)

* feat(s3): support batch request

1. support batch request

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): fix after merging

1. fix after merging

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>

* fix: fix potential infinite loop in logCache (#234)

* fix(s3Stream): prevent infinite loop when lastBlockStreamStartOffset is less or equal than startOffset on LogCache#get0

Signed-off-by: Shichao Nie <[email protected]>

* fix(s3Stream): check and create auto balancer metrics topic before consumer start

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat: stream safe trim/close/destory (#235)

Signed-off-by: Robin Han <[email protected]>

* feat(s3): add object retention delay time (#245)

1. add object retention delay time

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): fix objectPart not set null in some cases; add logIdent for writer (#255)

Signed-off-by: Curtis Wan <[email protected]>

* feat(core): add step control to auto balancer (#254)

* feat(core): add step control to auto balancer

Signed-off-by: Shichao Nie <[email protected]>

* style(s3Stream): fix checkstyle

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): prevent reassigning partition to inactive broker (#263)

* fix(core): prevent reassign partition to inactive broker

Signed-off-by: Shichao Nie <[email protected]>

* style(core): fix checkstyle

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat(s3): optimize batch request (#246)

* feat(s3): temp commit

1. temp commit

Signed-off-by: TheR1sing3un <[email protected]>

* feat(s3): optimize batch request handle process

1. optimize batch request handle process
2. support kv related batch
request

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): minor fix

1. minor fix

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): minor fix

1. minor fix

Signed-off-by: TheR1sing3un <[email protected]>

* fix(s3): fix checkstyle

1. fix checkstyle

Signed-off-by: TheR1sing3un <[email protected]>

* refactor(s3): remove `broker` concept in S3Stream, replace it with `node`

1. remove `broker` concept in S3Stream, replace it with `node`

Signed-off-by: TheR1sing3un <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>
Signed-off-by: Shichao Nie <[email protected]>
Co-authored-by: Shichao Nie <[email protected]>

* feat: handle S3client lifecycle (#264)

* feat(s3Stream): add lifecycle control for stream client

Signed-off-by: Shichao Nie <[email protected]>

* style(s3Stream): optimize log position

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat(s3Stream): use dedicated S3Operator for Compaction (#265)

Signed-off-by: Shichao Nie <[email protected]>

* feat(s3Stream): add trace log for S3 object write event (#266)

Signed-off-by: Shichao Nie <[email protected]>

* feat: parallel partition op (#272)

* feat: parallel partition operation

Signed-off-by: Robin Han <[email protected]>

* fix: persist clean shutdown mark when shutdown

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat: async append (#275)

* feat: async create segment

Signed-off-by: Robin Han <[email protected]>

* feat: async lazy stream

Signed-off-by: Robin Han <[email protected]>

* fix: fix https://github.com/AutoMQ/kafka-on-s3/issues/276

Signed-off-by: Robin Han <[email protected]>

* fix: fix unit test async operation fail

Signed-off-by: Robin Han <[email protected]>

* fix: fix bugspot

Signed-off-by: Robin Han <[email protected]>

* fix: fix https://github.com/AutoMQ/kafka-on-s3/issues/277

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat: pre-allocate log&time stream (#279)

Signed-off-by: Robin Han <[email protected]>

* feat(s3): limit object count in one object-deletion request (#286)

1. limit object count in one object-deletion request

Signed-off-by: TheR1sing3un <[email protected]>

* fix: E2E (#294)

* fix: remove hard_shutdown

Signed-off-by: Robin Han <[email protected]>

* fix: clean topic

Signed-off-by: Robin Han <[email protected]>

* fix: disable e2e auto create topic

Signed-off-by: Robin Han <[email protected]>

* fix: disable e2e auto create topic

Signed-off-by: Robin Han <[email protected]>

* fix: skip manual create consume offset

Signed-off-by: Robin Han <[email protected]>

* fix: suppress wait topic delete fail

Signed-off-by: Robin Han <[email protected]>

* fix: halt on KafkaException

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* fix: log cleaner (#297)

* fix: fix log cleaner issues/296

Signed-off-by: Robin Han <[email protected]>

* feat: disable E2E exit when fail

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* fix: fix stream trim (#301)

Signed-off-by: Robin Han <[email protected]>

* fix: delete kv firstly when destroying topicPartition (#303)

* fix: delete kv firstly when destroying topicPartition

Signed-off-by: Curtis Wan <[email protected]>

* fix: roll back kos_test_suite.yml

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* fix: fix log compaction read fail (#320)

* fix: fix log compaction read fail

Signed-off-by: Robin Han <[email protected]>

* fix: disable test_broker_failure unclean shutdown

Signed-off-by: Robin Han <[email protected]>

* fix: avoid pre-load

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat: update s3stream to 0.1.0-SNAPSHOT (#322)

* feat: update s3stream to 5.1.3-SNAPSHOT

Signed-off-by: Robin Han <[email protected]>

* chore: update s3stream to 0.1.0-SNAPSHOT

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* fix: show last N segments (#321)

Signed-off-by: Curtis Wan <[email protected]>

* feat: limit max retry delay (#323)

Signed-off-by: Robin Han <[email protected]>

* fix: init `CompactionManager` using an explicitly created `S3Operator` (#325)

Signed-off-by: Ning Yu <[email protected]>

* fix: do not warn inactive brokers (#324)

Signed-off-by: Curtis Wan <[email protected]>

* refactor: unite s3Operator for compaction tasks (#329)

Signed-off-by: Curtis Wan <[email protected]>

* feat(s3Stream): implements S3Stream metrics interface with yammer metrics (#333)

* feat(autobalancer): make auto balancer consumer backoff time configurable

Signed-off-by: Shichao Nie <[email protected]>

* feat(s3Stream): implements S3Stream metrics interface with yammer metrics

1. introduce yammer metrics implementation for S3Stream
2. implements metrics reporter to log S3Stream metrics periodically

Signed-off-by: Shichao Nie <[email protected]>

* refactor(s3Stream): change S3Stream dependency to 0.1.5

Signed-off-by: Shichao Nie <[email protected]>

* style(s3Stream): remove unused import

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* fix: temp fix time index out of bound (#332)

Signed-off-by: Robin Han <[email protected]>

* feat(s3Stream): support log metrics delta in s3stream metrics reporter (#337)

Signed-off-by: Shichao Nie <[email protected]>

* fix: fix issues312 - change topic name to id (#348)

Signed-off-by: Robin Han <[email protected]>

* fix: issues/312 treat KEY_NOT_EXIST as delete success (#349)

Signed-off-by: Robin Han <[email protected]>

* fix: do not delete kv if log apply failed (#350)

* fix: only delete kv if creation failed

Signed-off-by: Curtis Wan <[email protected]>

* fix: do not delete kv if log apply failed

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* fix: issue352, set stream slice end when roll segment (#353)

Signed-off-by: Robin Han <[email protected]>

* fix: shutdown order (#359)

* feat: update s3stream to 0.1.8-SNAPSHOT

Signed-off-by: Robin Han <[email protected]>

* fix: s3stream client shutdown order

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat(core): add configuration for compaction limitation (#356)

* feat(core): add configuration for compaction limitation

Signed-off-by: Shichao Nie <[email protected]>

* feat(core): change config name for compatibility

Signed-off-by: Shichao Nie <[email protected]>

* build(core): upgrade s3stream dependency to 0.1.9

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat(controller): check node match (#362)

* feat(controller): commit WAL object check node match

Signed-off-by: Robin Han <[email protected]>

* chore: upgrade s3stream

Signed-off-by: Robin Han <[email protected]>

* fix: fix unit test

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* chore: update s3stream to 0.1.11 (#365)

Signed-off-by: Robin Han <[email protected]>

* feat: save timeindex last entry in meta-stream; meta-stream compact i… (#357)

* feat: save timeindex last entry in meta-stream; meta-stream compact in close; fix producer-snapshot saving

Signed-off-by: Curtis Wan <[email protected]>

* fix: import control

Signed-off-by: Curtis Wan <[email protected]>

* refactor: add more comments and logs

Signed-off-by: Curtis Wan <[email protected]>

* refactor: roll back info in ElasticLog

Signed-off-by: Curtis Wan <[email protected]>

* refactor: check log level; add logIdent

Signed-off-by: Curtis Wan <[email protected]>

* fix: style problem

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* fix: concurrency problem in fetch (#370) (#375)

Signed-off-by: Curtis Wan <[email protected]>

* feat(core): implement getStreams interface for StreamManager (#376)

Signed-off-by: Shichao Nie <[email protected]>

* fix: build problem (#378)

Signed-off-by: Curtis Wan <[email protected]>

* feat(s3stream): add network throttle config and upgrade to 0.1.16 (#379)

* feat(s3stream): add network throttle config and upgrade to 0.1.16

Signed-off-by: Shichao Nie <[email protected]>

* build(s3stream): fix dependency typo

Signed-off-by: Shichao Nie <[email protected]>

* fix(s3stream): fix compaction constructor

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat: index cache (#382)

* feat(stream): add time index data cache

Signed-off-by: Robin Han <[email protected]>

* feat(stream): integrate file cache to time index

Signed-off-by: Robin Han <[email protected]>

* fix: fix unit test

Signed-off-by: Robin Han <[email protected]>

* feat: add pid to cache file

Signed-off-by: Robin Han <[email protected]>

* feat: set cache path

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat: fetch async (#386)

* refactor: reduce changes on raw codes

Signed-off-by: Curtis Wan <[email protected]>

* fix: style problem

Signed-off-by: Curtis Wan <[email protected]>

* fix: style problem; fix LEO in elasticLog fetch

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* fix(issu387): skip checkS3ObjectsLifecycle when not active (#388)

* fix(issu387): skip checkS3ObjectsLifecycle when not active

Signed-off-by: Robin Han <[email protected]>

* fix: fix unit test

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* chore(core): set default AutoBalancer network bandwidth to 100MB/s (#391)

Signed-off-by: Shichao Nie <[email protected]>

* feat: limit inflight append read request (#392)

Signed-off-by: Robin Han <[email protected]>

* fix: remove read limiter (#394)

Signed-off-by: Robin Han <[email protected]>

* chore: update s3stream to 0.1.18 (#404)

Signed-off-by: Robin Han <[email protected]>

* fix: retry request timeout (#408)

Signed-off-by: Robin Han <[email protected]>

* chore(core): set default AutoBalancer report interval to 10s (#409)

* chore(core): set default AutoBalancer report interval to 10s

Signed-off-by: Shichao Nie <[email protected]>

* chore(core): set default s3 metrics reporter interval to 60s

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): exclude broker from auto balancing when topic-partition is out of sync (#411)

Signed-off-by: Shichao Nie <[email protected]>

* fix: extending problem (#406) (#412)

* fix: extending problem (#406)

Signed-off-by: Curtis Wan <[email protected]>

* refactor: add comments

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* feat(s3stream): refine s3stream config (#414)

Signed-off-by: Shichao Nie <[email protected]>

* feat(s3stream): compatible with s3stream metrics name (#415)

Signed-off-by: Shichao Nie <[email protected]>

* refactor: remove slow fetch hint; rename esUnit test (#422)

Signed-off-by: Curtis Wan <[email protected]>

* refactor(issue429): rename wal object to sst object (#430)

* refactor(issue429): rename wal object to sst object

Signed-off-by: Robin Han <[email protected]>

* refactor: rename kafka config

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* refactor: rename SST to stream set object (#435)

* refactor: adaptor s3stream 0.3.0

Signed-off-by: Robin Han <[email protected]>

* refactor: rename config sst to stream set object

Signed-off-by: Robin Han <[email protected]>

* refactor: rename remaining sst to stream set object

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* fix(issue440): get lastOffst from records (#441)

* fix(issue440): get lastOffst from records

Signed-off-by: Robin Han <[email protected]>

* fix: unit test

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat(core): metrics refine & upgrade s3stream to 0.4.2-SNAPSHOT (#450)

Signed-off-by: Shichao Nie <[email protected]>

* feat(issus447): add failover controller (#454)

* feat(issues447): add failover controller

Signed-off-by: Robin Han <[email protected]>

* feat(issues447): add failover context image

Signed-off-by: Robin Han <[email protected]>

* feat(issues447): add failover switch

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* fix(issues447): test and fix (#455)

Signed-off-by: Robin Han <[email protected]>

* feat(s3stream): support thread pool status monitor (#456)

Signed-off-by: Shichao Nie <[email protected]>

* fix: inject ak and sk in ConfigUtils; adjust KAFKA_JDK_COMPATIBILITY_OPTS (#459)

Signed-off-by: Curtis Wan <[email protected]>

* feat(issues447): integrate serverless (#463)

Signed-off-by: Robin Han <[email protected]>

* fix(log): time metrics in nanos rather than millis (#464)

Signed-off-by: Ning Yu <[email protected]>

* feat(s3stream): upgrade to 0.5.5-SNAPSHOT (#467)

Signed-off-by: Shichao Nie <[email protected]>

* feat(issues471): isolate fast / slow read (#472)

* feat(issues471): isolate fast / slow read

Signed-off-by: Robin Han <[email protected]>

* fix: keep the style

Signed-off-by: Robin Han <[email protected]>

* fix: temp skip always success client test

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat(s3stream): update to 0.6.3 (#476)

- fix(kafka_issues475): do not log when fast fail

Signed-off-by: Robin Han <[email protected]>

* feat(issues447): add more failover log and update s3stream to 0.6.4 (#486)

Signed-off-by: Robin Han <[email protected]>

* fix(core): mark ElasticLogManager earlier to avoid local checkpoints (#494)

Signed-off-by: Curtis Wan <[email protected]>

* fix(core): update HW with LEO after creating log (#497)

* refactor: change enable method name

Signed-off-by: Curtis Wan <[email protected]>

* fix(core): update HW with LEO

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* feat(issues500): fetch use direct bytebuf (#503)

Signed-off-by: Robin Han <[email protected]>

* fix(core): avoid reading when checking shouldRoll (#504)

* fix(core): avoid reading when checking shouldRoll

Signed-off-by: Curtis Wan <[email protected]>

* refactor(core): log stream id and epoch

Signed-off-by: Curtis Wan <[email protected]>

* fix(tests): rolling is based on wall clock now

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* fix(issues505): get first batch timestamp from meta (#506)

Signed-off-by: Robin Han <[email protected]>

* fix(streamaspect): keep the rollingBaseTimestamp semantics (#511)

Signed-off-by: Robin Han <[email protected]>

* feat(s3): add s3 force style setting (#518)

Signed-off-by: Curtis Wan <[email protected]>

* fix: Duplicate configs of network bandwidth [issues 385] (#517)

* fix(s3stream): fix network util when capacity set to zero (#524)

Signed-off-by: Shichao Nie <[email protected]>

* fix(s3stream): initialize broker capacity to ignored value (#526)

Signed-off-by: Shichao Nie <[email protected]>

* fix(issues529): reduce group records to PooledMemoryRecords memory usage (#530)

Signed-off-by: Robin Han <[email protected]>

* fix: callback fail when partition closing (#536)

Signed-off-by: Robin Han <[email protected]>

* feat(issues543): use pooled bytebuf for PooledMemoryRecords (#545)

Signed-off-by: Robin Han <[email protected]>

* fix(issuses550): PooledMemoryRecords memory leak (#551)

Signed-off-by: Robin Han <[email protected]>

* feat(issues542): optimize S3ObjectsDelta memory usage (#547)

Signed-off-by: Robin Han <[email protected]>

* feat: await partition shutdown (#552)

Signed-off-by: Robin Han <[email protected]>

* fix: fix response earlier release (#555)

Signed-off-by: Robin Han <[email protected]>

* fix(core): solve NPE problem; avoid compact if replay failed (#561)

Signed-off-by: Curtis Wan <[email protected]>

* feat(s3stream): add OpenTelemetry support to s3stream metrics (#548)

Signed-off-by: Shichao Nie <[email protected]>

* test(metadata): fix testDeleteTooManyOneRequest (#579)

Signed-off-by: Curtis Wan <[email protected]>

* feat: meta json backward compatibility (#587)

Signed-off-by: Robin Han <[email protected]>

* feat(issues525): accelerate recovery from unclean shutdown (#596)

Signed-off-by: Robin Han <[email protected]>

* fix(issues598): fix test_replication_with_broker_failure fail (#599)

Signed-off-by: Robin Han <[email protected]>

* feat(core): introduce s3stream tracing (#610)

Signed-off-by: Shichao Nie <[email protected]>

* feat(issues602): compress stream set object data (#612)

Signed-off-by: Robin Han <[email protected]>

* feat(core): add telemetry docker compose scripts (#613)

Signed-off-by: Shichao Nie <[email protected]>

* chore: bump s3stream to 0.11.0-SNAPSHOT (#614)

Signed-off-by: Ning Yu <[email protected]>

* feat(core): change autobalancer capacity unit to bytes (#623)

Signed-off-by: Shichao Nie <[email protected]>

* docs(docker): refine README.md for telemetry (#626)

Signed-off-by: Shichao Nie <[email protected]>

* feat(s3stream): support config metrics level (#627)

Signed-off-by: Shichao Nie <[email protected]>

* feat(metadata): replace image map to delta map (#629)

Signed-off-by: Robin Han <[email protected]>

* feat(core): support transforming jmx metrics to OTLP (#632)

Signed-off-by: Shichao Nie <[email protected]>

* feat: bump s3stream to 0.14.0 (#637)

Signed-off-by: Robin Han <[email protected]>

* feat(core): support independent OTLP endpoint for trace (#638)

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): filter topics label for BrokerTopicMetrics (#640)

Signed-off-by: Shichao Nie <[email protected]>

* feat: stream without async (#641)

Signed-off-by: Robin Han <[email protected]>

* fix(s3stream): upgrade to 0.16.0-SNAPSHOT (#645)

Signed-off-by: Shichao Nie <[email protected]>

* feat(issues619): optimize checkpoint size (#646)

Signed-off-by: Robin Han <[email protected]>

* feat(telemetry): update grafana dashboard to fix label (#649)

Signed-off-by: Shichao Nie <[email protected]>

* fix(telemetry): fix mbeans match rules (#653)

Signed-off-by: Shichao Nie <[email protected]>

* feat(telemetry): support average stats for request time (#655)

Signed-off-by: Shichao Nie <[email protected]>

* feat: optimize s3stream metadata image memory usage (#661)

Signed-off-by: Robin Han <[email protected]>

* feat(issues662): make default config adapt to 2c16g (#663)

Signed-off-by: Robin Han <[email protected]>

* perf: return fast when the `fetchOffset` equals to the `confirmOffset` (#658)

Signed-off-by: Ning Yu <[email protected]>

* perf: check the last segment first to avoid calling `floorSegment` (#659)

Signed-off-by: Ning Yu <[email protected]>

* perf: allocate less in `readFromLocalLogV2` (#669)

Signed-off-by: Ning Yu <[email protected]>

* feat(tool): add admin tool to help user start AutoMQ easily (#670)

Signed-off-by: KaimingWan <[email protected]>

* feat(core): remove consumer group management for ab consumer (#671)

Signed-off-by: Shichao Nie <[email protected]>

* feat(core): add telemetry to release package (#675)

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): create internal topic on demand (#677)

Closes #674

Signed-off-by: Shichao Nie <[email protected]>

* feat(log): force compact meta when close (#676)

Signed-off-by: Robin Han <[email protected]>

* refactor(core): optimize scalability of autobalancer structure (#678)

* refactor(core): optimize scalability of autobalancer structure

Closes #667

Signed-off-by: Shichao Nie <[email protected]>

* style(core): remove unused imports

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat(core): add unit test for AnomalyDetector (#682)

Signed-off-by: Shichao Nie <[email protected]>

* feat: atomic failover feature (#693)

Signed-off-by: Robin Han <[email protected]>

* chore: bump s3stream to 0.18.0 (#698)

Signed-off-by: Ning Yu <[email protected]>

* feat(core): add kafka request time max metrics (#699)

Signed-off-by: Shichao Nie <[email protected]>

* refactor: rename module (#705)

Signed-off-by: KaimingWan <[email protected]>

* feat(issues665): clean up scale-in nodes' objects (#703)

* feat(issues665): clean up scale-in nodes' objects

Signed-off-by: Robin Han <[email protected]>

* fix: checkstyle

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* fix(log): make txn read async to avoid deadlock (#708)

Signed-off-by: Robin Han <[email protected]>

* refactor(core): simplify AutoBalancer goals (#712)

- use network usage instead of utilization for balancing
- remove capacity based balancing

Signed-off-by: Shichao Nie <[email protected]>

* feat(core): represents raw metric types with single byte (#717)

Signed-off-by: Shichao Nie <[email protected]>

* feat(metrics): introduce group commit offset metrics (#719)

* feat(metrics): introduce group commit offset metrics

Signed-off-by: Shichao Nie <[email protected]>

* fix(metrics): remove commit offset metrics when group is dead

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat(metrics): add more comprehensive jmx rules (#724)

Signed-off-by: Shichao Nie <[email protected]>

* fix: force to check AutoMQCreateTopicPolicy before creating a topic (#726)

Signed-off-by: Curtis Wan <[email protected]>

* fix: let kshell init holder (#727)

init holder in kshell

* feat(metrics): support OTLP http exporter (#728)

Signed-off-by: Shichao Nie <[email protected]>

* fix(log): file cache (#729)

Signed-off-by: Robin Han <[email protected]>

* fix(transaction): fix the async callback may cause missing abort txn (#734)

Signed-off-by: Robin Han <[email protected]>

* feat(e2e): add client tool core tests (#730)

* feat(e2e): add client tool core tests

Signed-off-by: Curtis Wan <[email protected]>

* fix(e2e): hard bounce for transaction test

Signed-off-by: Curtis Wan <[email protected]>

---------

Signed-off-by: Curtis Wan <[email protected]>

* fix: support ecs role (#733)

fix: now we use credential provider holder

Signed-off-by: KaimingWan <[email protected]>
Co-authored-by: shiguanxiong <[email protected]>

* fix(core): initialize goals before optimization (#737)

Signed-off-by: Shichao Nie <[email protected]>

* refactor(core): extract common methods to abstract class (#739)

Signed-off-by: Shichao Nie <[email protected]>

* feat(log): file cache support merge put (#740)

Signed-off-by: Robin Han <[email protected]>

* refactor(core): support customized ClusterModelSnapshot (#741)

Signed-off-by: Shichao Nie <[email protected]>

* feat(log): add txn index cache (#743)

Signed-off-by: Robin Han <[email protected]>

* fix(core): use partition metrics time as broker time (#744)

Signed-off-by: Shichao Nie <[email protected]>

* feat(log): timeindex api thread isolation (#745)

Signed-off-by: Robin Han <[email protected]>

* feat(core): refine exported metrics and dashboard (#748)

- use job & instance labels to match cluster id, node type and instance id
- append suffix to metrics name by default

Signed-off-by: Shichao Nie <[email protected]>

* fix(metrics): fix network metrics label value (#749)

Signed-off-by: Shichao Nie <[email protected]>

* fix: optimize and s3url and fix parse (#750)

* fix: add deprecated tips

Signed-off-by: KaimingWan <[email protected]>

* fix: make s3 ops bucket required

Signed-off-by: KaimingWan <[email protected]>

* fix: remove useless field auth method

Signed-off-by: KaimingWan <[email protected]>

* fix: rename parameter name

Signed-off-by: KaimingWan <[email protected]>

* fix: fix parse s3url args issue when involve credential provider holder

Signed-off-by: KaimingWan <[email protected]>

* fix: fix check style

Signed-off-by: KaimingWan <[email protected]>

---------

Signed-off-by: KaimingWan <[email protected]>

* fix(issues754): fix consume aborted txn (#755)

Signed-off-by: Robin Han <[email protected]>

* fix: don't deprecate s3.wal.path, plus a minor polish of s3url (#753)

Signed-off-by: daniel-y <[email protected]>

* fix(log): unit test (#756)

fix(log): keep log the same pattern as kafka

Signed-off-by: Robin Han <[email protected]>

* feat(log): full checkpoint based on dirty bytes (#760)

Signed-off-by: Robin Han <[email protected]>

* chore: convert license to bsl to accelerate open source innovation (#761)

Signed-off-by: daniel-y <[email protected]>

* refactor(config): remove useless configs in wal, and add iops config (#762)

refactor: remove useless configs in wal, and add iops config

Signed-off-by: Ning Yu <[email protected]>

* fix(metrics): refine grafana dashboards (#759)

Signed-off-by: Shichao Nie <[email protected]>

* fix(metrics): fix type (#763)

Signed-off-by: Shichao Nie <[email protected]>

* fix(isssue764): ControllerRequestSender stuck (#766)

* fix(isssue764): ControllerRequestSender stuck

Signed-off-by: Robin Han <[email protected]>

* fix: unit test

Signed-off-by: Robin Han <[email protected]>

---------

Signed-off-by: Robin Han <[email protected]>

* feat(metrics): change ot cardinality limit from 2000 -> 5000 (#767)

* feat(metrics): change ot cardinality limit from 2000 -> 10000

Signed-off-by: Shichao Nie <[email protected]>

* style(metrics): remove unused imports

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat(dashboard): refine dashboards (#768)

Signed-off-by: Shichao Nie <[email protected]>

* fix: rename broker-address to broker-list (#775)

Signed-off-by: daniel-y <[email protected]>

* chore: change STREAM_NOT_CLOSED log level to WARN (#776)

Signed-off-by: Robin Han <[email protected]>

* feat(telemetry): make metrics dashboard compatible with Aliyun (#778)

feat(telemetry): make metrics dashboard compatible with Aliyun prometheus

Signed-off-by: Shichao Nie <[email protected]>

* fix(telemetry): fix OTel collector http endpoint (#780)

Signed-off-by: Shichao Nie <[email protected]>

* fix(auth): throw an exception when failed to create a credential from env (#793)

Signed-off-by: Ning Yu <[email protected]>

* feat(core): verify stream epoch for stream object commit (#796)

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): prevent generate stream object record for noop object id (#797)

Signed-off-by: Shichao Nie <[email protected]>

* fix(issues798): checkpoint NPE (#800)

Signed-off-by: Robin Han <[email protected]>

* feat(issues801): stream trim only update stream metadata (#805)

Signed-off-by: Robin Han <[email protected]>

* feat(core): add metrics to monitor auto balancer metrics delay (#807)

Signed-off-by: Shichao Nie <[email protected]>

* fix(issues806): stream object leak (#808)

Signed-off-by: Robin Han <[email protected]>

* fix: range end offset isn't revertable (#809)

Signed-off-by: Robin Han <[email protected]>

* chore: rename s3ObjectRetention* to s3ObjectDeleteRetention for a more precise description (#810)

Signed-off-by: daniel-y <[email protected]>

* fix: set destroyed object size (#811)

Signed-off-by: Robin Han <[email protected]>

* fix(metrics): present metrics from active controller only (#815)

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): prevent anomaly detect exit on inactive controller (#816)

Signed-off-by: Shichao Nie <[email protected]>

* fix(issues817): txn index fetch out of bound (#818)

Signed-off-by: Robin Han <[email protected]>

* feat(core): add metrics to monitor s3 objects (#823)

* feat(core): add metrics to monitor s3 objects

Signed-off-by: Shichao Nie <[email protected]>

* feat(core): add s3 object panels to grafana dashboard

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): record s3 object metrics on active controller only (#824)

Signed-off-by: Shichao Nie <[email protected]>

* feat: add object ttl reach log (#825)

Signed-off-by: Robin Han <[email protected]>

* fix(core): catch exceptions on replaying records (#836)

* fix(core): catch exceptions on replaying records

Signed-off-by: Shichao Nie <[email protected]>

* feat(core): refactor AutoBalancerManager and fix unit tests

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* feat(core): refine grafana dashboards (#837)

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): fix auto balancer metrics delay time calculation (#838)

Signed-off-by: Shichao Nie <[email protected]>

* fix: log permanet fail (#839)

Signed-off-by: Robin Han <[email protected]>

* feat(core): redirect JUL log to sl4j and remove unused logging exporter (#843)

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): fix node id regex in broker dashboard (#841)

Signed-off-by: Shichao Nie <[email protected]>

* feat: record pooled record memory usage (#846)

Signed-off-by: Robin Han <[email protected]>

* fix(metrics): add label 'version' to kafka.request.count (#847)

Signed-off-by: Shichao Nie <[email protected]>

* feat(telemetry): add host name to OTel resource (#849)

Signed-off-by: Shichao Nie <[email protected]>

* feat(metrics): metrics on fetch limiters and executors (#848)

* feat(metrics): metrics on fetch limiters

Signed-off-by: Ning Yu <[email protected]>

* feat(metrics): metrics on fetch executors' queue size

Signed-off-by: Ning Yu <[email protected]>

* style: fix check style

Signed-off-by: Ning Yu <[email protected]>

---------

Signed-off-by: Ning Yu <[email protected]>

* feat(metrics): add buffer and thread metrics (#851)

* feat(telemetry): add direct memory panels (#853)

Signed-off-by: Shichao Nie <[email protected]>

* fix(telemetry): fix read ahead throughput panel unit (#854)

Signed-off-by: Shichao Nie <[email protected]>

* feat(metrics): rename DirectByteBufAlloc to ByteBufAlloc (#855)

* fix(telemetry): fix memory allocation metrics name (#856)

Signed-off-by: Shichao Nie <[email protected]>

* feat(metrics): enable buffer pools metrics (#857)

* fix(telemetry): fix jvm metrics (#859)

Signed-off-by: Shichao Nie <[email protected]>

* fix(telemetry): refine grafana dashboard (#860)

- deduplication for consumer lag calculation on partition reassignment
- add get/put object avg latency panels

Signed-off-by: Shichao Nie <[email protected]>

* fix(telemetry): add missing percentile metrics (#865)

* fix(telemetry): add missing percentile metrics

Signed-off-by: Shichao Nie <[email protected]>

* fix(telemetry): increase metric expiration time for collector

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: Shichao Nie <[email protected]>

* fix(telemetry): fix topic dashboard name (#868)

Signed-off-by: Shichao Nie <[email protected]>

* feat(issues871): log request info for stream object compaction (#872)

Signed-off-by: Robin Han <[email protected]>

* feat(core): merge multiple reassignments for same partition (#873)

Signed-off-by: Shichao Nie <[email protected]>

* feat(dashboard): add node level network metrics (#874)

Signed-off-by: Shichao Nie <[email protected]>

* feat(dashboard): set query interval to 30s to match default metrics reporter interval (#875)

Signed-off-by: Shichao Nie <[email protected]>

* fix(core): support alter auto balancer topic partitions (#876)

Signed-off-by: Shichao Nie <[email protected]>

---------

Signed-off-by: TheR1sing3un <[email protected]>
Signed-off-by: Curtis Wan <[email protected]>
Signed-off-by: Robin Han <[email protected]>
Signed-off-by: sc.nieh <[email protected]>
Signed-off-by: Shichao Nie <[email protected]>
Signed-off-by: Ning Yu <[email protected]>
Signed-off-by: KaimingWan <[email protected]>
Signed-off-by: daniel-y <[email protected]>
Co-authored-by: TheR1sing3un <[email protected]>
Co-authored-by: Robin Han <[email protected]>
Co-authored-by: TheR1sing3un <[email protected]>
Co-authored-by: Curtis Wan <[email protected]>
Co-authored-by: Shichao Nie <[email protected]>
Co-authored-by: Shichao Nie <[email protected]>
Co-authored-by: Yu Ning <[email protected]>
Co-authored-by: zhouyou9505 <[email protected]>
Co-authored-by: KamiWan <[email protected]>
Co-authored-by: shiguanxiong <[email protected]>
Co-authored-by: SSpirits <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants