indexer-alt: configure consistency retention in one place #20889
Conversation
## Description Add back a dedicated top-level configuration for consistency, to control the retention of the consistent range. Now that this range is controlled by the pruner, it can be configured by a `PrunerConfig` as well, rather than a dedicated `ConsistencyConfig`. The motivation for this is that the E2E testing setup needs to configure the consistent range, and it would be better to avoid having to enumerate all the consistent pipelines in the test harness because it would be easy to forget to update that list, whereas with this approach, we take advantage of the fact that there is an `add_consistent` macro to register the pipeline, and it will do the right thing. ## Test plan ``` sui$ cargo check sui$ cargo nextest run -p sui-indexer-alt sui$ cargo run -p sui-indexer-alt -- generate-config ```
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ |
amnn
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
There was a problem hiding this comment.
After looking at https://github.com/MystenLabs/sui-operations/pull/4795/files does that mean other pipelines that want pruning will do
[consistency]
retention = 3600
instead of
[pruner]
retention = 3600
[pipeline.coin_balance_buckets.pruner]
Or will we end up with two ways to essentially configure pruning?
What does the e2e testing setup look like w.r.t configuring the consistent range?
|
The The top-level In the example above, the consistent pipelines will all get retention It's unlikely that you would create a config this complex in practice though -- the idea is that sometimes you want to configure one retention for all pipelines being configured, sometimes you want to configure them individually, and for consistent pipelines, you don't want the choice -- they need to be the same. While we're talking about the ops PR, could you take a look at that as well? I'd love to be able to land this PR and that one together. |
…#20889) ## Description Add back a dedicated top-level configuration for consistency, to control the retention of the consistent range. Now that this range is controlled by the pruner, it can be configured by a `PrunerConfig` as well, rather than a dedicated `ConsistencyConfig`. The motivation for this is that the E2E testing setup needs to configure the consistent range, and it would be better to avoid having to enumerate all the consistent pipelines in the test harness because it would be easy to forget to update that list, whereas with this approach, we take advantage of the fact that there is an `add_consistent` macro to register the pipeline, and it will do the right thing. ## Test plan ``` sui$ cargo check sui$ cargo nextest run -p sui-indexer-alt sui$ cargo run -p sui-indexer-alt -- generate-config ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Description
Add back a dedicated top-level configuration for consistency, to control the retention of the consistent range. Now that this range is controlled by the pruner, it can be configured by a
PrunerConfigas well, rather than a dedicatedConsistencyConfig.The motivation for this is that the E2E testing setup needs to configure the consistent range, and it would be better to avoid having to enumerate all the consistent pipelines in the test harness because it would be easy to forget to update that list, whereas with this approach, we take advantage of the fact that there is an
add_consistentmacro to register the pipeline, and it will do the right thing.Test plan
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.