Support changing dimension schema in Auto Compaction #11874
Conversation
docs/configuration/index.md
Outdated
| @@ -988,7 +989,7 @@ The below is a list of the supported configurations for auto compaction. | |||
| |`chatHandlerNumRetries`|Retries for reporting the pushed segments in worker tasks.|no (default = 5)| | |||
|
|
|||
| ###### Automatic compaction granularitySpec | |||
| You can optionally use the `granularitySpec` object to configure the segment granularity of the compacted segments. | |||
| You can optionally use the `granularitySpec` object to configure the granularity and rollup of the compacted segments. | |||
There was a problem hiding this comment.
nit: These fields are defined below, so it seems redundant to mention them again here.
| existingDimensionsSpec.getDimensions() : | ||
| null; | ||
| if (!config.getDimensionsSpec().getDimensions().equals(existingDimensions)) { | ||
| return true; |
There was a problem hiding this comment.
nit: Please add a log line here for operators to know why the segment was selected for compaction similar to segmentGranularity above.
Similar comments for rollup + queryGranularity
| if (config.getDimensionsSpec() != null) { | ||
| final DimensionsSpec existingDimensionsSpec = lastCompactionState.getDimensionsSpec(); | ||
| // Checks for list of dimensions | ||
| if (config.getDimensionsSpec().getDimensions() != null) { |
There was a problem hiding this comment.
If config.getDimensionsSpec().getDimensions() == null and lastCompactionState.getDimensionsSpec() is non-null then we want needsCompaction to return false. Is that understanding correct?
There was a problem hiding this comment.
Yes. If config.getDimensionsSpec().getDimensions() == null then needsCompaction will return false regardless of lastCompactionState.getDimensionsSpec() value
Support changing dimension schema in Auto Compaction
Description
Changing dimensionsSpec is already supported in manual compaction task. This PR adds it to the auto compaction.
By changing dimensionsSpec, the user will be able to change the dimension ordering, remove dimension, and change dimension data type.
This PR has: