Remove the ability to create segments in v8 format #4420
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
b-slim
reviewed
Jun 17, 2017
| indexLoadersBuilder.put(i, legacyIndexLoader); | ||
| } | ||
| indexLoadersBuilder.put((int) V9_VERSION, new V9IndexLoader(columnConfig)); | ||
| indexLoaders = indexLoadersBuilder.build(); |
There was a problem hiding this comment.
marker for my self LGTM to this point.
jihoonson
approved these changes
Jun 23, 2017
| @@ -119,7 +116,7 @@ public Plumber findPlumber( | |||
| final Set<File> spilled = Sets.newHashSet(); | |||
|
|
|||
| // IndexMerger implementation. | |||
| final IndexMerger theIndexMerger = config.getBuildV9Directly() ? indexMergerV9 : indexMerger; | |||
| final IndexMerger theIndexMerger = indexMergerV9; | |||
There was a problem hiding this comment.
The theIndexMerger variable can be simply removed.
b-slim
reviewed
Jun 23, 2017
| case LONG: | ||
| builder.setValueType(ValueType.LONG); | ||
| builder.addSerde( | ||
| LongGenericColumnPartSerde.legacySerializerBuilder() |
There was a problem hiding this comment.
Then we do not need all those legacy Ser/Des ?
b-slim
reviewed
Jun 23, 2017
| ValueType type = valueTypes.get(metric); | ||
| switch (type) { | ||
| case LONG: | ||
| metWriters.add(new LongMetricColumnSerializer(metric, v8OutDir, ioPeon, metCompression, longEncoding)); |
There was a problem hiding this comment.
Same thing here do we still need MetricColumnSerializer ?
There was a problem hiding this comment.
Thanks for noticing, removed
b-slim
reviewed
Jun 23, 2017
| final DimensionHandler[] handlers = makeDimensionHandlers(mergedDimensions, dimCapabilities); | ||
| final List<DimensionMerger> mergers = new ArrayList<>(); | ||
| for (int i = 0; i < mergedDimensions.size(); i++) { | ||
| DimensionMergerLegacy merger = handlers[i].makeLegacyMerger( |
There was a problem hiding this comment.
This DimensionMergerLegacy can go too i guess
b-slim
reviewed
Jun 23, 2017
| /** | ||
| */ | ||
| public class IndexMerger | ||
| @ImplementedBy(IndexMergerV9.class) |
There was a problem hiding this comment.
do we expect that this class to be injected via other modules ? or it is to make code look nicer ?
There was a problem hiding this comment.
It is injected somewhere in indexing already. Other implementation is expected with segment format v10.
|
Overall LGTM there is some unused classes that can be removed. |
|
👍 |
|
LGTM, 👍 |
leventov
pushed a commit
to metamx/druid
that referenced
this pull request
Jul 19, 2017
commit 11f2eaa Author: dmitry.golitsyn <[email protected]> Date: Tue Jun 27 21:15:42 2017 +0300 Replace putIfAbsent with computeIfAbsent in DruidBalancer commit 574587b Author: dmitry.golitsyn <[email protected]> Date: Tue Jun 27 16:01:25 2017 +0300 Do not remove segment that should not be moved from currentlyMovingSegments (segments are removed by callbacks or not inserted) commit 7a261c8 Author: Jihoon Son <[email protected]> Date: Tue Jun 27 10:51:48 2017 +0900 Split travis test (apache#4468) commit 05d5868 Author: Roman Leventov <[email protected]> Date: Mon Jun 26 15:21:39 2017 -0500 Remove the ability to create segments in v8 format (apache#4420) * Remove ability to create segments in v8 format * Fix IndexGeneratorJobTest * Fix parameterized test name in IndexMergerTest * Remove extra legacy merging stuff * Remove legacy serializer builders * Remove ConciseBitmapIndexMergerTest and RoaringBitmapIndexMergerTest Do not remove segment that should not be moved from currentlyMovingSegments (segments are removed by callbacks or not inserted) Replace putIfAbsent with computeIfAbsent in DruidBalancer
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #4235
This PR changes the behaviour of:
HadoopConverterJob, which used to produce segments in v8 format. Now it produces segments in v9 format.ConvertSegmentTaskwhich delegates toIndexIO.convertSegment(), used to produce a segment in v8 format if given a segment in v9 format, that I believe was a bug.