simplification for spacesaver. before buckets were kept in an option str... #308
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…structure called bucketsOption to defer their computation if not needed. upon a more careful look it turns out this is not very often. the idea that buckets does not have to be calculated in case of repeated merges of SSManys was the original main motivation (since this is what a hadoop reducer will do), but the merge depends on min which depends on buckets in almost all situations, so bucketsOption doesn't help us much here. bucketsOption can be hurtful if an SSMany gets serialized and deserialized, since in that case the None gets serialized and buckets has to be recomputed if needed after deserialization.
| case None => SortedMap[Long, Set[T]]() ++ counters.groupBy(_._2._1).mapValues(_.keySet) | ||
| } | ||
| case class SSMany[T](capacity: Int, counters: Map[T, (Long, Long)], buckets: SortedMap[Long, Set[T]]) extends SpaceSaver[T] { | ||
| //assert(buckets.values.map(_.size).sum == counters.size) |
|
@koertkuipers sorry I lost track of this. We are about to publish. If you can pull from develop, get the merge green, I'll merge it. Sorry again for the delay. |
johnynek
added a commit
that referenced
this pull request
Jul 3, 2014
simplification for spacesaver. before buckets were kept in an option str...
|
Thank you, sir. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
...ucture called bucketsOption to defer their computation if not needed. upon a more careful look it turns out this is not very often. the idea that buckets does not have to be calculated in case of repeated merges of SSManys was the original main motivation (since this is what a hadoop reducer will do), but the merge depends on min which depends on buckets in almost all situations, so bucketsOption doesn't help us much here. bucketsOption can be hurtful if an SSMany gets serialized and deserialized, since in that case the None gets serialized and buckets has to be recomputed if needed after deserialization.