Sharded store #53
Merged
Sharded store #53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Current coverage is 64.44% (diff: 63.63%)@@ master #53 diff @@
==========================================
Files 26 27 +1
Lines 465 481 +16
Methods 0 0
Messages 0 0
Branches 74 80 +6
==========================================
+ Hits 297 310 +13
+ Misses 126 123 -3
- Partials 42 48 +6
|
brewkode
reviewed
Oct 12, 2016
| * Take care to not throw exceptions in this method. If it does, | ||
| * we propagate that error back to the service who invoked us. | ||
| */ | ||
| class ShardedStore(partitionsPerNode: Int, hashFn: Hash, createStore: (Int) => Store) extends Store with Logging { |
There was a problem hiding this comment.
Why is createStore() a function?
There was a problem hiding this comment.
We should be lazily creating a new store for each partition la?
|
Merging this since we need it for our finder benchmarks. Please share any feedback, I'll fix them in a separate PR. |
brewkode
reviewed
Oct 14, 2016
| when(hash.hash("1".getBytes)).thenReturn(1) | ||
| when(hash.hash("2".getBytes)).thenReturn(2) | ||
|
|
||
| val store1 = mock(classOf[Store]) |
There was a problem hiding this comment.
why can't we pass in-memory store and avoid the mocking here? Applies for all tests for ShardedStore
There was a problem hiding this comment.
Actually you're right, We could use that. I was thinking more from just interactions that's why I went with mocks. In this case we could use that too.
There was a problem hiding this comment.
Tracking it as part of #55
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For some use-cases where we can't have one large database (like RocksDB) for storing multi-terrabyte data. It would be great to have sharding of the
Storeand a wrapper store that manages it for us automatically.ShardedStoreextends Store, so it's a drop-in replacement for all of the existing stores and you would start having multiple shards automatically. If the existing store already has some data, they might have to be re-played or converted to the newer format. I would have written a tool for migration but since we still don't expose thescaninterface on store, it's not possible to write a generic utility that can do this for us.@brewkode Please check this out.