-
Notifications
You must be signed in to change notification settings - Fork 18
Morphium paged Iterator
Stephan Bösebeck edited this page Aug 29, 2014
·
2 revisions
Problem is, when dealing with huge tables or lots of data, you'd probably include paging to your queries. You would read data in chunks of for example 100 objects to avoid memory overflows. This is now available by Morphium. The new MorphiumIterator works as Iterable or Iterator - whatever you like. It's included in the Query-interface and can be used very easily:
Query<Type> q=morphium.createQueryFor(Type.class);
q=q.f("field").eq..... //whatever
for (Type t:q.asIterable()) {
//do something with t
}This creates an iterator, reading all objects from the query in chunks of 10... if you want to read them one by one, you only ned to give the chunk-size to the call:
for (Type t:q.asIterable(1)) {
//now reads every single Object from db
}You can also use the iterator as in the "good ol' days".
Iterator<Type> it=q.asIterable(100); //reads objects in chunks of 100
while (it.hasNext()) {
... //do something
}If you use the MorphiumIterator as the type it acutally is, you'd get even more information:
MorphiumIterator<Type> it=q.asIterable(100);
it.next();
....
long count=it.getCount(); //returns the number of objects to be read
int cursorPos=it.getCursor(); //where are we right now, how many did we read
it.ahead(5); //jump ahead 5 objects
it.back(4); //jump back
int bufferSize=it.getCurrentBufferSize(); //how many objects are currently stored in RAM
List<Type> lst=it.getCurrentBuffer(); //get the objects in RAMAttention: the count is the number of objects matching the query at the instanciation of the iterator. This ensures, that the iterator terminates. The Query will be executed every time the buffer boundaries are reached. It might cause unexpected results, if the sort of the query is wrong.
For example:
//created Uncached Objects with counter 1-100; value is always "v"
Query<UncachedObject> qu=morphium.createQueryFor(UncachedObject.class).sort("-counter");
for (UncachedObject u:qu.asIterable()) {
UncachedObject uc=new UncachedObject();
uc.setCounter(u.getCounter()+1);
uc.setValue("WRONG!");
MorphiumSingleton.get().store(uc);
log.info("Current Counter: "+u.getCounter()+" and Value: "+u.getValue());
}The output is as follows:
14:21:10,494 INFO [main] IteratorTest: Current Counter: 100 and Value: v
14:21:10,529 INFO [main] IteratorTest: Current Counter: 99 and Value: v
14:21:10,565 INFO [main] IteratorTest: Current Counter: 98 and Value: v
14:21:10,610 INFO [main] IteratorTest: Current Counter: 97 and Value: v
14:21:10,645 INFO [main] IteratorTest: Current Counter: 96 and Value: v
14:21:10,680 INFO [main] IteratorTest: Current Counter: 95 and Value: v
14:21:10,715 INFO [main] IteratorTest: Current Counter: 94 and Value: v
14:21:10,751 INFO [main] IteratorTest: Current Counter: 93 and Value: v
14:21:10,786 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:10,822 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:10,857 INFO [main] IteratorTest: Current Counter: 96 and Value: WRONG!
14:21:10,892 INFO [main] IteratorTest: Current Counter: 95 and Value: v
14:21:10,927 INFO [main] IteratorTest: Current Counter: 95 and Value: WRONG!
14:21:10,963 INFO [main] IteratorTest: Current Counter: 94 and Value: v
14:21:10,999 INFO [main] IteratorTest: Current Counter: 94 and Value: WRONG!
14:21:11,035 INFO [main] IteratorTest: Current Counter: 93 and Value: v
14:21:11,070 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,105 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:11,140 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:11,175 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:11,210 INFO [main] IteratorTest: Current Counter: 94 and Value: WRONG!
14:21:11,245 INFO [main] IteratorTest: Current Counter: 94 and Value: WRONG!
14:21:11,284 INFO [main] IteratorTest: Current Counter: 93 and Value: v
14:21:11,328 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,361 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,397 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,432 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:11,467 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:11,502 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:11,538 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:11,572 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,607 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,642 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,677 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,713 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,748 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:11,783 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:11,819 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:11,853 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:11,889 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:11,923 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,958 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:11,993 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:12,028 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:12,063 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:12,098 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,133 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,168 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,203 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,239 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:12,273 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:12,308 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:12,344 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:12,379 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:12,413 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,448 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,487 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,521 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,557 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,592 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:12,626 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:12,662 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:12,697 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:12,733 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,769 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,804 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,839 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,874 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,910 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:12,945 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:12,980 INFO [main] IteratorTest: Current Counter: 93 and Value: WRONG!
14:21:13,015 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:13,051 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,085 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,121 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,156 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,192 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,226 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,262 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,297 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:13,331 INFO [main] IteratorTest: Current Counter: 92 and Value: v
14:21:13,367 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,403 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,446 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,485 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,520 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,556 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,592 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,627 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,662 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:13,697 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,733 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,768 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,805 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,841 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,875 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,911 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,946 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:13,982 INFO [main] IteratorTest: Current Counter: 92 and Value: WRONG!
14:21:14,017 INFO [main] IteratorTest: Current Counter: 91 and Value: v
14:21:14,017 INFO [main] IteratorTest: Cleaning up...
14:21:14,088 INFO [main] IteratorTest: done...
The first chunk is ok, but all that follow are not. Fortunately count did not change or in this case, the iterator would never stop. Hence, if your collection changes while you're iterating over it, you might get inexpected results. Writing to the same collection within the loop of the iterator is generally a bad idea...
#Advanced Features
Since V2.2.5 the morphium iterator supports lookahead (prefetching). This means its not only possible to define a window size to step through your data, but also how many of those windows should be prefetched, while you step through the first one.
This works totally transparent for the user, its just a simple call to activate this feature:
theQuery.asIterable(1000,5); //window size 1000, 5 windows prefetch
Since 2.2.5 the morphium iterator is also able to be used by multiple threads simultaneously. This means, several threads access the same iterator. This might be useful for querying and alike. To use that, you only need to set setMultithreaddedAccess to true in the iterator itself:
MorphiumIterator<MyEntity> it=theQuery.asIterable(1000,15)
it.setMultithreaddedAccess(true);
Attention: Setting mutlithreaddedAccess to true will cause the iterator to be a bit slower as it has to do some things in a synchronized fashion.