Able to get queryCache to break

[natew]

Case

Getting these errors in node:

Seems to happen when upserting a bunch of stuff. The roughest approximation I have is to have a large amount (~80 rows) of items to upsert (lots of text and fields). Then go through in a loop and call upsert for all of them.

I could attempt to serialize this into a test case, but it is very tied into a deep area of the stack. I think the basic gist is: lots of data, upsert many rows at once.

Let me know I can try and narrow this down.

Info

• Environment: Node 8.4
• Adapter: memory

[natew]

This may be because not using atomicUpsert :) Checking

[natew]

Wonder if having upsert be atomic by default, and then having asyncUpsert being the backup would potentially be a better first-principles approach.

[pubkey]

Tried to reproduce with this test:

it('#278 insert and upsert many docs', async() => {
            console.log('-------');
            const db = await RxDatabase.create({
                name: util.randomCouchString(10),
                adapter: 'memory'
            });
            const collection = await db.collection({
                name: 'human',
                schema: schemas.primaryHuman
            });
            const docData = new Array(1000)
                .fill(0)
                .map(() => schemaObjects.simpleHuman());

            for (const doc of docData)
                await collection.insert(doc);

            const docData2 = clone(docData);
            docData2.forEach(doc => doc.lastName = doc.lastName + '1');

            for (const doc of docData2)
                await collection.upsert(doc);

            console.log(':::::');
            db.destroy();
        });

But could not reproduce.
I don't know if this is a bug or if you should use atomicUpsert. Can you help me writing a test?
Even if you should have used atomicUpsert, we should try to throw an usefull error here instead of the unreadable stacktrace :)

[pubkey]

Could reproduce and fix this now.
The problem:
The ChangeEventBuffer remembers the last x ChangeEvents (x=100 by default) and when you do RxQuery.exec() these events are used in the query-change-detection. If between last time .exec() was done and a new exec-run, too many change-events occured, the change-event-buffer is out of it's lower bound. This means that the query-change-detection can not be used and a re-execution of the query over the whole pouchdb must be done.

[natew]

Need some time, I actually tried disabling { waitForLeadership: false } due to other ticket yesterday, and then forgot that, and ended up down a very long rabbit hole to re-debug that bug. Took me a while to figure out that was still causing an issue.

Hoping I get some time today or later this week to properly give you a test case. I did get it to replicate (actually I couldn't get it not to bug recently, not sure if its a corrupted db or something else but it is throwing pointer errors with or without the queryCache enabled.

But will get a good test case soon.

Edit: see that you found a way to replicate, very cool.