API design: getMany(), key & value iterators, iterator.nextv()

[vweevers]

Jotting down thoughts about various feature requests, ideas and perf improvements - and how APIs for those could fit together.

db.getMany(keys[, options][, callback])

assert(db.supports.getMany)

for (const value of await db.getMany(['a', 'b']) {
  // value may be undefined
}

keyIterator = db.keys([options])

Inherits from AbstractIterator but yields keys instead of entries. Similar to db.createKeyStream() but without the overhead of streams (also in terms of browser bundle size; we might want to move streams to userland because they operate on iterators anyway, and you could potentially choose between readable-stream, core stream, streamx, pull stream).

assert(db.supports.keyIterators)
for await (const key of db.keys({ limit: 2, reverse: true }))

valueIterator = db.values([options])

assert(db.supports.valueIterators)
for await (const value of db.values({ gte: 'foo' }))

iterator.mode

A property by which consumers (like streams) can determine what results to expect. One of entries, keys, values.

iterator.nextv(size[, options][, callback])

Useful for optimized streams, but also just getting X entries. The highWaterMark option (of leveldown) still applies - if either size or hwm is reached then we stop.

const entries = await iterator.nextv(60) // [['key', 'value'], ...]
const keys = await keyIterator.nextv(60) // ['key', ...]
const values = await valueIterator.nextv(60) // ['value', ...]

Options could include end: true to automatically end the iterator.

const first = await iterator.nextv(1, { end: true })[0]

iterator.all([options][, callback])

Same as level-concat-iterator, but highly optimizable. In level-js it can also offer snapshot guarantees when we drop that from (regular use of) iterators. No options, that argument is just reserved.

// Same return type as db.getMany() but operating on a range
const array = await db.values({ gte: 'foo', limit: 1e3 }).all()

[vweevers]

The (internals of these) methods can benefit from each other. In leveldown for example:

• We can refactor the existing iterator to use a new struct (maybe call it Entry) that abstracts away key-value pairs and conversion to napi_value (either an array, string or buffer). Because we have benchmarks for iterators, we can verify there's no performance hit.
• This then reduces the amount of code needed for all() and also getMany()
• To implement keys() and values() we can add an internal mode enum that dictates whether Entry converts to an entry, key or value
• next() could be refactored to use nextv() internally, using a size of 1000. Later though, because this would change the data structure of iterator.cache which is an undocumented but public property (and I know some folks are using it).
• all() (which has to include cached results from next() as well) would then not have to deal with iterator.cache being in reverse; a simple .concat() might suffice.

[Raynos]

👍 for nextv() I’ve implemented this in userland.

[vweevers]

Done in abstract-level (without mode and without the end option). The getMany() method is also available in abstract-leveldown and levelup as it was added earlier.

Next steps:

1. Push and publish abstract-level implementations. Some will have specific problems to solve, like leveldown's caching, and I'll open issues on relevant repo's if needed.
2. Use nextv() and decide on default hwm read-stream#1 which can close Improve throughput of read streams by transferring multiple records at once community#70