diff --git a/README.md b/README.md index 9fc67cb..14b6308 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # level-js -> An [`abstract-leveldown`][abstract-leveldown] compliant store on top of [IndexedDB][indexeddb], which is in turn implemented on top of [LevelDB][leveldb] which brings this whole shebang full circle. +> An [`abstract-leveldown`][abstract-leveldown] compliant store on top of [IndexedDB][indexeddb]. [![level badge][level-badge]][awesome] [![npm](https://img.shields.io/npm/v/level-js.svg?label=&logo=npm)](https://www.npmjs.com/package/level-js) @@ -35,13 +35,11 @@ Here are the goals of `level-js`: - Store large amounts of data in modern browsers - Pass the full [`abstract-leveldown`][abstract-leveldown] test suite -- Support [`Buffer`][buffer] keys and values -- Support all key types of IndexedDB Second Edition -- Support all value types of the [structured clone algorithm][structured-clone-algorithm] except for `null` and `undefined` +- Support string and binary keys and values - Be as fast as possible -- Sync with [multilevel](https://github.com/juliangruber/multilevel) over ASCII or binary transports. +- ~~Sync with [multilevel](https://github.com/juliangruber/multilevel) over ASCII or binary transports.~~ -Being `abstract-leveldown` compliant means you can use many of the [Level modules][awesome] on top of this library. For some demos of it working, see [**@brycebaril**](https://github.com/brycebaril)'s presentation [Path of the NodeBases Jedi](http://brycebaril.github.io/nodebase_jedi/#/vanilla). +Being `abstract-leveldown` compliant means you can use many of the [Level modules][awesome] on top of this library. ## Example @@ -80,120 +78,17 @@ const value = await db.get('hello') ## Type Support -Unlike [`leveldown`][leveldown], `level-js` does not stringify keys or values. This means that in addition to strings and Buffers you can store almost any JavaScript type without the need for [`encoding-down`][encoding-down]. +Keys and values can be a string or binary. Supported binary types include [`Buffer`][buffer], `ArrayBuffer` or a view thereof (typed arrays). Any other type will be irreversibly stringified. The only exceptions are `null` and `undefined`. Keys and values of that type are rejected. -### Values +In order to sort string and binary keys the same way, for compatibility with `leveldown` and the larger ecosystem, `level-js` internally converts keys and values to binary before passing them to IndexedDB. If binary keys are not supported by the environment (like IE11) `level-js` falls back to `String(key)`. -All value types of the [structured clone algorithm][structured-clone-algorithm] are supported except for `null` and `undefined`. Depending on the environment, this includes: +If you desire non-destructive encoding (e.g. to store and retrieve numbers as-is), wrap `level-js` with [`encoding-down`][encoding-down]. Alternatively install [`level`][level] which conveniently bundles [`levelup`][levelup], `level-js` and `encoding-down`. Such an approach is also recommended if you want to achieve universal (isomorphic) behavior. For example, you could have [`leveldown`][leveldown] in a backend and `level-js` in the frontend. The `level` package does exactly that. -- Number, including `NaN`, `Infinity` and `-Infinity` -- String, Boolean, Date, RegExp, Array, Object -- ArrayBuffer or a view thereof (typed arrays); -- Map, Set, Blob, File, FileList, ImageData (limited support). - -In addition `level-js` stores [`Buffer`][buffer] values without transformation. This works in all target environments because `Buffer` is a subclass of `Uint8Array`, meaning such values can be passed to `IndexedDB` as-is. - -When getting or iterating binary values, regardless of whether they were stored as a `Buffer`, `ArrayBuffer` or a view thereof, values will return as a `Buffer`. This behavior can be disabled, in which case `ArrayBuffer` returns as `ArrayBuffer`, typed arrays return as typed arrays and `Buffer` returns as `Uint8Array`: +When getting or iterating keys and values, regardless of the type with which they were stored, keys and values will return as a Buffer unless the `asBuffer`, `keyAsBuffer` or `valueAsBuffer` options are set, in which case strings are returned. Setting these options is not needed when `level-js` is wrapped with `encoding-down`, which determines the optimal return type by the chosen encoding. ```js db.get('key', { asBuffer: false }) -db.iterator({ valueAsBuffer: false }) -``` - -If the environment does not support a type, it will throw an error which `level-js` catches and passes to the callbacks of `put` or `batch`. For example, IE does not support typed array values. At the time of writing, Chrome is the only browser that supports all types listed above. - -### Keys - -All key types of IndexedDB Second Edition are supported. Depending on the environment, this includes: - -- Number, including `Infinity` and `-Infinity`, but not `NaN` -- Date, except invalid (`NaN`) -- String -- ArrayBuffer or a view thereof (typed arrays); -- Array, except cyclical, empty and sparse arrays. Elements must be valid types themselves. - -In addition you can use [`Buffer`][buffer] keys, giving `level-js` the same power as implementations like `leveldown` and `memdown`. When iterating binary keys, regardless of whether they were stored as `Buffer`, `ArrayBuffer` or a view thereof, keys will return as a `Buffer`. This behavior can be disabled, in which case binary keys will always return as `ArrayBuffer`: - -```js -db.iterator({ keyAsBuffer: false }) -``` - -Note that this behavior is slightly different from values due to the way that IndexedDB works. IndexedDB stores binary _values_ using the structured clone algorithm, which preserves views, but it stores binary _keys_ as an array of octets, so that it is able to compare and sort differently typed keys. - -If the environment does not support a type, it will throw an error which `level-js` catches and passes to the callbacks of `get`, `put`, `del`, `batch` or an iterator. Exceptions are: - -- `null` and `undefined`: rejected early by `abstract-leveldown` -- Binary and array keys: if not supported by the environment, `level-js` falls back to `String(key)`. - -### Normalization - -If you desire normalization for keys and values (e.g. to stringify numbers), wrap `level-js` with [`encoding-down`][encoding-down]. Alternatively install [`level`][level] which conveniently bundles [`levelup`][levelup], `level-js` and `encoding-down`. Such an approach is also recommended if you want to achieve universal (isomorphic) behavior or to smooth over type differences between browsers. For example, you could have [`leveldown`][leveldown] in a backend and `level-js` in the frontend. The `level` package does exactly that. - -Another reason you might want to use `encoding-down` is that the structured clone algorithm, while rich in types, can be slower than `JSON.stringify`. - -### Sort Order - -Unless `level-js` is wrapped with [`encoding-down`][encoding-down], IndexedDB will sort your keys in the following order: - -1. number (numeric) -2. date (numeric, by epoch offset) -3. binary (bitwise) -4. string (lexicographic) -5. array (componentwise). - -You can take advantage of this fact with `levelup` streams. For example, if your keys are dates, you can select everything greater than a specific date (let's be happy and ignore timezones for a moment): - -```js -const db = levelup(leveljs('time-db')) - -db.createReadStream({ gt: new Date('2019-01-01') }) - .pipe(..) -``` - -Or if your keys are arrays, you can do things like: - -```js -const db = levelup(leveljs('books-db')) - -await db.put(['Roald Dahl', 'Charlie and the Chocolate Factory'], {}) -await db.put(['Roald Dahl', 'Fantastic Mr Fox'], {}) - -// Select all books by Roald Dahl -db.createReadStream({ gt: ['Roald Dahl'], lt: ['Roald Dahl', '\xff'] }) - .pipe(..) -``` - -To achieve this on other `abstract-leveldown` implementations, wrap them with [`encoding-down`][encoding-down] and [`charwise`][charwise] (or similar). - -#### Known Browser Issues - -IE11 and Edge yield incorrect results for `{ gte: '' }` if the database contains any key types other than strings. - -### Buffer vs ArrayBuffer - -For interoperability it is recommended to use `Buffer` as your binary type. While we recognize that Node.js core modules are moving towards supporting `ArrayBuffer` and views thereof, `Buffer` remains the primary binary type in the Level ecosystem. - -That said: if you want to `put()` an `ArrayBuffer` you can! Just know that it will come back as a `Buffer` by default. If you want to `get()` or iterate stored `ArrayBuffer` data as an `ArrayBuffer`, you have a few options. Without `encoding-down`: - -```js -const db = levelup(leveljs('mydb')) - -// Yields an ArrayBuffer, Buffer and ArrayBuffer -const value1 = await db.get('key', { asBuffer: false }) -const value2 = await db.get('key') -const value3 = value2.buffer -``` - -With `encoding-down` (or `level`) you can use the `id` encoding to selectively bypass encodings: - -```js -const encode = require('encoding-down') -const db = levelup(encode(leveljs('mydb'), { valueEncoding: 'binary' })) - -// Yields an ArrayBuffer, Buffer and ArrayBuffer -const value1 = await db.get('key', { valueEncoding: 'id' }) -const value2 = await db.get('key') -const value3 = value2.buffer +db.iterator({ keyAsBuffer: false, valueAsBuffer: false }) ``` ## Install @@ -268,16 +163,12 @@ To sustain [`Level`](https://github.com/Level) and its activities, become a back [indexeddb]: https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API -[leveldb]: https://github.com/google/leveldb - [buffer]: https://nodejs.org/api/buffer.html [awesome]: https://github.com/Level/awesome [abstract-leveldown]: https://github.com/Level/abstract-leveldown -[charwise]: https://github.com/dominictarr/charwise - [levelup]: https://github.com/Level/levelup [leveldown]: https://github.com/Level/leveldown @@ -285,5 +176,3 @@ To sustain [`Level`](https://github.com/Level) and its activities, become a back [level]: https://github.com/Level/level [encoding-down]: https://github.com/Level/encoding-down - -[structured-clone-algorithm]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm diff --git a/index.js b/index.js index 51f9dc7..e39a2d3 100644 --- a/index.js +++ b/index.js @@ -7,7 +7,8 @@ module.exports = Level var AbstractLevelDOWN = require('abstract-leveldown').AbstractLevelDOWN var inherits = require('inherits') var Iterator = require('./iterator') -var mixedToBuffer = require('./util/mixed-to-buffer') +var serialize = require('./util/serialize') +var deserialize = require('./util/deserialize') var setImmediate = require('./util/immediate') var support = require('./util/support') @@ -25,13 +26,31 @@ function Level (location, opts) { this.location = location this.prefix = opts.prefix || DEFAULT_PREFIX this.version = parseInt(opts.version || 1, 10) + + // Experimental, do not externally rely on this object yet. + // See Level/community#42. + this.supports = { + bufferKeys: Level.binaryKeys, + binaryKeys: Level.binaryKeys, + keys: { + string: true, + buffer: Level.binaryKeys, + arrayBuffer: Level.binaryKeys, + typedArray: Level.binaryKeys + }, + permanence: true, + snapshots: true, // For now (#86) + seek: false, // #178 + clear: false, // #175 + createIfMissing: false, + errorIfExists: false + } } inherits(Level, AbstractLevelDOWN) -// Detect binary and array key support (IndexedDB Second Edition) +// Detect binary key support (IndexedDB Second Edition) Level.binaryKeys = support.binaryKeys(indexedDB) -Level.arrayKeys = support.arrayKeys(indexedDB) Level.prototype._open = function (options, callback) { var req = indexedDB.open(this.prefix + this.location, this.version) @@ -93,11 +112,7 @@ Level.prototype._get = function (key, options, callback) { return callback(new Error('NotFound')) } - if (options.asBuffer) { - value = mixedToBuffer(value) - } - - callback(null, value) + callback(null, deserialize(value, options.asBuffer)) }) } @@ -131,27 +146,12 @@ Level.prototype._put = function (key, value, options, callback) { this.await(req, callback) } -// Valid key types in IndexedDB Second Edition: -// -// - Number, except NaN. Includes Infinity and -Infinity -// - Date, except invalid (NaN) -// - String -// - ArrayBuffer or a view thereof (typed arrays). In level-js we also support -// Buffer (which is an Uint8Array) (and the primary binary type of Level). -// - Array, except cyclical and empty (e.g. Array(10)). Elements must be valid -// types themselves. Level.prototype._serializeKey = function (key) { - if (Buffer.isBuffer(key)) { - return Level.binaryKeys ? key : key.toString() - } else if (Array.isArray(key)) { - return Level.arrayKeys ? key.map(this._serializeKey, this) : String(key) - } else { - return key - } + return serialize(key, this.supports.binaryKeys) } Level.prototype._serializeValue = function (value) { - return value + return serialize(value, true) } Level.prototype._iterator = function (options) { diff --git a/iterator.js b/iterator.js index 406a0b8..fca6cc7 100644 --- a/iterator.js +++ b/iterator.js @@ -5,7 +5,7 @@ var inherits = require('inherits') var AbstractIterator = require('abstract-leveldown').AbstractIterator var ltgt = require('ltgt') -var mixedToBuffer = require('./util/mixed-to-buffer') +var deserialize = require('./util/deserialize') var setImmediate = require('./util/immediate') var noop = function () {} @@ -23,6 +23,8 @@ function Iterator (db, location, options) { this._error = null this._transaction = null + this._keys = options.keys + this._values = options.values this._keyAsBuffer = options.keyAsBuffer this._valueAsBuffer = options.valueAsBuffer @@ -126,8 +128,17 @@ Iterator.prototype._next = function (callback) { var key = this._cache.shift() var value = this._cache.shift() - if (this._keyAsBuffer) key = mixedToBuffer(key) - if (this._valueAsBuffer) value = mixedToBuffer(value) + if (this._keys && key !== undefined) { + key = deserialize(key, this._keyAsBuffer) + } else { + key = undefined + } + + if (this._values && value !== undefined) { + value = deserialize(value, this._valueAsBuffer) + } else { + value = undefined + } setImmediate(function () { callback(null, key, value) diff --git a/package.json b/package.json index f33526a..e23c28a 100644 --- a/package.json +++ b/package.json @@ -29,8 +29,7 @@ "abstract-leveldown": "~6.0.1", "immediate": "~3.2.3", "inherits": "^2.0.3", - "ltgt": "^2.1.2", - "typedarray-to-buffer": "~3.1.5" + "ltgt": "^2.1.2" }, "devDependencies": { "airtap": "^2.0.0", diff --git a/test/custom-test.js b/test/custom-test.js index 81a7302..a46f69a 100644 --- a/test/custom-test.js +++ b/test/custom-test.js @@ -63,17 +63,30 @@ module.exports = function (leveljs, test, testCommon) { }) }) - test('put Buffer value, get Uint8Array value', function (t) { + test('put Buffer value, get string value', function (t) { var level = testCommon.factory() level.open(function (err) { t.notOk(err, 'no error') - level.put('key', Buffer.from('00ff', 'hex'), function (err) { + level.put('key', Buffer.from('abc'), function (err) { t.notOk(err, 'no error') level.get('key', { asBuffer: false }, function (err, value) { t.notOk(err, 'no error') - t.notOk(Buffer.isBuffer(value), 'is not a buffer') - t.ok(value instanceof Uint8Array, 'is a Uint8Array') - t.same(Buffer.from(value), Buffer.from('00ff', 'hex')) + t.is(value, 'abc') + level.close(t.end.bind(t)) + }) + }) + }) + }) + + test('put utf8 string value, get string value', function (t) { + var level = testCommon.factory() + level.open(function (err) { + t.notOk(err, 'no error') + level.put('key', '💩', function (err) { + t.notOk(err, 'no error') + level.get('key', { asBuffer: false }, function (err, value) { + t.notOk(err, 'no error') + t.is(value, '💩') level.close(t.end.bind(t)) }) }) @@ -96,17 +109,15 @@ module.exports = function (leveljs, test, testCommon) { }) }) - test('put Uint8Array value, get Uint8Array value', function (t) { + test('put Uint8Array value, get string value', function (t) { var level = testCommon.factory() level.open(function (err) { t.notOk(err, 'no error') - level.put('key', new Uint8Array(Buffer.from('00ff', 'hex').buffer), function (err) { + level.put('key', new Uint8Array(Buffer.from('abc').buffer), function (err) { t.notOk(err, 'no error') level.get('key', { asBuffer: false }, function (err, value) { t.notOk(err, 'no error') - t.notOk(Buffer.isBuffer(value), 'is not a buffer') - t.ok(value instanceof Uint8Array, 'is a Uint8Array') - t.same(Buffer.from(value), Buffer.from('00ff', 'hex')) + t.is(value, 'abc') level.close(t.end.bind(t)) }) }) @@ -129,16 +140,48 @@ module.exports = function (leveljs, test, testCommon) { }) }) - test('put ArrayBuffer value, get ArrayBuffer value', function (t) { + test('put ArrayBuffer value, get string value', function (t) { var level = testCommon.factory() level.open(function (err) { t.notOk(err, 'no error') - level.put('key', Buffer.from('00ff', 'hex').buffer, function (err) { + level.put('key', Buffer.from('abc').buffer, function (err) { t.notOk(err, 'no error') level.get('key', { asBuffer: false }, function (err, value) { t.notOk(err, 'no error') - t.ok(value instanceof ArrayBuffer, 'is a ArrayBuffer') - t.same(Buffer.from(value), Buffer.from('00ff', 'hex')) + t.is(value, 'abc') + level.close(t.end.bind(t)) + }) + }) + }) + }) + + // TODO: test putting+gettting: + // Buffer/Uint8Array/ArrayBuffer/string key with keyAsBuffer true/false + + test('put utf8 string key, get string key', function (t) { + var level = testCommon.factory() + level.open(function (err) { + t.notOk(err, 'no error') + level.put('💩', 'value', function (err) { + t.notOk(err, 'no error') + level.get('💩', { asBuffer: false }, function (err, value) { + t.notOk(err, 'no error') + t.is(value, 'value') + level.close(t.end.bind(t)) + }) + }) + }) + }) + + leveljs.binaryKeys && test('put ArrayBuffer key, get Buffer key', function (t) { + var level = testCommon.factory() + level.open(function (err) { + t.notOk(err, 'no error') + level.put(Buffer.from('00ff', 'hex').buffer, 'value', function (err) { + t.notOk(err, 'no error') + level.get(Buffer.from('00ff', 'hex'), { asBuffer: false }, function (err, value) { + t.notOk(err, 'no error') + t.same(value, 'value') level.close(t.end.bind(t)) }) }) @@ -194,8 +237,8 @@ module.exports = function (leveljs, test, testCommon) { t.ifError(err, 'no open error') db.batch([ - { type: 'put', key: Buffer.from([0]), value: 0 }, - { type: 'put', key: Buffer.from([1]), value: 1 } + { type: 'put', key: Buffer.from([0]), value: '0' }, + { type: 'put', key: Buffer.from([1]), value: '1' } ], function (err) { t.ifError(err, 'no batch error') @@ -204,8 +247,8 @@ module.exports = function (leveljs, test, testCommon) { t.ifError(err, 'no iterator error') t.same(entries, [ - { key: Buffer.from([0]), value: 0 }, - { key: Buffer.from([1]), value: 1 } + { key: Buffer.from([0]), value: '0' }, + { key: Buffer.from([1]), value: '1' } ], 'keys are Buffers') db.close(function (err) { diff --git a/test/index.js b/test/index.js index 16ded7e..e1afe01 100644 --- a/test/index.js +++ b/test/index.js @@ -28,7 +28,3 @@ suite(testCommon) // Additional tests for this implementation require('./custom-test')(leveljs, test, testCommon) -require('./structured-clone-test')(leveljs, test, testCommon) -require('./key-type-test')(leveljs, test, testCommon) -require('./key-type-illegal-test')(leveljs, test, testCommon) -require('./native-order-test')(leveljs, test, testCommon) diff --git a/test/key-type-illegal-test.js b/test/key-type-illegal-test.js deleted file mode 100644 index d51f0b6..0000000 --- a/test/key-type-illegal-test.js +++ /dev/null @@ -1,88 +0,0 @@ -/* global indexedDB */ - -'use strict' - -var support = require('../util/support') - -// Key types not supported by IndexedDB Second Edition. -var illegalTypes = [ - // Allow failure because IE11 treats this as a valid key. - { name: 'NaN Date', allowFailure: true, key: new Date(''), error: 'DataError' }, - { name: 'Error', key: new Error(), error: 'DataError' }, - { name: 'Function', key: function () {}, error: 'DataError' }, - { name: 'DOMNode', key: global.document, error: 'DataError' }, - { name: 'Boolean(true)', key: new Boolean(true), error: 'DataError' }, // eslint-disable-line - { name: 'Boolean(false)', key: new Boolean(false), error: 'DataError' }, // eslint-disable-line - { name: 'true', key: true, error: 'DataError' }, - { name: 'false', key: false, error: 'DataError' }, - { name: 'NaN', key: NaN, error: 'DataError' } -] - -// These are only tested if the environment supports array keys. -// Cyclical arrays are not tested because our #_serializeKey goes into a loop. -var illegalArrays = [ - // This type gets rejected by abstract-leveldown (and is also illegal in IDB). - { name: 'empty Array', key: [], message: 'key cannot be an empty Array' }, - - // These contain a valid element to ensure we don't hit an empty key assertion. - { name: 'Array w/ null', key: ['a', null], error: 'DataError' }, - { name: 'Array w/ undefined', key: ['a', undefined], error: 'DataError' }, - - { name: 'sparse Array', key: new Array(10), error: 'DataError' } -] - -module.exports = function (leveljs, test, testCommon) { - test('setUp', testCommon.setUp) - - if (support.test(['1'])(indexedDB)) { - illegalTypes = illegalTypes.concat(illegalArrays) - } - - illegalTypes.forEach(function (item) { - var skip = item.allowFailure ? 'pass' : 'fail' - var db - - test('open', function (t) { - db = testCommon.factory() - db.open(t.end.bind(t)) - }) - - test('put() illegal key type: ' + item.name, function (t) { - db.put(item.key, 'value', verify.bind(null, t)) - }) - - test('del() illegal key type: ' + item.name, function (t) { - db.del(item.key, verify.bind(null, t)) - }) - - test('get() illegal key type: ' + item.name, function (t) { - db.get(item.key, function (err) { - verify(t, /NotFound/.test(err) ? null : err) - }) - }) - - test('batch() put illegal key type: ' + item.name, function (t) { - db.batch([{ type: 'put', key: item.key, value: 'value' }], verify.bind(null, t)) - }) - - test('batch() del illegal key type: ' + item.name, function (t) { - db.batch([{ type: 'del', key: item.key }], verify.bind(null, t)) - }) - - test('close', function (t) { db.close(t.end.bind(t)) }) - - function verify (t, err) { - if (!err) { - t[skip]('type is treated as valid in this environment') - return t.end() - } - - if ('error' in item) t.is(err.name, item.error, 'is ' + item.error) - if ('message' in item) t.is(err.message, item.message, item.message) - - t.end() - } - }) - - test('tearDown', testCommon.tearDown) -} diff --git a/test/key-type-test.js b/test/key-type-test.js deleted file mode 100644 index 8a1d776..0000000 --- a/test/key-type-test.js +++ /dev/null @@ -1,119 +0,0 @@ -/* global indexedDB */ - -'use strict' - -var concat = require('level-concat-iterator') -var ta = require('./util/create-typed-array') -var support = require('../util/support') - -// All key types supported by IndexedDB Second Edition. -var types = [ - { type: 'number', key: -20 }, - { type: '+Infinity', key: Infinity }, - { type: '-Infinity', key: -Infinity }, - { type: 'string', key: 'test' }, - { type: 'Date', ctor: true, key: new Date() }, - { type: 'Array', ctor: true, allowFailure: true, key: [0, '1'] }, - { type: 'ArrayBuffer', ctor: true, allowFailure: true, key: ta(Buffer).buffer }, - { type: 'Int8Array', ctor: true, allowFailure: true, createKey: ta, view: true }, - { type: 'Uint8Array', ctor: true, allowFailure: true, createKey: ta, view: true }, - { name: 'Buffer', type: 'Uint8Array', ctor: true, allowFailure: true, key: ta(Buffer), view: true }, - { type: 'Uint8ClampedArray', ctor: true, allowFailure: true, createKey: ta, view: true }, - { type: 'Int16Array', ctor: true, allowFailure: true, createKey: ta, view: true }, - { type: 'Uint16Array', ctor: true, allowFailure: true, createKey: ta, view: true }, - { type: 'Int32Array', ctor: true, allowFailure: true, createKey: ta, view: true }, - { type: 'Uint32Array', ctor: true, allowFailure: true, createKey: ta, view: true }, - { type: 'Float32Array', ctor: true, allowFailure: true, createKey: ta, view: true }, - { type: 'Float64Array', ctor: true, allowFailure: true, createKey: ta, view: true } -] - -module.exports = function (leveljs, test, testCommon) { - var db - - test('setUp', testCommon.setUp) - test('open', function (t) { - db = testCommon.factory() - db.open(t.end.bind(t)) - }) - - types.forEach(function (item) { - var testName = item.name || item.type - - test('key type: ' + testName, function (t) { - var Constructor = item.ctor ? global[item.type] : null - var skip = item.allowFailure ? 'pass' : 'fail' - var input = item.key - - if (item.ctor && !Constructor) { - t[skip]('constructor is undefined in this environment') - return t.end() - } - - if (item.createKey) { - try { - input = item.createKey(Constructor) - } catch (err) { - t[skip]('constructor is not spec-compliant in this environment') - return t.end() - } - } - - if (!support.test(input)(indexedDB)) { - t[skip]('type is not supported in this environment') - return t.end() - } - - db.put(input, testName, function (err) { - t.ifError(err, 'no put error') - - db.get(input, { asBuffer: false }, function (err, value) { - t.ifError(err, 'no get error') - t.same(value, testName, 'correct value') - - var it = db.iterator({ keyAsBuffer: false, valueAsBuffer: false }) - - concat(it, function (err, entries) { - t.ifError(err, 'no iterator error') - t.is(entries.length, 1, '1 entry') - - var key = entries[0].key - var value = entries[0].value - - if (Constructor) { - var type = item.view ? 'ArrayBuffer' : item.type - var expected = '[object ' + type + ']' - var actual = Object.prototype.toString.call(key) - - if (actual === expected) { - t.is(actual, expected, 'prototype') - } else { - t[skip]('(de)serializing is not supported by this environment') - return t.end() - } - - if (item.view) { - t.ok(key instanceof ArrayBuffer, 'key is instanceof ArrayBuffer') - t.same(Buffer.from(new Constructor(key)), ta(Buffer), 'correct octets') - } else { - t.ok(key instanceof Constructor, 'key is instanceof ' + type) - t.same(key, input, 'correct key') - } - } else { - t.is(key, input, 'correct key') - } - - t.same(value, testName, 'correct value') - - db.del(input, function (err) { - t.ifError(err, 'no del error') - t.end() - }) - }) - }) - }) - }) - }) - - test('close', function (t) { db.close(t.end.bind(t)) }) - test('tearDown', testCommon.tearDown) -} diff --git a/test/native-order-test.js b/test/native-order-test.js deleted file mode 100644 index 4b7c2b1..0000000 --- a/test/native-order-test.js +++ /dev/null @@ -1,185 +0,0 @@ -'use strict' - -var concat = require('level-concat-iterator') - -module.exports = function (leveljs, test, testCommon) { - // Type sort order per IndexedDB Second Edition, excluding - // types that aren't supported by all environments. - var basicKeys = [ - // Should sort naturally - { type: 'number', value: '-Infinity', key: -Infinity }, - { type: 'number', value: '2', key: 2 }, - { type: 'number', value: '10', key: 10 }, - { type: 'number', value: '+Infinity', key: Infinity }, - - // Should sort naturally (by epoch offset) - { type: 'date', value: 'new Date(2)', key: new Date(2) }, - { type: 'date', value: 'new Date(10)', key: new Date(10) }, - - // Should sort lexicographically - { type: 'string', value: '"10"', key: '10' }, - { type: 'string', value: '"2"', key: '2' } - ] - - makeTest('on basic key types', basicKeys, function (verify) { - // Should be ignored - verify({ gt: undefined }) - verify({ gte: undefined }) - verify({ lt: undefined }) - verify({ lte: undefined }) - - verify({ gt: -Infinity }, 1) - verify({ gte: -Infinity }) - verify({ gt: +Infinity }, 4) - verify({ gte: +Infinity }, 3) - - verify({ lt: -Infinity }, 0, 0) - verify({ lte: -Infinity }, 0, 1) - verify({ lt: +Infinity }, 0, 3) - verify({ lte: +Infinity }, 0, 4) - - verify({ gt: 10 }, 3) - verify({ gte: 10 }, 2) - verify({ lt: 10 }, 0, 2) - verify({ lte: 10 }, 0, 3) - - verify({ gt: new Date(10) }, 6) - verify({ gte: new Date(10) }, 5) - verify({ lt: new Date(10) }, 0, 5) - verify({ lte: new Date(10) }, 0, 6) - - // IE 11 and Edge fail this test (yield 0 results), but only when the db - // contains key types other than strings (see strings-only test below). - // verify({ gte: '' }, 6) - - verify({ gt: '' }, 6) - verify({ lt: '' }, 0, 6) - verify({ lte: '' }, 0, 6) - - verify({ gt: '10' }, 7) - verify({ gte: '10' }, 6) - verify({ lt: '10' }, 0, 6) - verify({ lte: '10' }, 0, 7) - - verify({ gt: '2' }, 0, 0) - verify({ gte: '2' }, -1) - verify({ lt: '2' }, 0, -1) - verify({ lte: '2' }) - }) - - makeTest('on string keys only', basicKeys.filter(matchType('string')), function (verify) { - verify({ gt: '' }) - verify({ gte: '' }) - verify({ lt: '' }, 0, 0) - verify({ lte: '' }, 0, 0) - }) - - if (leveljs.binaryKeys) { - var binaryKeys = [ - // Should sort bitwise - { type: 'binary', value: 'Uint8Array.from([0, 2])', key: binary([0, 2]) }, - { type: 'binary', value: 'Uint8Array.from([1, 1])', key: binary([1, 1]) } - ] - - makeTest('on binary keys', basicKeys.concat(binaryKeys), function (verify) { - verify({ gt: binary([]) }, -2) - verify({ gte: binary([]) }, -2) - verify({ lt: binary([]) }, 0, -2) - verify({ lte: binary([]) }, 0, -2) - }) - } - - if (leveljs.arrayKeys) { - var arrayKeys = [ - // Should sort componentwise - { type: 'array', value: '[100]', key: [100] }, - { type: 'array', value: '["10"]', key: ['10'] }, - { type: 'array', value: '["2"]', key: ['2'] } - ] - - makeTest('on array keys', basicKeys.concat(arrayKeys), function (verify) { - verify({ gt: [] }, -3) - verify({ gte: [] }, -3) - verify({ lt: [] }, 0, -3) - verify({ lte: [] }, 0, -3) - }) - } - - if (leveljs.binaryKeys && leveljs.arrayKeys) { - makeTest('on all key types', basicKeys.concat(binaryKeys).concat(arrayKeys)) - } - - function makeTest (name, input, fn) { - var prefix = 'native order (' + name + '): ' - var db - - test(prefix + 'open', function (t) { - db = testCommon.factory() - db.open(t.end.bind(t)) - }) - - test(prefix + 'prepare', function (t) { - db.batch(input.map(function (item) { - return { type: 'put', key: item.key, value: item.value } - }), t.end.bind(t)) - }) - - function verify (options, begin, end) { - test(prefix + humanRange(options), function (t) { - t.plan(2) - - options.valueAsBuffer = false - concat(db.iterator(options), function (err, result) { - t.ifError(err, 'no concat error') - t.same(result.map(getValue), input.slice(begin, end).map(getValue)) - }) - }) - } - - verify({}) - if (fn) fn(verify) - - test(prefix + 'close', function (t) { - db.close(t.end.bind(t)) - }) - } -} - -function matchType (type) { - return function (item) { - return item.type === type - } -} - -function getValue (kv) { - return kv.value -} - -// Replacement for TypedArray.from() -function binary (bytes) { - var arr = new Uint8Array(bytes.length) - for (var i = 0; i < bytes.length; i++) arr[i] = bytes[i] - return arr -} - -function humanRange (options) { - var a = [] - - ;['gt', 'gte', 'lt', 'lte'].forEach(function (opt) { - if (opt in options) { - var target = options[opt] - - if (typeof target === 'string' || Array.isArray(target)) { - target = JSON.stringify(target) - } else if (Object.prototype.toString.call(target) === '[object Date]') { - target = 'new Date(' + target.valueOf() + ')' - } else if (Object.prototype.toString.call(target) === '[object Uint8Array]') { - target = 'Uint8Array.from([' + target + '])' - } - - a.push(opt + ': ' + target) - } - }) - - return a.length ? a.join(', ') : 'all' -} diff --git a/test/structured-clone-test.js b/test/structured-clone-test.js deleted file mode 100644 index fdde082..0000000 --- a/test/structured-clone-test.js +++ /dev/null @@ -1,221 +0,0 @@ -'use strict' - -var ta = require('./util/create-typed-array') - -function isDataCloneError (err) { - return err.name === 'DataCloneError' || err.code === 25 -} - -// level-js supports all types of the structured clone algorithm -// except for null and undefined (unless nested in another type). -var types = [ - { type: 'boolean', value: true }, - { type: 'number', value: -20 }, - { - type: 'NaN', - value: NaN, - test: function (value) { - // Replacement for Number.isNaN (for IE <= 11) - return typeof value === 'number' && isNaN(value) - } - }, - { type: '+Infinity', value: Infinity }, - { type: '-Infinity', value: -Infinity }, - { type: 'string', value: 'test' }, - { type: 'Boolean object', value: new Boolean(false) }, // eslint-disable-line - { type: 'String object', value: new String('test') }, // eslint-disable-line - { type: 'Date', ctor: true, value: new Date() }, - { type: 'RegExp', ctor: true, value: /r/g }, - { type: 'Array', ctor: true, value: [0, null, undefined] }, - { type: 'Object', ctor: true, value: { a: null, b: [undefined] } }, - { - type: 'Object', - name: 'Object (null prototype)', - ctor: true, - createValue: function () { - return Object.create(null) - } - }, - - { type: 'ArrayBuffer', ctor: true, allowFailure: true, value: ta(Buffer).buffer }, - { type: 'Int8Array', ctor: true, allowFailure: true, createValue: ta }, - - // Don't allow failure as this is the primary type for binary (Buffer) data - { type: 'Uint8Array', ctor: true, createValue: ta }, - { name: 'Buffer', type: 'Uint8Array', ctor: true, value: ta(Buffer) }, - - { type: 'Uint8ClampedArray', ctor: true, allowFailure: true, createValue: ta }, - { type: 'Int16Array', ctor: true, allowFailure: true, createValue: ta }, - { type: 'Uint16Array', ctor: true, allowFailure: true, createValue: ta }, - { type: 'Int32Array', ctor: true, allowFailure: true, createValue: ta }, - { type: 'Uint32Array', ctor: true, allowFailure: true, createValue: ta }, - { type: 'Float32Array', ctor: true, allowFailure: true, createValue: ta }, - { type: 'Float64Array', ctor: true, allowFailure: true, createValue: ta }, - { - type: 'Map', - ctor: true, - allowFailure: true, - createValue: function (Constructor) { - // Replacement for Map constructor arguments (for IE 11) - var value = new Constructor() - value.set('test', 123) - return value - }, - test: function (value) { - return value.get('test') === 123 - } - }, - { - type: 'Set', - ctor: true, - allowFailure: true, - createValue: function (Constructor) { - // Replacement for Set constructor arguments (for IE 11) - var value = new Constructor() - value.add(123) - return value - }, - test: function (value) { - return value.has(123) - } - }, - { - type: 'Blob', - ctor: true, - allowFailure: true, - createValue: function (Constructor) { - return new Constructor(['test']) - }, - test: function (value) { - // TODO. This test would be asynchronous. - return true - } - }, - { - type: 'File', - ctor: true, - allowFailure: true, - createValue: function (Constructor) { - return new Constructor(['test'], 'filename') - }, - test: function (value) { - // TODO. This test would be asynchronous. - return true - } - }, - { - type: 'FileList', - ctor: true, - allowFailure: true, - createValue: function () { - var input = global.document.createElement('input') - input.type = 'file' - return input.files - } - }, - { - type: 'ImageData', - ctor: true, - allowFailure: true, - createValue: function (Constructor) { - return new Constructor(1, 1) - }, - test: function (value) { - return value.data.length === 4 - } - } -] - -// Types that are not supported by the structured clone algorithm -var illegalTypes = [ - { name: 'Error', value: new Error() }, - { name: 'Function', value: function () {} }, - { name: 'DOMNode', value: global.document } -] - -module.exports = function (leveljs, test, testCommon) { - var db - - test('setUp', testCommon.setUp) - test('open', function (t) { - db = testCommon.factory() - db.open(t.end.bind(t)) - }) - - types.forEach(function (item) { - var testName = item.name || item.type - - test('structured clone: ' + testName, function (t) { - var ctor = item.ctor ? global[item.type] : null - var skip = item.allowFailure ? 'pass' : 'fail' - var input = item.value - - if (item.ctor && !ctor) { - t[skip]('constructor is undefined in this environment') - return t.end() - } - - if (item.createValue) { - try { - input = item.createValue(ctor) - } catch (err) { - t[skip]('constructor is not spec-compliant in this environment') - return t.end() - } - } - - db.put(testName, input, function (err) { - if (err && isDataCloneError(err)) { - t[skip]('serializing is not supported by the structured clone algorithm of this environment') - return t.end() - } - - t.notOk(err, 'no put error') - - db.get(testName, { asBuffer: false }, function (err, value) { - t.notOk(err, 'no get error') - - if (ctor) { - var expected = '[object ' + item.type + ']' - var actual = Object.prototype.toString.call(value) - - if (actual === expected) { - t.is(actual, expected, 'prototype') - t.ok(value instanceof ctor, 'instanceof') - } else { - t[skip]('deserializing is not supported by the structured clone algorithm of this environment') - return t.end() - } - } - - if (item.test) { - t.ok(item.test(value), 'correct value') - } else { - t.same(value, input, 'correct value') - } - - t.end() - }) - }) - }) - }) - - illegalTypes.forEach(function (item) { - test('structured clone (illegal type): ' + item.name, function (t) { - t.ok(item.value != null, 'got a value to test') - - db.put(item.name, item.value, function (err) { - t.ok(err, 'got an error') - t.ok(isDataCloneError(err), 'is DataCloneError') - - db.get(item.name, { asBuffer: false }, function (err, value) { - t.ok(/notfound/i.test(err), 'nothing was stored') - t.end() - }) - }) - }) - }) - - test('close', function (t) { db.close(t.end.bind(t)) }) - test('teardown', testCommon.tearDown) -} diff --git a/test/support-test.js b/test/support-test.js index b297c3e..de6af5f 100644 --- a/test/support-test.js +++ b/test/support-test.js @@ -14,16 +14,9 @@ module.exports = function (leveljs, test) { t.end() }) - test('mock arrayKeys support', function (t) { - t.ok(support.arrayKeys({ cmp: pos })) - t.notOk(support.arrayKeys({ cmp: neg })) - t.end() - }) - // Purely informational test('support', function (t) { t.pass('binary keys: ' + support.binaryKeys(indexedDB)) - t.pass('array keys: ' + support.arrayKeys(indexedDB)) t.end() }) } diff --git a/test/util/create-typed-array.js b/test/util/create-typed-array.js deleted file mode 100644 index 8947511..0000000 --- a/test/util/create-typed-array.js +++ /dev/null @@ -1,10 +0,0 @@ -'use strict' - -var bytes = [0, 127] - -// Replacement for TypedArray.from(bytes) -module.exports = function (TypedArray) { - var arr = new TypedArray(bytes.length) - for (var i = 0; i < bytes.length; i++) arr[i] = bytes[i] - return arr -} diff --git a/util/deserialize.js b/util/deserialize.js new file mode 100644 index 0000000..b999b54 --- /dev/null +++ b/util/deserialize.js @@ -0,0 +1,43 @@ +'use strict' + +var ta2str = (function () { + if (global.TextDecoder) { + var decoder = new TextDecoder('utf-8') + return decoder.decode.bind(decoder) + } else { + return function ta2str (ta) { + return ta2buf(ta).toString() + } + } +})() + +var ab2str = (function () { + if (global.TextDecoder) { + var decoder = new TextDecoder('utf-8') + return decoder.decode.bind(decoder) + } else { + return function ab2str (ab) { + return Buffer.from(ab).toString() + } + } +})() + +function ta2buf (ta) { + var buf = Buffer.from(ta.buffer) + + if (ta.byteLength === ta.buffer.byteLength) { + return buf + } else { + return buf.slice(ta.byteOffset, ta.byteOffset + ta.byteLength) + } +} + +module.exports = function (data, asBuffer) { + if (data instanceof Uint8Array) { + return asBuffer ? ta2buf(data) : ta2str(data) + } else if (data instanceof ArrayBuffer) { + return asBuffer ? Buffer.from(data) : ab2str(data) + } else { + return asBuffer ? Buffer.from(String(data)) : String(data) + } +} diff --git a/util/mixed-to-buffer.js b/util/mixed-to-buffer.js deleted file mode 100644 index 023f913..0000000 --- a/util/mixed-to-buffer.js +++ /dev/null @@ -1,9 +0,0 @@ -'use strict' - -var toBuffer = require('typedarray-to-buffer') - -module.exports = function (value) { - if (value instanceof Uint8Array) return toBuffer(value) - else if (value instanceof ArrayBuffer) return Buffer.from(value) - else return Buffer.from(String(value)) -} diff --git a/util/serialize.js b/util/serialize.js new file mode 100644 index 0000000..c80f4ff --- /dev/null +++ b/util/serialize.js @@ -0,0 +1,26 @@ +'use strict' + +var str2bin = (function () { + if (global.TextEncoder) { + var encoder = new TextEncoder('utf-8') + return encoder.encode.bind(encoder) + } else { + return Buffer.from + } +})() + +module.exports = function (data, asBuffer) { + if (asBuffer) { + if (Buffer.isBuffer(data)) { + return data + } else if (data instanceof Uint8Array) { + return data + } else if (data instanceof ArrayBuffer) { + return data + } else { + return str2bin(String(data)) + } + } else { + return String(data) + } +} diff --git a/util/support.js b/util/support.js index f5113d7..9b45277 100644 --- a/util/support.js +++ b/util/support.js @@ -12,4 +12,3 @@ exports.test = function (key) { } exports.binaryKeys = exports.test(new Uint8Array(0)) -exports.arrayKeys = exports.test([1])