Skip to content

Commit

Permalink
feat: add stringify method (#2)
Browse files Browse the repository at this point in the history
* chore: slight reorganization

* feat: rough draft of a serializer

* fix: make stringify spec-compliant

* chore: code cleanup and benchmarks

* chore: update readme

* fix: reject null/undefined in arrays

* chore: update readme

* fix: reject invalid dates
  • Loading branch information
cyyynthia authored May 22, 2023
1 parent 340636e commit 11e3a36
Show file tree
Hide file tree
Showing 16 changed files with 1,293 additions and 632 deletions.
96 changes: 87 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![License](https://img.shields.io/github/license/squirrelchat/smol-toml.svg?style=flat-square)](https://github.com/squirrelchat/smol-toml/blob/mistress/LICENSE)
[![npm](https://img.shields.io/npm/v/smol-toml?style=flat-square)](https://npm.im/smol-toml)

A small, fast, and correct TOML parser. smol-toml is fully(ish) spec-compliant with TOML v1.0.0.
A small, fast, and correct TOML parser and serializer. smol-toml is fully(ish) spec-compliant with TOML v1.0.0.

Why yet another TOML parser? Well, the ecosystem of TOML parsers in JavaScript is quite underwhelming, most likely due
to a lack of interest. With most parsers being outdated, unmaintained, non-compliant, or a combination of these, a new
Expand All @@ -23,12 +23,13 @@ smol-toml also passes all of the tests in https://github.com/iarna/toml-spec-tes

<details>
<summary>List of failed `toml-test` cases</summary>

These tests were done by modifying `primitive.ts` and make the implementation return bigints for integers. This allows
verifying the parser correctly intents a number to be an integer or a float.

*Ideally, this becomes an option of the library, but for now...*

The following tests are failing:
The following parse tests are failing:
- invalid/encoding/bad-utf8-in-comment
- invalid/encoding/bad-utf8-in-multiline-literal
- invalid/encoding/bad-utf8-in-multiline
Expand All @@ -44,11 +45,65 @@ The following tests are failing:

## Usage
```js
import { parse } from 'smol-toml'
import { parse, stringify } from 'smol-toml'

const doc = '...'
const parsed = parse(doc)
console.log(parsed)

const toml = stringify(parsed)
console.log(toml)
```

A few notes on the `stringify` function:
- `undefined` and `null` values on objects are ignored (does not produce a key/value).
- `undefined` and `null` values in arrays are **rejected**.
- Functions, classes and symbols are **rejected**.
- floats will be serialized as integers if they don't have a decimal part.
- `stringify(parse('a = 1.0')) === 'a = 1'`
- JS `Date` will be serialized as Offset Date Time
- Use the [`TomlDate` object](#dates) for representing other types.

### Dates
`smol-toml` uses an extended `Date` object to represent all types of TOML Dates. In the future, `smol-toml` will use
objects from the Temporal proposal, but for now we're stuck with the legacy Date object.

```js
import { TomlDate } from 'smol-toml'

// Offset Date Time
const date = new TomlDate('1979-05-27T07:32:00.000-08:00')
console.log(date.isDateTime(), date.isDate(), date.isTime(), date.isLocal()) // ~> true, false, false, false
console.log(date.toISOString()) // ~> 1979-05-27T07:32:00.000-08:00

// Local Date Time
const date = new TomlDate('1979-05-27T07:32:00.000')
console.log(date.isDateTime(), date.isDate(), date.isTime(), date.isLocal()) // ~> true, false, false, true
console.log(date.toISOString()) // ~> 1979-05-27T07:32:00.000

// Local Date
const date = new TomlDate('1979-05-27')
console.log(date.isDateTime(), date.isDate(), date.isTime(), date.isLocal()) // ~> false, true, false, true
console.log(date.toISOString()) // ~> 1979-05-27

// Local Time
const date = new TomlDate('07:32:00')
console.log(date.isDateTime(), date.isDate(), date.isTime(), date.isLocal()) // ~> false, false, true, true
console.log(date.toISOString()) // ~> 07:32:00.000
```

You can also wrap a native `Date` object and specify using different methods depending on the type of date you wish
to represent:

```js
import { TomlDate } from 'smol-toml'

const jsDate = new Date()

const offsetDateTime = TomlDate.wrapAsOffsetDateTime(jsDate)
const localDateTime = TomlDate.wrapAsLocalDateTime(jsDate)
const localDate = TomlDate.wrapAsLocalDate(jsDate)
const localTime = TomlDate.wrapAsLocalTime(jsDate)
```

## Performance
Expand All @@ -61,15 +116,20 @@ idea is to have a file relatively close to a real-world application.

The large TOML generator can be found [here](https://gist.github.com/cyyynthia/e77c744cb6494dabe37d0182506526b9)

| | smol-toml | @iarna/toml@3.0.0 | @ltd/j-toml | fast-toml |
|----------------|---------------------|-------------------|----------------|----------------|
| Spec example | **71,356.51 op/s** | 33,629.31 op/s | 16,433.86 op/s | 29,421.60 op/s |
| ~5MB test file | **3.8091 op/s** | *DNF* | 2.4369 op/s | 2.6078 op/s |
| **Parse** | smol-toml | @iarna/toml@3.0.0 | @ltd/j-toml | fast-toml |
|----------------|---------------------|-------------------|-----------------|-----------------|
| Spec example | **71,356.51 op/s** | 33,629.31 op/s | 16,433.86 op/s | 29,421.60 op/s |
| ~5MB test file | **3.8091 op/s** | *DNF* | 2.4369 op/s | 2.6078 op/s |

| **Stringify** | smol-toml | @iarna/toml@3.0.0 | @ltd/j-toml |
|----------------|----------------------|-------------------|----------------|
| Spec example | **195,191.99 op/s** | 46,583.07 op/s | 5,670.12 op/s |
| ~5MB test file | **14.6709 op/s** | 3.5941 op/s | 0.7856 op/s |

<details>
<summary>Detailed benchmark data</summary>

Tests ran using Vitest v0.31.0 on commit 04d233e351f9ae719222154ee2217aea8b95dbab
Tests ran using Vitest v0.31.0 on commit f58cb6152e667e9cea09f31c93d90652e3b82bf5

CPU: Intel Core i7 7700K (4.2GHz)

Expand All @@ -87,6 +147,16 @@ CPU: Intel Core i7 7700K (4.2GHz)
· smol-toml 3.8091 239.60 287.30 262.53 274.17 287.30 287.30 287.30 ±3.66% 10 fastest
· @ltd/j-toml 2.4369 376.73 493.49 410.35 442.58 493.49 493.49 493.49 ±7.08% 10 slowest
· fast-toml 2.6078 373.88 412.79 383.47 388.62 412.79 412.79 412.79 ±2.72% 10
✓ bench/stringifySpecExample.bench.ts (3) 1886ms
name hz min max mean p75 p99 p995 p999 rme samples
· smol-toml 195,191.99 0.0047 0.2704 0.0051 0.0050 0.0099 0.0110 0.0152 ±0.41% 97596 fastest
· @iarna/toml 46,583.07 0.0197 0.2808 0.0215 0.0208 0.0448 0.0470 0.1704 ±0.47% 23292
· @ltd/j-toml 5,670.12 0.1613 0.5768 0.1764 0.1726 0.3036 0.3129 0.4324 ±0.56% 2836 slowest
✓ bench/stringifyLargeMixed.bench.ts (3) 24057ms
name hz min max mean p75 p99 p995 p999 rme samples
· smol-toml 14.6709 65.1071 79.2199 68.1623 67.1088 79.2199 79.2199 79.2199 ±5.25% 10 fastest
· @iarna/toml 3.5941 266.48 295.24 278.24 290.10 295.24 295.24 295.24 ±2.83% 10
· @ltd/j-toml 0.7856 1,254.33 1,322.05 1,272.87 1,286.82 1,322.05 1,322.05 1,322.05 ±1.37% 10 slowest
BENCH Summary
Expand All @@ -99,6 +169,14 @@ CPU: Intel Core i7 7700K (4.2GHz)
2.12x faster than @iarna/toml
2.43x faster than fast-toml
4.34x faster than @ltd/j-toml
smol-toml - bench/stringifyLargeMixed.bench.ts >
4.00x faster than @iarna/toml
18.33x faster than @ltd/j-toml
smol-toml - bench/stringifySpecExample.bench.ts >
4.19x faster than @iarna/toml
34.42x faster than @ltd/j-toml
```

---
Expand All @@ -111,7 +189,7 @@ I initially reported this to the library author, but the author decided to
- b) [delete the issue](https://github.com/huan231/toml-nodejs/issues/12) when pointed out links to the NodeJS
documentation about the flag removal and standard resolution algorithm.

For the reference anyways, `toml-nodejs` (with proper imports) is ~8x slower on both benchmark with:
For the reference anyways, `toml-nodejs` (with proper imports) is ~8x slower on both parse benchmark with:
- spec example: 7,543.47 op/s
- 5mb mixed: 0.7006 op/s
</details>
49 changes: 49 additions & 0 deletions bench/stringifyLargeMixed.bench.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
/*!
* Copyright (c) Squirrel Chat et al., All rights reserved.
* SPDX-License-Identifier: BSD-3-Clause
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. Neither the name of the copyright holder nor the names of its contributors
* may be used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

import { bench } from 'vitest'
import { readFile } from 'fs/promises'
import { stringify as smolTomlStringify, parse } from '../src/index.js'
import { stringify as iarnaTomlStringify } from '@iarna/toml'
import { stringify as ltdJTomlStringify } from '@ltd/j-toml'

let obj = parse(
await readFile(new URL('./testfiles/5mb-mixed.toml', import.meta.url), 'utf8')
)

bench('smol-toml', () => {
smolTomlStringify(obj)
})

bench('@iarna/toml', () => {
iarnaTomlStringify(obj)
})

bench('@ltd/j-toml', () => {
ltdJTomlStringify(obj)
})
49 changes: 49 additions & 0 deletions bench/stringifySpecExample.bench.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
/*!
* Copyright (c) Squirrel Chat et al., All rights reserved.
* SPDX-License-Identifier: BSD-3-Clause
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. Neither the name of the copyright holder nor the names of its contributors
* may be used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

import { bench } from 'vitest'
import { readFile } from 'fs/promises'
import { stringify as smolTomlStringify, parse } from '../src/index.js'
import { stringify as iarnaTomlStringify } from '@iarna/toml'
import { stringify as ltdJTomlStringify } from '@ltd/j-toml'

let obj = parse(
await readFile(new URL('./testfiles/toml-spec-example.toml', import.meta.url), 'utf8')
)

bench('smol-toml', () => {
smolTomlStringify(obj)
})

bench('@iarna/toml', () => {
iarnaTomlStringify(obj)
})

bench('@ltd/j-toml', () => {
ltdJTomlStringify(obj)
})
7 changes: 4 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
{
"name": "smol-toml",
"version": "1.0.1",
"version": "1.1.0",
"keywords": [
"toml",
"parser"
"parser",
"serializer"
],
"description": "A small, fast, and correct TOML parser",
"description": "A small, fast, and correct TOML parser/serializer",
"repository": "[email protected]:squirrelchat/smol-toml.git",
"author": "Cynthia <[email protected]>",
"license": "BSD-3-Clause",
Expand Down
1 change: 1 addition & 0 deletions src/date.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ export default class TomlDate extends Date {
} else {
offset = match[3] || null
date = date.toUpperCase()
if (!offset) date += 'Z'
}
} else {
date = ''
Expand Down
105 changes: 105 additions & 0 deletions src/extract.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
/*!
* Copyright (c) Squirrel Chat et al., All rights reserved.
* SPDX-License-Identifier: BSD-3-Clause
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. Neither the name of the copyright holder nor the names of its contributors
* may be used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

import { parseString, parseValue } from './primitive.js'
import { parseArray, parseInlineTable } from './struct.js'
import { type TomlPrimitive, indexOfNewline, skipVoid, skipUntil, skipComment, getStringEnd } from './util.js'
import TomlError from './error.js'

function sliceAndTrimEndOf (str: string, startPtr: number, endPtr: number, allowNewLines?: boolean): [ string, number ] {
let value = str.slice(startPtr, endPtr)

let commentIdx = value.indexOf('#')
if (commentIdx > -1) {
// The call to skipComment allows to "validate" the comment
// (absence of control characters)
skipComment(str, commentIdx)
value = value.slice(0, commentIdx)
}

let trimmed = value.trimEnd()

if (!allowNewLines) {
let newlineIdx = value.indexOf('\n', trimmed.length)
if (newlineIdx > -1) {
throw new TomlError('newlines are not allowed in inline tables', {
toml: str,
ptr: startPtr + newlineIdx
})
}
}

return [ trimmed, commentIdx ]
}

export function extractValue (str: string, ptr: number, end?: string): [ TomlPrimitive, number ] {
let c = str[ptr]
if (c === '[' || c === '{') {
let [ value, endPtr ] = c === '['
? parseArray(str, ptr)
: parseInlineTable(str, ptr)

let newPtr = skipUntil(str, endPtr, ',', end)
if (end === '}') {
let nextNewLine = indexOfNewline(str, endPtr, newPtr)
if (nextNewLine > -1) {
throw new TomlError('newlines are not allowed in inline tables', {
toml: str,
ptr: nextNewLine
})
}
}

return [ value, newPtr ]
}

let endPtr
if (c === '"' || c === "'") {
endPtr = getStringEnd(str, ptr)
return [ parseString(str, ptr, endPtr), endPtr + +(!!end && str[endPtr] === ',') ]
}

endPtr = skipUntil(str, ptr, ',', end)
let slice = sliceAndTrimEndOf(str, ptr, endPtr - (+(str[endPtr - 1] === ',')), end === ']')
if (!slice[0]) {
throw new TomlError('incomplete key-value declaration: no value specified', {
toml: str,
ptr: ptr
})
}

if (end && slice[1] > -1) {
endPtr = skipVoid(str, ptr + slice[1])
endPtr += +(str[endPtr] === ',')
}

return [
parseValue(slice[0], str, ptr),
endPtr,
]
}
Loading

0 comments on commit 11e3a36

Please sign in to comment.