Performance yaml 1.10, yaml 2 and js-yaml

[gunters63]

I ran a simple benchmark (used https://gist.github.com/jeremychone/700b6ed721565e2488a2cff60b5ad749 because I was lazy), these are the results:

> node test.js write
json stringify 103ms
js-yaml stringify 7946ms
[email protected] stringify 3245ms
yaml@2 stringify 2442ms

> node test.js read
json parsing 94ms
js-yaml parsing 504ms
[email protected] parsing 2108ms
yamljs@2 parsing 9584ms

Some interesting observations:
js-yaml is much quicker than yaml in reading, much slower in writing
yamljs@2 got much slower in reading than [email protected] (19 times slower than js-yaml!), performance is in the same ballpark with writing.

[eemeli]

That is indeed interesting, but I don't think it's actually reflective of real-world performance.

The benchmark writes and reads once for each library a single YAML file with about 500 000 lines. That's obviously going to take a measurable amount of time and be different for different libraries, but is that really the size of file you're usually dealing with? If it is, is YAML really the right solution for you?

In my experience, most real-world files are at most a few hundred lines long, and most real-world usage patterns have significant other work happening around the parsing that effectively makes the time spent figuring out what exactly the stream of bytes is meant to represent so marginal that even a 100x difference in the speed of that operation becomes effectively irrelevant.

This isn't the case for all usages, of course, but I have not myself spent time explicitly benchmarking yaml. The measuring stick that I use to ensure that it's decent enough is the time it takes to run its own test suite of a bit over 3000 tests that read about 600 separate YAML files. Currently that's usually a bit under 2s on my own machine; a bit over 3s if I run jest --clearCache first. CI runs appear to take 8-20s for it.

If there is a real-world operation where you find yaml to be too slow, please do share some more details, and we can figure out how to address that.

[belgattitude]

@eemeli, I've just found this thread while trying to fix a 'small' issue in yarn.

Decided to try your library cause it's the only that would support what I'm looking for (preserving comments in a the yarnrc.yml).

While investigating, I was made aware that changing the parser from js-yaml to yaml would impact the performance of install (especially regarding lock file, a decently sized yaml file actually).

I don't have the exact numbers, but if you're looking for scenarios, here's some interesting comments

• [Bug] Preserve comments and styling in YAML configuration yarnpkg/berry#1463 (comment)

And the PR I'm investigating.

• [WIP] Study possibility of moving from js-yaml to yaml yarnpkg/berry#4135

Based on this https://github.com/yarnpkg/berry/blob/620a500d0f1e02656d20a0478b78af47f0a10a73/yarn.lock. parse with js-yaml gives approx 64ms vs 600ms for yaml (high end cpu).

I'm just letting a comment here: https://github.com/yarnpkg/berry/pull/4135/files#r810542028 if you're interested about the timings.

Just sharing a concrete example, certainly not for creating a benchmark comparison. Your AST approach is fantastic.

Thanks for working on it, it's genially useful.

[eemeli]

Yeah, handling lockfiles with thousands of entries does look like a use case where performance might really matter, and yaml might not be the best choice. It's built for power rather than speed, and simply does more with its input than e.g. js-yaml.

Out of interest, I extracted the relevant code from your test case to this:

// yarn-test.js
const { readFileSync } = require('fs')
const { parse } = require('yaml')

function test(num) {
  const src = readFileSync('./yarn.lock', 'utf8')
  const start = performance.now()
  let res
  for (let i = 0; i < num; ++i) {
    res = parse(src, {
      prettyErrors: false,
      uniqueKeys: false,
      schema: 'failsafe'
    })
  }
  const time = Math.round(performance.now() - start)
  return { count: Object.keys(res).length, time }
}

console.log(test(10))

To get a flame graph of what's happening there, this works:

npx 0x -o ./yarn-test.js

At a first glance, I'm not seeing any one really clear culprit for the processing time; it's more that yaml handles its input in stages: lexing, parsing to a CST, composing an AST, and then converting to JS. It all adds up.

[gunters63]

For now I just found it interesting that two different libraries implementing the same quite complex specification to the last bit (as both pass all official YAML tests) can have quite opposite performance characteristics for reading and writing YAML. I also was a bit surprised that JSON is so much faster. Sure the much more complex syntax is responsible, maybe also the three-tiered processing stage (represent/construct, serialize/compose, present/parse) of YAML.

I am currently starting using YAML files in a quite CPU constrained environment (an industrial controller with roughly the CPU power of a Raspberry 4). I don't know yet with how many YAML files the system will have to deal with, how big they will be and how often they will have to be read or written at the end, but performance is something I have to keep in mind. Maybe many of them can just be JSON files when there is no need for documentation resp. human editing or other features like type tags or aliasing.

That said, maybe there are some low-hanging fruits for performance improvements. In my experience there always are :).

[ingydotnet]

@gunters63 Given your specific situation, you might want to look into https://github.com/pantoniou/libfyaml which is very modern and performant. I don't know what your specific JS requirements are here, but libfyaml does compile to wasm. Its fy-tool binary might be useful for some quick benchmarks.

Here's some quick playing with a million character file:

> printf '%.0s- foo: ba\n' {1..100000} > /tmp/x.yaml
> ls -l /tmp/x.yaml
-rw-r--r-- 1 ingy ingy 1000000 Feb 14 06:59 /tmp/x.yaml
> time fy-tool < /tmp/x.yaml -m json >/dev/null

real	0m0.650s
user	0m0.559s
sys	0m0.098s
> fy-tool < /tmp/x.yaml -m json | wc
 300002  400002 2500003
> fy-tool < /tmp/x.yaml -m json | head -n5
[
  {
    "foo": "ba"
  },
  {

[gunters63]

@ingydotnet : Thanks for the pointer! Very interesting. I was more looking for Javascript and .Net YAML libraries because I am responsible for the (React) UI, the NodeJS web service and a .Net backend, but this will maybe be helpful for my teammates who are responsible for the native C++ realtime part of our software.

Wasm won't be helpful for me as the NodeJS web service will be the one dealing with YAML mostly, but maybe it can be used as a NAPI addon.

[ingydotnet]

@gunters63 libfyaml compiles to wasm and runs fine with NodeJS.
Here's a script and output of compiling fy-tool, using your test script to generate data and using the data to test with.
Unfortunately fy-tool compiled to wasm is the slowest here.
Obviously it's a very crude way to use libfyaml in JS, but it proves it works.
Perhaps someone will write a proper binding one day.

> cat libfyaml-wasm-test 
set -ex
git clone -q https://github.com/pantoniou/libfyaml
cd libfyaml/
./bootstrap.sh &>/dev/null
emconfigure ./configure &>/dev/null
emmake make &>/dev/null
emcc src/tool/fy-tool.c -g3 -I src/valgrind -I include -L src/.libs -l fyaml -o fy-tool.js -s ALLOW_MEMORY_GROWTH=1 &>/dev/null
wget -q https://gist.githubusercontent.com/jeremychone/700b6ed721565e2488a2cff60b5ad749/raw/b827675c53f3838dd6097e42e6887e1f2b19474c/yaml-vs-json-perf.js
perl -p0i -e 's/router/{router}/' yaml-vs-json-perf.js
npm install cmdrouter fs-extra yamljs js-yaml &>/dev/null
node yaml-vs-json-perf.js write >/dev/null
: Native fy-tool
time fy-tool --mode json < data-yamljs.yaml > /dev/null
: WASM fy-tool
time node fy-tool --mode json < data-yamljs.yaml > /dev/null
: NPM yaml.js
time node -e 'console.log(JSON.stringify(require("yaml").parse(require("fs").readFileSync(0,"utf-8")),null,2))' < data-yamljs.yaml >/dev/null

> bash libfyaml-wasm-test 
+ git clone -q https://github.com/pantoniou/libfyaml
+ cd libfyaml/
+ ./bootstrap.sh
+ emconfigure ./configure
+ emmake make
+ emcc src/tool/fy-tool.c -g3 -I src/valgrind -I include -L src/.libs -l fyaml -o fy-tool.js -s ALLOW_MEMORY_GROWTH=1
+ wget -q https://gist.githubusercontent.com/jeremychone/700b6ed721565e2488a2cff60b5ad749/raw/b827675c53f3838dd6097e42e6887e1f2b19474c/yaml-vs-json-perf.js
+ perl -p0i -e 's/router/{router}/' yaml-vs-json-perf.js
+ npm install cmdrouter fs-extra yamljs js-yaml
+ node yaml-vs-json-perf.js write
+ : Native fy-tool
+ fy-tool --mode json

real	0m1.918s
user	0m1.797s
sys	0m0.120s
+ : WASM fy-tool
+ node fy-tool --mode json

real	0m8.690s
user	0m9.211s
sys	0m0.421s
+ : NPM yaml.js
+ node -e 'console.log(JSON.stringify(require("yaml").parse(require("fs").readFileSync(0,"utf-8")),null,2))'

real	0m5.782s
user	0m9.984s
sys	0m0.544s

[ingydotnet]

Hmmm. I just noticed that yaml.js is slowest on user time.
I don't really have a good sense of what that represents here.
I would guess that the last node process must have run on multiple cores since the user time exceeds real time.
It's a crude test for sure.

[gunters63]

I didn't know yet Node.js can use wasm. Maybe I experiment a bit myself, also maybe try to use libfyaml as a NAPI addon. But this gets now slowly off-topic :).