Slow performance with large arrays

[tedeschia]

I need to validate some large arrays (about 100,000 items). When there are lots of validation errors, the validation process takes a long time to run.

I reproduced the issue in CodeSandbox: https://codesandbox.io/s/yup-performance-test-6jxn5m?file=/src/index.js

There I defined an array schema of required numbers and run the validation for some datasets to measure the performance. This is the result:

Validation of 5000 items took 169.6 ms
Validation of 10000 items took 320.1 ms - 88.7% increase vs last fun
Validation of 20000 items took 985.1 ms - 207.7% increase vs last fun
Validation of 40000 items took 5,034.8 ms - 411.1% increase vs last fun

It seems to be an exponential increase of time to process the validations.

Doing some profiling, it seems that the issue is in this line of ValidationError class:
this.inner = this.inner.concat(err.inner.length ? err.inner : err);

[jquense]

That would probably be pretty easy to optimize if you want to take a stab at it?

[jquense]

Alternatively, you could at abortEarly: true to fail fast, since it's unlikely you need 40,000 of probably the same error

[tedeschia]

abortEarly is not an option for me. It's nos necessarily all the same errors (this case was just an example) and I need to give the user a feedback for every row.

I tried with this change and it was solved:

const innerErrors = err.inner.length ? err.inner : [err];
this.inner.splice(this.inner.length, 0, ...innerErrors);

do you want me to send a PR?

I'm also measuring other bottlenecks and found that ValidationError constructor also takes its time (considering thousands of it) at:
super()
Error.captureStackTrace(this, ValidationError)
I'm wondering if I could implement any kind of "minimalistic" version of ValidationError class, in case of people (like me) that prioritize performance over functionallity (I'm just interested in error message, and don't use stack trace at all).
...or maybe this only happens in dev mode? I'm not sure.

[jquense]

I'd definitely be open to a PR for both, maybe we could toggle the stack traces off via an option, tbh the stack isn't really useful for validation error

[tedeschia]

With the proposed optimizations in #2044, the performance is significantly improved, and there is no exponential performance degradation when the array grows.
But anyway, it takes some time to perform the validations. In my tests, an array of 60000 items takes about 4 seconds to get validated, and during this process, the UI gets blocked.
I'm wondering if processing the validations in chunks, freeing the thread between them, could benefit the user experience even when the validation needs some time to process. I would explore this if you consider it a good idea.

[tedeschia]

Hi @jquense, this is me again with this performance improvement!
I just sent the PR #2142 adding a new ValidationError class to be used when disableStackTrace=true, so it keeps compatibility with previous versions, while improving error creation performance when disableStackTrace is used.
Please could you review it?
Regards!