Crashes while looking for helper files in huge project subdirectories

[transcranial]

Currently, ava looks for helper files in all project subdirectories excluding node_modules:

https://github.com/avajs/ava/blob/master/lib/ava-files.js#L90-L95

This can overwhelm ava when there exists subdirectories that are huge and/or with complicated fs hierarchies:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [ava]
 2: 0x126389c [ava]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [ava]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [ava]
 5: v8::internal::Factory::NewInternalizedStringImpl(v8::internal::Handle<v8::internal::String>, int, unsigned int) [ava]
 6: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [ava]
 7: v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, bool*, v8::internal::LookupIterator::Configuration) [ava]
 8: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [ava]
 9: 0x126964c063a7
Aborted (core dumped)

Allowing the specification of exclusion globs for helper files would help prevent this. We reverted back to v0.17.0 for now.

Environment:
node v7.4.0
linux 4.2.0-34-generic
ava v0.18.1

[sindresorhus]

Approximately how many files/directories and how deep? Do you specify any arguments to $ ava or any AVA config in package.json?

Could you try removing

ava/lib/ava-files.js

Lines 141 to 143 in b886c5b

	cache: Object.create(null),
	statCache: Object.create(null),
	realpathCache: Object.create(null),

and see if it helps?

Could you try to narrow down to which patterns are causing this?

I find it strange that just globbing for files would cause this. AVA must be reading in too much data somewhere.

[transcranial]

Removing those lines make no noticeable difference. Changing my package.json config, including removing it, seems to make no difference either. I'm able to reproduce with a new barebones project, but with the problematic subdirectory copied over.

Interestingly, it seems to be the cumulative effect of all 4 glob patterns, and not any one in particular. Leaving 1 pattern, tests pass but takes 11s. Leaving 2, pass but takes 18s. Leaving 3, pass but takes 35s. With all 4, takes 55s and then exits with the error above. Adding the subdirectory manually to excludePatterns of handlePaths in findTestHelpers solves the issue and tests pass in 1s. Folder maxdepth is 3. Stats:

$ find dev/ -type d | wc -l
17630
$ find dev/ -type f | wc -l
107270

The folder is already in .gitignore. Would it make sense to automatically exclude those in .gitignore in addition to node_modules?

[sindresorhus]

The folder is already in .gitignore. Would it make sense to automatically exclude those in .gitignore in addition to node_modules?

Yes, that should be done regardless. Could you open a separate issue for that?

[novemberborn]

This is caused by optimistically transpiling helper files. With RFC 001 we'll only transpile them when they're required in the workers.

[novemberborn]

With RFC 001 we'll only transpile them when they're required in the workers.

Clearly this hasn't happened yet.

I think we should change how files patterns are interpreted, and then add support for helpers patterns:

• files patterns that match files cause those files to be treated as tests, not helpers
  • We support negation patterns to refine selections if necessary
  • We exclude {project-dir}/node_modules by default, though ideally you can override this by providing a more specific path
• files patterns that match directories mean that containing files are treated as tests, unless they're inside fixtures and helpers directories (that are children of the matched directories), or start with _ (regardless of how deeply they are nested)

This makes test file matching more understandable. We can then add support for helper patterns, so you can match those and we transpile them.

A utility command that prints out what files are matched and how they're treated would also be useful.

(Originally from #1320 (comment))

[wmertens]

There are a number of issues related to this and the fix is very simple: clear/customize the helper matcher patterns.

Would it not be possible to roll out a quick fix for this by making a helperPatterns option that replaces the default one? For people that transpile everything they can then clear that array.

[maschwenk]

I see this is a thoroughly discussed issue. Is there any workaround outside of forking and emptying that const array?