[WIP] Build optimizations

[divmain]

This PR is probably worth looking at now. All tests pass, although I do intend to add additional tests. I also need to pull in latest canary; please advise on direction for the question at the end of this PR description, and that'll inform how I pull in latest canary commits.

Measuring wins was difficult; I was looking for a good React project (or projects) that could be used to test performance changes (both for this PR and others), but without much success. I put together fuzzponent for this purpose. Given a seed and a recurse-depth, it'll generate a large tree of React component JS files, PRNG-style.

Performance has definitely improved with the changes in this PR. To test, I generated a project of 3021 React components with fuzzponent -d 2 -s 206. If you run that command, you should get the same components.

• No cache (avg): 20.64 seconds
• Cold cache (avg): 17.76 seconds (14% improvement)
• Warm cache (avg): 12.19 seconds (41% improvement)

All times were measured w/ time next build .. I haven't yet investigated dev build times; I don't expect to see significant changes. Cold cache times improved because server build step improved (client build warmed the cache).

The loader exposes an API for a non-FS-backed cache. This could be an in-memory cache, Redis, or something else.

(no longer relevant)

There are a ton of commits showing here - that is because I preserved the git commit history for babel-loader when pulling it in with:

git filter-branch -f --tree-filter '
    if [ -d ./src ]; then
        find . -depth 1 -not -name "src" -not -name ".git" -not -name "LICENSE" -print0 | xargs -0 rm -fr
        mkdir -p packages/next/build/webpack/loaders
        mv src packages/next/build/webpack/loaders/next-babel-loader
        mv packages/next/build/webpack/loaders/next-babel-loader/index.js packages/next/build/webpack/loaders/next-babel-loader/babel-loader.js
        if [ -f ./LICENSE ]; then
            mv LICENSE packages/next/build/webpack/loaders/next-babel-loader/
        fi
    else 
        find . -depth 1 -not -name ".git" -print0 | xargs -0 rm -fr
    fi
' HEAD

I can squash all of that if you don't see value in having history for those files.

@timneutkens @rauchg

[timneutkens]

We'll squash the PR, so I guess keeping the history here is fine either way. However I think we're pinging everyone that ever contributed to babel-loader with this as the PR is showing 56 participants

[divmain]

Squashed the commits to remove all the extra participants (sorry about that!). Will address other feedback later today. Thanks!

[vjpr]

@divmain

From what I gather, this PR implements a custom babel webpack loader with a file system cache.

Since webpack5 will have built-in persistent caching (https://twitter.com/TheLarkInn/status/1069246795031576576), will we still need this?

I am keen to have an FS-backed cache for next.js asap though, so this looks great!

[vjpr]

@divmain Also, would this loader be useful for other webpack projects?

Maybe its better to create a separate repo for it.

Also, not sure how close it is to being merged and released, but it might be an idea to override the next-babel-loader in next.config.js with this one for testing (easier than working on a forked next.js).

[vjpr]

Just occurred to me that this is will not really improve things more than babel's cacheDirectory, it would just allow more caching options such as a redis db that could be shared with a team.

[wtgtybhertgeghgtwtg]

I'd go with make-dir since it's already in the dependency tree via find-cache-dir.

[timneutkens]

I'm going to close this PR for now, we'll revisit it 🔜