Parcel 2: BundleGraph and Default Bundler plugin #2401

devongovett · 2018-12-12T07:42:12Z

This is an implementation of the default bundler plugin for Parcel 2, along with the infrastructure in core necessary to support it. It implements code splitting, splitting by asset type, asset deduping, and bundle splitting.

Closes #2139. Related to #885.

Adds a BundleGraph to core, and changes the Bundler plugin API to return one. Bundler plugins accept the complete asset graph, along with an empty bundle graph instance to fill in. This way we don't need to expose the constructor to plugins. A BundleGraph contains nodes of type bundle_group and bundle.
Each Bundle object now has an AssetGraph in it instead of just an array of assets. This gives packagers access to the dependencies and other information in the graph for that bundle.
Adds a bunch of helper methods that will be useful for bundler plugins to BundleGraph and AssetGraph, along with some traversal methods to Graph itself.
Changes existing JS packager to use the AssetGraph within the bundle to write its assets. This also included changing the way asset outputs were read from the cache to do it within the packager via asset.getOutput() rather than beforehand in the PackagerRunner.

The default bundler traverses the asset graph and creates a node of type bundle_group for every entry point and async dependency. Connected to these are bundle nodes, which each contain a subset of the full asset graph. The point of the bundle_group nodes is to group things that would be loaded in parallel, e.g. JS + CSS, or JS and other bundle split JS.

Child bundle groups can share dependencies with their parents, so assets that would be duplicated are removed from children. When a dependency with an isolated environment (e.g. a web worker) is hit where it would be impossible to access parent dependencies, it is connected to the root of the bundle graph instead. This way it will have no parents and thus not share anything with other bundles.

Within each bundle node is a sub asset graph containing only the assets in that bundle. These asset graphs inside bundles start from one or more assets and point to the rest of the subgraph for those assets. When an async dependency is hit that creates a separate bundle, the dependency resolution is replaced with the bundle_group node from the bundle graph for this dependency, which gives packagers access to the referenced bundles that should be loaded.

The above example shows three async dependencies, two of which depend on react and lodash. Since those are large libraries and they are shared, they are extracted out into their own separate bundle which is added to the original bundle groups to be loaded in parallel. This allows them to be cached independently of the original bundles, decreasing the overall application size.

Within those async bundles, the separated assets which are no longer in the original bundles are replaced with asset_reference nodes, which give packagers access to the original assets without including them or their dependencies in the bundle.

The full rules for the default bundler are as follows:

If an async or entry dependency is seen, start a new bundle group.
If an asset is a different type than the current bundle, make a parallel bundle in the same bundle group.
If an asset is already in a parent bundle in the same entry point, exclude from child bundles.
If an asset is only in separate isolated entry points (e.g. workers, different HTML pages), duplicate it.
If the sub-graph from an asset is >= 30kb, and the number of parallel requests in the bundle group is < 5, create a new bundle containing the sub-graph.
If two assets are always seen together, put them in the same extracted bundle.
Consider larger split bundles before smaller ones.

Feedback on the API for AssetGraph and BundleGraph would be appreciated! One idea I had was to make bundler plugins composable, but I will explore that in a future PR. 😄

devongovett · 2018-12-12T07:44:31Z

packages/core/core/src/AssetGraph.js

+      id: 'bundle:' + asset.id,
+      type: asset.type,
+      assetGraph: graph,
+      filePath: 'bundle.' + BUNDLECOUNT++ + '.js'


Will be replaced by namer plugins

devongovett · 2018-12-12T07:45:19Z

packages/core/core/src/AssetGraph.js

@@ -220,6 +226,96 @@ export default class AssetGraph extends Graph {
    }
  }

+  getDependencies(asset: Asset): Array<Dependency> {


How do we feel about adding all of these helper methods meant for external plugin consumption to the existing AssetGraph class used in core?

I'd rather not expose the raw AssetGraph. Ideally we'd wrap it and only expose what we want the plugins to use.

Yeah I agree with that. Let's get this PR in first and then discuss what the interface we want to expose is.

devongovett · 2018-12-12T07:46:54Z

packages/core/core/src/PackagerRunner.js

@@ -27,6 +29,14 @@ export default class PackagerRunner {
  }

  async writeBundle(bundle: Bundle) {
+    // deserialize asset graph from JSON


Would be awesome to have a better way of serializing and deserializing complex objects from JSON like this...

devongovett · 2018-12-12T07:48:51Z

packages/core/core/src/BundleGraph.js

+const getBundleGroupId = (bundleGroup: BundleGroup) =>
+  'bundle_group:' + bundleGroup.dependency.id;
+
+export default class BundleGraph extends AssetGraph {


Meant to make this extend from Graph instead of AssetGraph obviously, but AssetGraph has the nice dumpGraphViz function. Will move that to Graph later.

padmaia · 2018-12-12T21:52:02Z

packages/core/core/src/Graph.js

+
+  traverseAncestors(
+    startNode: Node,
+    visit: (node: Node, context?: any, traversal: TraversalContext) => any


Most of the other traversal methods have visit as the first arg, is there a reason why this one is different?

well... in the other ones it is optional, whereas this one it is required. So I felt like it looked nicer to have the callback at the end in this case.

I would probably prefer consistency, but I realize it's a nitpick ;)

padmaia · 2018-12-13T00:16:10Z

All in all, this looks really good! The graph visualizations really help explain what's going on. My one constructive criticism would be that it got a little confusing following how the bundle asset graphs are morphed by the bundler. I think the idea we talked about of exposing a wrapper or a different asset graph class to the bundler plugin would naturally help with this. It might help to strip out nodes that aren't really necessary for bundling, like files and transformer requests. Would be less cognitive load for bundler plugin writers and would be nicer to introspect.

padmaia · 2018-12-13T00:21:16Z

packages/transformers/js/src/JSTransformer.js

-      }
+      // if (GLOBAL_RE.test(asset.code)) {
+      //   walk.ancestor(asset.ast.program, insertGlobals, asset);
+      // }


Will this be uncommented eventually?

yeah... that transform is currently broken

devongovett · 2018-12-13T00:46:13Z

Yeah I was thinking about removing transformer requests and file nodes from the asset graphs inside bundles as well. Would definitely make things simpler. Need to figure out how to do the graph transforms for that when creating a bundle.

One idea I was also thinking about was how to make bundler plugins more composable so you could use the default one in addition to a third party plugin without the third party one needing to reimplement all of the features of the default. We could possibly do this by making the input to bundler plugins only a bundle graph instead of also the original asset graph. The initial bundle graph passed in would contain a single bundle containing all of the assets, and then the bundler plugins would take that bundle graph and modify it to include other bundles. This way you could chain multiple bundlers together by passing the bundle graph to each one to modify. Thoughts?

padmaia · 2018-12-13T17:50:05Z

Hmm, do you have an example of when that would be useful? Seems like it would help if you want to break the bundles apart even more, but not very helpful if you want to rearrange what is in the bundles.

devongovett · 2018-12-14T02:43:10Z

Yeah that's true. If we broke apart the default bundler slightly into its three phases so they could be composed together that might make it easier to keep parts and overrides parts. e.g. keep the dynamic import handling, but override the bundle splitting behavior. idk, I haven't fully thought it through yet.

devongovett · 2018-12-16T22:04:49Z

Going to merge this to get some stuff in. Will go back and refactor some of this later.

devongovett added 16 commits December 11, 2018 22:36

Default bundler

854e45b

Move duplicated assets into their own bundles

318f221

Handle assets of different types

0bdfb0b

Move isolated bundles to a separate tree

7c0b7e6

Support skipping children and stopping graph traversal

afc78c3

Combine bundles when assets are always seen together

0be126f

Read from cache in Asset getOutput method

c715266

Clean up code and write bundles with packagers

bdd3bf7

More cleanup

92e413a

Move BundleGraph to core

4c749e5

Pass a BundleGraph instance to Bundler plugins

609bfbd

Nicer bundle node descriptions

5eadaf2

Add more helper methods to BundleGraph

ac9d27f

Clean up API for js packager

cf1c040

Use getEntryAssets

e10e042

Remove console.log

72ed8c6

devongovett commented Dec 12, 2018

View reviewed changes

padmaia reviewed Dec 12, 2018

View reviewed changes

padmaia reviewed Dec 13, 2018

View reviewed changes

devongovett merged commit d683eeb into v2-work-so-far Dec 16, 2018

devongovett deleted the default-bundler branch December 16, 2018 22:05

devongovett mentioned this pull request Dec 17, 2018

[RFC] Automatically extract modules shared across multiple bundles into their own bundle #885

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parcel 2: BundleGraph and Default Bundler plugin #2401

Parcel 2: BundleGraph and Default Bundler plugin #2401

devongovett commented Dec 12, 2018

devongovett Dec 12, 2018

devongovett Dec 12, 2018

padmaia Dec 12, 2018

devongovett Dec 12, 2018

devongovett Dec 12, 2018

devongovett Dec 12, 2018

padmaia Dec 12, 2018

devongovett Dec 12, 2018

padmaia Dec 13, 2018

padmaia commented Dec 13, 2018

padmaia Dec 13, 2018

devongovett Dec 13, 2018

devongovett commented Dec 13, 2018

padmaia commented Dec 13, 2018

devongovett commented Dec 14, 2018

devongovett commented Dec 16, 2018

Parcel 2: BundleGraph and Default Bundler plugin #2401

Parcel 2: BundleGraph and Default Bundler plugin #2401

Conversation

devongovett commented Dec 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

padmaia commented Dec 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devongovett commented Dec 13, 2018

padmaia commented Dec 13, 2018

devongovett commented Dec 14, 2018

devongovett commented Dec 16, 2018