[RFC] Parcel 2: Loader Design

[devongovett]

This issue is to discuss the design of loaders in Parcel 2. It will be closed once resolved and a new one will be opened for the implementation.

Loader plugins in Parcel 2 are responsible for generating runtime modules that know how to load assets of various types over the network, e.g. JS, CSS, WASM, etc. Loaders are added to the bundle graph just before packaging, once code split points are known. There are loader plugins defined in the .parcelrc config for various types of assets, and their generate method is called to generate an asset to add to the graph for each bundle reference.

The problem with loaders is that they are pretty JavaScript specific, at least how they are currently defined. They load bundles of some type into a JS context. This is somewhat limiting potentially if we want Parcel to support more target languages, like native or something. They are also the only the only plugin type that is specific to JS in Parcel core.

I looked into implementing loaders in the BundlerRunner, just after naming and before packaging. There were a few issues:

1. We need to modify the asset graph in each bundle in order to add loader modules and other runtime modules necessary to make them work. There is not a very obvious place for them to go in the main asset graph as they are not explicit dependencies. I attached them with edges from the bundle node in an individual bundle's asset graph, which is kinda weird but works. Would love to find a better design for that. This does mean that these dependencies are not watched (they are not in the main asset graph), but that would probably be ok I think.
   
2. The runtime which actually calls loaders needs to get injected into the bundle as well (bundle-loader.js in the above graph). An individual loader only knows how to load one bundle, but multiple bundles may need to be loaded in parallel for a bundle group, and we'd want to cache in order to not load the same bundle twice. The runtime handles that. However, the runtime is pretty specific to the packager's implementation of the module system - needs to hook into module registration etc. Seems like the runtime should actually be injected by the packager...
3. We can't really do this in the packager though, since packagers run in workers and we need to modify the bundle graph and have access to parcel core (resolver + transformers). One idea would be to just pre-bundle the runtime and include that in the packager outside the asset graph altogether. That might be kinda limiting and annoying though. And the loaders themselves would still need to get added to the graph somehow.

One idea is to generalize the idea of loaders into "runtime" plugins, which run after bundlers, and can inject any sort of runtime into a bundle. For example, there would be a JS runtime plugin which would inject loaders for JS, the runtime for loaders, the HMR runtime, etc. It still seems like packagers and runtimes are tightly coupled though. The way the packager writes the module system would have an effect on how the runtime could work.

Any other ideas would be greatly appreciated. I think this is the last big piece we need to figure out for Parcel 2!

cc. @padmaia @jamiebuilds

[devongovett]

Explored this a bunch more and came to the following conclusions. There are basically 3 ways we could go about this in order to keep Parcel core language agnostic.

1. Add an additional level of nesting to loaders, and add "runtime" plugins. Runtime plugins receive a bundle as input and can use an API to add generated assets to the bundle's asset graph. Those assets would get run through the same transformer pipeline as the rest of the assets, but would be generated rather than come from a file.

   "loaders": {
     "*.js": {
       "*.js": "@parcel/loader-js-js
     }
   },
   "runtime": {
     "*.js": ["@parcel/runtime-js-hmr", "@parcel/runtime-js-loaders"],
   }
   

   • Pros: Runtime plugins can do anything they want, not restricted to just HMR and loaders.
   • Cons: Two plugin types rather than one, runtime plugins somehow need access to the config in order to access loader plugins or loaders need to be added to the graph in core and references need to be passed to runtimes?

2. Loaders as options to runtime plugins. Runtime plugins still access the graph and can modify it as they see fit.

   "runtime": {
     "*.js": [
       "@parcel/runtime-js-hmr",
       ["@parcel/runtime-js-loaders", {
         "loaders": {
           "*.js": "@parcel/loader-js",
           "*.css": "@parcel/loader-css"
         }
       }]
     ],
   }

   • Pros: Only one plugin type (maybe?)
   • Cons: What do loader packages consist of? Are they plugins? If so, then why are they options? If not, then what are they and how do they generate code? How do the options get merged in configs?

3. Specific runtime plugins. In this case, runtime plugins return an asset, which is added to the graph according to the specific runtime type they are (e.g. HMR as an entry, loaders attached to bundles, etc.).

   "runtime": {
     "*.js": {
       "hmr": "@parcel/runtime-js-hmr",
       "loader": "@parcel/runtime-loader-js",
       "loaders": {
         "*.js": "@parcel/loader-js",
         "*.css": "@parcel/loader-js-css"
       }
     }
   }

   • Pros: Only one plugin type. Both return a single asset's contents rather than mutating in whatever way they want. Easier to write and document.
   • Cons: Less flexible/extensible for custom runtimes. How the loader runtime accesses the loaders themselves is still a question. "loader" vs "loaders" is confusing.

I think number 3 is probably the best of the above options, but there are still some unanswered questions. I've prototyped it here. Let me know what you think.

[jamiebuilds]

Thinking about this more, we might actually be trying to offer too much in terms of configuration.

What if instead we shipped a single "runtime" (plugin) that is responsible for loading every asset type?

{
  "runtime": "@parcel/runtime-web"
}

It would have to implement logic to load every possible asset type. Luckily, in terms of the final transforms assets, there aren't that many unique asset types you load on the web:

• HTML
• CSS
• JS
• JSON
• Static Content: Images, Videos, Fonts, Misc Downloads (PDFs and such), etc.

A singular runtime would have a lot more control than what we can offer through configuration. It'd be easier to ship a @parcel/runtime-electron or @parcel/runtime-react-native or whatever as well.

[devongovett]

Yeah that could possibly work. Perhaps it is too rare a case to need to customize the way bundles are loaded on demand. Parcel 1 has:

• JS
• WASM
• HTML (which is really a generic plain text loader)
• CSS

We could allow multiple runtime plugins to run as well. That way people could add their own in addition to the default if they wanted.

I wonder if we should support multiple runtime plugins that depend on the target environment. That way you could have a multi-target build (e.g. web + electron), and have it go through different runtime plugins.

{
  "runtime": {
    "browser": ["@parcel/runtime-browser"],
    "node": ["@parcel/runtime-node"],
    "electron": ["@parcel/runtime-electron"]
  }
}

Thoughts?

[jamiebuilds]

I wonder if we should support multiple runtime plugins that depend on the target environment ...

{
  "runtime": {
    "browser": ["@parcel/runtime-browser"],
    "node": ["@parcel/runtime-node"],
    "electron": ["@parcel/runtime-electron"]
  }
}

I'm not sure, because there you are talking about targets where you have "main/module/browser/electron" which have no meaning besides their associated environment... Technically you could redefine the "electron" target to have an environment that has nothing to do with electron.

Really this configuration is more aligned with the "environment" than it is the "target", so you'd have one of two options that I can see:

1. Put it in the environment config

{
  "targets": {
    "electron": {
      "engines": { "electron": ">=2.x" },
      "runtime": "@parcel/runtime-electron"
    }
  }
}

2. Associate it with the environment config (somehow ... not like this:)

{
  "runtime": {
    "...if electron in engines...": "@parcel/runtime-electron",
    "...if has browserlist....": "@parcel/runtime-browser",
    "...if node in engines...": "@parcel/runtime-node"
  }
}

We could allow multiple runtime plugins to run as well. That way people could add their own in addition to the default if they wanted.

Do you mean this separately from the runtime.browser/node/electron bit above? Or do you mean accepting an array of runtimes?

[devongovett]

Each bundle has an associated environment, which gets propagated from the target, and which can possibly change at an async import (e.g. "browser" -> "web-worker"). This is the environment's context, which represent the globals and such that are available in that environment. This is basically what a runtime plugin would need to target.

It's not based on the name given by the user in package.json which is meaningless as you say, but is resolved based on engines specified there or static analysis. The ones we have so far are here.

Do you mean this separately from the runtime.browser/node/electron bit above? Or do you mean accepting an array of runtimes?

Accepting an array of runtimes. I don't think there is a reason not to?

[jamiebuilds]

Each bundle has an associated environment, which gets propagated from the target, and which can possibly change at an async import (e.g. "browser" -> "web-worker"). This is the environment's context, which represent the globals and such that are available in that environment. This is basically what a runtime plugin would need to target.

Oh I understand.

Accepting an array of runtimes. I don't think there is a reason not to?

Depends on what you want to allow runtimes to do. If you just want a fallback system of loaders like the "resolver" plugins, I think that's fine.

[devongovett]

Depends on what you want to allow runtimes to do.

I could see someone wanting to inject some custom code not even related to loaders or hmr into every bundle for some purpose. Basically runtime plugins are an opportunity to add things to a bundle in some way.

[ZWkang]

runtime mean loader can control all the bundle?

[devongovett]

I prototyped the above idea here. Overall, I like it but it ended up with some duplication between the browser and node runtimes. The actual loaders are all different, but the runtime that calls the loaders is the same, along with the code that inserts it into the bundle. HMR, when it comes, will probably be similar but slightly different.

I guess we could do a single runtime package with all the loaders in it, e.g. @parcel/runtime-js instead of per environment ones, but I kinda liked that idea. Alternatively, we could keep @parcel/runtime-browser and @parcel/runtime-node and add a util package with the shared code. Thoughts?

[padmaia]

I guess we could do a single runtime package with all the loaders in it, e.g. @parcel/runtime-js instead of per environment ones, but I kinda liked that idea.

I think the good idea was the idea of a runtime plugin that is resolved by environment. Seems okay to me to have the same plugin resolve for different environments if those environments are mostly the same. Doesn't seem like we would gain much from splitting it into 3 different packages, at least at this point.

[devongovett]

Created a PR here: #2534