Feature Request: Manual component code splitting

[alexlande]

Summary

I'm working on an app that would benefit from component-based code splitting that works with static rendering. I tweeted about it here: https://twitter.com/gatsbyjs/status/1007760243596513280. Basically, I'd like to be able to code split components using import() while rendering them in statically built pages.

The result would be that the HTML for a page contains the lazy loaded component's markup and the app isn't hydrated on the client until components used on the page are loaded. After the initial page load, other pages would lazy load these components as you navigate through the app.

Code splitting already works, so preloading before rendering and statically rendering code split components would be the new behavior here.

Tools

react-static handles this use case with react-universal-component. Usage example: https://github.com/nozzle/react-static/blob/master/examples/dynamic-imports/src/containers/About.js

I typically use react-loadable for component code splitting. That project has pretty comprehensive instructions on server rendering, but there are a lot of moving parts-- a babel plugin, a webpack plugin, a component to capture a list of rendered loadable components, and preload methods to be used on the client and the server to ensure that the initial render + hydration works.

Motivation

The project I'm working on is a large site where page customization is driven by components rather than templates. There's only a single template, but authors using the connected CMS can mix and match a large number (100+) of content components that fit into slots in the template. This means that Gatsby's existing template code splitting doesn't help us much, and if we were to load all of the components in the template the resulting JS bundle would be huge.

[KyleAMathews]

This is definitely something we want to support. Question is how :-) This space seems like a moving target so not entirely sure what we want to do. Also question is can we support react-loadable/react-universal-component in plugins as that'd be ideal as we'd decouple core from supporting these different options.

[alexlande]

Yep, agreed that a plugin would be best. I ran into two issues looking at this that make me unsure if it needs deeper integration than a plugin can do:

1. react-loadable has an async preloadAll method used to preload components in Node, that needs to be called before rendering. I think this would require async replaceRenderer support along the lines of WIP: Add async SSR data fetching #2423 if you were going to do it through a plugin.
2. loadable and universal-component both have methods that take the webpack stats object and return a list of modules to load for the static render-- not sure if this would conflict with Gatsby or would fit in to the plugin architecture somewhere. https://github.com/jamiebuilds/react-loadable#mapping-loaded-modules-to-bundles and https://github.com/faceyspacey/react-universal-component#flushing-for-ssr.

[KyleAMathews]

Re 1) Totally ok to either extend existing APIs or add a new API. Re 2) we already use data from the stats object e.g.

gatsby/packages/gatsby/src/cache-dir/static-entry.js

Line 157 in 2e55615

let chunks = get(stats, fetchKey)

We filter it down before writing it out at

gatsby/packages/gatsby/src/utils/webpack.config.js

Lines 214 to 240 in 2e55615

	apply: function(compiler) {
	compiler.hooks.done.tapAsync(
	`gatsby-webpack-stats-extractor`,
	(stats, done) => {
	let assets = {}
	for (let chunkGroup of stats.compilation.chunkGroups) {
	if (chunkGroup.name) {
	let files = []
	for (let chunk of chunkGroup.chunks) {
	files.push(...chunk.files)
	}
	assets[chunkGroup.name] = files.filter(
	f => f.slice(-4) !== `.map`
	)
	}
	}
	const webpackStats = {
	...stats.toJson({ all: false, chunkGroups: true }),
	assetsByChunkName: assets,
	}
	fs.writeFile(
	path.join(`public`, `webpack.stats.json`),
	JSON.stringify(webpackStats),
	done
	)

Would you like to PR a plugin + any core changes necessary for either/both of those? That'd be super nice!

[alexlande]

I'd love to, but it might be a while until I'm able to get to it. I'll take a shot at it, but you should leave the help wanted tag on this in the meantime in case anyone else is able to take it 😄

[DylanVann]

Exact same situation here. One template, flexible component based content. So I end up having one file with:

const convert = (typename, props) => {
  if (typename === 'Image') return <Image {...props} />
  if (typename === 'Video') return <Video {...props} />
  // etc.
}

So then all the components get included for every page chunk instead of just ones that are actually used.

[ronanlevesque]

Same here. This would be the perfect solution to handle inline SVGs when combined with https://github.com/smooth-code/svgr.

[gatsbot]

Old issues will be closed after 30 days of inactivity. This issue has been quiet for 20 days and is being marked as stale. Reply here or add the label "not stale" to keep this issue open!

[gatsbot]

This issue is being closed due to inactivity. Is this a mistake? Please re-open this issue or create a new issue.

[Undistraction]

@DylanVann @ronanlevesque I know this is an old issue, but did you come up with a good solution? I'm facing the same problem - I'm using [rehype-react] to map custom html tags in Mardown to react components, which is creating a bottlenecked file with imports for all possible components, meaning they all end up in the same chunk.

[ryaninvents]

@Undistraction I was able to implement custom components using MDX, a custom context, and useEffect to pass imported components to the layout root. I don't have the code in front of me but might be able to sketch it out if you're interested

[ryaninvents]

@Undistraction Here's a sandbox if you want to check it out.

[jrestall]

Exact same situation here. One template, flexible component based content. So I end up having one file with:

const convert = (typename, props) => {
  if (typename === 'Image') return <Image {...props} />
  if (typename === 'Video') return <Video {...props} />
  // etc.
}

So then all the components get included for every page chunk instead of just ones that are actually used.

@DylanVann Did you ever find a solution to this? I've got the exact same situation where a Headless CMS configures pages from hundreds of possible components so I need to code split each page otherwise the one template page bundle becomes huge.

I'm thinking I'll need to dynamically generate during the build a different Page.tsx template file for each page and insert only imports for the components that are used in that configured page. Does this sound like a reasonable approach @KyleAMathews?

[jrestall]

For those coming across this issue, as an update to the above, I did end up solving this for the OrchardCore CMS. It builds pages dynamically from potentially 100's of widgets so I did end up creating a template file dynamically for each page that is defined in the CMS.

The main code can be found at

• https://github.com/jrestall/gatsby-orchardcore/blob/master/packages/gatsby-theme-orchardcore-pages/gatsby-node.js#L34
• https://github.com/jrestall/gatsby-orchardcore/blob/master/packages/gatsby-theme-orchardcore-flows/gatsby-node.js#L60

Currently waiting on #15196 to be merged since the themes use a custom node API. @KyleAMathews

[manuelJung]

can we re-open this issue since #15196 won't be merged? this is currently the only feature that stops me from using gatsby in production (since component-based codesplitting is a must have for the projekt).

[jrestall]

Hey @manuelJung #15196 isn't necessary to solve this issue, it was only related to how I wanted to implement my Gatsby themes, which I've now changed to using a IContentBuilder interface that each theme uses to contribute to building each page.

To fix this issue, if you have similar requirements as me, look into how I'm dynamically building a template file for each page based on it's discovered dependencies (for me this is the widgets used on each page).

[manuelJung]

Hey @jrestall, this is a really clever solution but i feels a bit hacky. I'm having thousands of routes and this would lead to tousands of template files. I've tried to write a plugin with loadable-components, but it requires the bundling and rendering to happen in the same context (even if it would work, we would see a lot of loading-spinners when we navigate) so i don't think manual code splitting is doable in a traditional way (loadable-components, react-loadable, react-universal-components...)

But i have an idea for a more gatsby-ish way: We could extends the createPage action like this:

actions.createPage({
  path: `/my-path`,
  component: path.resolve(__dirname, 'src/templates/MyComponent.js'),
  modules: {
    WidgetA: path.resolve(__dirname, 'src/widgets/WidgetA.js'),
    WidgetB: path.resolve(__dirname, 'src/widgets/WidgetB.js'),
    ....
  }
})

these module-paths will be added to the page-data.json (like the componentChunkName). Within gatsby/cache-dir/load.js we would also load all modules when we execute loadPage. These Components will be injected in the template where we can render them:

// templates/MyComponent.js

export default ({data, modules}) => {
  const {WidgetA, WidgetB} = modules

return (
  <div>
    <WidgetA/>
    <WidgetB/>
  </div>
)
}

The infrastructure to do this already exists and the best thing is, that all Widgets will get preloaded by default. That means there would be no loading-spinner ever, since the route renders only when all modules are fetched. That way we could add thousands of widget without affecting the main bundle size. With React-Context we could make these widgets accessible to all sub-components

I think this could enable very flexible system-architectures. Technically it would also be able to allow these widgets to fire their own (dynamic) graphql queries since theys are already known during the bootstrap-phase (but that's a bit out of scope now)

[jrestall]

@manuelJung, I'd be keen to see a working solution as that does seem cleaner, keep us updated!