[gatsby-plugin-mdx] Plugins can't create nodes properly

[flogy]

Description

Sub-plugins of gatsby-plugin-mdx that are included via gatsbyRemarkPlugins array can't properly create nodes. The nodes are only created if cache was cleared before or changes were made to the MDX files. If nothing has changed the sub-plugin's code is not executed and thus no nodes can be created.

Steps to reproduce

▶️ Minimal Reproduction project: https://github.com/flogy/gatsby-remark-plugins-caching-issue

• Start the project and check that the test node is available in GraphiQL.
• Stop the project and start it again without clearing the cache or changing any MDX file.
• See that the node is no longer available in GraphiQL.
• Stop the project and change a MDX file or clear the cache.
• Start the project and see that the node is available again.

For testing that nodes can't be created in gatsby-node.js instead (which I tried as workaround as described later in this issue):

• Start the project and see that the console output from the gatsby-node.js never appears in the console.

Expected result

Having a possibility to create the required nodes consistently.

Actual result

The nodes are only created if cache was cleared before or changes were made to the MDX files. If nothing has changed the sub-plugin's code is not executed and thus no nodes can be created.

Suggestions on how to fix this

I can imagine those three possibilities. The latter two are more like workarounds. Maybe there are even more and cleaner solutions?

• gatsby-plugin-mdx could look for a second exported function (besides the default one) in the sub-plugins that would be called even if all information are taken from the cache. In this function, I could then create my nodes.
• gatsby-plugin-mdx could allow disabling the cache.
• Allow sub-plugins to also have a gatsby-node.js that is executed. Like that, sub-plugins could save all the information they would like to create nodes with to the cache and the gatsby-node.js - which would be executed on every start - could then get that information from the cache and create the nodes. I actually thought I could use this option, as gatsby-remark-images seems to also work with it's gatsby-browser.js, but in my experiments it never executed my gatsby-node.js when I had my sub-plugin inside the gatsbyRemarkPlugins array. When I took it out and added it to the main plugins of Gatsby, it worked.

Environment

 System:
    OS: Windows 10 10.0.18362
    CPU: (8) x64 Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
  Binaries:
    Node: 10.18.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.16.0 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 6.13.7 - C:\Program Files\nodejs\npm.CMD
  Languages:
    Python: 3.5.1 - C:\Program Files\Python35\python.EXE
  Browsers:
    Edge: 44.18362.449.0
  npmPackages:
    gatsby: ^2.15.24 => 2.18.18
    gatsby-image: ^2.2.21 => 2.2.37
    gatsby-mdx-tts: 0.0.2 => 0.0.2
    gatsby-plugin-emotion: ^4.1.14 => 4.1.18
    gatsby-plugin-fixhash: 0.0.4 => 0.0.4
    gatsby-plugin-google-analytics: ^2.1.31 => 2.1.31
    gatsby-plugin-linkedin-insight: ^1.0.1 => 1.0.1
    gatsby-plugin-manifest: ^2.2.18 => 2.2.34
    gatsby-plugin-mdx: ^1.0.74 => 1.0.74
    gatsby-plugin-netlify: ^2.1.30 => 2.1.30
    gatsby-plugin-offline: ^3.0.9 => 3.0.30
    gatsby-plugin-prefetch-google-fonts: ^1.4.3 => 1.4.3
    gatsby-plugin-react-helmet: ^3.1.8 => 3.1.18
    gatsby-plugin-react-svg: ^2.1.2 => 2.1.2
    gatsby-plugin-robots-txt: ^1.5.0 => 1.5.0
    gatsby-plugin-sharp: ^2.2.25 => 2.3.10
    gatsby-plugin-sitemap: ^2.2.19 => 2.2.24
    gatsby-plugin-typescript: ^2.1.9 => 2.1.23
    gatsby-remark-external-links: 0.0.4 => 0.0.4
    gatsby-remark-images: ^3.1.25 => 3.1.39
    gatsby-source-filesystem: ^2.1.28 => 2.1.43
    gatsby-transformer-sharp: ^2.2.17 => 2.3.9
    gatsby-transformer-yaml: ^2.2.15 => 2.2.20

Thanks in advance!

[pieh]

Hey @flogy!

The gatsby-node.js hooks not being executed is related to this issue - #21592 (particularly last paragraph after "---edit:" touches on it)

As for creating nodes in <subplugin>/index.js - that's by design - this will be called only on new or updated nodes. Also if you are creating nodes there, you should ALWAYS set parent to be id of MDX node (otherwise nodes that you create there will be garbage collected). Also note that the nodes that you create there should be derived from Mdx node in some way (it should not be totally unrelated node).

So correct way to handle this is suggestion number 3 (which require #21592 to be handled)

[flogy]

Hi @pieh

Thanks for your fast reply!

Would the created nodes actually survive a restart if they were linked to the Mdx nodes as children? If yes, this might be my preferred solution.

I just tryed this but unfortunately I was not able to link them to the parent node (ID of the parent node seems to be fakeNodeIdMDXFileABugIfYouSeeThis 😅). But that's another issue - I opened a separate issue here: #21818. I also created a new commit to the minimal reproduction project where I now also demonstrate the non-working node linking.

If not, I'd go with option 3, but it feels a bit like a workaround to me.

[pieh]

Would the created nodes actually survive a restart if they were linked to the Mdx nodes as children? If yes, this might be my preferred solution.

They should be preserved, but you might also need to use createParentChildLink action together with creating Node with parent set to markdownNode.id. That's how all transformers work right now - for example

gatsby/packages/gatsby-transformer-json/src/gatsby-node.js

Lines 23 to 34 in 4bc595b

	const jsonNode = {
	...obj,
	id,
	children: [],
	parent: node.id,
	internal: {
	contentDigest: createContentDigest(obj),
	type,
	},
	}
	createNode(jsonNode)
	createParentChildLink({ parent: node, child: jsonNode })

But yeah, getting "fake id" is a blocker to make that happen and I don't have insight into this yet (thanks for opening separate issue for it!)

[flogy]

Very cool! Looking forward to try this again as soon as we have a solution for the "fake id" problem. Using the createParentChildLink didn't do the trick - as you expected - because of that issue.

[LekoArts]

Hi!

I'm closing this as a stale issue as in the meantime Gatsby 4 and related packages were released. You can check our Framework Version Support Page to see which versions currently receive active support. If this is a feature request, please create a discussion as we moved feature requests from issues to GitHub Discussions.

Please try the mentioned issue on the latest version (using the next tag) and if you still see this problem, open a new bug report. It must include a minimal reproduction.

Thanks!