plugins defined by themes can't be resolved in some cases

[pieh]

Description

Gatsby is currently always using project directory when resolving absolute paths for plugins:

gatsby/packages/gatsby/src/bootstrap/load-plugins/load.js

Lines 87 to 102 in 9b2d51b

	const requireSource =
	rootDir !== null
	? createRequireFromPath(`${rootDir}/:internal:`)
	: require
	// If the path is absolute, resolve the directory of the internal plugin,
	// otherwise resolve the directory containing the package.json
	const resolvedPath = slash(
	path.dirname(
	requireSource.resolve(
	path.isAbsolute(pluginName)
	? pluginName
	: `${pluginName}/package.json`
	)
	)
	)

The rootDir is always project root even is given plugin was declared by a theme and a theme has it in list of dependencies.

Confirmed after adding some custom debug logs (that's example from using our e2e-tests/themes/production-runtime - in particular I added marker for gatsby-plugin-page-creator declared by theme we use (https://github.com/gatsbyjs/gatsby/blob/master/e2e-tests/themes/gatsby-theme-about/gatsby-config.js) - look for from-theme (hello) in output. It still uses gatsby/e2e-tests/themes/production-runtime as root, but it should use path to a theme (gatsby-theme-about) as root:

success open and validate gatsby-configs - 0.043s
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby/dist/internal-plugins/dev-404-page" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby/dist/internal-plugins/load-babel-config" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby/dist/internal-plugins/internal-data-bridge" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby/dist/internal-plugins/prod-404" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby/dist/internal-plugins/webpack-theme-component-shadowing" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "gatsby-plugin-page-creator" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "gatsby-plugin-page-creator" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / from-theme (hello)
trying to resolve "gatsby-theme-about" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "gatsby-plugin-page-creator" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug
trying to resolve "/Users/misiek/dev/gatsby/e2e-tests/themes/node_modules/gatsby-plugin-page-creator/index.js" from "/Users/misiek/dev/gatsby/e2e-tests/themes/production-runtime" / No debug

Steps to reproduce

It's not quite clear how to reproduce when this cause actual build/develop errors. I have some reproduction ( https://github.com/pieh/gatsby-default-with-gatsby-theme-blog-version-missmatch ) but problem here there is another issue (gatsby-plugin-mdx specific, where fixing this error alone won't solve the issue in the reproduction - that's because gatsby-plugin-mdx uses custom field gatsbyRemarkPlugins to define subplugins that is not processed by gatsby core - there will be separate issue for that).

So only way to verify fix is manual logging I think (Unless we are able to create theme that would remark instead of mdx and maybe similar thing would happen)

How to fix

Without doing major refactoring of gatsby-config merging and plugin loading I think we can fix this issue by adding additional field during gatsby-config traversal ( https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/bootstrap/load-themes/index.js ) that would specify gatsby-config location where given plugin instance was declared. This field would need to be ignore when we merge duplicate plugins ( https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/utils/merge-gatsby-config.js#L52-L55 ) and finally used in load-plugins section to correctly use createRequireFromPath using that field instead of rootDir always (might need to look out for main gatsby-config being special case)

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜

[pieh]

This was fixed by #23696

There is still not fixed issue described in #21592