Unexpected Rendering Engines validation failure (Gatsby v4)

[gkossakowski]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

After migrating to Gatsby v4, build fails with:

failed Validating Rendering Engines - 0.641s

 ERROR #98001  WEBPACK

Built Rendering Engines failed validation failed validation.

Please open an issue with a reproduction at https://github.com/gatsbyjs/gatsby/issues/new for more help



  Error: Generated engines use disallowed import "babel-runtime/helpers/asyncToGenerator". Only allowed imports are to Node.js builtin modules or engines internals.

I don't know Gatsby's internals but it looks like that import is intentional and coming from Gatsby itself: https://github.com/gatsbyjs/gatsby/pull/33030/files#diff-53a98e8932749e1788143dca477bd89b34e3f53d2db77569afac21ddede8f8a4R36

Is that an indication of a bug in Gatsby?

Reproduction Link

https://github.com/gkossakowski/gkossakowski.github.io/tree/gatsby-v4

Steps to Reproduce

1. Checkout the repo
2. Run gatsby build

Expected Result

Builds successfully.

Actual Result

Error: Generated engines use disallowed import "babel-runtime/helpers/asyncToGenerator". Only allowed imports are to Node.js builtin modules or engines internals.

Environment

System:
    OS: macOS 12.1
    CPU: (10) arm64 Apple M1 Pro
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 17.2.0 - /opt/homebrew/bin/node
    Yarn: 3.1.1 - /opt/homebrew/bin/yarn
    npm: 8.1.4 - /opt/homebrew/bin/npm
  Languages:
    Python: 3.7.6 - /Users/grek/opt/miniconda3/bin/python
  Browsers:
    Chrome: 96.0.4664.110
    Safari: 15.2
  npmPackages:
    gatsby: ^4.4.0 => 4.4.0
    gatsby-cli: ^4.4.0 => 4.4.0
    gatsby-link: ^4.4.0 => 4.4.0
    gatsby-plugin-catch-links: ^4.4.0 => 4.4.0
    gatsby-plugin-feed: ^4.4.0 => 4.4.0
    gatsby-plugin-flow: ^3.4.0 => 3.4.0
    gatsby-plugin-google-gtag: ^4.4.0 => 4.4.0
    gatsby-plugin-manifest: ^4.4.0 => 4.4.0
    gatsby-plugin-netlify: next => 4.0.0-next.0
    gatsby-plugin-netlify-cms: ^6.4.0 => 6.4.0
    gatsby-plugin-netlify-cms-paths: ^1.3.0 => 1.3.0
    gatsby-plugin-offline: ^5.4.0 => 5.4.0
    gatsby-plugin-optimize-svgs: ^1.0.5 => 1.0.5
    gatsby-plugin-react-helmet: ^5.4.0 => 5.4.0
    gatsby-plugin-sass: ^5.4.0 => 5.4.0
    gatsby-plugin-sharp: ^4.4.0 => 4.4.0
    gatsby-plugin-sitemap: ^5.4.0 => 5.4.0
    gatsby-remark-autolink-headers: ^5.4.0 => 5.4.0
    gatsby-remark-copy-linked-files: ^5.4.0 => 5.4.0
    gatsby-remark-external-links: 0.0.4 => 0.0.4
    gatsby-remark-images: ^6.4.0 => 6.4.0
    gatsby-remark-katex: ^6.4.0 => 6.4.0
    gatsby-remark-prismjs: ^6.4.0 => 6.4.0
    gatsby-remark-relative-images: ^2.0.2 => 2.0.2
    gatsby-remark-responsive-iframe: ^5.4.0 => 5.4.0
    gatsby-remark-smartypants: ^5.4.0 => 5.4.0
    gatsby-source-filesystem: ^4.4.0 => 4.4.0
    gatsby-transformer-remark: ^5.4.0 => 5.4.0
    gatsby-transformer-sharp: ^4.4.0 => 4.4.0

Config Flags

No response

[iChenLei]

Debug ScreenRecord

@gkossakowski I cloned your repo and debug this issue

Gatsby internal Load check

packages/gatsby/src/utils/validate-engines/child.ts

export async function validate(directory: string): Promise<void> {
  // intercept module loading and validate no unexpected imports are happening
  // @ts-ignore TS doesn't like accessing `_load`
  mod._load = (request: string, parent: mod, isMain: boolean): any => {

packages/gatsby/src/schema/graphql-engine/bundle-webpack.ts

    externals: [
      `cbor-x`, // optional dep of lmdb-store, but we are using `msgpack` (default) encoding, so we don't need it
      `babel-runtime/helpers/asyncToGenerator`, // undeclared dep of yurnalist (but used in code path we don't use)
      `electron`, // :shrug: `got` seems to have electron specific code path
      mod.builtinModules.reduce((acc, builtinModule) => {
        if (builtinModule === `fs`) {
          acc[builtinModule] = `global _actualFsWrapper`
        } else {
          acc[builtinModule] = `commonjs ${builtinModule}`
        }

        return acc
      }, {}),
    ],

Analysis

`babel-runtime/helpers/asyncToGenerator`, // undeclared dep of yurnalist (but used in code path we don't use)

Added above code via PR #33030 , to avoid bundle babel asyncToGenerator to final js bundle. But it break build when gatsbyjs user using gatsby-plugin which contain babel asyncToGenerator code. For example: gatsby-plugin-netlify-cms-paths

cc @vladar @wardpeet

[redabacha]

tldr: try removing gatsby-cli and gatsby-recipes from dependencies / devDependencies in your package.json

i've also encountered this issue in a similar way on gatsby v4.4.0. in my case, having gatsby-cli installed as a devDependency (would also break as a regular dependency i think) was the trigger to the following error when doing a normal build:

success Building production JavaScript and CSS bundles - 34.872s
⠇ Building Rendering Engines
<w> [webpack.cache.PackFileCacheStrategy] Serializing big strings (1574kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
<w> [webpack.cache.PackFileCacheStrategy] Serializing big strings (1574kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
success Building Rendering Engines - 34.603s
success Building HTML renderer - 22.735s
success Execute page configs - 1.325s
failed Validating Rendering Engines - 1.060s

 ERROR #98001  WEBPACK

Built Rendering Engines failed validation failed validation.

Please open an issue with a reproduction at https://github.com/gatsbyjs/gatsby/issues/new for more help



  Error: Module parse failed: Unexpected token (32:1)
  You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
  | export const createSchemaCustomization = ({
  |   actions: { createTypes },
  > }: CreateSchemaCustomizationArgs) => {
  |   createTypes(`
  |     type MagnoliaFlag implements Node @dontInfer {
  
  - index.js:199071 Object.<anonymous>
    /home/reda/Documents/Projects/hbi-content/.cache/query-engine/index.js:199071:7
  
  - index.js:690826 __webpack_require__
    /home/reda/Documents/Projects/hbi-content/.cache/query-engine/index.js:690826:42
  
  - index.js:189292 Module.<anonymous>
    /home/reda/Documents/Projects/hbi-content/.cache/query-engine/index.js:189292:101
  
  - index.js:690826 __webpack_require__
    /home/reda/Documents/Projects/hbi-content/.cache/query-engine/index.js:690826:42
  
  - index.js:690960 
    /home/reda/Documents/Projects/hbi-content/.cache/query-engine/index.js:690960:27
  
  - index.js:691138 
    /home/reda/Documents/Projects/hbi-content/.cache/query-engine/index.js:691138:3
  
  - index.js:691143 Object.<anonymous>
    /home/reda/Documents/Projects/hbi-content/.cache/query-engine/index.js:691143:12
  
  - loader:1101 Module._compile
    node:internal/modules/cjs/loader:1101:14
  
  - loader:1153 Object.Module._extensions..js
    node:internal/modules/cjs/loader:1153:10
  
  - loader:981 Module.load
    node:internal/modules/cjs/loader:981:32
  
  - loader:822 originalModuleLoad
    node:internal/modules/cjs/loader:822:12
  
  - child.ts:51 Function._module.default._load
    [hbi-content]/[gatsby]/src/utils/validate-engines/child.ts:51:16
  
  - loader:1005 Module.require
    node:internal/modules/cjs/loader:1005:19
  
  - helpers:102 require
    node:internal/modules/cjs/helpers:102:18
  
  - child.ts:70 Object.validate
    [hbi-content]/[gatsby]/src/utils/validate-engines/child.ts:70:29
  
  - child.js:65 process.messageHandler
    [hbi-content]/[gatsby-worker]/dist/child.js:65:32

the error suggests that the query-engine bundle does not like to see typescript code in a local plugin but i think that's a separate issue. the problem i have is that the project where this error happened is not using any ssg/dsg features so it should have never had needed to come to this step to build or validate engines anyway.

from what i can tell, the shouldGenerateEngines utility here is responsible for determining whether the engine bundles should be built + validated or not.

after a bit of debugging in vscode, i can see the generate variable that is used by that utility being mistakenly set to true by the component features emitter here where the redux action looks like this:

{
  "type": "SET_COMPONENT_FEATURES",
  "payload": {
    "componentPath": "/home/reda/Documents/Projects/hbi-content/node_modules/gatsby-recipes/src/providers/gatsby/page.js",
    "serverData": false,
    "config": true
  }
}

which is quite strange as we have no direct dependency on gatsby-recipes and it looks like that package has now been deprecated in gatsby v4.5.0 (unreleased at the time of this post). well it turns out that it's a dependency of gatsby-cli (at least in v4.4.0) which is likely why that it's appearing in the action up above.

$ yarn why gatsby-recipes
└─ gatsby-cli@npm:4.4.0
   └─ gatsby-recipes@npm:1.4.0 (via npm:^1.4.0)

and if you look at the source code of page.js in gatsby-recipes, you can see that it's clearly not intended to be a page component. so why has gatsby arrived to the conclusion that it is?

well i'm not entirely sure how it gets to this point but the file-parser has a small set of criteria of what might constitute a page component and the page.js file in gatsby-recipes satisfies all of it, namely it contains the text config and graphql. hence gatsby will parse the file and find that there is an export named config and dispatch the SET_COMPONENT_FEATURES action with config as true resulting in the unnecessary build step.

while i think this will be fixed in v4.5.0 as the gatsby-cli package no longer depends on gatsby-recipes, it is interesting to note how seemingly innocent files like this could trip up the build process. could it be worthwhile to make sure that the config and even getServerData exports are actually functions of some kind when traversing the ast here? i can make a pr for this validation if that's potentially useful.

my gatsby info for good measure:

  System:
    OS: Linux 5.15 Fedora Linux 35 (Workstation Edition)
    CPU: (32) x64 AMD EPYC-Milan Processor
    Shell: 5.1.8 - /bin/bash
  Binaries:
    Node: 16.13.0 - /tmp/xfs-7c3ddd76/node
    Yarn: 3.1.1 - /tmp/xfs-7c3ddd76/yarn
    npm: 8.1.0 - /usr/bin/npm
  Languages:
    Python: 3.10.1 - /usr/bin/python
  Browsers:
    Chrome: 96.0.4664.110
    Firefox: 95.0.2
  npmPackages:
    gatsby: ^4.4.0 => 4.4.0
    gatsby-background-image: ^1.6.0 => 1.6.0
    gatsby-cli: ^4.4.0 => 4.4.0
    gatsby-legacy-polyfills: ^2.4.0 => 2.4.0
    gatsby-plugin-feed: ^4.4.0 => 4.4.0
    gatsby-plugin-image: ^2.4.0 => 2.4.0
    gatsby-plugin-manifest: ^4.4.0 => 4.4.0
    gatsby-plugin-postcss: ^5.4.0 => 5.4.0
    gatsby-plugin-s3: ^0.3.8 => 0.3.8
    gatsby-plugin-sharp: ^4.4.0 => 4.4.0
    gatsby-plugin-styled-jsx: ^5.4.0 => 5.4.0
    gatsby-source-filesystem: ^4.4.0 => 4.4.0
    gatsby-transformer-sharp: ^4.4.0 => 4.4.0

i hope this information is helpful 🙏

[LekoArts]

Hey @redabacha - thanks for that deep dive and explanation. Really appreciate it!
We haven't seen this in our testing + many other people haven't seen this issue as generally speaking people don't add gatsby-cli as a dependency (as it's not necessary and our docs also don't teach this).

4.5.0 will go out on January 11 and we try to keep babel traversal/file parsing as light as possible (as it gets costly) so for now I'd like to keep the AST checking as-is.

OP, can you please try the suggestion?

[gkossakowski]

Thanks for speedy responses!

I've tried removing the dependency on gatsby-cli but I'm hitting what appears to be an issue with yarn (v3.1.1):

❯ gatsby build

 ERROR

There was a problem loading the local build command. Gatsby may not be installed in your site's "node_modules" directory. Perhaps you need to run "npm install"? You might need to delete your "package-lock.json"
as well.

I tried running yarn install to no avail. Then, I tried

❯ yarn gatsby build
success open and validate gatsby-configs, load plugins - 1.187s
success onPreInit - 0.007s
success delete worker cache from previous builds - 0.002s
success initialize cache - 0.063s
success copy gatsby files - 0.054s
success Compiling Gatsby Functions - 0.136s
success onPreBootstrap - 0.149s
success createSchemaCustomization - 0.014s
success Checking for changed pages - 0.001s
success source and transform nodes - 0.846s
info Writing GraphQL type definitions to /Users/grek/coding/gkossakowski.github.io/.cache/schema.gql
success building schema - 0.547s
success createPages - 0.252s
success createPagesStatefully - 0.218s
info Total nodes: 251, SitePage nodes: 43 (use --verbose for breakdown)
success Checking for changed pages - 0.000s
success Cleaning up stale page-data - 0.014s
success onPreExtractQueries - 0.001s
success extract queries from components - 0.647s
success write out redirect data - 0.011s
success Build manifest and related icons - 0.124s
success onPostBootstrap - 0.128s
info bootstrap finished - 5.959s
success write out requires - 0.015s
failed Building production JavaScript and CSS bundles - 13.217s

 ERROR #98123  WEBPACK

Generating JavaScript bundles failed

ObjectMiddleware.register: serializer for mini-css-extract-plugin/dist/CssModule/null is already registered

Is this a related issue or a separate one? Should I downgrade to the classic yarn?

[LekoArts]

I don't think I've seen this error recently, so if you can I'd appreciate it that you try it with yarn 1 or npm install --legacy-peer-deps? Thanks!

[gkossakowski]

I tried yarn v1, and got further, and I'm hitting some gatsby's incompatibilities now:

"gatsby-plugin-sitemap" threw an error while running the onPostBuild lifecycle:

Page array from `query` wasn't found at `data.allSitePage.nodes`.
Fix the custom query or provide a custom `resolvePages` function.
https://www.gatsbyjs.com/plugins/gatsby-plugin-sitemap/#api-reference


  59 |
  60 |             _context.next = 13;
> 61 |             return Promise.resolve(resolvePages(queryRecords)).catch(function (err) {
     |                                    ^
  62 |               return reporter.panic(_internals.REPORTER_PREFIX + " Error resolving Pages", err);
  63 |             });
  64 |

File: node_modules/gatsby-plugin-sitemap/gatsby-node.js:61:36



  Error: Page array from `query` wasn't found at `data.allSitePage.nodes`.
  Fix the custom query or provide a custom `resolvePages` function.
  https://www.gatsbyjs.com/plugins/gatsby-plugin-sitemap/#api-reference

This seems to be a rabbit hole deeper than I want to go down. My underlying issue is that I'm using a starter that's no longer maintained and I need to migrate to something else, but that's out of scope for this issue.

[joakimmillen]

I'm just going to drop this of here since i might be useful for someone debugging this.

We managed to get this error by simply installing [email protected], not actually using it anywhere but just adding it to the dependancy list and installing it.

[interglobalmedia]

Hi,
I am getting the same error. And I am using gatsby 4.4.0. I had the gatsby-cli and removed it. But that did not fix my issue. Maybe gatsby 4.5.0 release will help resolve this issue.

[interglobalmedia]

Hi, actually I saw that I did not successfully remove the yarn package (I'm used to npm!) the first time. I tried again, and now everything works as expected. And for anyone who might have issues with node-sass, don't upgrade for the moment above 6.0.0. It will throw errors as well. Thanks!

[LekoArts]

We've released 4.5 that removed gatsby-recipes from the CLI, so even if you install it in your project (which you won't need to do) this should be fixed.

[SebastianMera]

It is not fixed. You should re-open this.

[ivanteso]

I'm trying to use the Deferred Static Generation (DSG) on some templates, but as soon as I add defer: true on createPage, I get a very similar error. Node 16.13.0

"gatsby": "^4.9.1",
"gatsby-plugin-catch-links": "^4.9.0",
"gatsby-plugin-exclude": "^1.0.2",
"gatsby-plugin-feed": "^4.9.0",
"gatsby-plugin-manifest": "^4.9.0",
"gatsby-plugin-netlify": "^4.1.0",
"gatsby-plugin-polyfill-io": "^1.1.0",
"gatsby-plugin-preconnect": "^1.3.0",
"gatsby-plugin-react-helmet": "^5.9.0",
"gatsby-plugin-remove-fingerprints": "0.0.2",
"gatsby-plugin-remove-serviceworker": "^1.0.0",
"gatsby-plugin-sitemap": "^5.9.0",
"gatsby-plugin-styled-components": "^5.9.0",
"gatsby-source-filesystem": "^4.9.0",
"gatsby-source-graphql": "^4.9.0",
"gatsby-transformer-json": "^4.9.0",
"gatsby-transformer-remark": "^5.9.0",
"gatsby-plugin-force-trailing-slashes": "^1.0.6",
"gatsby-plugin-root-import": "^2.0.8",
"gatsby-plugin-sharp": "^4.9.0",
"gatsby-plugin-webpack-bundle-analyser-v2": "^1.1.25",
"babel-preset-gatsby": "^2.1.0",

Not exactly sure what's going on or if it's somehow related to the original issue. Happy to provide more context!

[OoDeLally]

@ivanteso Did you find a solution ? I have the same problem. Without any defer it builds flawlessly.
As soon as I add a defer, it fails with

failed Validating Rendering Engines - 0.805s

 ERROR #98001  WEBPACK

Built Rendering Engines failed validation failed validation.

Please open an issue with a reproduction at https://github.com/gatsbyjs/gatsby/issues/new for more help

  Error: Module parse failed: Unexpected token (6:25)
  You may need an appropriate loader to handle this file type, currently no loaders are configured to pr
  ocess this file. See https://webpack.js.org/concepts#loaders
  | import { COLORS } from "../constants"
  | 
  > export const defaultTheme: DefaultTheme = {
  |   fontFamily: '"Nunito Sans", sans-serif',
  |   headerBgUrl: "url(/bg.jpg)",

[sensedrive]

@OoDeLally I am experiencing the same.

[developer-bart]

I experience a similar issue only it failed on @vercel/webpack-asset-relocator-loader. For me turning off createSchemaCustomization in gatsby-node did the trick. Hope it helps.

This was my error: