not all nodes available in sourceNodes within gatsby-node.js

[chrisdrackett]

Description

I'm getting markdown files from my file system and creating nodes. I'd like to do work on these nodes after they are all created and read that this is possible via sourceNodes within gatsby-node.js. The problem I'm running into is one set of my markdown files don't seem to be available at this step. I do see them in graphiql, but if I run a log during sourceNodes some of my nodes are not there.

The docs say:

If you define this hook in gatsby-node.js it will be called exactly once after all of your source plugins have finished creating nodes.

Steps to reproduce

• add multiple gatsby-source-filesystem plugins
• add gatsby-transformer-remark
• add the following to gatsby-node.js:

exports.sourceNodes = ({ actions, getNodes, getNode }) => {
  const { createNodeField } = actions

  console.log(getNodes().filter((node) => node.internal.type === `File`).length)
  console.log(
    getNodes().filter((node) => node.internal.type === `MarkdownRemark`).length
  )
}

Expected result

The count for File nodes should match the count for MarkdownRemark nodes.

Actual result

The File node count is higher than MarkdownRemark

Environment

  System:
    OS: macOS 10.14.2
    CPU: (12) x64 Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 11.3.0 - /usr/local/bin/node
    Yarn: 1.12.3 - /usr/local/bin/yarn
    npm: 6.4.1 - /usr/local/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 71.0.3578.98
    Safari: 12.0.2
  npmPackages:
    gatsby: ^2.0.80 => 2.0.82
    gatsby-image: ^2.0.20 => 2.0.25
    gatsby-plugin-manifest: ^2.0.13 => 2.0.13
    gatsby-plugin-offline: ^2.0.16 => 2.0.20
    gatsby-plugin-react-helmet: ^3.0.2 => 3.0.5
    gatsby-plugin-sharp: ^2.0.17 => 2.0.17
    gatsby-source-filesystem: ^2.0.8 => 2.0.12
    gatsby-transformer-remark: ^2.1.17 => 2.1.17
    gatsby-transformer-sharp: ^2.1.10 => 2.1.10
  npmGlobalPackages:
    gatsby-cli: 2.4.8

[KyleAMathews]

Transformer nodes won't be done necessarily by the time your code is called. You should use onCreateNode to extend nodes.

[chrisdrackett]

the issue is I'm trying to map nodes to each other, but I assume its possible that during onCreateNode the related nodes I'm looking for might not have been created yet?

I'm working off the example here: #3129 (comment)

[abohannon]

I'm also running into this issue. Trying to map two markdown files together, but it's a toss up whether Gatsby will 'see' both. Most often Gatsby only loads one of the nodes and I get null/undefined errors. And unless I'm misunderstanding the docs, sourceNodes() when defined in gatsby-node.js should only fire once all nodes are created, so there shouldn't be any race conditions, right? https://www.gatsbyjs.org/docs/node-apis/#sourceNodes

Would love to see a more straightforward API that allows mapping markdown files!

[0x6e6562]

To get a deterministic order, can you partition your gatsby-source-filesystem configuration to ensure pre-requisite nodes exist before dependent nodes?

I have a scenario in onCreateNode where I need all image files to have been previously processed into nodes so that they can be referenced.

To ensure ordering, I partitioned the configuration of the source plugin like this:

plugins: [
  'gatsby-plugin-react-helmet',
  {
    resolve: `gatsby-source-filesystem`,
    options: {
      name: 'img',
      path: `${__dirname}/src/pages/img/`
    }
  },
  {
    resolve: `gatsby-source-filesystem`,
    options: {
      path: `${__dirname}/src/pages`,
      name: 'pages',
      ignore: ['**/img/**'],
    },
  },
]

This will tag each node with the name attribute - in your onCreateNode callback this tag is available via the sourceInstanceName field on each node.

It seems like nodes get fed through to the onCreateNode callback in partitioned blocks.

I'm not sure whether the gatsby-source-filesystem plugin guarantees this implicit ordering behavior though - it would be nice to know this.

[0x6e6562]

As an aside, I've been trying to achieve a two-pass process so that I can use intermediate node query results to enrich other nodes - see #11092

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[chrisdrackett]

Is there no official way to bind data two ways? :'(

[gatsbot]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

Thanks again for being part of the Gatsby community!