Will latests fixs will be embeded in a release ?

[emaulandi]

Thank you for this great plugin !

I was wondering if there were any plans of releasing a new version of the npm package that will embed the latest fix that will persist unchanged nodes between two runs (if I understood it correctly) ?

[alvis]

Hi @emaulandi can you elaborate more about the problem you’re facing?

I’m planning to work on it again to support the recently and finally released image/file api. So I’m keen to fix any issues there.

[emaulandi]

My notion database have 48 pages/nodes that I want to access the data to display content and so on.

• After a gatsby clean all 48 nodes are detected as newEntities and 48 nodes are processed / created, I have all the data I need.
• When I rerun the build (either locally with gatsby develop or when deploying the site with Netlify with cache enabled), there is no newEntites detected (normal) but only 10 updatedEntities (maybe those with the lastest edited date, I am not sure because I did not updated them meanwhile) and all other nodes that are not detected as updated are not processed. In that case I miss a bunch of data.

I think this is covered by the latest fix you commited on the repo ?

Let me know if the explanation is not clear or if you need further elements.

[alvis]

@emaulandi Thanks. It's very clear. I just release 1.0.2 together with a minor fix.
Check it out and let us know if it resolves your issue.

[emaulandi]

Thanks a lot for your reactivity ! Just updated the package.

It seems that I run into the same problem where nodes that are still there is the cache but have no changes are not processed.
There are present in the oldMap and newMap (node.js) but are not detected neither as new entities or as updated entities (oldMap.get(id) is true and oldEntity.digest !== newEntity.digest is false) and thus this.createNode is never called on them (if I understood node.js correctly, which might not be the case :) )

[alvis]

Let me check with my example repo. I encountered the exact same problem as you and therefore the fix. Let me get back to you by today. ;)

[alvis]

@emaulandi It turns out I got the usage of touchNode wrong. I thought nodes only need to be touched when they were first created, but it isn't. All nodes have to be touched at every run. I have released a newer version to address this issue. Please try and let me know :)

[emaulandi]

It works like a charm ! Thank you so much for this really fast update and help !

[emaulandi]

Sorry, I might not have checked this fully. After few redeploys, I found the same problem. From the plugin infos during build I can see that I have X nodes added and Y nodes processed (touchNode was apply on those Y nodes, Y being the total numbers of nodes). But then still only the X nodes appear in the graphql query allNotionPage { nodes { }}.

From your example repo, does all processed nodes (touched nodes) which where not added are available for query and use ?

[alvis]

It's an interesting found. I managed to reproduce your issue. If you don't have any pages on notion got updated, it works fine.
However, if you get just a page updated and you're building with cache, some NotionPage nodes could disappear even they've been touched.

It should be caused by my understanding about touchNode. I tried to call it before or after createNode, but no luck. Some NotionPage nodes still get missing when the source on Notion get updated. Weird.

Get look into it more tomorrow.

[alvis]

@emaulandi It turns out that the bug is introduced unintentionally by including child nodes' content in the hash of a database node. Now I believe it's fixed. You can try the fix via the following and let me know if I can merge and release a newer version. :)

npm install alvis/gatsby-source-notion#pull/2/head

[emaulandi]

Thanks again for the reactivity !
I tested locally and running 2 consecutive deploys on Netlify and I think it works fine !
With no cache, all nodes are created and the following runs I can see [gatsby-source-notion] keeping [X being all] nodes, I have all the nodes in graphql and all nodes are used to build pages an so on. 🙏🙏🙏

[alvis]

@emaulandi Could you try updating some pages or adding new ones?
In theory, it should show keeping [x for the number of unchanged nodes] nodes together with added/updated [y] nodes?

[emaulandi]

So I tried running multiple times gatsby develop locally with the following scenario :

Actions : Empty cache
Run 1 / Status OK
info [gatsby-source-notion] added 50 nodes
info [gatsby-source-notion] keeping 0 nodes

---

Run 2 / Status OK
info [gatsby-source-notion] keeping 50 nodes

---

Actions : Update 1 node in notion
Run 3 / Status OK
info [gatsby-source-notion] updated 1 nodes
info [gatsby-source-notion] keeping 49 nodes

---

Actions :

• Update 1 other node in notion
• Add 1 node in notion

Run 3 / Status KO (only two nodes available : the new and updated one)
info [gatsby-source-notion] added 1 nodes
info [gatsby-source-notion] updated 2 nodes
info [gatsby-source-notion] keeping 48 nodes

---

Run 4 / Status KO (only two nodes available : the new and updated one)
info [gatsby-source-notion] keeping 51 nodes

---

Actions : Delete 2 nodes
Run 5 / Status KO There was an error in your GraphQL query: Cannot query field "allNotionPage" on type "Query".
info [gatsby-source-notion] updated 1 nodes
info [gatsby-source-notion] removed 2 nodes
info [gatsby-source-notion] keeping 48 nodes

---

Actions : gatsby clean
Run 6 / Status OK
info [gatsby-source-notion] added 49 nodes
info [gatsby-source-notion] keeping 0 nodes

[alvis]

Hi @emaulandi thanks so much for helping to test.

Sorry. My silly head. I forgot adding a new page would trigger an update to the database node too.
It should be fixed with the latest verion.

npm install alvis/gatsby-source-notion#pull/2/head

[emaulandi]

No worries ! Thank you for you help.

It seems that adding a node is still problematic :

• Run 1 OK : added 49 nodes, keeping 0 nodes
• Run 2 OK : keeping 49 nodes
• update 1 node > Run 3 OK : updated 1 nodes, keeping 48 nodes
• add 1 node > Run 4 KO only the added node is present in graphql : added 1 nodes, updated 1 nodes, keeping 48 nodes (Even when not updating a node in the the previous step, both added and updated are triggered)
• delete 1 node : Run 5 KO Cannot query field "childrenNotionPage" on type "NotionDatabase", Cannot query field "childrenNotionPage" on type "NotionDatabase"

I also tried cleaning the cache after adding 2 dummy nodes, running a first time and then deleting the 2 nodes. In that case, no problem.

[alvis]

Thanks for testing. Just wondering if you've deleted node_modules and package-lock.json and reinstalled everything again before running the test? The info seems to be in the old format.

[emaulandi]

Yes, I did it again but I have the same result.

Regarding infos, I think you've deleted "processed" to be replace by "keeping" ?
I can see the computeNodeGraph for example in in nodes_modules/gatsby-source-notion that was added in the last update I think ?

[alvis]

It turns out the root of issue is deeper than I thought. The problem is that whenever a database node need to get updated either by changing its title or adding a page to it, all its children will get deleted. Neither touchNode nor createParentChildLink has any effect at all.

In the official doc, it says

Because all children nodes are derived from their parent, when a parent node is deleted or changed, Gatsby deletes all of the child nodes (and their child nodes, and so on) with the expectation that they’ll be recreated again by transformer plugins. This is done to ensure there are no nodes left over that were derived from older versions of data but shouldn’t exist any longer.

It seems to be long standing problem of Gatsby, see

• [gatsby-plugin-mdx] Plugins can't create nodes properly · Issue #21840
• How to recreate/update gatsby source nodes · Issue #22611

I'm not expecting any change from Gatsby to solve the issue, so I wrote an update to recreate any missing nodes automatically instead. The latest update can be installed via the following.

npm install alvis/gatsby-source-notion#c97d301

[emaulandi]

Thank you for the detailed explanations.
With this update, when adding / deleting a node, all unchanged nodes are preserved.

[alvis]

@emaulandi The fix is now released as 1.0.4. Thanks for helping again :)