Question: Any tips for speeding up building/update schema steps?

[benedfit]

Is there any way to help Gatsby know the expected schema for the nodes I'm creating so that it doesn't need to infer them?

I currently have quite complex node structures and it's resulting in both these steps taking about 30 seconds each to complete?

[pieh]

Currently no way to do this, but I have started doing some work on assisting schema creation. Originally it was focused on enforcing schema types but this could be extended to take over schema creation completely for sure. If you are interested - check master...pieh:schema_wip (just note that this is big undertaking and I didn't have time recently to work on this, so nothing was done in last month).

I am actually interested in your complex example if you can share it - I try do some performance comparison between current code and changes I propose. So far I was using https://github.com/freeCodeCamp/guides which has 2764 markdown files but they all are similar and doesn't seem to affect total build time that much (schema creation take 1s-1.5s when I build locally)

[benedfit]

Sure, here is the site in question: https://github.com/avaclancheuk/destiny-clan-warfare. Just a heads up if you run it, it is calling out to a lot of API endpoints for data, so the source and transform nodes step is currently taking a very long time. I'm working on getting a smaller subset of data for development purposes, but currently, it will take about 10-15 mins to finish a build and create the 1000s of pages. I'm currently on a big optimisation kick as I was blowing up Netlify deployments by having so many pages change between deployments

[KyleAMathews]

Sounds like something pathological in your data. I've never seen schema creation take longer than ~1.5seconds.

[KyleAMathews]

BTW, one quick way of speeding up sourcing is to parallelize fetches in your sourceNodes https://github.com/avaclancheuk/destiny-clan-warfare/blob/beta/gatsby-node.js

E.g. await Promise.all([fetch(...), fetch(...)])

[benedfit]

@KyleAMathews Thanks for the parallelization tip, I'll give that a shot. Could complex nested child arrays with lots of data be a potential issue for my schema creation?

[KyleAMathews]

Could complex nested child arrays with lots of data be a potential issue for my schema creation?

Potentially. It's hard to say without profiling your exact data. There's other sites with lots of nested data that doesn't have problems.

[benedfit]

ok, thanks. I'll start digging and see if I can find the bottlenecks

[benedfit]

I have managed to speed up sourceNodes no end by parallelizing my fetches. I now need to dig into the node creation further to find the bottle necks

[benedfit]

So createNodes is my next adventure. For one particular type of node, I'm creating 12500+ nodes. This appears to be what's slowing things down so much, and is what's affecting the schema building/updating

[benedfit]

It would appear that the sheer volume of nodes is what's causing the bottlenecks. I'm not investigating alternate storage methods that bypass graphql