JavaScript heap out of memory

[TemporalAgent7]

Description

I believe gatsby develop may have a memory leak. I'm seeing the same thing on different machines (16Gb and 32Gb RAM); the graphql queries step runs until the node process reaches roughly ~1.5GB memory usage then crashes with a message like:

success open and validate gatsby-configs — 0.005 s
success load plugins — 0.131 s
success onPreInit — 0.283 s
success initialize cache — 0.026 s
success copy gatsby files — 0.071 s
success onPreBootstrap — 0.009 s
success source and transform nodes — 5.291 s
success building schema — 0.623 s
success createPages — 0.608 s
success createPagesStatefully — 0.025 s
success onPreExtractQueries — 0.000 s
success update schema — 0.478 s
success extract queries from components — 0.127 s
⠄ run graphql queries — 310/728 2.51 queries/second
<--- Last few GCs --->

[10208:0000029380BC1810]   139036 ms: Scavenge 1374.2 (1423.1) -> 1373.3 (1424.1) MB, 2.2 / 0.0 ms  (average mu = 0.317, current mu = 0.262) allocation failure
[10208:0000029380BC1810]   139039 ms: Scavenge 1374.3 (1424.1) -> 1373.5 (1424.6) MB, 2.0 / 0.0 ms  (average mu = 0.317, current mu = 0.262) allocation failure
[10208:0000029380BC1810]   139043 ms: Scavenge 1374.4 (1424.6) -> 1373.6 (1425.1) MB, 2.1 / 0.0 ms  (average mu = 0.317, current mu = 0.262) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0000003945650361]
Security context: 0x02be3421d8c1 <JSObject>
    1: getArgumentValues(aka getArgumentValues) [000000DDB6E15DD9] [D:\work\datacore\node_modules\graphql\execution\values.js:~140] [pc=0000003945D68053](this=0x01fdf16025b1 <undefined>,0x00a7f208e0c9 <Object map = 00000012DDA4A641>,0x0291110fc0a1 <Object map = 0000003241B432F1>,0x0049891fc611 <Object map = 000001DC64B665C1>)
    2: completeObjectValue(aka...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF788862A1A v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+4858
 2: 00007FF788807E16 uv_loop_fork+80918
 3: 00007FF7888089F0 uv_loop_fork+83952
 4: 00007FF788C8AC1E v8::internal::FatalProcessOutOfMemory+798
 5: 00007FF788C8AB57 v8::internal::FatalProcessOutOfMemory+599
 6: 00007FF788F63D74 v8::internal::Heap::RootIsImmortalImmovable+14788
 7: 00007FF788F599B4 v8::internal::Heap::CollectGarbage+7556
 8: 00007FF788F58088 v8::internal::Heap::CollectGarbage+1112
 9: 00007FF788F619F7 v8::internal::Heap::RootIsImmortalImmovable+5703
10: 00007FF788F61A76 v8::internal::Heap::RootIsImmortalImmovable+5830
11: 00007FF788B22531 v8::internal::Factory::NewFillerObject+49
12: 00007FF78927D28A v8::internal::StoreBuffer::StoreBufferOverflow+26826
13: 0000003945650361

gatsby build does succeed; the node process maintains a constant memory usage (of around 320Mb) and constantly pegs a single core of the CPU while running 727 GraphQL queries for 10+ minutes, but it eventually completes successfully.

I suspect I'm doing something silly with GraphQL mappings (I only just started using Gatsby and GraphQL a few days ago) - I'm trying to join 3 json sources. If there's an "optimizer" / "analyzer" for GraphQL that could point me to my mistakes, please let me know.

I'm also looking into options for increasing Node's V8 memory allocation in case that contributes to the problem.

Steps to reproduce

git clone https://github.com/TemporalAgent7/datacore
cd datacore
npm install
gatsby develop

Expected result

No crashes.

Actual result

V8 out of memory.

Environment

  System:
    OS: Windows 10
    CPU: (12) x64 Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
  Binaries:
    Yarn: 1.12.3 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 6.4.0 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: 44.17763.1.0
  npmPackages:
    gatsby: ^2.1.31 => 2.1.32
    gatsby-plugin-typescript: ^2.0.11 => 2.0.11
    gatsby-source-filesystem: ^2.0.24 => 2.0.25
    gatsby-transformer-json: ^2.1.10 => 2.1.10
    gatsby-transformer-remark: ^2.3.2 => 2.3.3

[pieh]

Hey @TemporalAgent7
In develop we do hold query results in memory. It seems like page query results (at least for "crew" paths) are very large -
. So those piles up for that develop vs build memory usage difference.

So while this out of memory should definitely be fixed, I think You should look into your queries, because pulling >700KB of data for each site seems pretty heavy?

[TemporalAgent7]

I trimmed the queries down to only the needed fields, and the situation didn't really improve. Now the build sometimes fails on Netlify (due to their 15 minute timeout); the most expensive step is still the run graphql queries (709.850 s — 726/726 1.02 queries/second).

I have a many-to-many relationship between 2 large JSON lists, and that currently results in duplication across the pages. Do you have any recommendations about how I could be optimizing that with GraphQL and mappings?

Or am I better off dumping the JSON files in output, and loading them dynamically?

Thank you!

[TemporalAgent7]

FWIW, I ended up uploading and fetching the other JSON at runtime, which reduced the query step duration 100x (from 12 minutes to 7 seconds); it looks like mapping joins (in particular self-referencing joins) are terrible performance-wise.

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[vinutha93bnvs]

Hey,

i have site with quite large data to be queried from contentful.
Need some quick fix.
Please help

[DSchau]

@vinutha93bnvs could you please open a new issue with more detail and a reproduction if you'd like additional help? Thank you!

As far as this issue--we believe it to be mostly closed with some changes related to stringifying data that @pieh implemented in #10732 which seems to scale Gatsby much more effectively.

As such--closing this out. Thanks for all the help/info everyone!