Memory leak?

[stuartlynn]

Running geojson2mvt on a large region tends to crash with the javascript heap size limit being reached. I think something is leaking memory in there. Plot shows heapsize vs tile number. You can see the Garbage collection working but its not grabbing everything.

Not sure if this is an issue in geojson2mvt or one of the required libraries but investigating.

[gijs]

This tool is exactly what I need! Too bad I'm having similar issues with enormous GeoJSON files.

Maybe we can use streaming instead of parse() and parse the geojson file line by line?

var geoJson = JSON.parse(fs.readFileSync(filePath, "utf8"));

[chriswhong]

Hi all, could either of you let us know the enormity of the data you're trying to bring in? Are we talking hundreds of thousands or millions of vertices? I would love to test as well and see if I can help.

[chriswhong]

@stuartlynn for my education, can you tell me how you produced that heap size chart? Is there a node package that does it, or did you pull the data somehow and chart it elsewhere?

[gijs]

The GeoJSON file I'm attempting to convert is 1.08 GB. It contains every farming parcel in the Netherlands. I guess we're in the millions of vertices then?

[chriswhong]

I feel like any workflow dealing with files that big will be riddled with problems. I think the tools that can generate tiles from a database might be a better option for a dataset that large. https://github.com/mapbox/awesome-vector-tiles

[stuartlynn]

Hey @chriswhong it's super easy

process.memoryUsage().heapUsed

super useful. Trying to get heap dumps to see what's leaking.

I don't think its the size of the data in memory, for example, my file is just 77mb. It's the number of tiles you want that's the problem I think.

Every time through the inner loop that generates the tiles, something is allocating memory that's getting lost and not deallocated properly. It might be in one of the deps rather than your code. Trying to track it down.

[stuartlynn]

Ok looks like it might be all the fs.writeSync calls. Looking at the heap the system objects account for the huge size.

Going to try and rewrite it a little to use async write and potentially a pool of workers? See if that helps

[stuartlynn]

also huge thank you to @rochoa for showing me how to debug node better. If you use

node --debug --inspect convert.js

It will give you a URL in chrome you can use with the standard chrome tools to debug

[chriswhong]

@stuartlynn Thanks for the tips, and the worker pool sounds like a good idea, I look forward to seeing how you go about it.

[SKalt]

Looks like geojson-vt's internal tile generation is caching all generated tiles (see geojson-vt#L82). I'm currently rolling my own implementation to recursively save lower-zoom tiles, then delete them from the index along with the parent tile.

[gijs]

FYI We've moved to using the ST_AsMVT function in PostGIS to generate our vector tile pyramids.

[stuartlynn]

That's what I ended up doing as well. Would still be good to have a version of geojson2mvt that can handle large files. @SKalt happy to help with that implementation if you need it.

[SKalt]

I hadn't heard about the ST_AsMVT function, that's extremely useful. @stuartlynn Thanks! The work is at skalt/store-mvt currently. I'm using the issues tab for my to-do list until the project is ready to be published.