Extracting all raster tiles from a PMTiles file

[diehl]

I'm looking for a way to extract all the raster tiles in the terrarium PMTiles archive to a hierarchical directory structure for processing. I have yet to find a way to accomplish this via the command line. Any suggestions would be greatly appreciated!

[wipfli]

Maybe try tile-join with the --output-to-directory command option. https://github.com/felt/tippecanoe?tab=readme-ov-file#tile-join

[diehl]

Looks like that only works for vector tiles.

[bdon]

The older python CLI has a command to do this: https://github.com/protomaps/PMTiles/blob/main/python/bin/pmtiles-convert#L49

For the use case of processing every tile, I do not believe exporting to a filesystem directory is the right solution because for large tilesets the # of files/inodes will be a bottleneck. Maybe there is some way to use the python or go libraries to do your operation in memory on an input archive and write out an output pmtiles.

[diehl]

@bdon To add some additional color, I've been thinking about using Coiled's infrastructure for parallel processing of the tiles in the cloud. Having the tiles as a directory of files makes it easy to parallelize the computation using their infrastructure. I'm realizing as I write this that for this particular example of processing the Mapzen terrain tiles, maybe the best bet is to process the tiles directly from S3, since the hierarchical directory structure I seek is already there. That still leaves open the question of how I turn that transformed dataset back into a PMTiles file.

Stepping back for a moment to consider a more general solution... if we have a PMTiles file containing raster tiles and we want to incrementally process each tile and write out a new PMTiles file with the transformed rasters, is there an obvious way to perform this general operation? I hadn't considered asking until now whether there's an efficient way to incrementally add data to the output PMTiles file, thereby keeping the storage footprint of the pipeline well bounded.

[bdon]

Having the tiles all on S3 is one way but you're paying a latency overhead requesting them each at once. If you had to write a custom program for parallel processing you could do something like:

1. use deserialize_directory to get all tile IDs and offsets in memory
2. split the # of tiles into N equal sized parts
3. Request a stream of tile contents from each worker process/machine, since if clustered=true they are contiguous, which will let you batch many tiles into a single network I/O operation.
4. Write each stream of tiles out to a separate S3 sink file along with serialized information about their IDs and offsets (using pickle, etc)

You would then need a single process to merge N output files into a single archive and write a final directory.

I don't know of any existing obvious ways to do this in parallel.

[diehl]

Thanks @bdon - much appreciated!