Using PMTiles for tile diffs

[nyurik]

This is mostly to document what we discussed at FOSS4G 2022, not necessarily if this is the right approach, or if pmtiles should implement this at all.

Summary

The goal is to able to create compact diffs between different revisions of tilesets. The tileset itself could be .pmtiles or .mbtiles files, or some other format. Ideally it should be possible to apply multiple diffs at once.

Diff Storage structure

PMTiles could be used for the diff storage, with each data blob (tile) representing the new tile content. Skipped tile IDs are unchanged.

Advanced diffs

Once the simple diffing is implemented, we may want to have a per layer diffing, so that if all layers but one is the same, the diff would contain just that one layer, and the full tile will be reconstructed when the diff is applied.

Tileset data structures

Using PMTiles as a diff shipping format does not mean the actual stored data (tileset) will have to be pmtiles. When merging a tileset with the diff(s), if tileset is mbtiles, sqlite will make it relatively efficient. If the tileset is pmtiles, a new copy would have to be created (requiring extra 100+GB of temp storage), but the merging algorithm might possibly be simpler and faster (unconfirmed).

[bdon]

Thanks, my intention is to keep this out of the scope of V3 but it is an interesting application - we should leave this issue open for now

thoughts are:

1. how is this need addressed by existing tooling (MBtiles)?
2. is it achievable within PMTiles by, for example, using ghost sections of the archive to store a offset>hash mapping?

[bdon]

See #228 (comment) for discussion on RSync binary diff strategy for pmtiles