404 with some (big?) pmtiles

[eddy-geek]

TL;DR I can't get tiles out of go-pmtiles v0.5 for some files which work well through pmtiles-js.

I suspect it is linked with the file having leaf directories.

File	Size	JS - test URL	GO - test URL
Bugianen_Cervino	0.2 GB	🆗	🆗 wtracks C-go
Bugianen	3.5 GB (sorry!)	🆗 wtracks B-js	⛔ wtracks B-go
eslo14_walps	1.5 G	🆗	⛔

Context

I am integrating some pmtiles maps in https://opoto.github.io/wtracks/

After merging pmtiles support (opoto/wtracks#21), I have working, but excruciatingly slow pmtiles-js-on-HTTP1 maps (that we discussed in protomaps/PMTiles#16). For example IT Bugianen (find it in More... menu or though the custom link above)

Now I'm trying to evaluate performance on go-pmtiles on the same map, but it does not work (metadata works 404 on tiles I tried).
See the links to download/test the files above. I including a valid file to show that the issue is not how my server setup on montagne.top.

To test on wtracks, search for a covered area for both maps eg "Aosta".
Or test with curl:

curl -i https://www.montagne.top/maki/Bugianen_Cervino/16/34100/23382.jpg
HTTP/1.1 200 OK
curl -i https://www.montagne.top/maki/Bugianen/16/34100/23382.jpg
HTTP/1.1 404 Not Found
curl -I https://www.montagne.top/tile/eslo14_walps/16/34100/23382.png
HTTP/1.1 404 Not Found

I can try to debug it myself at some point, but figured maybe it's a known issue with big files?

[eddy-geek]

reproduced with master

go build
./go-pmtiles serve ./
# working
curl -I http://localhost:8080/Bugianen_Cervino/16/34100/23382.jpg
HTTP/1.1 200 OK
# broken
curl http://localhost:8080/Bugianen/metadata ; echo
{"bounds": "6.592,44.024,8.789,46.498", "name": "Bugianen", "type": "baselayer", "version": "1.1", "description": "https://tartamillo.wordpress.com/bugianen/", "format": "jpg"}
curl -I http://localhost:8080/Bugianen/16/34100/23382.jpg
HTTP/1.1 404 Not Found

Adding some logs in reader.go ParseDirectory:

logger.Printf("Got %d entries, %d leaves, LeafZ=%d.", len(the_dir.Entries), len(the_dir.Leaves), maxz)

working:

2021/12/18 04:56:56 reader.go:78: Got 8525 entries, 0 leaves
2021/12/18 04:56:56 loop.go:143: fetched Bugianen_Cervino 0-512000
2021/12/18 04:56:56 loop.go:143: fetched Bugianen_Cervino 62692493-33382
2021/12/18 04:56:56 loop.go:304: served Bugianen_Cervino/16/34100/23382 in 3.220239m

broken:

2021/12/18 04:56:36 reader.go:78: Got 0 entries, 3 leaves, LeafZ=7.
2021/12/18 05:36:16 reader.go:85: Leaf {Z:7 X:66 Y:45}
2021/12/18 05:36:16 reader.go:85: Leaf {Z:7 X:66 Y:46}
2021/12/18 05:36:16 reader.go:85: Leaf {Z:7 X:67 Y:45}
2021/12/18 04:56:36 loop.go:143: fetched Bugianen 0-512000

[eddy-geek]

working

pmtiles-show Bugianen_Cervino.pmtiles
root entries: 8525

broken

pmtiles-show Bugianen.pmtiles
root entries: 3

So "broken" has 3 leaf directories.

But since it has 0 entries we fal into

        if len(root_value.directory.Entries) == 0 {
		return 404, headers, nil
	}

So, I don't think I see leaf-directory support in go-pmtiles?

If I remove this block of code, it works 🏄

[bdon]

Do you have this working successfully with the python pmtiles-convert? that implementation supports writing leaf directories, while the go one doesn't yet.

[bdon]

nvm, I misread that - the go-pmtiles writer library doesn't support leaf directories yet, but the reader should. let me look more into it

[eddy-geek]

start with my very last comment -- it works, modulo the regression introduced by 00d3f47

I guess it's also unusual that my files end up with 0 entries in root dir, because they don't have low (<7) zoom levels.

[bdon]

OK, I understand it now. You're right that the go reader code should be permissive of the root directory having no actual tile entries and only leaf directories.

However, it does seem non-optimal to organize things that way - did you create the PMTiles archive via the python converter? can you share your original MBTiles or other source to see if the algorithm can be adapted to produce a more optimized directory layout?

[eddy-geek]

did you create the PMTiles archive via the python converter?

yes

can you share your original MBTiles

here

[eddy-geek]

fix deployed on the server, so there's real-life testing possible with the links above now, and it's indeed quite faster than pmtiles-js thx to parallelisation & caching :-)

[eddy-geek]

about the not optimized code - adressed in #23

this is the code in pmtiles/writer.py that you think is "not optimized" (not such a big issue IMO): you're hardcoding a "base_zoom = 7", which is only only optimal for a whole world map.
instead "7" could be replaced with min zoom that fits all tiles

by the way, funnny you did a fix for the corresponding issue in pmtile-convert: protomaps/PMTiles@f439b3b "handle case where there are not root tiles, only leaf directories"

[bdon]

The go-pmtiles tool now writes the new and improved spec v3 format, which does not have any concept of a fixed leaf zoom.

You can migrate v2 archives using pmtiles convert old.pmtiles new.pmtiles.

There is still an outstanding issue of how to best organize the root directory to use the fixed root request size of 16,384 bytes efficiently in #16.