Add .mbtiles support #549

nyurik · 2023-01-08T07:18:50Z

Merge after #548 and #517

Adds a new .mbtiles backend, without the grid support. Uses extensive tile content detection, i.e. if the content is gzipped, png, jpeg, gif, webp.

From CLI, can be as easy as adding a path to a directory that contains a .mbtiles file (works just like pmtiles support)

# All *.mbtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures

From configuration file, the path can be specified in a number of ways (same as pmtiles)

mbtiles:
  paths:
    # scan this whole dir, matching all *.mbtiles files
    - /dir-path
    # specific mbtiles file will be published as mbtiles2 source
    - /path/to/mbtiles2.mbtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/mbtiles.mbtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/mbtiles.mbtiles

Fixes #494

* All tests and internal code now uses ST_TileEnvelope function * Remove `tile_bbox` * Rename test function sources for clarity - this will be needed in a subsequent PR to add other function tests

Can handle the `getmvt(z,x,y) -> [bytea,md5]` case and several others.

* All table sources now use the same system as functions * Proper table name escaping * Prepared statements might speed things up, but this is not certain because we prepare on each get request. At least now we can optimize in one place.

BREAKING: * srid is now the same type as PG -- i32 * renamed config vals `table_sources` and `function_sources` into `tables` and `functions` Fixes: * predictable order of instantiation * bounding boxes computed in parallel for all tables (when not configured) * split up discovery and query creation - this way user overrides happen before the final query is generated * more proper name escaping * lots of test improvements

* Fix id column issue * Detect if postgis is newer than 3.1 and use st_tileenvolope margin param

* Use `PathBuf` instead of `String` where dealing with files * Parse keep_alive as u64 * More config tests to crash if martin output contains warnings or errors

* Since this is a library, all errors should have a strongly typed enum. * table bounds computing function was not escaping identifiers * table bounds computation was also silently ignoring all errors

codecov-commenter · 2023-01-08T07:30:00Z

Codecov Report

Base: 58.65% // Head: 57.75% // Decreases project coverage by -0.89% ⚠️

Coverage data is based on head (051c632) compared to base (ba65e34).
Patch coverage: 59.47% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #549      +/-   ##
==========================================
- Coverage   58.65%   57.75%   -0.90%     
==========================================
  Files          27       29       +2     
  Lines        2242     2507     +265     
==========================================
+ Hits         1315     1448     +133     
- Misses        927     1059     +132

Impacted Files	Coverage Δ
src/args/connections.rs	`77.50% <ø> (ø)`
src/lib.rs	`41.64% <ø> (-1.59%)`	⬇️
src/mbtiles/mod.rs	`0.00% <0.00%> (ø)`
src/pmtiles/mod.rs	`27.63% <0.00%> (-1.95%)`	⬇️
src/utils/utilities.rs	`50.00% <ø> (-9.68%)`	⬇️
src/config.rs	`58.53% <36.36%> (-1.74%)`	⬇️
martin-tile-utils/src/lib.rs	`73.68% <60.00%> (-0.74%)`	⬇️
src/srv/server.rs	`76.26% <66.66%> (-2.24%)`	⬇️
martin-mbtiles/src/lib.rs	`72.37% <72.37%> (ø)`
src/args/pg.rs	`84.05% <100.00%> (ø)`
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

maxammann · 2023-01-08T11:26:55Z

I'm looking right now for a mbtiles server so this comes just in time 😅

I'll test this out now, for a code review the PR is maybe too large. Maybe you can leave comments where you wish reviews.

nyurik · 2023-01-08T12:36:21Z

Thanks, let me know if you run into any issues. The reason this pr is so big is because it includes two other prs. If you review them in order, they will be much smaller. @stepankuzmin said he will try to get to them today

stepankuzmin · 2023-01-08T18:52:32Z

src/file_config.rs

+              - /dir-path
+              - /path/to/pmtiles2.pmtiles
+            sources:
+                pm-src1: /tmp/pmtiles.pmtiles


Probably we should also add the *.mbtiles test here?

there is no point because this test is not specific to pmtiles - it decodes a sub-element under pmtiles: or mbtiles: keys. I will change the filenames to make it clearer.

maxammann · 2023-01-08T21:02:50Z

Not sure where it comes from but I have an emptry string in the tiles array:

maxammann · 2023-01-08T21:12:28Z

mbtiles/src/lib.rs

+    pub async fn get_metadata(&self) -> MbtResult<Metadata> {
+        let mut res = Metadata {
+            tilejson: tilejson! {
+                tiles: vec![String::new()],


I think this adds an empty string in the tiles array.

Thanks, I'll need to fix this, and also handle the x-forwarded-for header (there is a ticket for that). I'll have time tonight to look at it, in a few hrs

That header works for me already. It places the correct host and protocol

which header? We currently handle x-rewrite-url, which is a really bad one from the security perspective. See #504

ewagstaff · 2023-02-17T17:18:35Z

@nyurik is there a good source to read more about what this change allows? Would this mean MapLibre can display data from a static .mbtiles file without the need to spin up a map server?

nyurik · 2023-02-17T19:30:52Z

@ewagstaff this change simply allows mbtiles file to be Martin's source of data, just like Postgres and .pmtiles files. Martin itself is a map server, but if your data is in a file format, Martin would not need any additional servers. If you want completely serverless, take a look at pmtiles.

nyurik added 30 commits December 6, 2022 21:41

Require PostGIS 3 and use ST_TileEnvelope

226014b

* All tests and internal code now uses ST_TileEnvelope function * Remove `tile_bbox` * Rename test function sources for clarity - this will be needed in a subsequent PR to add other function tests

move sql files to ensure the order

d3887e4

Support z,x,y and record-returning func

aa84b15

Can handle the `getmvt(z,x,y) -> [bytea,md5]` case and several others.

Massive update to tables

8410826

* All table sources now use the same system as functions * Proper table name escaping * Prepared statements might speed things up, but this is not certain because we prepare on each get request. At least now we can optimize in one place.

wip

8354ec1

Margin, fix id fields

9ab7e9e

* Fix id column issue * Detect if postgis is newer than 3.1 and use st_tileenvolope margin param

Use non-lowercase table name

8125242

Use non-lowercase table name

380d044

fix casing

6fa7779

minor cleanup

caef7bc

fix mixed case functions

12c42a2

remove Xyz::new(), makes it more complicated

ac02930

tmp

8ab9457

Allow auto discovery per schema

3a873e2

minor panic fix

918b97c

wip

e46e8bf

[WIP] implement pmtiles support

b7c2ff8

wip

dcb9ce2

Catch errors in tests, minor fixes

85f4841

* Use `PathBuf` instead of `String` where dealing with files * Parse keep_alive as u64 * More config tests to crash if martin output contains warnings or errors

Catch errors in tests, minor fixes

4fc0071

* Use `PathBuf` instead of `String` where dealing with files * Parse keep_alive as u64 * More config tests to crash if martin output contains warnings or errors

make sure there are no errors or warnings during test

130e0f5

Catch errors in tests, minor fixes

55fdebf

* Use `PathBuf` instead of `String` where dealing with files * Parse keep_alive as u64 * More config tests to crash if martin output contains warnings or errors

Catch errors in tests, minor fixes

577eaad

* Use `PathBuf` instead of `String` where dealing with files * Parse keep_alive as u64 * More config tests to crash if martin output contains warnings or errors

make sure there are no errors or warnings during test

a2d5a77

Merge remote-tracking branch 'origin/multi-pg' into multi-pg

9a5361e

merge

7d21034

Strongly-type errors, fix bounds escaping

c39262f

* Since this is a library, all errors should have a strongly typed enum. * table bounds computing function was not escaping identifiers * table bounds computation was also silently ignoring all errors

Merge branch 'errors' into multi-pg

9c003a3

Merge branch 'multi-pg' into pmtiles

d12c4be

nyurik added 6 commits January 8, 2023 01:34

Merge branch 'pmtiles' into mbtiles

3cecc1a

tests

35921c4

test tilejsons

ad22324

Merge branch 'refactoring' into pmtiles

fa7e90b

tilejson tests

b7e017d

Merge branch 'pmtiles' into mbtiles

f76fa66

nyurik requested a review from stepankuzmin January 8, 2023 07:18

bump pmtiles

bb3aa7f

nyurik added 4 commits January 8, 2023 09:34

Merge branch 'main' into refactoring

453da8c

Merge branch 'refactoring' into mbtiles

275c6ad

Merge branch 'refactoring' into pmtiles

71d1839

Merge branch 'pmtiles' into mbtiles

20d7001

stepankuzmin approved these changes Jan 8, 2023

View reviewed changes

maxammann reviewed Jan 8, 2023

View reviewed changes

nyurik added 6 commits January 8, 2023 19:04

Merge branch 'main' into pmtiles

3d670a7

Merge branch 'pmtiles' into mbtiles

a6c251e

rm unknown data type, minor rfctr

85c26d3

re-enable empty-srid test

e3d2c3b

fix tilejson in mbt, move to v3.0.0

30fa3e8

readme, cleanup

051c632

nyurik merged commit 9e5ed2f into maplibre:main Jan 9, 2023

nyurik deleted the mbtiles branch January 9, 2023 05:10

nyurik mentioned this pull request Feb 7, 2023

Update to tilejson 3.0 #465

Closed

brendan-ward mentioned this pull request Jun 6, 2023

mbtileserver vs Martin Comparison? consbio/mbtileserver#170

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add .mbtiles support #549

Add .mbtiles support #549

nyurik commented Jan 8, 2023

codecov-commenter commented Jan 8, 2023 •

edited

Loading

maxammann commented Jan 8, 2023

nyurik commented Jan 8, 2023

stepankuzmin Jan 8, 2023

nyurik Jan 9, 2023

maxammann commented Jan 8, 2023

maxammann Jan 8, 2023

nyurik Jan 8, 2023

maxammann Jan 9, 2023

nyurik Jan 9, 2023

ewagstaff commented Feb 17, 2023

nyurik commented Feb 17, 2023

Add .mbtiles support #549

Add .mbtiles support #549

Conversation

nyurik commented Jan 8, 2023

codecov-commenter commented Jan 8, 2023 • edited Loading

Codecov Report

maxammann commented Jan 8, 2023

nyurik commented Jan 8, 2023

stepankuzmin Jan 8, 2023

Choose a reason for hiding this comment

nyurik Jan 9, 2023

Choose a reason for hiding this comment

maxammann commented Jan 8, 2023

maxammann Jan 8, 2023

Choose a reason for hiding this comment

nyurik Jan 8, 2023

Choose a reason for hiding this comment

maxammann Jan 9, 2023

Choose a reason for hiding this comment

nyurik Jan 9, 2023

Choose a reason for hiding this comment

ewagstaff commented Feb 17, 2023

nyurik commented Feb 17, 2023

codecov-commenter commented Jan 8, 2023 •

edited

Loading