Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: improve diff cmd with copy opts #1183

Merged
merged 1 commit into from
Feb 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 4 additions & 3 deletions docs/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@
- [Recipes](recipes.md)
- [Tools](tools.md)
- [martin-cp bulk tile generation](martin-cp.md)
- [MBTiles Info and Metadata](mbtiles-meta.md)
- [MBTiles Copying / Diffing](mbtiles-copy.md)
- [MBTiles Validation](mbtiles-validation.md)
- [MBTiles Metadata](mbtiles-meta.md)
- [MBTiles Schemas](mbtiles-schema.md)
- [Copying MBTiles](mbtiles-copy.md)
- [Diffing/Patching MBTiles](mbtiles-diff.md)
- [Validating MBTiles](mbtiles-validation.md)
- [Development](development.md)
35 changes: 6 additions & 29 deletions docs/src/mbtiles-copy.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,13 @@ mbtiles copy normalized.mbtiles dst.mbtiles \

## `mbtiles copy --diff-with-file`

Copy command can also be used to compare two mbtiles files and generate a delta (diff) file. The diff file can be applied to the `src_file.mbtiles` elsewhere, to avoid copying/transmitting the entire modified dataset. The delta file will contain all tiles that are different between the two files (modifications, insertions, and deletions as `NULL` values), for both the tile and metadata tables.

There is one exception: `agg_tiles_hash` metadata value will be renamed to `agg_tiles_hash_in_diff`, and a new `agg_tiles_hash` will be generated for the diff file itself. This is done to avoid confusion when applying the diff file to the original file, as the `agg_tiles_hash` value will be different after the diff is applied. The `apply-diff` command will automatically rename the `agg_tiles_hash_in_diff` value back to `agg_tiles_hash` when applying the diff.
This option is identical to using [`mbtiles diff ...`](mbtiles-diff.md). The following commands two are equivalent:

```shell
mbtiles copy src_file.mbtiles diff_file.mbtiles \
--diff-with-file modified_file.mbtiles
mbtiles diff file1.mbtiles file2.mbtiles diff.mbtiles

mbtiles copy file1.mbtiles diff.mbtiles \
--diff-with-file file2.mbtiles
```

## `mbtiles copy --apply-patch`
Expand All @@ -33,28 +33,5 @@ Copy a source file to destination while also applying the diff file generated by

```shell
mbtiles copy src_file.mbtiles dst_file.mbtiles \
--apply-patch diff_file.mbtiles
```

## `mbtiles apply-patch`

Apply the diff file generated from `copy` command above to an mbtiles file. The diff file can be applied to the `src_file.mbtiles` elsewhere, to avoid copying/transmitting the entire modified dataset.

Note that the `agg_tiles_hash_in_diff` metadata value will be renamed to `agg_tiles_hash` when applying the diff. This is done to avoid confusion when applying the diff file to the original file, as the `agg_tiles_hash` value will be different after the diff is applied.

```shell
mbtiles apply-patch src_file.mbtiles diff_file.mbtiles
```

#### Applying diff with SQLite

Another way to apply the diff is to use the `sqlite3` command line tool directly. This SQL will delete all tiles from `src_file.mbtiles` that are set to `NULL` in `diff_file.mbtiles`, and then insert or update all new tiles from `diff_file.mbtiles` into `src_file.mbtiles`, where both files are of `flat` type. The name of the diff file is passed as a query parameter to the sqlite3 command line tool, and then used in the SQL statements. Note that this does not update the `agg_tiles_hash` metadata value, so it will be incorrect after the diff is applied.

```shell
sqlite3 src_file.mbtiles \
-bail \
-cmd ".parameter set @diffDbFilename diff_file.mbtiles" \
"ATTACH DATABASE @diffDbFilename AS diffDb;" \
"DELETE FROM tiles WHERE (zoom_level, tile_column, tile_row) IN (SELECT zoom_level, tile_column, tile_row FROM diffDb.tiles WHERE tile_data ISNULL);" \
"INSERT OR REPLACE INTO tiles (zoom_level, tile_column, tile_row, tile_data) SELECT * FROM diffDb.tiles WHERE tile_data NOTNULL;"
--apply-patch diff.mbtiles
```
50 changes: 35 additions & 15 deletions docs/src/mbtiles-diff.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,45 @@

## `mbtiles diff`

Diff command compares two mbtiles files `A` and `B`, and generates a diff (delta) file.
If the diff file is [applied](mbtiles-copy.md#mbtiles-apply-patch) to `A`, it will produce `B`.
The diff file will contain all tiles that are different between the two files
(modifications, insertions, and deletions as `NULL` values), for both the tile and metadata tables.
The only exception is `agg_tiles_has` metadata value. It will be renamed to `agg_tiles_hash_in_diff` and a
new `agg_tiles_hash` will be generated for the diff file itself.
Copy command can also be used to compare two mbtiles files and generate a delta (diff) file. The diff file can be [applied](#mbtiles-apply-patch) to the `src_file.mbtiles` elsewhere, to avoid copying/transmitting the entire modified dataset. The delta file will contain all tiles that are different between the two files (modifications, insertions, and deletions as `NULL` values), for both the tile and metadata tables.

There is one exception: `agg_tiles_hash` metadata value will be renamed to `agg_tiles_hash_in_diff`, and a new `agg_tiles_hash` will be generated for the diff file itself. This is done to avoid confusion when applying the diff file to the original file, as the `agg_tiles_hash` value will be different after the diff is applied. The `apply-diff` command will automatically rename the `agg_tiles_hash_in_diff` value back to `agg_tiles_hash` when applying the diff.

```shell
# This command will comapre `a.mbtiles` and `b.mbtiles`, and generate a new diff file `diff.mbtiles`.
mbtiles diff a.mbtiles b.mbtiles diff.mbtiles
# This command will compare `file1.mbtiles` and `file2.mbtiles`, and generate a new diff file `diff.mbtiles`.
mbtiles diff file1.mbtiles file2.mbtiles diff.mbtiles

# If diff.mbtiles is applied to a.mbtiles, it will produce b.mbtiles
mbtiles apply-diff a.mbtiles diff.mbtiles b2.mbtiles
# If diff.mbtiles is applied to file1.mbtiles, it will produce file2.mbtiles
mbtiles apply-diff file1.mbtiles diff.mbtiles file2a.mbtiles

# b.mbtiles and b2.mbtiles should now be the same
# file2.mbtiles and file2a.mbtiles should now be the same
# Validate both files and see that their hash values are identical
mbtiles validate b.mbtiles
[INFO ] The agg_tiles_hashes=E95C1081447FB25674DCC1EB97F60C26 has been verified for b.mbtiles
mbtiles validate file2.mbtiles
[INFO ] The agg_tiles_hashes=E95C1081447FB25674DCC1EB97F60C26 has been verified for file2.mbtiles

mbtiles validate file2a.mbtiles
[INFO ] The agg_tiles_hashes=E95C1081447FB25674DCC1EB97F60C26 has been verified for file2a.mbtiles
```

## `mbtiles apply-patch`

Apply the diff file generated with the `mbtiles diff` command above to an MBTiles file. The diff file can be applied to the `src_file.mbtiles` that has been previously downloaded to avoid copying/transmitting the entire modified dataset again. The `src_file.mbtiles` will modified in-place. It is also possible to apply the diff file while copying the source file to a new destination file, by using the [`mbtiles copy --apply-patch`](mbtiles-copy.md#mbtiles-copy---apply-patch) command.

Note that the `agg_tiles_hash_in_diff` metadata value will be renamed to `agg_tiles_hash` when applying the diff. This is done to avoid confusion when applying the diff file to the original file, as the `agg_tiles_hash` value will be different after the diff is applied.

mbtiles validate b2.mbtiles
[INFO ] The agg_tiles_hashes=E95C1081447FB25674DCC1EB97F60C26 has been verified for b2.mbtiles
```shell
mbtiles apply-patch src_file.mbtiles diff_file.mbtiles
```

#### Applying diff with SQLite

Another way to apply the diff is to use the `sqlite3` command line tool directly. This SQL will delete all tiles from `src_file.mbtiles` that are set to `NULL` in `diff_file.mbtiles`, and then insert or update all new tiles from `diff_file.mbtiles` into `src_file.mbtiles`, where both files are of `flat` type. The name of the diff file is passed as a query parameter to the sqlite3 command line tool, and then used in the SQL statements. Note that this does not update the `agg_tiles_hash` metadata value, so it will be incorrect after the diff is applied.

```shell
sqlite3 src_file.mbtiles \
-bail \
-cmd ".parameter set @diffDbFilename diff_file.mbtiles" \
"ATTACH DATABASE @diffDbFilename AS diffDb;" \
"DELETE FROM tiles WHERE (zoom_level, tile_column, tile_row) IN (SELECT zoom_level, tile_column, tile_row FROM diffDb.tiles WHERE tile_data ISNULL);" \
"INSERT OR REPLACE INTO tiles (zoom_level, tile_column, tile_row, tile_data) SELECT * FROM diffDb.tiles WHERE tile_data NOTNULL;"
```
1 change: 1 addition & 0 deletions martin/src/srv/server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ pub fn new_server(config: SrvConfig, state: ServerState) -> MartinResult<(Server
.workers(worker_processes)
.run()
.err_into();

Ok((Box::pin(server), listen_addresses))
}

Expand Down
Loading
Loading