Cloud Optimized GeoTIFF (COG) support

[sharkAndshark]

Add support for Cloud Optimized GeoTIFF (COG).

CLI

martin  /path/to/cogs/file.tif  /path/to/directory

Conf file

...
cogs:
  paths:
    # scan this whole dir, matching all *.tif files
    - /dir-path
    # specific cog file will be published as cog source
    - /path/to/file1.tif
  sources:
   # named source matching source name to a single file
    cog-src1:  /path/to/file.tif

It seems that the conf sections about PMTiles and MBTiles are very similar. I don't know maybe we could merge them into a single section?

files:
  paths:
    - /dir1
    - /path/to/file1.ext
  sources:
    source1: /path/to/file.ext

Things to track:

• Rgb/Rgba with 8 bits per sample
  done by Add basic cog support #1590
• Auto-discovery for both *.tif and *.tiff
  done by Autodiscover *.tiff files in addition to *.tif #1635
• Gray and 8 bits per sample
• Other combination of color-type and bits per sample
• Common compression method which tiff crate not supported. More research needs to do here.
  • NONE
  • PACKBITS
  • Deflate
  • LZW
  • ModernJPEG
  • ZSTD (since Support ZSTD compression method for reading image-rs/image-tiff#256)
  • LZMA
• Do not ignore sub-file which is mask, more research needs to do here
• Remote COG file like S3

[nyurik]

I like the idea, thanks! One aspect that I am not too certain about is combining things into files -- one of highly requested features for files it to serve them from http -- in order words, pmtiles and i suspect COG files will reside in some cloud bucket, and Martin will access them with the HTTP range requests.

This means that we would need to tell the type of that file without relying on file extension. We could try to auto-detect it - e.g. get first few bytes and detect the type, but it won't be relevant to mbtiles, so in short - much easier to keep things clean and separate.

[sharkAndshark]

Some refs:

• TIFF spec
• The BigTIFF File Format
• GeoTIFF spec
• COG spec
• COG format details
• How to recognize a COG and how to create a proper one
• cogdumper of mapbox

[sharkAndshark]

I've built a simple C# prototype without gdal, and am working on to port it to Rust.
The config would be separated as @nyurik suggested.

[sharkAndshark]

Tif with Non-Square pixel

Square
On X-Axis Or Y-Axis 1 pixel = 10 meters
Pixel Size = (10.000000000000000,-10.000000000000000)

Non-Square
On X it's 10, On Y it's 11
Pixel Size = (10.000000000000000,-11.000000000000000)

If it's non-square, we should adjust the output PNG size based on the resolution of X or Y and we should tell the user the resolution we picked in our TileJSON.

PseudoCode

let (resolution_x, resolution_y) = get_tiff_resolutions();
let ratio = resolution_x / Math.abs(resolution_y); // we always pick resolution_x as the resolution
let output_tile_size = (tif_tile_size[0], tif_tile_size[1] / ratio);

I'm not sure how often we encounter TIF files with non-square pixels. We will handle this when needed.

[sharkAndshark]

The tile-schema and tile-json

Clients needs to know the tile-schema(like min_zoom, max_zoom,tile-size, extent, resolutions array...) in tiff file to display them.

We must add these in tile json and it's the most high priority.

And Another to be noted is:

Should we handle the google-schema specially?

The tile-schema could be GoogleMapsCompatible(see gdal doc).

If it's google compatible, Should we do a transformation internally, like if we have 5 layers in a GoogleMapsCompatible tiff file:

Add a google compatible in tile-json when it's tiff file.

{
    ...,
   "min_zoom": 10,
   "max_zoom": 14,
   "googleMapsCompatibled": "true",
    ...
}

sub-file	not-specially	specially
sub-file 0	user should call /4/x/y	user should call /14/x/y
sub-file 1	user should call /3/x/y	user should call /13/x/y
....	....	....

[sharkAndshark]

I find a better subfiles organization description: https://docs.ogc.org/per/21-025.html#toc33 @nyurik @CommanderStorm

[sharkAndshark]

And a tool to help view the internal details of a tiff file, exiftool. The output of this tools is really helpful.

sudo apt update
sudo apt install libimage-exiftool-perl
exiftool ./rgba_u8_nodata.tiff  -a -U  -g1

[sheecegardezi]

Some refs:

* [TIFF spec](https://download.osgeo.org/libtiff/doc/TIFF6.pdf)

* [The BigTIFF File Format](https://www.awaresystems.be/imaging/tiff/bigtiff.html)

* [GeoTIFF spec](https://docs.ogc.org/is/19-008r4/19-008r4.html)

* [COG spec](https://github.com/cogeotiff/cog-spec/blob/master/spec.md)

* [COG format details](https://gdal.org/drivers/raster/cog.html#file-format-details)

* [How to recognize a COG and how to create a proper one](https://cogeotiff.github.io/rio-cogeo/Is_it_a_COG/)

* [cogdumper of mapbox](https://github.com/mapbox/COGDumper/blob/master/cogdumper/cog_tiles.py)

none of these actually define COG data format the spec seem incomplete

[sheecegardezi]

I think i have created valid cog with alpha channel and 8 bit:

gdalwarp -of cog -dstalpha -srcnodata 0  -ot BYTE -b 1 -b 2 -b 3  -overwrite -multi --config GDAL_CACHEMAX 9000 -wm 9000 -co BLOCKSIZE=256 -co BIGTIFF=YES -co COMPRESS=DEFLATE -co PREDICTOR=YES -co NUM_THREADS=ALL_CPUS data.tif cog.tif

I dont think martin support serving COG:

docker run --rm -it -p 3000:3000 -v "$(pwd):/workspace" ghcr.io/maplibre/martin /workspace/cog.tif
[2025-01-25T12:12:21Z INFO  martin] Starting Martin v0.14.2
[2025-01-25T12:12:21Z INFO  martin] Config file is not specified, auto-detecting sources
[2025-01-25T12:12:21Z ERROR martin] Unrecognizable connection strings: ["/workspace/cog.tif"]

Is it possible to get a sample cog that martin supports:

[CommanderStorm]

@sheecegardezi you are not running a martin version with basic COG support.
If you want to use this feature, you will have to go nightly and use the :main tag.

See this issue for further context:

• Releasing v0.15.0 #1632

[sheecegardezi]

@CommanderStorm thanks for picking that up! I just tested it works like a charm with nightly build.

For completeness following worked:

gdalwarp -t_srs EPSG:3857 -of cog -dstalpha -srcnodata 0  -ot BYTE -b 1 -b 2 -b 3  -overwrite -multi --config GDAL_CACHEMAX 9000 -wm 9000 -co BLOCKSIZE=256 -co BIGTIFF=YES -co COMPRESS=DEFLATE -co PREDICTOR=2 -co LEVEL=9 -co NUM_THREADS=ALL_CPUS data.tif cog.tif
docker run --rm -it -p 3000:3000 -v "$(pwd):/workspace" ghcr.io/maplibre/martin:main /workspace/cog.tif

[sharkAndshark]

I will add tile schema of COG to tile json. It's not convenient for clients to load COG without it. Traveling, I will be back next day.