Add code profiling benchmarks for STAC queries dependent on COG assets on S3

[abarciauskas-bgse]

We would like to compare performance of tiling COGs with Zarr to help data providers decide which format better suits their overall needs.

Tasks:

• Determine COGs to profile - must be stored on S3 w/o authentication or having a role-based policy we can use
• Build code to seed a STAC catalog with a collection and items
• Write a simplified tile function which includes the step of registering a query (mosaic id) and then generates tiles using it

Questions:

• If we compare this with a tile function using XarrayReader from rio-tiler is it really comparing apples and apples since usually registering a mosaic query and then generating a tile requires 2 HTTP calls (although you can re-use the mosaic id)

[vincentsarago]

@abarciauskas-bgse comparing COG and Zarr is a great idea (even if the Zarr format is not specifically designed for this). I don't think you need to go all the way down to Mosaic (with titiler-pgstac). Already comparing rio_tiler.io.Reader and the reader from titiler-xarray should be good enough IMO.

As soon as you enter the Mosaic side you add:

• database connection
• multithreading
• more network influence (for each of the files)

so the comparison will be harder 🤷. I think we should first focus on non-mosaic comparison before adding much more complexity.

[abarciauskas-bgse]

I agree it will be more straight forward to just compare COG with Zarr, however @sharkinsspatial made the point that it's a bit like comparing apples and oranges because when applications need to create image tiles for a given set of user parameters (bounding box and time step), in COG-land first there needs to be a determination of which COGs to tile and then to tile them.

So to compare more similar operations you have:

1. COGs: You have a metadata read from pgSTAC followed by the COG data read
2. Zarr: you have the metadata read from the zarr metadata files and then the data read.

[vincentsarago]

I see, you don't have to use titiler-pgstac then, you can just use https://cogeotiff.github.io/rio-tiler/mosaic/

so to be clear, you're considering that COG are usually small and maybe sparse dataset (multiple scenes, spatially distributed) while Zarr will usually be big (covering the entire earth).

If the goal is to help choosing the data format between COG and Zarr I don't really understand why we should compare two different kind of dataset 🤷

[abarciauskas-bgse]

@vincentsarago

you're considering that COG are usually small and maybe sparse dataset (multiple scenes, spatially distributed) while Zarr will usually be big (covering the entire earth).

I think this is a good point, if possible, I would like to find or create some dataset in COG and Zarr which is the same underlying data but represented in these 2 different formats.

That may be where I start :)

[sharkinsspatial]

To provide a bit of context around this, here are my comments from the Slack thread

The rationale was to make performance comparisons a bit more fair. When we use titiler-xarray, in order to look up the data chunks we need to create a tile, xarray needs to load all the zarr archive’s consolidated metadata and use that to determine where the chunks are in storage and how to access them. When we render a tile from a single COG this “index lookup” isn’t necessary because we already know the COG where we are reading chunks from.

This makes comparing performance of requesting a tile from a single COG to requesting a tile via unfair and is sort of an apples to oranges comparison. To make the comparison more even (apples to apples) I suggested we use a comparison with titiler-pgstac and titiler-xarray since both suffer the overhead of an “index lookup”. titiler-pgstac is also the COG discovery model we are using in all our real world applications as well so we should be doing performance comparison using this baseline.

[abarciauskas-bgse]

There is a public archive of COGs generated for CMIP6 for a VEDA related dashboard, using NetCDFs we have stored in S3 as well. There is a public archive is here: https://nex-gddp-cmip6-cog.s3.us-west-2.amazonaws.com/ and we have the original NetCDFs in S3 as well. So we can:

• Seed a pgSTAC collection of these files (working on this now)
• Write profiling code for titiler-pgstac for a series of tiles
• Create a kerchunk reference for the NetCDFs in S3
• Write profiling code for titiler-xarray for the same series of tiles (this is already started in sketch profiling developmentseed/titiler-xarray#4)

I'm going to start on this for https://nex-gddp-cmip6-cog.s3.us-west-2.amazonaws.com/index.html#monthly/CMIP6_ensemble_median/tas/ (2,808 entries)

[abarciauskas-bgse]

To be more specific about how I think this is apples and apples

1. Assume you are starting with a known collection
2. Metadata read step
   a. For COG: you register a query with pgstac for the collection id, variable and time step of interest
   b. For Zarr: You read consolidated metadata and "lazily load" the variable and time step (slice) of interest
   c. For now - I'm going to store both of these metadata components (pgstac and kerchunk reference) locally, but it might be of interest to move pgSTAC and kerchunk to the cloud...
3. Generate tiles
   a. COG: use mosaic ID to read chunks from COGs on S3
   b. Zarr: Use zarr metadata to read chunks from NetCDFs or Zarrs on S3

[vincentsarago]

Note: https://github.com/sm-Fifteen/asgi-server-timing-middleware this might be interesting for profiling the application