Support loading Basis-enabled Replica files

[mosra]

Motivation and Context

ℹ️ Depends on #334, which is still yet-to-be-merged.

I converted the *.hdr files from apartment_0 from the Replica Dataset to *.basis using the convert-hdr.sh script from here (needs features from Magnum master) and this is the result:

1. The original raw RGB16F pixel data have 6.7 GB and most GPUs would expand them to RGBA16F during upload due to alignment requirements, increasing the memory use way over 8 GB (I'm guessing from the fact that e.g. Metal or D3D doesn't have 48-bit types at all). I didn't do any experiments with EXR compression right now, I assume it could lead to some file size savings. Among other uncompressed options there are formats like RGB9E5 that are half of size of RGBA16F and could still have acceptable quality while preserving HDR.
2. Conversion of the raw HDR data to a 16-bit uncompresed EXR using MiniExrImageConverter and then using ImageMagick to a 8-bit (LDR) PNG with exposure adjustment reduces the storage requirements to about 600 MB, but then again expanding them to full RGBA8 when uploading to the GPU will eat over 4 GB of memory. So it's a 1:2 memory use reduction, but we unfortunately lose the HDR aspect.
3. Conversion of that 8-bit PNG using BasisImageConverter results in 46 MB of data on-disk, which is pretty cool. Then, expanding that to 4x4 ASTC will make each 16x16 block occupy 128 bits, which is a 1:6 memory usage reduction compared to the original RGB16F (and 1:8 compared to RGBA16F). Unfortunately Basis only supports LDR formats right now.
4. If we would want GPU texture compression and HDR, the BC6h format would be the choice. (There are also ASTC HDR formats, but the support is quite spotty ATM -- only Intel drivers on Windows, apparently.) This one compresses each 4x4 block to 128 bits, so the same GPU memory savings as with Basis, but with HDR on top. Tell me if I should pursue this path further and try compressing the textures to the BC6h format.

How Can This Be Tested

Download apartment_0.basis.zip (46 MB) and extract it next to the *.hdr files in apartment_0. Load the model with the viewer. It should report what format it will decode Basis to (it's ASTC 4x4 in my case, it would be BCn on Mac) and then load the *.basis files instead of *.hdr (comparatively quite faster) and then show basically the same view as with the *.hdr files. Please tell me what's the perceived quality / artifacts compared to what you're used to seeing -- I used just the default conversion parameters, so there's definitely room to improve.

[msbaines]

LGTM but can we add a test.

[mosra]

I would love to, but as far as I can recall, there aren't any tests for loading of the "vanilla" Replica files either (@bigbike can you confirm/deny?), so I don't know where to start.

[bigbike]

I would love to, but as far as I can recall, there aren't any tests for loading of the "vanilla" Replica files either (@bigbike can you confirm/deny?), so I don't know where to start.

No, we do not have any CI tests for replica model yet. Feel free to create one. I sincerely appreciate it.
I believe the "office_1" is the simplest model with least number of triangles.

[bigbike]

Based on the description, it seems item 4) is our only option at this point. I understand there is a significant gain on the memory usage, I am wondering how much do we loss in terms of visual quality.

[erikwijmans]

I agree with using option 4) here. The ptex textures in replica allow you to increase visual quality and leverage the HDR features to change exposure on the fly. If a user is concerned with memory usage, the quad-colored plys do still look good at low resolutions and use very little GPU memory.

[mosra]

Based on the discussion above, I'll revisit this with a BC6H in a DDS container instead of Basis (though I'm not ruling out the possibility of Basis itself supporting that some day). Currently I'm fully focused on the assert import APIs, will jump on the BC6H encoding right after.

For the Replica tests -- should I add one similarly to the other Python tests and provide a "ground truth" numpy file for it? Or ... wouldn't it make more sense to have the rendering test on the C++ side (less moving parts, better diagnostic output etc.)? Comparing against a ground truth image probably won't check all corner cases of the PTex rendering, but doing that would mean (I assume) handcrafting some test data. So at least something :)

[erikwijmans]

Up to you on tests front! I would be happy with either one :)

[mosra]

Actually ... 😅 Are you (or anybody else, @bigbike?) able to add the tests? Because for me it means attempting to verify the output of code that I didn't write and don't even have an exact idea what should be expected (e.g., those artifacts related to disabled use of buffer textures on macOS and such).

Thanks in advance :)

[eundersander]

Closing this stale PR.