EXT_texture_webp extension proposal

[OmarShehata]

This is a proposed draft of an extension to support WebP textures following the discussion here: #1503.

Convenient markdown link

Since extensions cannot redefine existing properties, this adds a property on an Image object that mirrors the base image object, but allows the WebP format. This simplifies implementation (see PRs in gltf-pipeline and CesiumJS. The latter includes a sample model).

Although I just noticed the MSFT_texture_dds extension allows uri to point to a dds file, even though the base spec says it must be a PNG or JPEG. If this is allowed then this extension could potentially be simplified.

The only things I'm unsure of are:

• It's strange to define a mimeType inside the extension object, given that it will always be image/webp. But defining it there simplifies the implementation since clients can just treat it as a regular image.

[emackey]

I had an offline conversation with @OmarShehata and @pjcozzi where I voiced some of Zell's fragmentation concerns in #1503 (comment). The response, if I'm summarizing correctly, was that WebP has much faster decode & render than the core formats, so for systems like 3D Tiles where lots of images are being streamed down, the decode speed becomes as much of a win (or even more) as the network bandwidth reduction.

Is the intention here that WebP would never be used for "normal" glTFs in the wild, such as from the Blender exporter and other tools? Would it be strictly contained to just 3D Tiles models? Or would other engines see the benefit of WebP and want to adopt this as well?

I'm just wondering if the README needs to more clearly state the intended use, so that this doesn't needlessly become a source of fragmentation in the wider ecosystem of models.

[donmccurdy]

Is the intention here that WebP would never be used for "normal" glTFs in the wild, such as from the Blender exporter and other tools? Would it be strictly contained to just 3D Tiles models? Or would other engines see the benefit of WebP and want to adopt this as well?

And, would the answers to the questions above still be true if CTTF were complete today?

It would clearly by ideal if CTTF were universally the best choice for texture data, but I don't know enough about the expected compression rates, transcoding times, and quality to make that claim.

Using an AGI_ prefix might be the most conservative choice here, but if we expect others to adopt glTF+WebP even after CTTF is complete, I'm open to other ideas.

[lexaknyazev]

the base spec says it must be a PNG or JPEG

As with other "enum" JSON types, the schema allows image.mimeType to be an arbitrary string, so DDS extension design is correct.

I would suggest moving the extension object to texture, while keeping image objects simple. Basically, replicating DDS extension.

[lexaknyazev]

@donmccurdy
It's safe to say that storage / transmission cost of WebP is better or the same as CTTF. It's also safe to say that WebP is definitely better than JPEG if you are willing to restrict runtime to WebGL and ignore Safari clients (Firefox will have WebP support later this year).

CTTFv1 will provide smaller GPU memory footprint for opaque natural images. Mileage may vary for images with alpha channel, text or vector shapes (think UI elements).

So, there's room for both, especially with regards to Cesium ecosystem.

[emackey]

...DDS extension design is correct.

I would suggest moving the extension object to texture, while keeping image objects simple. Basically, replicating DDS extension.

Thanks @lexaknyazev. Can we predict that the CTTF extension will be the same?

@OmarShehata I know you've got PRs open across a number of repos now. Is it possible to change the extension to work like DDS? It helps consistency of implementing multiple file types, and it helps GLB packing/unpacking since they don't need to understand the innards of each file type extension.

[lexaknyazev]

Can we predict that the CTTF extension will be the same?

Yes, I'm going to use image the same way for CTTF for the exact same reason - uri/bufferView packing.

[OmarShehata]

Thanks for all the feedback so far! I've revised the draft and updated the implementation and sample model in the CesiumJS PR.

I'm just wondering if the README needs to more clearly state the intended use, so that this doesn't needlessly become a source of fragmentation in the wider ecosystem of models.

I do agree here, that using this for everyday models would destroy a glTF's cross-platform nature. I added a note in the Best Practices section to warn about this (wording suggestions welcome).

At this point, I think the reason to go for a EXT_ as opposed to CESIUM_ is that I believe this would be useful to multiple vendors given the use case described in #1503 (comment)

For situations like the work I do at Facebook, where we generate client-specific delivery versions of assets (or where folks are able and willing to host a fallback JPEG version online), having a formalised way to use WebP is inarguably powerful. Asset size is of overwhelming concern for us.

[donmccurdy]

Thanks for the clarifications, sounds good. :)

[emackey]

Looks good so far. Two minor comments.

[OmarShehata]

@emackey thanks! I took care of your comments and added a link to webp_js as another decoder library that is potentially more complete/has better examples than libwebpjs.

[emackey]

This looks like an exact match for DDS now, which is great for consistency of image extensions. I think it's good to merge.

[emackey]

@lexaknyazev Did you want to take one last look before this gets merged?

[donmccurdy]

Definitely a nit, but, why EXT_image_webp rather than EXT_texture_webp?

[OmarShehata]

Definitely a nit, but, why EXT_image_webp rather than EXT_texture_webp?

Oh, yeah. It was just that way because of the initial draft. I'll update this now.

[OmarShehata]

@emackey @lexaknyazev this should be ready.

[emackey]

@lexaknyazev Is this good to merge?

[pjcozzi]

@OmarShehata can you please open a new PR to add this extension to the list: https://github.com/KhronosGroup/glTF/blob/master/extensions/README.md

[chase-moskal]

@lexaknyazev

[...] if you are willing to restrict runtime to WebGL and ignore Safari clients (Firefox will have WebP support later this year).

i made a webp polyfill that allows webp to work in safari today. it runs the webp decoder clientside, in the browser. you exchange clientside computational effort for large bandwidth savings. much room for optimizations in the library, but it gets the job done today

• webp-hero — clientside browser polyfill for webp

eventually i'd be interested to see if i could integrate this for usage in web game engines like babylon, maybe even cache results to indexdb or something.. just wanted to drop a line in case somebody needs something like this. i started webp-hero initially to make 100MB of PNGs a more reasonable 5MB, which made an infeasible image-heavy website project suddenly feasible in the real world, webp really shines with it's ability to represent transparency in a lossy alpha channel

[aaronfranke]

@OmarShehata Where can we find test glTF files that use WebP textures?

[OmarShehata]

@aaronfranke this was the one originally used when drafting this extension: https://github.com/CesiumGS/cesium/blob/bd8be76b7707ddb9c858ae39538c18064c218f98/Specs/Data/Models/Box-Textured-Webp/CesiumBoxWebp.gltf

[donmccurdy]

This should work to convert any glTF with PNG or JPEG textures:

npm install --global @gltf-transform/cli

gltf-transform webp input.glb output.glb 

It uses pretty stock compression settings at the moment. Using --slots, --quality, and --lossless flags you could experiment further: higher quality or lossless modes for normal maps may be worthwhile.