Proposal: Open3DGC mesh compression extension schema

[kainino0x]

mesh_compression_open3dgc proposal & mock-up

I have put together this proposal, with lots of Patrick Cozzi's advice, in order to try to solve the problem of how to specify and used compressed mesh data in glTF. This is based on the extension specification and implementation work of Fabrice Robinet, which is in turn based on the Open3DGC algorithm and library work by Khaled Mamou. It is designed to support meshes, but not animations.

This proposal should not require any changes to the core glTF specification, and most or all changes are within "extensions" objects. This avoids churn in the core specification, and also has future-proofing benefits in format flexibility: While the three libraries I've looked at (Open3DGC, OpenCTM, webgl-loader) all work similarly, there could easily be other libraries which would be very difficult to shoehorn into this structure.

The semantics of this extension are also designed to provide a good design baseline for possible mesh compression extensions. Though I haven't thoroughly researched on the structure of other compression libraries, they should be able to use similar schemata. Some changes may be necessary to this extension schema, as it has not been implemented or formalized, but I will try to maintain structural compatibility with other compression libraries.

Please reply to this issue with any feedback you might have on this proposal.

(cc #208 #230)

Rendered (Gist for change tracking purposes)

[pjcozzi]

Thanks for this detailed proposal, @kainino0x.

I'm going to take a second pass through this, but here are my initial comments (all are minor):

• We'll talked about having a generic mesh compression extension framework and then specific compression extensions. Here, you have just used the generic glTF extension framework. Perhaps this is a better approach since it gives each compression extension full control over how it fits into the glTF schema, but your extension still provides a concrete example that may form the foundation for other extensions. What do you think?
• To be consistent with glTF naming, extensions.GLTF_mesh_compression_open3dgc.decompressedBuffer should be plural, e.g., the top-level glTF buffers property is not buffer.
• GLTF_mesh_compression_open3dgc - we can drop the GLTF prefix. Perhaps mesh_compression_open3dgc or open3dgc_mesh_compression.

[--------------model1---------------][--------------model2---------------]

Should model be primitive or mesh?

[----indices----][positions][normals]

What about texture coordinates and application-specific attributes? Are they not really important to this spec since the decompressor returns each attribute in a separate array?

[pjcozzi]

I took a second pass through this, and have no new questions. This looks like a great start, and perhaps we will only have minor tweaks once we do the implementation.

[kmammou]

One comment regarding “Open3DGC … is designed to support meshes, but not animations.”

The Open3DGC library could also be used to encode skinning/bone-based animations:

• Use DynamicVectorEncoder/ DynamicVectorDecoder to compress time-varying matrices/quaternions/translations
• Compress animations weights as additional attributes associated with the mesh vertices.

Morph animations could be compressed as follows:

• Compress morph targets as separate 3D meshes
• Use DynamicVectorEncoder/ DynamicVectorDecoder to compress morph target weights

[kainino0x]

@kmammou Sorry, I should clarify. I meant to say that this proposal does not support Open3DGC's animation compression. I will edit that paragraph.

[kmammou]

Oh. Thanks for the clarification :)!
--Khaled

2015-07-20 11:56 GMT-04:00 Kai Ninomiya [email protected]:

@kmammou https://github.com/kmammou Sorry, I should clarify. I meant to
say that this proposal does not support Open3DGC's animation compression. I
will edit that paragraph.

—
Reply to this email directly or view it on GitHub
#398 (comment).

[pjcozzi]

@kmammou do you have any other feedback on this proposal?

[kmammou]

Everything looks fine to me.

On Mon, Jul 20, 2015, 17:27 Patrick Cozzi [email protected] wrote:

@kmammou https://github.com/kmammou do you have any other feedback on
this proposal?

—
Reply to this email directly or view it on GitHub
#398 (comment).

[pjcozzi]

Thanks @kmammou. @kainino0x do you have a specific questions about the compression algorithm or libraries @kmammou may be able to help with?

[kainino0x]

I have implemented a compressed glTF loader in Cesium using the above spec (that is, for now, without any support for animations). This implementation took about 4 hours of work on Cesium. Some of the glTF properties were unneeded (because they appear in the compressed file header), so I removed them (see gist changelog).

cesium#2901 changes

[kainino0x]

@kmammou I was not able to load the binary Open3DGC format in Cesium using o3dgc.js. I wasn't able to find any open issues about this. By any chance, do know what could be going wrong? I get an error like

TypeError: processConfig[config] is not a function

because config is 16, or 12, or whatever (but there are only 10 possible connectivity configurations!)

[kmammou]

It seems that the decoded values are wrong. Is this problem specific to one
model? Do you see the same issue with Open3DGC-ASCII?

2015-07-27 11:21 GMT-04:00 Kai Ninomiya [email protected]:

@kmammou https://github.com/kmammou I was not able to load the binary
Open3DGC format in Cesium using o3dgc.js. I wasn't able to find any open
issues about this. By any chance, do know what could be going wrong? I get
an error like

TypeError: processConfig[config] is not a function

because config is 16, or 12, or whatever (but there are only 10 possible
connectivity configurations!)

—
Reply to this email directly or view it on GitHub
#398 (comment).

[kainino0x]

@kmammou I see this with all of the models that I have tried with Open3DGC-binary. With Open3DGC-ASCII, everything works fine. Of course, it could be a problem with the converter, but I don't know how (we are using params.SetStreamType();).

I did notice one comment in the converter (probably written by @fabrobinet?):

//Open3DGC binary is disabled

but I didn't understand why that comment was there; everything seems fine.

[kmammou]

@kainino0x You can find below some examples showing Open3D-binary streams decoding:
http://khaledmammou.com/o3dgc_new/index.php?model=hand_binary.s3d
http://khaledmammou.com/o3dgc_new/index.php?model=ben_binary.s3d
http://khaledmammou.com/o3dgc_new/index.php?model=wooddoll_binary.s3d

The same examples with Open3D-ascii streams decoding:
http://khaledmammou.com/o3dgc_new/index.php?model=hand_ascii.as3d
http://khaledmammou.com/o3dgc_new/index.php?model=ben_ascii.as3d
http://khaledmammou.com/o3dgc_new/index.php?model=wooddoll_ascii.as3d

The original models in OBJ format are available here
http://khaledmammou.com/o3dgc_new/hand.obj
http://khaledmammou.com/o3dgc_new/ben.obj
http://khaledmammou.com/o3dgc_new/wooddoll.obj

The Open3D-binary streams
http://khaledmammou.com/o3dgc_new/hand_binary.s3d
http://khaledmammou.com/o3dgc_new/ben_binary.s3d
http://khaledmammou.com/o3dgc_new/wooddoll_binary.s3d

The Open3D-ascii streams
http://khaledmammou.com/o3dgc_new/hand_ascii.as3d
http://khaledmammou.com/o3dgc_new/ben_ascii.as3d
http://khaledmammou.com/o3dgc_new/wooddoll_ascii.as3d

The GZipped Open3D-ascii streams
http://khaledmammou.com/o3dgc_new/hand_ascii.txt.gzip
http://khaledmammou.com/o3dgc_new/ben_ascii.txt.gzip
http://khaledmammou.com/o3dgc_new/wooddoll_ascii.txt.gzip

The decoded models in OBJ format are available here
http://khaledmammou.com/o3dgc_new/hand_binary_dec.obj
http://khaledmammou.com/o3dgc_new/ben_binary_dec.obj
http://khaledmammou.com/o3dgc_new/wooddoll_binary_dec.obj
http://khaledmammou.com/o3dgc_new/hand_ascii_dec.obj
http://khaledmammou.com/o3dgc_new/ben_ascii_dec.obj
http://khaledmammou.com/o3dgc_new/wooddoll_ascii_dec.obj

The table below summarizes the compression results obtained with
Open3DGC-binary/ascii.

Model	Uncompressed	Open3DGC-binary	GZipped Open3DGC-ASCII
Ben	1828 KB	194 KB	204 KB
Hand	355 KB	39 KB	46 KB
Wooddoll	142 KB	20 KB	21 KB

The uncompressed size was computed as follows:
Uncompressed size (bytes) = #indices * 2 + # positions * 12 + # normals *
12 + # texture coord * 8

[kainino0x]

(@pjcozzi @fabrobinet @tfili?)

I have some general questions which are not closely related to compression, but which came up in implementation work.

• Based on the mapping between glTF and COLLADA, it seems like primitives isn't really well translated. I think primitives isn't a list of primitives, but a list of primitives objects (COLLADA's triangles, lines, etc.) So perhaps it should be called primitiveSets, submeshes or similar?
• What is the motivation for the way mesh and primitive are separated? This looks like it's based on COLLADA, but the reasoning doesn't seem to carry over: In COLLADA, vertex data is inside a mesh, so if multiple triangles objects want to use common vertex data, they must be on the same mesh. In glTF, vertex data is global, so glTF primitives don't seem to offer any benefit over glTF meshes.
  • nodes can list multiple meshes, so a mesh isn't just a collection.
  • glTF mirrors COLLADA's architecture: a set of accessors per triangles object. While it might be a little annoying to implement in the converter, I would advocate changing core glTF to look more like the following:

"meshes": {
  "mesh1": {
    "name": "mesh1",
    "attributes": { "POSITION": "acc1", "NORMAL": "acc2", ... },
    "primitives": [ { "indices": "acc3", "material": "mat1", "primitiveType": 4 }, ... ]
  }
}

Or, alternatively, to completely dissolve the primitives object.

"meshes": {
  "mesh1": {
    "name": "mesh1",
    "attributes": { "POSITION": "acc1", "NORMAL": "acc2", ... },
    "indices": "acc3",
    "material": "mat1",
    "primitiveType": 4
  }
}

The former happens to match more nicely with Open3DGC and with sample data that I have been looking at.

[kainino0x]

@kmammou Thanks! I will take a look at those and see if I can track down the difference.

[pjcozzi]

So perhaps it should be called primitiveSets, submeshes or similar?

I'm open to changing this. Perhaps submeshes is OK. Somewhat related: #341

What is the motivation for the way mesh and primitive are separated?

I'm OK with either of @kainino0x's suggestions. The later is a bit cleaner if we can justify the simplified design doesn't remove useful structure from content, and implement it in the converter.

@fabrobinet any insight into the current design? In practice, is it useful to have multiple primitives for a mesh? If so, is the attribute flexibility actually useful?

Note that the alternative here is for this extension to require that all primitives in a mesh have the same attributes.

[fabrobinet]

It is very important to match what you get in authoring tools. You typically have multiples primitives each associated with a material that are considered as a whole (the mesh). Please don't change this.
Also you should be using 0.9 and use mode as updated since #341.

[kainino0x]

@fabrobinet Thanks for the reply. I'm definitely fine with keeping multiple primitives per mesh. As far as I understand, when an authoring tool outputs multiple primitives per mesh, all of the primitives will always use the same attributes. So do you think it would be reasonable to move attributes to be a mesh property?

[fabrobinet]

That is not true, you may have say 2 primitives: 1 with a material that is a plain diffuse color, 1 using a texture. In which case you end up with 1 primitive relying on texture coords(uv) attribute and not the other one. So it's that typically positions are always shared in a primitive it is not necessarily true for the other attributes.

[kainino0x]

@fabrobinet I see. Although going from Blender to COLLADA to glTF that doesn't seem to happen, it certainly could in general. How about this: Would it be okay to require that no two primitives on the same mesh use different accessors for the same semantic? (This could still simplify the representation.)

If we don't want to assume that generally, then I will still disable mesh compression for meshes which have more than one accessor for one semantic, and everything will be fine there.

However, if there is no restriction on how primitives work, I still don't see how they add any expressiveness over just meshes. From an engine point of view, I imagine it would be nice to bind attributes once per mesh and draw once per primitive. But if primitives can have any attributes, then every attribute results in both binding and drawing.

EDIT: found a case for which it probably makes sense to keep the current system: for meshes with more than 2¹⁶ vertices, vertex data will have to be separate for each group of 2¹⁶ vertices, which probably should each be a primitive of a mesh. Given this I'm fine with keeping it as it is

[smalcom]

Maybe will interest you: variant of implementation
assimp/assimp#972

[pjcozzi]

@smalcom very cool, did you test this with any loaders? Also, any chance you are interested in turning the current ideas into a full glTF extension?

[smalcom]

did you test this with any loaders?

No, only with glTF.

full glTF extension

yes, interesting. Let me some time, i prepare doc.

[pjcozzi]

did you test this with any loaders?
No, only with glTF.

Not sure what this means. Did you load the compressed model into any rendering engine?

full glTF extension
yes, interesting. Let me some time, i prepare doc.

Go for it! Awesome.

[smalcom]

Did you load the compressed model into any rendering engine?

load into assimp_qt_viewer
https://github.com/assimp/assimp/tree/master/tools/assimp_qt_viewer

[pjcozzi]

Great discussion. Replaced by #874.