Tangent-basis workflow for getting 100% correct normal-mapping #1252
Comments
If the glTF includes tangents, then the normal maps should be in whatever tangent space is defined by the normal and tangent vectors + bitangent sign. If the glTF does not include tangents, the spec says this: https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#meshes
Tech art is telling me that Substance Painter currently does not write out tangents when exporting glTF files, so the normal maps needs to be in MikkTSpace. I'm not sure what Substance Painter is doing here. If you bring a model generated with MikkTSpace algorithms without tangents into Unity, for example, and set the import settings to generate tangents, then the normal maps should match. |
|
|
|
The same is true for BabylonJS sandbox though BabylonJS uses a slightly different tangent space calculation in the shader. |
|
Thanks for the info and fast reply.
We are writing a custom renderer (and the Maya2glTF exporter) and would
like to use glTF as our mesh format, so we need to understand all the
details.
So far using the MikkTspace source code from Blender does not result in
correct renderings. This code also accepts an `angularThreshold` to smooth
the tangents, reducing the number of vertex splits, so this argument must
be the same in the baker and renderer, but is unknown to us. So if tangents
are not in the glTF file, it is ambiguous to know how to call the
MikkTspace code...
However even when the mesh contains normals and tangents, one
implementation (unreal) computes the bitangent in the pixel shader, and
others (unity, glTF/pbr) in the vertex shader. I guess the latter is the
suggested approach?
You write: "For best results, the mesh triangles should also be processed
using default MikkTSpace algorithms." But the MikkTspace code I found does
not process the mesh, it just returns tangents. You most likely mean that
the vertices should be split correctly to deal with different tangents for
the same normal? This is unclear to me.
The glTF spec also has a pending TODO, it does not specify how to deal with
TBN frames after morphing and skinning. Do you have any guidelines for
this? Otherwise different implementations will render differently when the
meshes are deformed.
Last but not least, the glTF PBR code can also generate tangents on the fly
in the fragments shader. However these tangents do not seem to be the same
as the offline MikkTspace ones. Should they be?
Thanks a lot!
Peter
…On Tue, 20 Feb 2018, 18:33 Gary Hsu, ***@***.***> wrote:
What is the correct way to bake normal-maps for GLTF / PBR?
If the glTF includes tangents, then the normal maps should be in whatever
tangent space is defined by the normal and tangent vectors + bitangent
sign. If the glTF does not include tangents, the spec says this:
https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#meshes
Implementation note: When tangents are not specified, client
implementations should calculate tangents using default MikkTSpace
algorithms. For best results, the mesh triangles should also be processed
using default MikkTSpace algorithms.
Implementation note: When normals and tangents are specified, client
implementations should compute the bitangent by taking the cross product of
the normal and tangent xyz vectors and multiplying against the w component
of the tangent: bitangent = cross(normal, tangent.xyz) * tangent.w
Tech art is telling me that Substance Painter currently does not write out
tangents when exporting glTF files, so the normal maps needs to be in
MikkTSpace. I'm not sure what Substance Painter is doing here. If you bring
a model generated with MikkTSpace algorithms without tangents into Unity,
for example, and set the import settings to generate tangents, then the
normal maps should match.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1252 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACR1b8cnc_tdZlDg3PH1vg9g87j_LJFJks5tWwHigaJpZM4SMUDY>
.
|
We should perhaps put this in the spec, but I would expect to use the default tangent space from mikktspace.h. According to the comments, the default fAngularThreshold is 180 degrees which disables the threshold.
I think this is an implementation detail. It shouldn't matter where the bitangent is calculated.
The blender wiki has a nice explanation along with links to the original paper and accompanying code. What code are you referring to?
Not yet. Ideally, this section needs to be filled out with a reference implementation, but I'm not sure I expect everyone to do the same thing for this.
Yes, ideally. The spec defines this in an implementation note because not everyone will be able to do this due to performance reasons or something else. For example, this is in my backlog to investigate for BabylonJS if we can do this on load. Right now, if you want the normal map of a glTF asset to render identically across the ecosystem, the glTF asset must include tangents. |
|
I will ask out artist if I can share his extreme test model here, so we can discuss this with an example. It was based on a polycount article, so I am pretty sure he will allow this. Yes I am using the MikkT code from that blender page. I am currently using it in my Maya2glTF exporter, for testing. Regarding the bitangent in the pixel shader or not, I believe this does make a small difference, since substance painter explicitly allows you to specify this. Normal maps for Unreal and Unity are slightly different it seems, although both use MikkT these days. My tests also indicate that the glTF PBR pixel shader generated tangents are not identical to the MikkT tangents. Again, I must triple check this. Unity is not using the default threshold of 180, at least that is what our tests seem to indicate. Actually I was unable to get the same tangents using any MikkT threshold. It is off course possible that my code is not using MikkT correctly, since I cannot find any unit tests for MikkT :-) I also found derivative maps to be interesting, but somehow these did not become popular... |
This is the expected behavior; the glTF PBR sample doesn't calculate the proper MikkT tangents. Also, regarding calculating bitangent in the pixel shader vs the vertex shader; I believe in some cases the bitangent interpolation across the face of a triangle could result in a different basis than recalculating the bitangent at each pixel. I don't know which is the ideal case here (probably pixel shader) nor how significant the difference can be at it's most extreme. |
Is this überhaupt possible to do in a fragment shader? If one looks at the MikkTspace, this is rather complicated, recombining triangles into groups, smoothing, re-ordering, etc... Maybe derivative maps are better for pure fragment shader bump mapping, since these do not require a TBN frame at all, but somehow these did not become popular... |
I don't think it's reasonable to reconstruct a MikkT space tangent in a pixel shader. If someone wanted to tackle building a good js implementation to do load-time MikkT calculation of tangents (or if this already exists) that would be the right route to getting a MikkT-correct WebGL implementation. |
|
@bghgary |
|
@lexaknyazev If two vertices of the same triangle have different bi-tangent signs, this would mean that the triangle is both mirrored and not-mirrored in texture-space. IMO that is an invalid triangle. |
Yep. I'd propose to disallow them. |
|
@snagy Actually an ASM.JS port of the MikkT-space C code would be straightforward I guess? |
What do you want to adjust? Vertices of the same triangle should have the same |
|
Something along this:
|
|
+1 Sounds good to me. |
ghost
mentioned this issue
|
Related: mrdoob/three.js#15749 We're adding support for stored tangents in three.js. When tangents are absent, we still generate them in the shader, and cannot use MikkT there. A good JS or WASM MikkT implementation would be valuable IMO. |
+1 We have an issue about this for Babylon here BabylonJS/Babylon.js#3345, though it's not high priority at the moment. Note that it doesn't have the be the full MikkT algorithm as we only need the tangents. The meshes should have already been processed with MikkT algorithm except with the tangents omitted. |
|
I've found quite a few graphics engineers to be a little confused/irritated by this aspect of glTF, as the spec quite clearly requests MikkT, but all (?) the WebGL engines use the dxdt in-shader differentiation approach to compute them. When there's a pragmatic consensus implied by the all the de-facto standard implementations, and it's at such odds with what the spec says, folks don't quite know what to think. Maybe some language clarification here? |
|
ThreeJS support for stored tangents was released today, in r102. When tangents are missing, we continue to use dxdt for lack of a MikkT implementation (and some concerns about the practicality of adding one). One change that may help the situation would be a robust offline implementation for adding the |
It is true that WebGL engines currently use dxdt approaches when tangents are not present, but they use different dxdt approaches. For example, Babylon's implementation comes from http://www.thetenthplanet.de/archives/1180. Last I checked, three uses a different one. |
|
I'm still hoping to get an offline implementation working, to add a |
|
Related: #1637. |
|
Just to put the offer on the table: if someone can figure out how to compile https://github.com/mmikk/MikkTSpace to WASM (or port it to AssemblyScript?), I'll very gladly do the remaining work of integrating it into an offline CLI tool for adding tangents to glTF files. 🙏 |
|
That article raises a great point about per-vertex vs per-pixel bitangent computation. I think we should have a sample model that highlights the difference (and also ensure that the spec is clear about it). /cc @emackey |
|
Quick update on this issue — I've been able to find three implementations of the MikkTSpace standard so far:
I've written WebAssembly bindings for the Rust implementation — this was, at least for me, considerably easier than writing bindings for the C code. The WebAssembly build is working, with a total size of about 18kb minzipped. I haven't published it to npm yet, and need to do some more work packaging it for that purpose, but you can track the progress here: https://github.com/donmccurdy/mikktspace-js. Once that is done it will be included in donmccurdy/glTF-Transform#175, so that tangents can be added to arbitrary glTF models with: Fully agreed with @lexaknyazev that it would be helpful to have a sample demonstrating correct handling of these issues clearly. I do not think that we (three.js) will be using MikkTSpace by default at runtime — it's an extra 18kb, requires unwelding all mesh primitives, and doesn't differ visibly from our derivative implementation in GLSL for most assets (with some notable exceptions). For those assets where the MikkTSpace tangents are required, we can point users to the necessary steps to generate them. |
Is this the expected loader behavior of a glTF viewer? That, in the case of receiving glTF content without tangents, it should unweld the mesh [in the process of generating tangents] in order to render the model in a spec-compliant way? |
No, most are using screen-space derivatives to calculate tangents on the fly. |
|
The spec has this implementation note:
It isn't possible to generate MikkTSpace tangents with an existing index list, so this is implicitly saying client implementations should unweld the mesh vertices too. In practice, at least for WebGL, no one does this. First, a web-friendly MikkTSpace implementation did not exist until recently. Second, the performance cost is non-trivial, and there's little point in paying it unless you know that screen-space derivatives will give the wrong results for a particular model. Perhaps we should rephrase this suggestion when moving to more normative language? /cc @lexaknyazev |
|
If dfdt does become part of the spec, clarification on whether to normalize the tangent space would also be useful. For example, the sample viewer appears to always normalize both supplied and computed tangents. This appears to be in conflict with the MikkTSpace website which states:
|
|
Would it be possible to create a tangent vertex baker that would result in similar tangent and bitangent to the screenspace derivitives that are calculated on the fly (At least visually if not mathematically). This would give the best of both worlds. An accurate normal map baker, with fast fragment calculated tangents and bitangents. Would simple UV based tangent calculation "fit" with fragment calculated derivatives? |
|
Unfortunately I think there would be a couple problems with that:
|
|
Luckily xNormal and Substance both have tangent calculation plugins. I think I'll take a stab at looking at creating a xNormal plugin for triangulated meshes. I'll share my results on whether it works or not. |
|
Something that might be of interest: I've just spent a few days working out how to generate MikkTSpace tangents for a mesh that's already correctly welded (which, if the baker uses MikkTSpace, it should be). I'm not sure how it would perform in JS, but unlike the full algorithm it's linear time and doesn't need extra storage. |
|
Please continue the discussion in #2056. |
Tools like Substance Painter have templates for targetting Unity 5 and Unreal 4, to make sure that the tangent-space normal-map baking uses exactly the same shaders as in these engines (so that the same tangent-basis is used).
What is the correct way to bake normal-maps for GLTF / PBR?
We tried some extreme models, but can't seem to get correct results.
Did anyone succeed in using e.g. Blender + Substance Painter for this?
Thanks,
Peter Verswyvelen
The text was updated successfully, but these errors were encountered: