Better outlining for solid geometry

[kring]

I was optimistic that #8646 would be sufficient for outlining solid geometry (e.g. untextured buildings), but it's not quite. It's pretty good (not great) when the volumes are entirely convex, but outlines in concave parts of the model get swallowed. For example:

There should be an outline at the red arrow, but there's not because the outline completely fails the depth test against the two adjacent faces.

I think the best solution to this problem is to bias the depth of the outlines toward the camera so that they are visible. But we can't bias them too much, or outlines on back faces will show through the front faces.

WebGL's built-in polygonOffset doesn't quite work here, for two reasons:

• polygonOffset can only bias triangle depth, not line depth. Biasing triangle depth is not ideal because it can interfere with pickPosition. Biasing outlines can too, but they're a far less important part of the scene.
• Fixed-function polygonOffset doesn't work with log depth, so we have to implement it ourselves. (partially done, perhaps poorly, as part of Fix log depth woes #8600)

Some people advocate using the stencil buffer to render outlines. But no approach I've seen or can think of is able to avoid showing outlines "through" solid geometry when multiple outlined faces are rendered in a single draw call.

So here's my half-baked idea for properly biasing outlines:

• Render a line (i.e. GL_LINES primitive) along the edge of each outlined face. An edge connecting two faces will be rendered twice. One draw call, lots of line primitives.
• Include the face normal as a vertex attribute. This is easy because we need the normals for rendering the faces anyway, and we're likely re-using the same vertex data for the outlines.
• In the fragment shader for an outline:
  • Find the ray to a diagonally-adjacent screen-space pixel, e.g. 1 pixel away from the center of the screen (I think) in both the X and Y directions from the current fragment.
  • Find the intersection of the ray with the plane defined by the fragment eye space position and the eye space face normal.
  • Subtract the Z coordinates of the original fragment eye space position and the ray-plane intersection computed above. This is the change in eye-space depth for a one pixel change along the face. Biasing the depth closer to the camera by this value should guarantee that the outline passes the depth test against fragments from the solid face.

I haven't implemented this or completely convinced myself that it will work, yet.

[kring]

I've implemented the above, and it's pretty good! Except when faces are nearly edge-on with the camera, in which case it's a bit too eager:

In the solid-outlines branch.

[kring]

Also I think it should be possible to do this computation in the vertex shader instead of the fragment shader to improve performance, but I'm going to get it working well in the fragment shader before I try that.

[kring]

Ok after spending way too much time on this, I've gotten it pretty good, but as expected it's impossible to completely avoid artifacts with this approach of rendering separate lines with a depth bias.

For example:

Note the "tails" sticking into the solid geometry, and the generally rubbish look here due to lack of anti-aliasing.

We can fix both problems with a single-pass technique as described in these papers:

Two Methods for Antialiased Wireframe Drawing with Hidden Line Removal
Fast and versatile texture-based wireframe rendering

Both should offer much better performance than rendering GL_LINES in a separate command, and look nicely anti-aliased, too. However, they're both tricky (and less efficient) to implement without geometry shaders, which sadly WebGL lacks.

[kring]

Artifacts go the other way too, where the line isn't solid when viewed on an angle:

(Note that these screenshots are on a 4K monitor and being blown up to a normal DPI, so they do exaggerate the problem somewhat)

[kring]

I'm very interested if anyone has any ideas or knows of any other good papers for outline rendering.