add defines to deal with GL vs. DX v-coordinate direction

[mbechard]

There is probably another nicer way to fix this, but when implementing this
algorithm in GL in my engine it wasn't working correctly.
It seemed that adjusting for the fact that +V is up in GL vs.
+V is down in DX could fix the issue.
Doing A/B comparisons with the sample app and my GL app along the way, it
did indeed fix the issue.
Maybe flipping the Area/Search textures and flipping my inputs could
also fix it, but doing it this way keeps all the inputs and outputs
consistent with the existing engine code.

[iryoku]

Thanks for the patch, it is a very good find!

I wonder if flipping the texcoord.y coordinate in the vertex shader could achieve the same results? If you can try this out, I could add a comment to documentation for OpenGL users. Otherwise I will merge your patch.

[mbechard]

It may partially work but I think we’d still need to invert the coordinates for the Area/Search map since I assume those are 0-1 lookups. Also it would result in the output from each stage flipping the output vertically making debugging/integration quite confusing.

From: Jorge Jimenez [mailto:[email protected]]
Sent: Saturday, February 28, 2015 7:58 AM
To: iryoku/smaa
Cc: mbechard
Subject: Re: [smaa] add defines to deal with GL vs. DX v-coordinate direction (#5)

Thanks for the patch, it is a very good find!

I wonder if flipping the texcoord.y coordinate in the vertex shader could achieve the same results? If you can try this out, I could add a comment to documentation for OpenGL users. Otherwise I will merge your patch.

—
Reply to this email directly or view it on GitHub #5 (comment) . https://github.com/notifications/beacon/ADbBQcG7ZkBi_mcbF6CPjjQcNf2AsgQCks5nwbK_gaJpZM4DnVYp.gif

[biller23]

Or maybe we could transpose the "search" tex and add the vertical flipping only in SMAASearchLength() before the return statement (and the mad). This is the method I used so far, adding "e.g = -e.g" in the end of SMAASearchLength().
The "search" tex is the smaller of the two.

[mbechard]

Does that work? The first thing I noticed when I implemented this was that the edge texture was wrong since its looking for up/left pixels but it was instead getting down pixels due to the V difference and its offset. Maybe it all works out in the end but I wasn't familiar enough with the algorithm to know.

[biller23]

This old SMAA-OpenGL demo (https://github.com/scrawl/smaa-opengl) use this technique.

[mbechard]

What about the area texture? I found I had to invert the lookup into that as well.

[biller23]

It's strange I know, but I left the area texture as it is. It's possible that I'm doing something wrong.
I'm using SMAA 1X, I don't know if it works in other technique.

I made the change in this fork if you want to test if it really works or I'm doing something wrong: https://github.com/biller23/smaa

There are 2 commits (master...biller23:master):

1. I added searchTexBytesGL[] in SearchTex.h
2. I updated the shader func SMAASearchLength()

screenshot : http://i.imgur.com/Ha1Lfyn.png

EDIT:
Ok, maybe I'm doing something wrong.
Even without my modification (texture and shader) the code seems to work!?
I'm a little confused now :)

[mbechard]

Hey, I just tried those two modifications in my code and it didn't give the correct results. I do notice in your opengl example the width/height of the search tex has changed so maybe there is another difference there that isn't accounted for in those 2 changes?

[biller23]

I dont use https://github.com/scrawl/smaa-opengl (this demo use old textures), I use the latest SMAA.hlsl shader and latest AreaTex.h\SearchTex.h textures (https://github.com/iryoku/smaa), and strangely everything seems to work right.

Don't ask me why, I really cannot find the answer :)
Previously I was using the modification I made, only now I discovered that I can use SMAA out of the box...

Search texture loading:
internalFormat : GL_COMPRESSED_RED_RGTC1
w, h : SEARCHTEX_WIDTH, SEARCHTEX_HEIGHT
format : GL_RED
type : GL_UNSIGNED_BYTE;

Area texture loading:
internalFormat : GL_COMPRESSED_RG_RGTC2
w, h : AREATEX_WIDTH, AREATEX_HEIGHT
format : GL_RG
type: GL_UNSIGNED_BYTE

This is a comparison http://imgur.com/a/cfLDs

[mbechard]

Hey, take a look at your source image put through my solution:
http://i.imgur.com/p3RcxaR.jpg
Notice on his right index finger where its touching the lantern handle, it its smoother in my version than your example file. So I think some edges are being missed without this patch.

(Editd to remove mention of .jpg as I realize the imgur files were png.)

[biller23]

Yes, maybe it's smoother (but I can see random discontinuities in different places anyway).
I used SMAA x1 with ULTRA settings and SMAAColorEdgeDetectionPS().

When you said "it wasn't working correctly" I thought it was more broken...

[mbechard]

Ah, here is a better comparison then. My previous one was SMAA x1 on High with LuminanceEdge. Here it is with Ultra using ColorEdge:
http://imgur.com/KBdC3kJ

Really the only thing that matters though is getting identical results to the reference algorithm. I'll check on Monday what it gives for this image.

[mbechard]

Here's a screencap of the image sent through the reference application:
http://imgur.com/xZpphCO

[biller23]

My bad :)

[mbechard]

No problem, thanks for the alternative to try out.

[iryoku]

Sorry for the delay, I've been out of town for a while. Thanks for sharing these tests!

Comparing biller23 image:
http://imgur.com/a/cfLDs

With reference images from mbechard:
http://imgur.com/xZpphCO

It looks to me like the differences cannot be explained by v-coordinate mistmatches (otherwise all vertical lines should be broken). It looks more like a different SMAA configuration (most likely threshold difference). Is the reference demo screenshot from mbechard using the same SMAA version as the OpenGL version from biller23?

Thinking about this, I think it should work regardless of the v-coodinate direction, as long as all passes sample in the same way. Not sure why previous versions didn't worked out of the box.

[mbechard]

Using the code untouched I get something similar from the edge pass except it is detecting edges below instead of above which makes sense. However the blend weights pass ends up returning mostly 0 for everything. Things only started to work when I started to apply the inverts.

[iryoku]

That is using latest version as well?

Could we share the shaders we are all using (SMAA and HLSL caller code) to try to discover what's going on? It's strange that it works out of the box for biller23.

[mbechard]

Ya I did the git clone just a few weeks ago. The source I'm using is the source posted in my merge request. At the very least it does seem like the area and/or search texture lookups would need to be inverted since a input lookup value of 0 will result in different results in DX vs. GL.
I don't doubt there is a more minimal solution than my suggestion, but as far as I can tell this solution gives identical results to the reference code in DX.

[clusty]

Hey guys,

Sorry to barge in, but I am experiencing some issues on OS X with OpenGL.
Some of my extracted edges are off by one resulting in some really bad antialiasing results. Using mbechard, patch just resulted in shifting of the pattern. Here are 2 screenshots of the original image as well as the extracted edges:
https://www.dropbox.com/sh/36myqp26au7ey51/AADjmIj2hL98xmDOVR6QRlHMa?dl=0

I could also post the code since I had to further tweak it a little since I am stuck using GLSL 1.2
Any help is welcomed.

[turol]

We've created a new OpenGL demo which uses SMAA: https://github.com/turol/smaaDemo . It uses more modern OpenGL than the previous one and includes both image loader and its own test scene.

We also tried to port it to WebGL so it could be demoed in a browser but GLSL ES defeated us for now.

The demo uses @mbechard's fixed shader and also flips search and area textures. There was no way that I could find to make it work with just one of those changes.

This demo also has debug output which looks identical to D3D version. It should be useful for debugging.

[turol]

There's now a new version of my SMAA demo with the following features:

• Vulkan renderer in addition to OpenGL
• Y coordinate convention and shading language have been separated from each other
• Depth edge detection has been fixed on OpenGL, possibly also predicated tresholding but that is untested