[d3d8] Add shadow perspective divide workaround for Splinter Cell

[AlpyneDreams]

Splinter Cell seems to expect its shaders to do a perspective divide when sampling shadow maps even though it never sets D3DTTFF_PROJECTED. As far I'm aware, this game never worked properly except for on legacy D3D8 drivers for Windows XP with earlier GeForce cards, so I have to assume this was a driver hack. At some point D3D8 drivers or the frontend were updated to match the D3D9 behavior, breaking the game entirely. This document from NVIDIA says that the XYZ coordinates are divided by W (it calls them STR and Q), but never mentions D3DTTFF_PROJECTED.

Without the perspective divide, shadows are broken in some (but not all) scenes rendering the game unplayable as they need to be visible for stealth. To further complicate things, the shader also seems to expect that the second texcoord (texcoord 1) also has perspective divide applied, even though texture 1 is a non-depth light cookie texture so it's not as simple as applying perspective divide for all depth textures.

What this hack does is it will simply force perspective divide for stages 0 and 1 when a depth texture/shadow map is bound to stage 0, and ignore any pleas from the game to set the texture transform flags back to zero while the shadow map is bound. This fixes shadows in the training mission:

I have not been able to find any indicators the game might provide to the driver to enable a hack somehow (e.g. a FOURCC code) and a deep dive on the web has returned nothing. I was not able to find any nonstandard texture stage states passed in, and the shaders don't seem to have any special flags or extensions used. It has been nearly impossible to fully know if the game is doing anything like that because the game crashes on my machine when using renderdoc, apitrace, and d3d8to9 on Linux, so my information is incomplete. Therefore, this hack might not necessarily perfectly emulate what those drivers are doing but it makes the game work.

If someone could get a d3d8 apitrace of the game running with shadows enabled, we might be able to work out a bit more about what's going on here. I have attached a relevant save game file that would need to be traced:

• ShadowyCrates.zip

This PR fixes #4257

[WinterSnowfall]

I've confirmed the shadow maps do look properly now in Splinter Cell. As for Splinter Cell: Pandora Tomorrow (the second game in the series), it does not appear to need this hack at all. Its shadow maps seem fine with just the modified dref scale, which we already apply on master.

I guess we'd need someone to play Pandora Tomorrow more extensively to be sure, but it's not going to be me, as I'm not a fan of stealth games.

In any case, this seems fine for now 👍 .

[mirh]

Could the accompanying sample help? There was also one for opengl (paper), which was then ported back to dx too.
(also I think the xbox guys may have had similar doubts? and I found some old snippet about that)

[WinterSnowfall]

Could the accompanying sample help? There was also one for opengl (paper), which was then ported back to dx too.

I don't see any magic flags/states used anywhere in those samples, so I'm afraid it doesn't add anything to what we already know.

Edit: @AlpyneDreams might be noteworthy to remark that the sample is setting:

	d3dDevice->SetTextureStageState(0, D3DTSS_TEXTURETRANSFORMFLAGS,
									D3DTTFF_COUNT4 | D3DTTFF_PROJECTED);

As one would expect. It's still odd Splinter Cell doesn't use D3DTTFF_PROJECTED.

(also I think the xbox guys may have had similar doubts? and I found some old snippet about that)

Neither seem related to Nvidia (hardware) shadow maps though, which is really the topic at hand here.

[AlpyneDreams]

I believe the shadows are a direct port of the Xbox version and that's why they only worked on very specific hardware that was similar enough to the Xbox at the time.

[AlpyneDreams]

@WinterSnowfall Specifically, the game's readme says this:

Why does Splinter Cell have a special mode for NV2x/NV3x graphic chips?

Splinter Cell was originally developed on XBOX™. Features only available on NV2x chips
were used and it was decided to port them to the PC version even if these chips would be the
only one able to support them. Considering the lighting system of XBOX™ was well validated,
it was easy to keep that system intact.

Splinter Cell Dynamic lighting system

Splinter Cell shadow system is a major part of the game. On NV2x/NV3x hardware, it runs
using a technique called Shadow Buffers. This technique is rendering the scene from every
shadow casting light and store a depth buffer that represent each pixel viewed by this light
source. Each pixel has an X, Y, Z coordinate in the light system and these coordinates can be
transformed, per pixel, in the viewer coordinate system. It’s then easy to compare with the actual
depth stored in the Z buffer to figure out if the pixel viewed by the camera is the same or is
occluded by the pixel viewed by the light. If they are the same, it means the pixel is lighted, if
the light pixel is in front of the viewer pixel, it means the pixel is in the shadow. On all other
current hardware, the game is using another technique called projected shadows

[mirh]

Mhh so.. It appears indeed like dege also reported some kind of baffling situation, even though in the following page he claimed to have discovered some whatever proper quirk.
I even tried to check UD3DRenderDevice::SupportsShadowBuffer in Ghidra, but I have hardly ever seen a more badly decompiled function (not gonna lie though, I saw a 0x200 value in the first conditional which is the value for D3DFMT_D24S8 and I lighted up).

https://developer.download.nvidia.com/GPU_Programming_Guide/GPU_Programming_Guide.pdf#page=53
Holy fettucini baloney, I think I have scored.

[WinterSnowfall]

I think I have scored.

Early NVIDIA drivers (version 45.23 and earlier) implicitly assumed that shadow maps were to be projected. This behavior changed with NVIDIA drivers 52.16 and later—programmers now need to explicitly set the appropriate texture stage flags.

Similarly, when using the ps1.1-1.3 shader models, drivers version 52.16 and later now require that the projective flag is explicitly set for the texture-stage sampling the shadow-map (for example, SetTextureStageState(0, D3DTSS_TEXTURETRANSFORMFLAGS, D3DTTFF_PROJECTED).

Well that settles it, yes. It was a driver hack, one that was most likely deprecated between the two games, which is why Pandora Tomorrow doesn't appear to need it. Nice find.

@AlpyneDreams Might be worth updating the comments to match these findings, so that we don't forget why the hack is needed. Otherwise, it looks even more justified now.