Reduce shadow update frequency for distant real-time shadows

[Calinou]

Related to #2744, which is about implementing a LOD system for lights. This proposal is complementary and can be implemented independently.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

Real-time light shadows are always updated at full frequency (once every frame), even when there's no good reason to do so. While Godot caches shadow maps that don't move over frames, it still has to perform this caching check every frame, which has a cost.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

reduz proposed that we update distant shadows less frequently to improve performance. Since distant shadows occupy a lower portion of the screen and are less detailed, the lower update frequency will be less noticeable.

This can be done with DirectionalLight3D, but likely also with OmniLight3D and SpotLight3D as well.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

For distant directional shadows, we can update the distant splits less often. For instance, when using PSSM 4 Splits (the default), we can:

• Update the first split every frame.
• Update the second split every 2 frames.
• Update the third split every 3 frames.
• Update the fourth and last split every 4 frames.

For PSSM 2 Splits (a more low-end/mobile-friendly option), we can:

• Update the first split every frame.
• Update the second split every 2 frames.

For OmniLights and SpotLights, we can use a heuristic based on the distance, the light's radius and the camera FOV. Lights covering a small percentage of the screen could have their shadows updated less often.

We can expose those update frequencies in the Project Settings, so that these can be adjusted depending on users's needs. For instance, on very low-end setups, you may wish to force all shadows to update only once every 2 frames. Inversely, if you can spare the processing power and want to keep the current behavior, you could set all update frequencies to happen every frame as before.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, as shadow rendering mechanisms are part of the renderer.

Is there a reason why this should be core and not an add-on in the asset library?

This is core rendering functionality.

[m4nu3lf]

I would also provide a way to reduce the highest framerate for shadowmaps to be a fraction of the current framerate. I modified Godot 3 so that all OmniLights and SpotLight update code is only run once every other frame and I can't notice the delay whilst getting a huge speedup with lots of lights.

[Zireael07]

@m4nu3lf: Can you please make a PR?

[m4nu3lf]

@m4nu3lf: Can you please make a PR?

I could but the change needs work (and testing) because as of now it is too hacky.

@Calinou has anybody started working on this feature? Is it meant for Godot 3 too?

[Calinou]

@Calinou has anybody started working on this feature? Is it meant for Godot 3 too?

Not that I know of. Feel free to open a PR for both master and 3.x 🙂

I wouldn't make half-framerate updating the default for omni and spot lights (except on mobile perhaps), but it's interesting to have as an option.

[m4nu3lf]

After lots of testing, I think implementing the feature I proposed is not worth it as fast-moving lights close to the camera can create quite annoying artifacts. Instead, it's better first to implement the proper LOD solution described in this issue. If this is not enough, we could implement a per-light option to update it at a fraction of the framerate.

[Calinou]

@m4nu3lf Could you upload your work in a branch somewhere? It'd be interesting to use a base nonetheless. Also, if someone really needs this in their project right now, they could make use of the feature you implemented. Thanks in advance 🙂

[m4nu3lf]

@Calinou I think I don't have the code anymore, but given it was about three lines of code, I can easily describe it.
I didn't implement the settings part. It was hardcoded in the visual server.
I just added a boolean flag to the class and here I was flipping it, and if the value was true, I was skipping the block. That's basically it.

[Calinou]

@Calinou I think I don't have the code anymore, but given it was about three lines of code, I can easily describe it. I didn't implement the settings part. It was hardcoded in the visual server. I just added a boolean flag to the class and here I was flipping it, and if the value was true, I was skipping the block. That's basically it.

Thanks for the pointers! I have an initial implementation for 3.x here: https://github.com/Calinou/godot/tree/shadow-add-frameskip-option-3.x

I couldn't figure out how to update DirectionalLight shadows less frequently yet. Based on my testing (with broken rendering), it could speed up shadow rendering very significantly if you use a day-night cycle (which is usually quite slow).

[m4nu3lf]

@Calinou I had a look at the code. I think that the problem with directional shadow map is that their transform is updated elsewhere. And even if you skip the rendering block, their matrix will move with the camera causing incorrect results.
If you look here:
https://github.com/godotengine/godot/blob/3.x/servers/visual/visual_server_scene.cpp#L2382
and you follow the function here:
https://github.com/godotengine/godot/blob/3.x/drivers/gles3/rasterizer_scene_gles3.cpp#L4481
and again here:
https://github.com/godotengine/godot/blob/3.x/drivers/gles3/rasterizer_scene_gles3.cpp#L2685
you get to the place where the matrix is updated.
We should skip that as well somehow. But I think it might take some refactoring to get there.
Hope this helps!

[m4nu3lf]

Hum, on the other hand, it seems like that transform comes from the light structure and it is updated by this call here:
https://github.com/godotengine/godot/blob/3.x/servers/visual/visual_server_scene.cpp#L2674.
What happens if you just skip the directional light update block above the one for spot/omni lights?

[Calinou]

What happens if you just skip the directional light update block above the one for spot/omni lights?

I tried skipping out this entire block, but it caused flickering: https://github.com/godotengine/godot/blob/3.x/servers/visual/visual_server_scene.cpp#L2644-L2676

[m4nu3lf]

I wonder if there is some other dependence on the camera matrix. I'll give it another look soon.

[m4nu3lf]

@Calinou It turns out the issue with directional lights' shadow maps was a pretty dumb one. Here is the solution
If we don't populate the list inside the block every frame with the directional lights with shadows, the rasterizer won't renderer them at all for the given frame, causing the flickering.

[Ansraer]

Just wanted to point out that the proposed scheduling won't really work. To quickly recap:

Update the first split every frame.
Update the second split every 2 frames.
Update the third split every 3 frames.
Update the fourth and last split every 4 frames.

With this, we would have the following:

Split 1
Split 1, Split 2
Split 1, Split 3
Split 1, Split 2, Split 4
Split 1
Split 1, Split 2, Split 3
Split 1
Split 1, Split 2, Split 4
Split 1, Split 3
Split 1, Split 2
Split 1
Split 1, Split 2, Split 3, Split 4
...

Which is less than ideal. Our frame time has become inconsistent, while the worst frames are still just as slow. One of the following would be better:

Option A	Option B
Split 1, Split 4	Split 1, Split 2
Split 2, Split 3	Split 1, Split 3
Split 1, Split 4	Split 1, Split 2
Split 2, Split 3	Split 1, Split 4

[jitspoe]

I could really use this with omni and spot lights! Ideally, there would be some sort of configurable (like a quality setting that could be exposed to players) for number of updates per frame, and then a priority system would be applied to determine which ones to update that frame. I had to do something similar for my enemy movement (hopefully a temp fix until physics performance issues are fixed) and went with something like this:

	var priority := delta
	var player_to_body := player_pos.direction_to(body.global_position)
	var dot := player_look_dir.dot(player_to_body)
	var distance := player_pos.distance_to(body.global_position)
	var dir_priority := 2.0 + dot
	priority *= 1.0 / distance
	priority *= dir_priority
	body.update_priority += priority

So basically the priority numbers keep accumulating every frame, and stuff that's closer and in the look direction of the player gets a higher priority number added each frame so they update more frequently. It's a bit more complicated in the case of lights, as they could be behind the player, or you could have a far away light with a large radius casting big shadows, but I think a simple implementation could go a long way.

[jitspoe]

After digging into things a bit for this issue godotengine/godot#97472, I think there are a few key things we could do to help performance:

1. Avoid atlas re-allocating. Currently shadows try to find the "best" atlas location, which often results in other shadows being bumped out of their best location, then we're bouncing back and forth even on a completely static scene.
2. Limit number of updates per frame (basically this proposal). This could have a lot of artifacts with self-shadowing, but may not be noticed on distant objects.
3. Pre-bake static geometry into its own buffer (basically a second atlas) and only render dynamic stuff then merge buffers.
4. Limit bounding boxes of omni lights based on static geometry / occlusion, so for example here, if A moves, we'd update the shadows, but no need to update them for B be cause it's behind a wall and would be in shadow already. We could limit the bounds to the green rectangle:
   
5. A bit more involved, but if there were some way to bake a static lightmap per light then cast shadows from just realtime stuff that cancel out that lightmap, that would probably be the most optimal method.
6. Not all 6 shadows of the omni light need to update every frame. For example, if we have moving models in the lower left here, we don't need to update the shadows in the upper right:
   

[clayjohn]

@jitspoe See godotengine/godot#77683. A lot of what you cover comes naturally when splitting between dynamic and static shadows. That being said, with rendering shadows you walk a fine line when you try to bookkeep too much. If we have to do pairing and culling on a per-face basis for the light, then you will quickly find that the cost of culling far outweighs the cost of just rendering the extra shadows to the shadowmap.

I think that two things will make the biggest difference and will make any other optimization so small they son't be worth doing:

1. Splitting dynamic objects from static objects
2. improving our atlasing strategy so there is less "stealing" going on and the shadows are more cooperative with each other.

Once we do these I think the others won't be worth the effort.