SubViewPort.get_texture.get_image makes Godot stutter/jitter

[sanderfoobar]

Godot version

4.0.2-stable

System information

Ubuntu 22.04, RTX 3080, Vulkan API 1.3.224 - Forward+, OpenGL API 3.3.0 NVIDIA 525.60.13, X11, AMD 5900x

Issue description

Each of these spikes is exactly 0.2sec apart and caused by the get_image() call in the following line:

# runs every 0.2sec
get_node("%subviewport").get_texture().get_image()

This micro-stutter is noticeable especially when walking around, moving with the mouse (FPS character, etc)

Some things I have noticed:

1. The size of the subviewport seems of no influence
2. The draw mode of the subviewport seems of no influence, for example DEBUG_DRAW_UNSHADED would still stutter
3. I tracked it down to the following call:

godot/drivers/vulkan/rendering_device_vulkan.cpp

Line 2844 in 278fc75

_flush(true);

I patched it with:

if(tex->width != 127)
    _flush(true);

and set my subviewport size width to 127 so it doesn't _flush() for my specific case and the micro-stutter is gone. A flat line:

p.s: that FPS counter is mangohud, e.g: mangohud ./bin/godot.linuxbsd.editor.x86_64

Steps to reproduce

1. Create a 3D scene
2. Create a SubViewport with a camera inside it
3. Get the texture of the subviewport via get_texture()
4. Call get_image() on that texture
5. Observe a stutter

Or download my minimal reproduction project

Minimal reproduction project

stutter_image_data.zip

[lawnjelly]

I'm not super familiar with 4.x or this bit of code, but reading back from GPU to CPU often causes a pipeline stall. The GPU is working on a queue, in a different place, sometimes different frame to the CPU. As soon as you ask to read something back "immediately", the GPU has to finish everything in the queue before the state is correct to read back. This can even mean rendering one or multiple frames. Even requesting a single pixel can cause this.

[Calinou]

A flat line:

That flat line is oddly bumpy. I think the stutter isn't fully resolved here. 😶

This isn't something that can be fully resolved without implementing hardware readbacks, which have 2-3 frames of latency. We should probably implement a way to perform those from a script, but it's not trivial.

[sanderfoobar]

Thanks for the info. My context is the following:

• https://www.youtube.com/watch?v=wYPjKQpLG6g&t=455s
• https://www.youtube.com/watch?v=3Z8hTdD79To&t=241s

These 2 tutorials cover stealth mechanics; detecting if the player is hidden in shadows/darkness. Both tutorials repeatedly call get_image() and then walk the pixels for their luminance value. If the pixel is dark, then the player "is not visible". I am pursuing this method because I don't know another way to establish if the player is visible at runtime (taking into account dynamic lighting).

For the above use-cases the timing is not super important, as detection may happen every 200ms or so. It would perhaps be nice if we could get the image but it doesn't have to be "right now" in favor of not stuttering Godot. I am also wondering if I can solve this by staying within the GPU and measuring pixels via a compute shader.

[smix8]

If it is still the same as with Godot 3 then Viewport.get_texture().get_image() also does a full image copy so the frame stutter gets more intense the higher the rendertarget resolution.

In Godot 3 it was basically impossible to have acceptable frame rate while requiring texture reads on rendertargets for e.g. deformable terrain or other effects that require textures at higher resolution to be detailed enough. Last time I tested in Godot 4 beta it still had all the same issues.

[lawnjelly]

These 2 tutorials cover stealth mechanics; detecting if the player is hidden in shadows/darkness. Both tutorials repeatedly call get_image() and then walk the pixels for their luminance value.

There's a lot of possible ways of detecting shadows in stealth games, I've done it before but am no expert.

I suspect many use smoke and mirrors (e.g. artist placed areas / scripting, because gameplay scripting can be critical), and not what you think. I haven't watched these videos, but reading back from GPU is not a good way of doing this imo. I had a few people ask this for my lightmap module (i.e. just reading the lightmap), but just because your feet are in shadow doesn't mean the rest of your body is.

Some options off the top of my head (whether level is static or dynamic will affect choices):

• Spatial partitioning scheme for shadow / non shadow (manual or automatic)
• Light probes
• CPU ray tracing
• Shadow map stored on CPU and sampled on CPU

[Zireael07]

Yeppers, I saw those pixel-based tutorials but in the end I went with just raycasting and some trigonometry. Works like a charm <3

[sanderfoobar]

Perhaps this micro-stuttering only occurs on my Ubuntu machine (nvidia driver issue?) I tried another OS, same hardware:

• Windows 10 Enterprise LTSC 64-bit (10.0, Build 19044)
• nvidia driver 528.49

And got a pretty stable frametime (with the project included in OP):

I also tried on my laptop (Ubuntu 21, thinkpad X1 Gen9 - random intel GPU) and it felt pretty smooth.

Maybe someone with Linux + official Nvidia drivers + mangohud can try & confirm my test project.

[DigitallyTailored]

As a workaround, it may be possible to first copy the viewport texture to a separate texture, and only retrieve the image data after waiting a few frames so that the queue has time to finish.

I gave this a stab in my project where I'm having this issue and unfortunately the stutter seems to remain.

[nubunto]

just stumbled upon this, in fact simply adding a subviewport to a scene causes it to stutter dramatically.
does not happen on gl_compatibility though.

[Calinou]

just stumbled upon this, in fact simply adding a subviewport to a scene causes it to stutter dramatically.

Can you upload a minimal reproduction project for that? If the SubViewport is redrawing continuously, it's possible that your CPU or GPU can't keep up with the increased demands of rendering two viewports at once instead of just one.