Create SKSurface from GRBackendRenderTarget to show Vulkan 3D graphics

[abenedik]

What would you like to be added

We were able to provide a basic support for Uno platform for Ab4d.SharpEngine - Vulkan based 3D rendering engine. This means that 3D scene that is rendered by Ab4d.SharpEngine can be shown in an Uno platform app (using Skia backend).

Currently, this is done by copying the bytes from a rendered texture to a SKBitmap and then showing this in a class that is derived from SKCanvasElement.

This works but is very inefficient because the rendered texture is copied from the GPU memory to the CPU memory (to SKBitmap) and then Skia sends this back to the GPU to show it in the app.

It would be much better to preserve the rendered texture on the GPU and somehow tell Skia to show that (this is also possible by Avalonia - see AvaloniaUI/Avalonia#9925).

I tried to achieve that by creating a GRBackendRenderTarget object from a Vulkan texture. But when I call SKSurface.Create method and pass the GRBackendRenderTarget, no surface is created.

See the source code for that:
https://github.com/ab4d/Ab4d.SharpEngine.Samples/blob/features/UnoPlatformSharedTexture/Ab4d.SharpEngine.Samples.UnoPlatform/Ab4d.SharpEngine.Samples.UnoPlatform/SharedTextureTestPage.xaml.cs

This can be tested by cloning the "features/UnoPlatformSharedTexture" branch of the Ab4d.SharpEngine.Samples on GitHub (https://github.com/ab4d/Ab4d.SharpEngine.Samples/tree/features/UnoPlatformSharedTexture). After cloning, open the Ab4d.SharpEngine.Samples.UnoPlatform.sln solution file in the Ab4d.SharpEngine.Samples.UnoPlatform folder.

In the SharedTextureTestPage.xaml.cs file there are comments where I was investigating what is going on when the SKSurface.Create method is called. It seems that this calls sk_surface_new_backend_render_target (from SkiaSharp) and then native WrapBackendRenderTarget is called. This then does some validations if the GrBackendRenderTarget is compatible with the GrRecordingContext. I assume that one of this checks returns false and this returns null from the Create method.

Maybe the problem is that I have created an ad-hoc context by calling GRContext.CreateGl(). It would be probably better to use the context of the App, but I do not know if it is possible to get that.

I do not know how to proceed. It would be great if someone with a better knowledge of the Uno platform internals could check this and provide a way to show a shared texture.

This would add Uno platform as another platform that is fully supported by Ab4d.SharpEngine. This would also allow any other third-party rendering engine to show its rendered textures in Uno platfom.

Note:
To test how the integration with SKBitmap works (coping to CPU), start the app by showing the MainPage - uncomment the line 44 and comment line 45 in App.xaml.cs.

Why is this needed

No response

For which platform

Skia (WPF), Skia (Linux X11)

Anything else we need to know?

Note:
Ab4d.SharpEngine is a commercial library. However the next version will allow getting a free license for open-source projects.

[jeromelaban]

Thanks for the report!

In practice, you're using the right integration point with SKCanvasElement, but the reason why the interop is not working is not clear at this point.

Would you be able to provide a sample of this specific scenario using SkiaSharp directly, maybe using a blank WPF app? There may be some creation sequence in Uno that makes it incorrect that we could fix to get this scenario running.

[abenedik]

Based on your info I tried to update the SkiaSharpSample from SkiaSharp repo. I created a custom class that was derived from SKGLElement and there I had the OpenGL backend context. I tried to create the SKSurface from GRBackendRenderTarget that was created from GRVkImageInfo. And this did not work - just as with my Uno platform sample.

However, with a lot of trouble, I was able to debug the native Skia code and there the validate_backend_render_target returns false because the backends are not the same. The following method:

GrGLFormat AsGLFormat(const GrBackendFormat& format) {
    if (format.isValid() && format.backend() == GrBackendApi::kOpenGL) {
        const GrGLBackendFormatData* data = get_and_cast_data(format);
        SkASSERT(data);
        return GrGLFormatFromGLEnum(data->asEnum());
    }
    return GrGLFormat::kUnknown;
}

returns GrGLFormat::kUnknown because** format.backend() != GrBackendApi::kOpenGL. After that the validation fails because the formats are not compatible.

So, it seems to me that it is not possible to create a SkSurfce on OpenGL backend from a GRBackendRenderTarget that is created from a Vulkan Image.

But I have only very basic knowledge of Skia, so maybe there is another way. At least, Avalonia is able to show Vulkan surface - there I pass a shared handle (created by vkGetMemoryWin32HandleKHR on Windows or vkGetMemoryFdKHR on Linux) from a Vulkan image and then Avalonia is able to show the shared texture.

It seems that it is possible to create an OpenGL texture from windows shared handle by calling glImportMemoryWin32HandleEXT. But I do not know how to use that in Skia.

Another possibility that I am thinking about is to create an OpenGL texture (in Skia) and somehow create a shared handle from that. Then pass the shared handle to Ab4d.SharpEngine - I could use that to create a Vulkan texture from the shared handle and then render to that. Again, I do not know how to do that.

[jeromelaban]

Thanks for the update. It may be that the backend used by Avalonia is not using OpenGL but something else, which allows for that specific feature. We intend to provide ways to use other renderers in the future, which may help to get further in having this capability.

[abenedik]

Avalonia is using OpenGL backend by default, see:

But Avalonia app can be also started by using Vulkan backend (Skia build for Vulkan), and that makes sharing textures from Ab4d.SharpEngine even easier.

When you will provide a way to integrate a third-party rendered into Uno Platform, please contact me and I will be glad to test that and help you if needed.