Pass pointer address of texture instead of colors32

[elblogbruno]

I believe the repo is awesome, but having some FPS or milliseconds improvement is a good thing to develop
So , one thing in my mind for some time is that in unity you can get the texture image pointer address and I ask if it could be passed directly,so mediapipe access it instead of Colors32 that is a very expensive operation. I simply had the idea don't know how to implement it tough.

Thanks and great job.
Bruno

[elblogbruno]

@homuler here there's more info about how to get the pointer adress https://docs.unity3d.com/ScriptReference/Texture.GetNativeTexturePtr.html

[homuler]

@elblogbruno Thank you for reporting the issue.

Currently, DemoGraph#PushInput copies GPU buffer to ImageFrame (CPU memory) and copies ImageFrame data to GPU buffer again. The good thing is that it needs less native APIs (so I could implement fast), but those operations are unnecessary and expensive too as you pointed out.

I'll change the code to pass the GL texture name to MediaPipe directly, but it looks difficult to do with the current architecture (because Unity has to know when to release which GL texture), so it may take time.

[elblogbruno]

Glad my comment was useful for you and that it can be implemented in the near future!
I believe in the URL I had passed previously, there are algorithm examples.

Cordially
Bruno

[homuler]

Finally...we can now pass the texture's native pointer (or texture name) to MediaPipe (OpenGL ES only).

MediaPipeUnityPlugin/Assets/MediaPipe/Examples/Scripts/DemoGraph.cs

Lines 54 to 63 in 18e00a8

	var glTextureName = textureFrame.GetNativeTexturePtr();
	return gpuHelper.RunInGlContext(() => {
	var glContext = GlContext.GetCurrent();
	var glTextureBuffer = new GlTextureBuffer((UInt32)glTextureName, textureFrame.width, textureFrame.height,
	textureFrame.gpuBufferformat, textureFrame.OnRelease, glContext);
	var gpuBuffer = new GpuBuffer(glTextureBuffer);
	return graph.AddPacketToInputStream(inputStream, new GpuBufferPacket(gpuBuffer, timestamp));
	});

We can also ask MediaPipe to render the output to a Unity Texture directly (cf. OfficialDemoGPU)

But in fact OfficialDemoGPU does not perform so well as the official APK, I'll continue to investigate to improve the performance.

Memo (performance tips)

• Build a GpuBuffer object from the input texture's name and pass it to MediaPipe (OpenGL ES only)
• Use CalculatorGraph#ObserveOutputStream and get output packets asynchronously (cf. OfficialDemoCPU)

[elblogbruno]

I can't wait to try it! Glad the idea was able to be implemented, have you noticed improvement?
Thanks for all!
Great job

[elblogbruno]

I am getting better performance (50-60fps 10ms) on OfficialDemoGPU than hand graph

Record_2020-12-31-11-10-11_34f41116383684cf98aa807a41aeab0b.mp4