TextureFramePool issue when using Vulkan on Android breaks graph

[NathanielEssenberg]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.04):
• Target Platform: Android
• Mobile Device (if the target is Android or iOS): Android 11
• MediaPipeUnityPlugin version or commit id: 0.8.6
• Bazel version:
• GCC/G++ version:
• Android NDK version (if the target platform is Android): 21
• Xcode version (if the target platform is iOS):

Describe the current behavior
The demo scene works fine on Android (and in Editor), when built from the original demo project in the repo. However, when switching renderer to Vulkan, the detection does not work.
The same is true for my own project, in which I only import MediaPipeUnity as a package.
It seems to me like there is an issue with the camera textures, maybe on the native plugin side?

Describe the expected behavior
I hope Vulkan can be supported for this project :)

Steps to reproduce the issue
Build the desktop demo scene for Android using the Vulkan Graphics API :)

Full logs
This is the interesting part (from logcat):

08-13 08:48:30.664 17748 17775 I Unity : Graph Changed: Pose Tracking
08-13 08:48:30.664 17748 17775 I Unity : GraphSelectorController:OnValueChanged(Dropdown)
08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.Events.UnityAction1:Invoke(T0) 08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.Events.UnityEvent1:Invoke(T0)
08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.UI.Dropdown:OnSelectItem(Toggle)
08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.Events.UnityAction1:Invoke(T0) 08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.Events.UnityEvent1:Invoke(T0)
08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.EventSystems.EventFunction1:Invoke(T1, BaseEventData) 08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.EventSystems.ExecuteEvents:Execute(GameObject, BaseEventData, EventFunction1)
08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.EventSystems.StandaloneInputModule:ProcessTouchPress(PointerEventData, Boolean, Boolean)
08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.EventSystems.StandaloneInputModule:ProcessTouchEvents()
08-13 08:48:30.664 17748 17775 I Unity : UnityEngine.EventSystems.StandaloneInputModule:Process()
08-13 08:48:30.664 17748 17775 I Unity :
08-13 08:48:30.665 17748 17775 I Unity : Gpu resources are already initialized
08-13 08:48:30.665 17748 17775 I Unity : SceneDirector:StartGraph()
08-13 08:48:30.665 17748 17775 I Unity : SceneDirector:ChangeGraph(GameObject)
08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.Events.UnityAction1:Invoke(T0) 08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.Events.UnityEvent1:Invoke(T0)
08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.UI.Dropdown:OnSelectItem(Toggle)
08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.Events.UnityAction1:Invoke(T0) 08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.Events.UnityEvent1:Invoke(T0)
08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.EventSystems.EventFunction1:Invoke(T1, BaseEventData) 08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.EventSystems.ExecuteEvents:Execute(GameObject, BaseEventData, EventFunction1)
08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.EventSystems.StandaloneInputModule:ProcessTouchPress(PointerEventData, Boolean, Boolean)
08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.EventSystems.StandaloneInputModule:ProcessTouchEvents()
08-13 08:48:30.665 17748 17775 I Unity : UnityEngine.EventSystems.StandaloneInputModule:Process()
08-13 08:48:30.665 17748 17775 I Unity :
08-13 08:48:30.760 17748 17775 I Unity : pose_detection is saved to /storage/emulated/0/Android/data/com.DefaultCompany.MediaPipeUnityPlugin/files/Cache/pose_detection.bytes (length=2961104)
08-13 08:48:30.760 17748 17775 I Unity : Mediapipe.AssetBundleManager:WriteCacheFile(TextAsset, String, Boolean)
08-13 08:48:30.760 17748 17775 I Unity : PoseTrackingGraph:PrepareDependentAssets()
08-13 08:48:30.760 17748 17775 I Unity : DemoGraph:Initialize()
08-13 08:48:30.760 17748 17775 I Unity : DemoGraph:Initialize(GpuResources, GlCalculatorHelper)
08-13 08:48:30.760 17748 17775 I Unity : d__24:MoveNext()
08-13 08:48:30.760 17748 17775 I Unity : UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
08-13 08:48:30.760 17748 17775 I Unity :
08-13 08:48:30.802 17748 17775 I Unity : pose_landmark_full is saved to /storage/emulated/0/Android/data/com.DefaultCompany.MediaPipeUnityPlugin/files/Cache/pose_landmark_full.bytes (length=6431504)
08-13 08:48:30.802 17748 17775 I Unity : Mediapipe.AssetBundleManager:WriteCacheFile(TextAsset, String, Boolean)
08-13 08:48:30.802 17748 17775 I Unity : DemoGraph:Initialize()
08-13 08:48:30.802 17748 17775 I Unity : DemoGraph:Initialize(GpuResources, GlCalculatorHelper)
08-13 08:48:30.802 17748 17775 I Unity : d__24:MoveNext()
08-13 08:48:30.802 17748 17775 I Unity : UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
08-13 08:48:30.802 17748 17775 I Unity :
08-13 08:48:30.802 17748 17775 I Unity : Loaded dependent assets
08-13 08:48:30.802 17748 17775 I Unity : DemoGraph:Initialize()
08-13 08:48:30.802 17748 17775 I Unity : DemoGraph:Initialize(GpuResources, GlCalculatorHelper)
08-13 08:48:30.802 17748 17775 I Unity : d__24:MoveNext()
08-13 08:48:30.802 17748 17775 I Unity : UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
08-13 08:48:30.802 17748 17775 I Unity :
08-13 08:48:36.523 17748 17928 W Unity : The released texture does not belong to the pool
08-13 08:48:36.523 17748 17928 W Unity : TextureFramePool:OnTextureFrameRelease(UInt64, IntPtr)
08-13 08:48:36.523 17748 17928 W Unity :
08-13 08:48:36.613 17748 17928 W Unity : The released texture does not belong to the pool
08-13 08:48:36.613 17748 17928 W Unity : TextureFramePool:OnTextureFrameRelease(UInt64, IntPtr)
08-13 08:48:36.613 17748 17928 W Unity :
08-13 08:48:36.683 17748 17928 W Unity : The released texture does not belong to the pool
08-13 08:48:36.683 17748 17928 W Unity : TextureFramePool:OnTextureFrameRelease(UInt64, IntPtr)
08-13 08:48:36.683 17748 17928 W Unity :
08-13 08:48:36.748 17748 17928 W Unity : The released texture does not belong to the pool
08-13 08:48:36.748 17748 17928 W Unity : TextureFramePool:OnTextureFrameRelease(UInt64, IntPtr)
08-13 08:48:36.748 17748 17928 W Unity :
08-13 08:48:36.814 17748 17928 W Unity : The released texture does not belong to the pool
08-13 08:48:36.814 17748 17928 W Unity : TextureFramePool:OnTextureFrameRelease(UInt64, IntPtr)

Additional context
Add any other context about the problem here.

[homuler]

The same is true for my own project, in which I only import MediaPipeUnity as a package.
It seems to me like there is an issue with the camera textures, maybe on the native plugin side?

I think the plugin code itself (source files under Packages/com.github.homuler.mediapipe) works on Android with Vulkan, but sample scenes does not work as you pointed out.

This is mainly because DemoGraph tries to pass native texture (e.g. OpenGL ES texture) directly to MediaPipe on Android.
However, MediaPipe does not support Vulkan API, so when you use Vulkan, you need to copy pixel data to CPU first and then pass it to MediaPipe (this is how the sample scenes work on UnityEditor).

Unfortunately, I don't have enough time to fix this issue soon.
To circumvent it, please remove code blocks that are compiled only for Android and make your app run the same code as it runs on UnityEditor.

For example, this code block should be removed.

MediaPipeUnityPlugin/Assets/Mediapipe/Samples/Scripts/DemoGraph.cs

Lines 76 to 85 in a754416

	#if UNITY_ANDROID && !UNITY_EDITOR
	if (IsGpuEnabled()) {
	lock (frameLock) {
	currentTextureFrame = textureFrame;
	currentTextureName = textureFrame.GetNativeTexturePtr();
	return gpuHelper.RunInGlContext(PushInputInGlContext);
	}
	}
	#endif

[NathanielEssenberg]

I understand, thank you :).

I have a question, in this case:
I would like to try changing some of the MediaPipe source files. I see that Bazel pulls the MediaPipe source directly from the Google repo.
For example, I am trying to modify some code in the mediapipe gpu folder, so I pulled the folder and placed it as a subfolder in mediapipe_api/gpu. I am now trying to figure out which paths to modify in the BUILD files, and how. Unfortunately I am new to Bazel...

Do you think it is okay to try this, or should I try to replace MediaPipe completely instead of just one folder? Maybe it is easier to replace MediaPipe completely instead of a single folder?

[homuler]

I am now trying to figure out which paths to modify in the BUILD files, and how.

If you created subdirectories, then probably each directories should have BUILD file.
Each BUILD file contains build targets and when you build them, they generate object files.
To include those object files into native libraries (e.g. libmediapipe_c.so), please update below deps.

MediaPipeUnityPlugin/mediapipe_api/BUILD

Lines 4 to 44 in b8ea5c1

	cc_library(
	name = "mediapipe_c",
	deps = [
	":calculators",
	"//mediapipe_api/external:glog",
	"//mediapipe_api/external:protobuf",
	"//mediapipe_api/external:stdlib",
	"//mediapipe_api/framework:calculator",
	"//mediapipe_api/framework:calculator_graph",
	"//mediapipe_api/framework:output_stream_poller",
	"//mediapipe_api/framework:timestamp",
	"//mediapipe_api/framework/formats:classification",
	"//mediapipe_api/framework/formats:detection",
	"//mediapipe_api/framework/formats:image_frame",
	"//mediapipe_api/framework/formats:landmark",
	"//mediapipe_api/framework/formats:rect",
	"//mediapipe_api/framework/port:logging",
	"//mediapipe_api/graphs/object_detection_3d/calculators:model_matrix",
	"//mediapipe_api/modules/face_geometry/protos:face_geometry",
	"//mediapipe_api/util:resource_util",
	] + select({
	"@com_google_mediapipe//mediapipe/gpu:disable_gpu": [],
	"//conditions:default": [
	"//mediapipe_api/gpu:gl_base",
	"//mediapipe_api/gpu:gl_context",
	"//mediapipe_api/gpu:gl_calculator_helper",
	"//mediapipe_api/gpu:gl_texture_buffer",
	"//mediapipe_api/gpu:gpu_buffer",
	"//mediapipe_api/gpu:gpu_buffer_format",
	"//mediapipe_api/gpu:gpu_shared_data_internal",
	],
	}) + select({
	"@com_google_mediapipe//mediapipe/gpu:disable_gpu": [],
	"@com_google_mediapipe//mediapipe:ios": [],
	"//conditions:default": [
	"//mediapipe_api/gpu:egl_surface_holder",
	],
	}),
	visibility = ["//visibility:public"],
	alwayslink = True,
	)

Do you think it is okay to try this, or should I try to replace MediaPipe completely instead of just one folder? Maybe it is easier to replace MediaPipe completely instead of a single folder?

There are 2 ways to achieve this.

1. Fork mediapipe and pull it in place of the original repository.
2. Create files under mediapipe_api and apply patch files if necessary (just like this plugin does).

If you'd like to make a significant change, it'd be easier to fork the repository (1).
However, if you'd only like to add code and not to change existing code, I think it's better not to change mediapipe itself (2), because it's not easy to maintain the forked repository (and in addition, you may also need to change this plugin's code).

You can see how this plugin modifies existing features here.

[homuler]

Fixed by #296