Pose tracking latency with versions higher than 0.8.4 mediapipe

[ROBYER1]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.04): Android 10, IOS 14, Windows 10
• Target Platform: Android/IOS/Windows Unity Editor
• Mobile Device (if the target is Android or iOS): iPhone 7, iPad 10.2 (2020), iPad Pro (2020), Xperia XZ3, various windows 10 PCs
• MediaPipeUnityPlugin version or commit id: Any commit after 24576ce

Describe the current behavior
Testing across various devices, we have found that the pose tracking has latency on all devices after a short period of time when using any commit higher than 24576ce when we build Mediapipe 0.8.5 and above, mainly on 0.8.6 where 3d pose points were added to the pose graph. We tested in editor on high end windows pc, on android phones and IOS devices where the latency would always sneak in after a while and the pose would be very out of sync. Others mentioned it here and we also found it with the Holistic graph too

When we roll back to 0.8.4, the latency is good again and the sample can run for hours on mobiles and desktops.

Describe the expected behavior

Mediapipe 0.8.6 pose detection should run without latency as it does on 0.8.4, if this is an issue with the actual Mediapipe plugin and not your implementation, I have already raised an issue with Google here

Steps to reproduce the issue
I would recommend to test Mediapipe 0.8.6 from here a754416
And test Mediapipe 0.8.4 from here 24576ce

Build mediapipe then build and run the sample, set to Pose tracking in Unity Editor on Windows 10/Android 10 device/IOS device and have it run for longer than a minute, eventually you should see something like this other person's recording we found, the pose tracking becomes out of sync, however we do not get flickering like that in our tests, just the latency is very bad.

[LGuilhermeRodrigues]

I've uploaded an unlisted video here, maybe it can help.

I think I had the same issue, it was clear that it was happening in the build for windows

While in the unity editor and android it seemed that the delay was smaller even if I keep the app opened for 7 minutes

System information:

• OS Platform and Distribution: Windows 11 i9 CPU
• Target Platform: Windows (CPU) / Windows Unity Editor (CPU) / Android 64 (GPU)
• Mobile Device: Samsung Galaxy Note 10+
• MediaPipeUnityPlugin: chore(release): 0.6.2

[homuler]

I pushed a new commit (d58ab72).
The flickering problem (#219) may be getting bigger on some devices, but I think the latency has been improved.

[ROBYER1]

Terrific work, thanks homuler, I will close this issue now. For a current project I am sticking with 0.8.4 as in the end we didn't need the 3d pose points but this is a relief to see if I need to use more recent versions of mediapipe.

[ROBYER1]

Just tested out the push, rebuilt and the flickering is very bad now despite the latency being fixed. Seems almost unusable - going to stick with 0.8.4 Mediapipe for now as it actually works good despite not having the fancy 3d landmark points

[homuler]

In fact, we need to wait for a few frames here to ensure the input image has been processed

MediaPipeUnityPlugin/Assets/Mediapipe/Samples/Graphs/PoseTracking/Scripts/PoseTrackingGraph.cs

Lines 54 to 56 in d58ab72

	if (poseLandmarksStreamPoller.QueueSize() > 0) {
	poseLandmarks = FetchNextPoseLandmarks();
	}

like this.

// pseudo code
yield return new WaitUntil(() => isTimeout() || poseLandmarksStreamPoller.QueueSize() > 0);
if (poseLandmarksStreamPoller.QueueSize() > 0) {
  poseLandmarks = FetchNextPoseLandmarks();
}

But this will blocks the main thread, so if the output is null, I think it's better to ignore it until a certain time has passed.

I'm completely rewriting the sample app (https://github.com/homuler/MediaPipeUnityPlugin/tree/new-sample), and this problem will be fixed there, because the implementation does not block the main thread.

MediaPipeUnityPlugin/Assets/Mediapipe/Samples/Scenes/Pose Tracking/PoseTrackingSolution.cs

Lines 72 to 77 in 9cba17c

	if (runningMode == RunningMode.Async) {
	graphRunner.OnPoseDetectionOutput.AddListener(OnPoseDetectionOutput);
	graphRunner.OnPoseLandmarksOutput.AddListener(OnPoseLandmarksOutput);
	graphRunner.OnPoseWorldLandmarksOutput.AddListener(OnPoseWorldLandmarksOutput);
	graphRunner.StartRunAsync(imageSource).AssertOk();
	} else {

[ROBYER1]

I was curious what was changing in the new sample branch you were working on, I will have to test it out early next week if there is any issue with flickering or latency in my tests I will feed back here. Thanks for your hard work on this plugin