(0.8.6) Holistic tracking has latency after 30-60 secs, works fine on previous versions

[ROBYER1]

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in MediaPipe): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Windows 10, Android 11
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Xperia XZ3
• Programming Language and version ( e.g. C++, Python, Java): C#
• MediaPipe version: 0.8.6
• Bazel version (if compiling from source): 3.7.2
• Solution ( e.g. FaceMesh, Pose, Holistic ): Holistic

Describe the current behavior:
in 0.8.4 the tracking in Holistic is fine but in 0.8.5 and 0.8.6 we are noticing the tracking begins to be out of sync greatly with the tracked person after 30-60 seconds. An example video of this is here from this issue on the Unity Mediapipe plugin repository

We are testing in Windows desktop builds, android builds and within Unity Editor itself using that repository

Describe the expected behavior:
Tracking after 30-60 seconds with Holistic should stay stable, I suspect this issue has crept in around the 3d world tracking points for Holistic. This issue is not device specific and happens on High end desktop/laptops as well as android phones in our tests.

Standalone code to reproduce the issue:
Please use the build steps for windows/android in this repo in Unity any version of Unity > 2020.3 version, run the Assets\Mediapipe\Samples\Scenes\DesktopDemo Scene and switch the graph to holistic.

Works fine using this commit

The latency issue shows when using this commit

[sgowroji]

Hi @ROBYER1, You are using the different repo https://github.com/homuler/MediaPipeUnityPlugin, And that might be using MediaPipe. But the above repo is not maintained by MediaPipe. Could you please have a look. Thanks!

[ROBYER1]

Hi @sgowroji I am aware the Unity Mediapipe repo isn't maintained by Google but I am just switching out the built Mediapipe libraries in my tests and finding that 0.8.4 Holistic tracking is fine after a minute or so but any version higher has some latency and lag after a minute.

I can't be the only person with this issue - see it here:
https://photos.google.com/share/AF1QipOX7Nd_pdFU7AbA6t8BYIwcvHe5IHAO39cP6L2fdMCeb4izXjtXCRGpHwiFpJ2aDQ?key=eVlfSVFXV3JNWVpaWG1tcjFtdHFBWEhDcTN5TFlB

[ROBYER1]

Found the same issue when deploying Mediapipe 0.8.6 to Iphone 7 and and Xperia XZ3 as well as running on windows desktop in Unity editor. Rolling back to 0.8.4 solved the latency issues that crept in. I would suspect there is a regression - see the other issues that mention this above my comment also as it is not just me.

[ROBYER1]

Closing this as it was an issue with the implementation we were using, however there is an increasing issue of flickering on some devices, possibly due to the weight of the model increasing.
homuler/MediaPipeUnityPlugin#256 (comment)

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No