Suddenly freezes

[peke-tictactoe]

OS Platform and Distribution (e.g., Linux Ubuntu 20.04, WSL2): mac OS Monterey(Using Apple M1 max PC)
Target (e.g. desktop cpu, android arm64):desktop cpu
Bazel version: 4.2.2
Python version: 3.10.1
GCC/G++ version: 13.0.0
Unity version: 2021.2.7f1

Describe the current behavior
After starting the Start scene, Unity will do pose detect and other processes normally for a while, but suddenly Unity will freeze and stop working. There are no errors.
Describe the expected behavior
I don't want it to freeze.
Steps to reproduce the issue
After the build, I start Unity, change the Preferable Inference Mode to CPU, and then start the start scene. Other than that, I did not do anything else.

Full logs
No errors.

Additional context
I thought it might be a problem with my PC's camera, so I tried the web camera separately from the built-in camera, but it froze.

[homuler]

Please upload the Editor.log.

[peke-tictactoe]

Editor.log

[homuler]

Waiting for WebCamTexture to start
UnityEngine.StackTraceUtility:ExtractStackTrace () (at /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/StackTrace.cs:37)
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
Mediapipe.Unity.MemoizedLogger:Log (Mediapipe.Logger/LogLevel,object) (at Assets/Mediapipe/Samples/Common/Scripts/MemoizedLogger.cs:181)
Mediapipe.Logger:Log (Mediapipe.Logger/LogLevel,object) (at Packages/com.github.homuler.mediapipe/Runtime/Scripts/Util/Logger.cs:156)
Mediapipe.Logger:LogVerbose (object) (at Packages/com.github.homuler.mediapipe/Runtime/Scripts/Util/Logger.cs:205)
Mediapipe.Unity.WebCamSource/<WaitForWebCamTexture>d__50:MoveNext () (at Assets/Mediapipe/Samples/Common/Scripts/ImageSource/WebCamSource.cs:253)
UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr) (at /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/Coroutines.cs:17)

(Filename: Assets/Mediapipe/Samples/Common/Scripts/ImageSource/WebCamSource.cs Line: 253)

TimeoutException: Failed to start WebCam
  at Mediapipe.Unity.WebCamSource+<WaitForWebCamTexture>d__50.MoveNext () [0x00082] in /Users//MediaPipeUnityPlugin/Assets/Mediapipe/Samples/Common/Scripts/ImageSource/WebCamSource.cs:260 
  at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [0x00020] in /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/Coroutines.cs:17 

No errors.

It looks like a TimeoutException is thrown.
You say it freezes suddenly, but it actually freezes when you switch scenes, doesn't it?
Can you give me more information about the camera you are using?

[peke-tictactoe]

It stops suddenly while I'm using it, not when I switch the scene. The camera I'm using is the built-in camera of my mac.

[homuler]

Other than that, I did not do anything else.

But actually you switched scenes, right?
Does this mean that the camera stopped while running FaceDetectionSolution and then tried to switch scenes?

the built-in camera of my mac.

Can you at least tell me the model (e.g. MacBook Pro 14-inch (2021))? (and the resolution if possible)

[peke-tictactoe]

Switch the scene and it will work fine for a while. Sometimes it stops at FaceDetectionSolution even without switching scenes. I'm sorry for my lack of words.
The model is MacBook Pro 16-inch (2021) Resolution is 3456 x 2234

[m-rlin]

The same happened to me, and when I checked, it stopped when TextureFrame called GetNativeTexturePtr.

[homuler]

Thanks! > @merlin-rtzr

Texture.GetNativeTexturePtr seems to block the main thread and judging from the system call (I used dtruss), it looks like a deadlock is occurring.
Currently, I cannot determine if this is a bug in Unity or if I'm calling the API incorrectly.

[homuler]

I confirmed that the following script can reproduce the issue (in an almost empty scene), and now I suspect there is a problem with the Unity implementation.

using System.Collections;
using UnityEngine;
using UnityEngine.UI;

using Mediapipe.Unity;

public class FreezeTest : MonoBehaviour
{
    [SerializeField] RawImage screen;
    [SerializeField] int width;
    [SerializeField] int height;
    [SerializeField] int fps;

    WebCamTexture webCamTexture;
    Color32[] pixelData;

    IEnumerator Start()
    {
        var webCamDevice = WebCamTexture.devices[0];
        webCamTexture = new WebCamTexture(webCamDevice.name, width, height, fps);
        webCamTexture.Play();

        yield return new WaitUntil(() => webCamTexture.width > 16);

        var texture = new Texture2D(width, height, TextureFormat.RGBA32, false);
        pixelData = new Color32[width * height];
        screen.texture = texture;

        while (true)
        {
          webCamTexture.GetPixels32(pixelData);
          texture.SetPixels32(pixelData);
          texture.Apply();
          Debug.Log(texture.GetNativeTexturePtr());
          yield return new WaitForEndOfFrame();
        }
    }

    void Destroy()
    {
        webCamTexture.Stop();
    }
}

[Magistrix93]

I confirmed that the following script can reproduce the issue (in an almost empty scene), and now I suspect there is a problem with the Unity implementation.

There is a workaround or we have to wait for a new release with the fix?

[homuler]

It's easy to avoid this issue since it only exists in the sample app code (not in the plugin code).
Simply put, you can just safely skip TextureFrame#ChangeNameFrom call because this method is currently useless except on Android (it's required to build a GpuBufferPacket).

However, fixing it is another thing, and I'm still thinking about what to do.
To be honest, if it's a Unity bug and will be fixed, I think the current implementation is the most consistent.

[Magistrix93]

It's easy to avoid this issue since it only exists in the sample app code (not in the plugin code). Simply put, you can just safely skip TextureFrame#ChangeNameFrom call because this method is currently useless except on Android (it's required to build a GpuBufferPacket).

However, fixing it is another thing, and I'm still thinking about what to do. To be honest, if it's a Unity bug and will be fixed, I think the current implementation is the most consistent.

Hi, I tried as you suggested commenting lines 181 and 187 from TextureFrame

public void SetPixels32(Color32[] pixels)
        {
            //var oldName = GetTextureName();
            
            _texture.SetPixels32(pixels);
            _texture.Apply();
            _nativeTexturePtr = IntPtr.Zero;


            //ChangeNameFrom(oldName);
        }

but now I'm getting this error that stops all async flow of mediapipe:

ArgumentException: Another instance has the same name
Mediapipe.Unity.TextureFrame.RegisterInstance (Mediapipe.Unity.TextureFrame textureFrame) (at Assets/Mediapipe/Samples/Common/Scripts/ImageSource/TextureFrame.cs:296)

[homuler]

@Magistrix93 It's already fixed (cf. #505), so please pull the latest commit.