How to get hand landmarks or other data from graphs?

[elblogbruno]

Hello,
I have seen issue #5, and where you point to are private methods. I do want to access data from my script, am I following a bad philosophy of getting the data out?
Thanks
Cordially,
Bruno

[homuler]

Currently, classes under Examples are not supposed to be extended or used directly (they are implemented for reference).
This plugin is intended to proved lower level APIs and enable users to customize and run graphs in C#.

It might be better if it provides some highe-level APIs or wrapper classes for graphs at least, but it doesn't for now (actually, there are 2 or 3 ways to retrieve outputs from graphs, and you should select one when you implement them).

If you want those wrapper classes, I'll take it as a feature request and examine later.

[elblogbruno]

Well maybe I can do my work on creating my own API for the plugin, as I only want to access hand tracking data, for example, I will see what I can do!
Thanks
Great job!
Bruno

[elblogbruno]

As I believe the official demo graph on android, the texture is drawn directly from the mediapipe plugin. There's any way to access the landmark data?
Thanks

[homuler]

It's almost same as the other examples.
You can get them synchronously

class SomeGraph : DemoGraph {
  public Status StartRun() {
    ...
    outputStreamPoller = graph.AddOutputStreamPoller<List<NormalizedLandmarkList>>("hand_landmarks")
        .ConsumeValueOrDie();
    outputPacket = new NormalizedLandMarkListVectorPacket();
    ...
  }

  public override void RenderOutput(WebCamScreenController screenController, TextureFrame textureFrame) {
    var landmarks = FetchNext(outputStreamPoller, outputPacket, "hand_landmarks");
    // do something

    // Note that the above method blocks the thread, and if the output packet is empty, it never returns (the process hangs).
    // see https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/Examples/Graphs/HandTracking/Scripts/HandTrackingGraph.cs
  }
}

or asynchronously.

// see https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/Examples/Graphs/OfficialDemo/Scripts/OfficialDemoDesktop.cs

class SomeGraph : DemoGraph {
  public override Status StartRun() {
    ...
    graph.ObserveOutputStream<NormalizedLandmarkListVectorPacket, List<NormalizedLandmarkList>>(
        "hand_landmarks", OutputCallback, out outputCallbackHandle).AssertOk();
    ...
  }

  private Status OutputCallback(NormalizedLandmarkListVectorPacket packet) {
    using (var statusOrLandmarks = packet.Consume()) {
      if (statusOrLandmarks.ok) {
        using (var landmarks = statusOrLandmarks.ConsumeValueOrDie()) {
          // do something
        }
      }
    }

    return Status.Ok();
  }
}

[elblogbruno]

Oh thanks!!

[elblogbruno]

And can I get only the landmarks and forget about the image?, The thing is I am using ARFoundation, and don't know how I will get both working.
From ARFoundation I can access the images of the camera from the CPU, i can feed those to mediapipe somehow?

Thanks!

[homuler]

Is it that you'd like to know how to send input images to MediaPipe?
If you have image data on CPU, then you can send them to MediaPipe as an ImageFramePacket.
https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/Examples/Scripts/DemoGraph.cs#L48-L53

[elblogbruno]

Currently on ARFoundation I can access images like this
https://docs.unity3d.com/Packages/[email protected]/manual/cpu-camera-image.html
I get a pointer to the texture. Can I pass it to the plugin, or I pass the texture2D?
Thanks

[homuler]

If you can get NativeArray<byte>, you can build an ImageFrame.
https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/SDK/Scripts/Framework/Formats/ImageFrame.cs#L28

[elblogbruno]

Ok thanks I will try that!