How to save/recall a gesture for later use #2
Comments
|
They way I used it was simply but is not avaliable atm (discussed later). I just used the LeapMotion Scene to record as binary assets the gestures I needed, then during the loading proccess I just loaded all the frames using the LeapRecorder class and finally I added the gesture using this method (I think you knew this already): As you are trying to use the JSON feature you should know that the recorder implemented inside the LeapTrainer is untested since I didn't needed to use it. You're getting the error because the gesture is NOT being recorded at all.
LeapTrainer trainer;
bool isTraining = false;
void Start()
{
if (trainer)
{
// These are DELEGATES so one adition is enought to receive callbacks
trainer.OnGestureCreated += Trainer_OnGestureCreated;
trainer.OnGestureRecognized += Trainer_OnGestureRecognized;
trainer.OnGestureUnknown += Trainer_OnGestureUnknown;
trainer.OnTrainingCountdown += Trainer_OnTrainingCountdown;
trainer.OnTrainingStarted += Trainer_OnTrainingStarted;
trainer.OnTrainingComplete += Trainer_OnTrainingComplete;
// This will only be needed in case you change the parameter inside LeapTrainer
// because youll be able to track when a training gesture is recorded and notify the
// repetitions left.
// int trainingGestures = 1; // The number of gestures samples that collected during training
trainer.OnTrainingGestureSaved += Trainer_OnTrainingGestureSaved;
}
}
private void Trainer_OnGestureCreated(string name, bool trainingSkipped)
{
Debug.Log("Gesture " + name + " has been created");
}
private void Trainer_OnTrainingCountdown(int countdown)
{
Debug.Log("Training will start in " + countdown + " seconds");
}
private void Trainer_OnTrainingStarted(string name)
{
Debug.Log("Started training gesture " + name);
}
private void Trainer_OnTrainingComplete(string name, List<List<Point>> gestures, bool isPose)
{
Debug.Log("Finished training gesture " + name);
}
private void Trainer_OnGestureRecognized(string name, float value, Dictionary<string, float> allHits)
{
Debug.Log("Gesture " + name + " recognized");
}
private void Trainer_OnGestureUnknown(Dictionary<string, float> allHits)
{
Debug.Log("No gesture recognized...");
// However you can still work with the hits in case I detect some recording are underperforming...
// For instance: if you have a recording that the best match is a value of 0.5 but the threshold is 0.7 you'll
// never be notified, but you'll still receive the value here.
}
private void Trainer_OnTrainingGestureSaved(string name, List<List<Point>> gestures)
{
Debug.Log("Training gesture recorded... X more to go.");
}
void Update()
{
if (Input.GetButtonDown("Fire3") && trainer != null)
{
startTraining();
}
}
void startTraining()
{
if (trainer == null || isTraining) return;
isTraining = true;
Debug.Log("Requesting new training...");
trainer.Create("TestGest", false);
}
For the new recording scene:
For the runtime load:
I'll try to implement something for this as soon as I have some time but anyway if you're going for the first, let me know if you succeed since I don't have a LeapMotion at all. Bests! |
|
@Victorma Thank you for your help again here. I feel like this could be seriously useful for AR implementations like ours. |
|
@Victorma I wanted to let you know since you said you are trying to make a new recording scene, this will be an issue to overcome with any attempt. If I succeed, I will share it for sure. EDIT that wasnt so bad, and I learned new things. I thought I would have to make it another byte file from scratch which is why I worried. JsonUtility took care of it just fine, now I have a working Orion LeapRecorder. Ill post it tomorrow with a recorder scene once the code is cleaned up. In the future, it probably wouldnt be too hard to just merge the LeapRecorder method into the main LeapClassifier's create method, rather than using the exact method roboleary used. . |
|
@bridgewaterrobbie that is great! Even if its not byte json solution is better for most of the cases because it has better portabillity. Whenever you have the updated version of LeapRecorder I'll be very pleased if you share it with me in a PR here 😄 Thanks for your effort! |
|
@Victorma I intend to integrate leaprecorder into LeapTrainer later, to make it a bit more cohesive. That might take a while though. Is there any reason that you know of that the solution from roboleary (when working) is superior to the method LeapRecorder uses of just capturing all the frames for later analysis? Otherwise I think Ill strip his unimplemented stuff out, and replace it with LeapRecorder's functionality. |
|
if successful please let me know because, since am new to unity 3D-- am I appreciate all your works; On Thu, Oct 20, 2016 at 4:52 AM, bridgewaterrobbie <[email protected]
|
|
@mugabi0978 First download LeapClassifier with the new LeapRecorder from github (My link is here: https://github.com/bridgewaterrobbie/LeapTrainer.CSharp , Victorma will probably pull it into the main one later) Use the scenes I made to record a series of actions to a file for later. You can use the playback mode (Z by default) to check your recording if you want. Note that the playback here may be a bit faster, but it wont matter here. Next open your own scene up. You will need to instantiate an instance of LeapTrainer as well as LeapRecorder. Use LeapRecorder to load in the file you made earlier with Load. This will load all of the frames to your instance of LeapRecorder. Now call LeapTrainer.loadFromFrames, passing in LeapRecorder.GetFrames(Returns a list of the Frame objects). This will create the gesture for the LeapTrainer to recognize. Finally, use delegate methods like Victorma showed above to act on a gesture being recognized. I haven't done that last step yet, but I imagine switch statements would be your friend here. |
|
Whenever i run the scene i get this error below:- The referenced script on this Behaviour (Game Object 'LeapHandController') If possible, please can you make a video to illustrate your statements i think that could be a easier way to learn what you are saying. Thank You. On Thu, Oct 20, 2016 at 10:44 AM, bridgewaterrobbie <
|
|
@mugabi0978 EDIT Im going ahead and pushing the small fix needed too. |
|
Ok i downloaded last update and no errors...thanks for the update.... On Thu, Oct 20, 2016 at 11:48 AM, powell lorenz [email protected]
|
|
@bridgewaterrobbie related to: #2 (comment) Now I'm heading to review the PR and merge it ^^ Also, thanks for your cooperation with @mugabi0978 @mugabi0978, I'll try to update SignAdventure with the new updates here and I'll let you know. |
|
Wow, it would be my pressure if you do @Victorma Am really awaiting for that anytime from now;- And also please update the readme...to match people like us. (unity 3d @bridgewaterrobbie https://github.com/bridgewaterrobbie and @Victorma On Thu, Oct 20, 2016 at 12:25 PM, Victorma Perez Colado <
|
|
Ok, I think I am one line of code from being done, but this is not quite working yet. I am now getting an error in GeometricalMatcher, at line ~80 (Ive added some debug statements) The line is Argument is out of range. My best guess: Something about how Orion works is formatting the data differently enough to anger GeoMatcher. This difference is not signifigant enough to make stored V2 frames complain (As evidenced by your SignAdventure still recognizing frames from Orion and matching them to your V2 Leap Recorder) I am trying to figure this out now, but I figured I would post here now too. It seems that the Recognize function always either returns 0, or has this error. Thank you again. |
|
@bridgewaterrobbie I don't know what could it be... I wasn't able to make Orion SDK work in Sign Adventure because of the deserialization issue. Nevertheless, this wednesday I'm having the Leap Motion so I'll be able to test this and fix it. |
|
@Victorma no worries, not like you are permenently responsible for your code on github. Any help is appreciated, but we will take care e of anything you don't want to (though we wouldn't have realized the create method was not working with your help admittedly) I'll make a pull request for anything we discover. The issue is either in how the point array is getting passed to match, or something about how Orion handles frames is exposing a bug in geometric matched. I'm going to reference the original $p to see if that can fix things |
Hey sorry to bother you again, we have been trying to get up and running but running into a few issues that we think you might be able to solve quickly. We tried looking through signadventure, but havent had much luck there since it seems you use a different process, and we were unable to find the methods you used to initially create the gestures.
We are trying to create a gesture, then save it for later use with .toJSON, restoring it with .fromJSON. In sign adventure, you seem to use a .byte file to recall old gestures, but we can't figure out the path you took to load them from the excercise prefab into LeapClassifier, nor how to create a new byte file.
Also we have one very strange thing happen as we call toJson. Despite checking that the Gestures dictionary does contain the key with "this.gestures.ContainsKey(gestureName)", the code crashes immediately after with the line "var gestures = this.gestures[gestureName];" with a
We also have not managed to get a gesture to be recognized after creating it. Our code looks like this:
Again, sorry for the bother and we appreciate any feedback/solutions you have time to give.
The text was updated successfully, but these errors were encountered: