Add support for input and gestures for hand controllers (#552)

* Delete baked lighting settings for base * Fix ControllerPopupWindow NRE * Implement internal is grabbing update * Disable hand meshing by default and assign tracked poses to test profile * Add IsValidCollision extension to colliders * Feature/pni change requests (#577) * Update ColliderExtensions.cs * Update CollisionExtensions.cs * updated submodules * updated sdk * updated sdk * Cleanup namespace * Better fallback pointer pose calculation * Introduce hand rendering mode * Implement runtime rendering mode switch * Update submodules * Start new hands examples room * Add canvas UI and 3D Buttons station * Show button stations result * Implement set rendering mode station * Added more stations * Update submodules * updated wmr checkout * updated sdk * Update submodules * Add additive scene loader for hands demo scene * Restore base scene * Add simple physics demo station * Cleanup namespace * Cleanup hand examples a bit * Update stations * Implement runtime bounds mode selection * Update submodules * Update Ultraleap checkout * Update physics station * Update submodule * Update submodule * Fix UWP certficate error * Moved examples to examples repo * reset editor window options for hands * Implement pinch strength * Add pinch strength to hand controller interface * Update submoduules * Update SDK * Use array instead of list for tracked poses * Restructure pose recognition code * Improved variable naming * Add root pose to hand data * Set root pose for simulation * Fix joint rotations * Update SDK * Remove base converter and introduce Post Processor * Update data provider to use post processor * Optimize recognition * Introduce grip strength * Enable setters for post processor * Minor refactoring in post processor * Removed NONE joint * Rename joints * Update submodules * Extend default hand mappings * Sync simulation profiles * Reset popup window options for hands * Hand input and poses change request (#616) * Cleaned up RecordedHandJoint(s) to remove reference values removed null unity object lifetime bypass added better exceptions when starting simulated hand controller data provider and Simulated hand data converter * fixed missing line * updated sdk checkout * Implement grip pose * Adjust processor to joint enum renamings * updated sdk checkout * Introduce hand finger enum * Rename pose recognizer and implement frame throttle * Minor docs update * Implement finger curls * Add Oculus (Desktop) package back in for now * Disable debug log for grip calculations * updated tracked pose processor compare (#617) aesthetic and spell checking in hand controller Co-authored-by: Dino Fejzagic <[email protected]> * Implement IMixedRealityHandController.TryGetFingerCurlStrength * Update SDK * Disable recognition for now * Add index curl constraint to isgripping detection * updated sdk checkout * updated SDK * updated sdk checkout * Implement recorded pose baking and update inspector * Update SDK * Remove handedness from hand data * Update Oculus * Fix baked data not persisted * Update SDK * Fix pose recognizer handedness issue * Improve pose processor * Fix IsPointing set to false when pinching * Update submodules * Update examples checkout * Do not pass tracked pose scriptable object outside of data provider * Update examples * Update examples * Fix potential NRE Co-authored-by: Stephen Hodgson <[email protected]> * Fix potential casting error Co-authored-by: Stephen Hodgson <[email protected]> * Fix formatting * Use for instead of foreach * Revert metro package version * Rename HandControllerPoseDefinition to HandControllerPoseProfile * Revert to ReordableList for tracked poses * Change HandData to be a struct * Update submodule * Update example module * Fix invalid simulated hand position when camera rotation is changed * Fix generic pointer pose invalid when camera rotated * Remove redundant properties in hand mesh data * Update submodules * Only update hand state if hand is tracked * Remove obsolete pose extensions * Introduce hand utilities for pose estimation * Update submodules * Implement TryGetHandMeshData * Remove HandData input event * Update submodule * Update submodules * Implement relativeTo paramter for TryGetJointPose * Fix IsPointing * Use TrackingState instead of bool for tracking state * Fix missed occurence of IsTracked * Update Oculus * Fix pointer pose * Update Oculus * Fix fallback pointer pose * Fix pose recognizer left/right issue * Update submodules * Remove tracked pose mapping (coming back later) * Fix hand velocity calculation * Fix local to world joint translation * Fix simulation mappings * Fix IsPointing after teleport * Update submodule * Fix simulated hand not reset on tracking loss * Fix simulated hand position after teleport * Update SDK * Update SDK * Update examples module * fixed some typos * updated oculus checkout * Update example checkout * Reset hand controller options window * Update submodules * fixed submodule checkouts * Remove tracked poses from hand data provider interface * Cache bounds arrays * Allow a bit more tolerance wit IsPointing * Update submodules * Update example * Update modules * Update submodules * Update oculus * Update WMR * Clean up gizmo meshes * Remove duplicate vertices in gizmo meshes * Update submodules and wmr package * Only apply default head height if not managed by device * Update submodules * Introduce IHandDataPostProcessor * Update submodules * Update WMR submodule Co-authored-by: Stephen Hodgson <[email protected]> Co-authored-by: Simon (Darkside) Jackson <[email protected]>
XRTK · Sep 22, 2020 · 24a757e · 24a757e
1 parent 870bb59
commit 24a757e
Show file tree

Hide file tree

Showing 68 changed files with 3,237 additions and 1,488 deletions.
diff --git a/Submodules/Examples b/Submodules/Examples
diff --git a/Submodules/Lumin b/Submodules/Lumin
diff --git a/Submodules/Oculus b/Submodules/Oculus
diff --git a/Submodules/SDK b/Submodules/SDK
diff --git a/Submodules/WindowsMixedReality b/Submodules/WindowsMixedReality
diff --git a/XRTK-Core/Packages/com.xrtk.core/Editor/ControllerPopupWindow.cs b/XRTK-Core/Packages/com.xrtk.core/Editor/ControllerPopupWindow.cs
@@ -130,6 +130,7 @@ public static void Show(MixedRealityControllerMappingProfile profile, Serialized
             }
 
             window = (ControllerPopupWindow)CreateInstance(typeof(ControllerPopupWindow));
+            window.currentControllerName = profile.ControllerType?.Type?.Name;
 
             window.currentControllerName = profile.ControllerType?.Type?.Name;
             window.titleContent = new GUIContent($"{window.currentControllerName} {handednessTitleText}Input Action Assignment");

diff --git a/...or/Data/Controllers/Simulation/Hands.meta → ...k.core/Editor/Data/Controllers/Hands.meta b/...or/Data/Controllers/Simulation/Hands.meta → ...k.core/Editor/Data/Controllers/Hands.meta
diff --git a/...ackages/com.xrtk.core/Editor/Data/Controllers/Hands/HandControllerPoseProfileInspector.cs b/...ackages/com.xrtk.core/Editor/Data/Controllers/Hands/HandControllerPoseProfileInspector.cs
@@ -0,0 +1,159 @@
+// Copyright (c) XRTK. All rights reserved.
+// Licensed under the MIT License. See LICENSE in the project root for license information.
+
+using UnityEditor;
+using UnityEngine;
+using XRTK.Definitions.Controllers.Hands;
+using XRTK.Definitions.Utilities;
+using XRTK.Editor.Extensions;
+using XRTK.Editor.Profiles;
+using XRTK.Extensions;
+using XRTK.Providers.Controllers.Hands;
+
+namespace XRTK.Editor.Data.Controllers.Hands
+{
+    [CustomEditor(typeof(HandControllerPoseProfile))]
+    public class HandControllerPoseProfileInspector : BaseMixedRealityProfileInspector
+    {
+        private SerializedProperty id;
+        private SerializedProperty description;
+        private SerializedProperty isDefault;
+        private SerializedProperty keyCode;
+        private SerializedProperty data;
+
+        // Baked data.
+        private SerializedProperty didBake;
+        private SerializedProperty isGripping;
+        private SerializedProperty fingerCurlStrengths;
+        private SerializedProperty gripStrength;
+
+        private static readonly GUIContent generalSettingsHeader = new GUIContent("General Settings");
+        private static readonly GUIContent simulationSettingsHeader = new GUIContent("Simulation Settings");
+        private static readonly GUIContent bakeSettingsHeader = new GUIContent("Bake Settings");
+        private static readonly GUIContent infoHeader = new GUIContent("Info");
+        private static readonly GUIContent bakeButtonContent = new GUIContent("Generate Baked Data");
+        private static readonly GUIContent updateBakeButtonContent = new GUIContent("Update Baked Data");
+        private bool infoExpanded = false;
+
+        protected override void OnEnable()
+        {
+            base.OnEnable();
+
+            id = serializedObject.FindProperty(nameof(id));
+            description = serializedObject.FindProperty(nameof(description));
+            isDefault = serializedObject.FindProperty(nameof(isDefault));
+            keyCode = serializedObject.FindProperty(nameof(keyCode));
+            data = serializedObject.FindProperty(nameof(data));
+
+            // Baked data.
+            didBake = serializedObject.FindProperty(nameof(didBake));
+            isGripping = serializedObject.FindProperty(nameof(isGripping));
+            fingerCurlStrengths = serializedObject.FindProperty(nameof(fingerCurlStrengths));
+            gripStrength = serializedObject.FindProperty(nameof(gripStrength));
+        }
+
+        public override void OnInspectorGUI()
+        {
+            RenderHeader("This profile defines a hand pose that can be recognized at runtime and trigger input events.");
+
+            serializedObject.Update();
+
+            description.isExpanded = EditorGUILayoutExtensions.FoldoutWithBoldLabel(description.isExpanded, generalSettingsHeader);
+            if (description.isExpanded)
+            {
+                EditorGUI.indentLevel++;
+
+                EditorGUILayout.PropertyField(id);
+                EditorGUILayout.PropertyField(description);
+                EditorGUILayout.PropertyField(data);
+
+                EditorGUI.indentLevel--;
+            }
+
+            EditorGUILayout.Space();
+
+            isDefault.isExpanded = EditorGUILayoutExtensions.FoldoutWithBoldLabel(isDefault.isExpanded, simulationSettingsHeader);
+            if (isDefault.isExpanded)
+            {
+                EditorGUI.indentLevel++;
+
+                EditorGUILayout.PropertyField(isDefault);
+                EditorGUILayout.PropertyField(keyCode);
+
+                EditorGUI.indentLevel--;
+            }
+
+            EditorGUILayout.Space();
+
+            didBake.isExpanded = EditorGUILayoutExtensions.FoldoutWithBoldLabel(didBake.isExpanded, bakeSettingsHeader);
+            if (didBake.isExpanded)
+            {
+                if (!didBake.boolValue)
+                {
+                    EditorGUILayout.HelpBox($"This hand pose hasn't been baked yet. In order for the pose to be recognized when running your application you need to bake it," +
+                    $" which will precompute important data and save performance.", MessageType.Warning);
+                }
+
+                if (GUILayout.Button(didBake.boolValue ? updateBakeButtonContent : bakeButtonContent))
+                {
+                    BakePoseData();
+                }
+            }
+
+            EditorGUILayout.Space();
+
+            infoExpanded = EditorGUILayoutExtensions.FoldoutWithBoldLabel(infoExpanded, infoHeader);
+            if (infoExpanded)
+            {
+                EditorGUI.indentLevel++;
+
+                if (didBake.boolValue)
+                {
+                    EditorGUILayout.LabelField($"Baked Is Gripping:\t\t{isGripping.boolValue}");
+                    EditorGUILayout.LabelField($"Baked Grip Strength:\t\t{gripStrength.floatValue}");
+                    EditorGUILayout.LabelField($"Baked Thumb Curl:\t\t{fingerCurlStrengths.GetArrayElementAtIndex((int)HandFinger.Thumb).floatValue}");
+                    EditorGUILayout.LabelField($"Baked Index Curl:\t\t{fingerCurlStrengths.GetArrayElementAtIndex((int)HandFinger.Index).floatValue}");
+                    EditorGUILayout.LabelField($"Baked Middle Curl:\t\t{fingerCurlStrengths.GetArrayElementAtIndex((int)HandFinger.Middle).floatValue}");
+                    EditorGUILayout.LabelField($"Baked Ring Curl:\t\t{fingerCurlStrengths.GetArrayElementAtIndex((int)HandFinger.Ring).floatValue}");
+                    EditorGUILayout.LabelField($"Baked Little Curl:\t\t{fingerCurlStrengths.GetArrayElementAtIndex((int)HandFinger.Little).floatValue}");
+                }
+                else
+                {
+                    EditorGUILayout.HelpBox("Bake the hand pose to see baked information here.", MessageType.Info);
+                }
+
+                EditorGUI.indentLevel--;
+            }
+
+            serializedObject.ApplyModifiedProperties();
+        }
+
+        private void BakePoseData()
+        {
+            // Initialize the hand data using the joint information from the recorded pose.
+            var poseDefinition = target as HandControllerPoseProfile;
+            var handData = poseDefinition.ToHandData();
+
+            // Intialize processors needed.
+            var gripPostProcessor = new HandGripPostProcessor();
+
+            // Process the hand data, most hand data processors
+            // will ignore the hand data if it is not tracked, so we
+            // have to temporarily fake it's tracking state and then reset it.
+            handData.TrackingState = Definitions.Devices.TrackingState.Tracked;
+            gripPostProcessor.PostProcess(Handedness.Right, handData);
+            handData.TrackingState = Definitions.Devices.TrackingState.NotTracked;
+
+            isGripping.boolValue = handData.IsGripping;
+            gripStrength.floatValue = handData.GripStrength;
+            fingerCurlStrengths.ClearArray();
+            fingerCurlStrengths.arraySize = handData.FingerCurlStrengths.Length;
+            for (int i = 0; i < handData.FingerCurlStrengths.Length; i++)
+            {
+                fingerCurlStrengths.GetArrayElementAtIndex(i).floatValue = handData.FingerCurlStrengths[i];
+            }
+
+            didBake.boolValue = true;
+        }
+    }
+}
diff --git a/...edHandControllerPoseDataInspector.cs.meta → ...andControllerPoseProfileInspector.cs.meta b/...edHandControllerPoseDataInspector.cs.meta → ...andControllerPoseProfileInspector.cs.meta
diff --git a/XRTK-Core/Packages/com.xrtk.core/Editor/Data/Controllers/Simulation.meta b/XRTK-Core/Packages/com.xrtk.core/Editor/Data/Controllers/Simulation.meta
diff --git a/...core/Editor/Data/Controllers/Simulation/Hands/SimulatedHandControllerPoseDataInspector.cs b/...core/Editor/Data/Controllers/Simulation/Hands/SimulatedHandControllerPoseDataInspector.cs
diff --git a/XRTK-Core/Packages/com.xrtk.core/Editor/Data/EditorWindowOptions.json b/XRTK-Core/Packages/com.xrtk.core/Editor/Data/EditorWindowOptions.json
@@ -484,32 +484,6 @@
                 true
             ]
         },
-        {
-            "Controller": "MixedRealityHandController",
-            "Handedness": 1,
-            "InputLabelPositions": [
-                {
-                    "x": 413.0,
-                    "y": 125.0
-                }
-            ],
-            "IsLabelFlipped": [
-                false
-            ]
-        },
-        {
-            "Controller": "MixedRealityHandController",
-            "Handedness": 2,
-            "InputLabelPositions": [
-                {
-                    "x": 3.0,
-                    "y": 125.0
-                }
-            ],
-            "IsLabelFlipped": [
-                true
-            ]
-        },
         {
             "Controller": "OculusTouchController",
             "Handedness": 1,
@@ -711,102 +685,6 @@
                 false
             ]
         },
-        {
-            "Controller": "SimulatedMixedRealityHandController",
-            "Handedness": 1,
-            "InputLabelPositions": [
-                {
-                    "x": 480.0,
-                    "y": 80.0
-                },
-                {
-                    "x": 480.0,
-                    "y": 100.0
-                },
-                {
-                    "x": 480.0,
-                    "y": 120.0
-                },
-                {
-                    "x": 480.0,
-                    "y": 140.0
-                },
-                {
-                    "x": 480.0,
-                    "y": 160.0
-                },
-                {
-                    "x": 480.0,
-                    "y": 180.0
-                },
-                {
-                    "x": 480.0,
-                    "y": 200.0
-                },
-                {
-                    "x": 480.0,
-                    "y": 220.0
-                }
-            ],
-            "IsLabelFlipped": [
-                false,
-                false,
-                false,
-                false,
-                false,
-                false,
-                false,
-                false
-            ]
-        },
-        {
-            "Controller": "SimulatedMixedRealityHandController",
-            "Handedness": 2,
-            "InputLabelPositions": [
-                {
-                    "x": -55.0,
-                    "y": 80.0
-                },
-                {
-                    "x": -55.0,
-                    "y": 100.0
-                },
-                {
-                    "x": -55.0,
-                    "y": 120.0
-                },
-                {
-                    "x": -55.0,
-                    "y": 140.0
-                },
-                {
-                    "x": -55.0,
-                    "y": 160.0
-                },
-                {
-                    "x": -55.0,
-                    "y": 180.0
-                },
-                {
-                    "x": -55.0,
-                    "y": 200.0
-                },
-                {
-                    "x": -55.0,
-                    "y": 220.0
-                }
-            ],
-            "IsLabelFlipped": [
-                true,
-                true,
-                true,
-                true,
-                true,
-                true,
-                true,
-                true
-            ]
-        },
         {
             "Controller": "OculusTouchOpenVRController",
             "Handedness": 2,