Improved support for overlay input

[CircuitLord]

This is a feature request to be able to hide the steamvr laser pointer for custom overlays.

My problem is that I would like to be able to have a custom pointer to interact with my overlays, since the default steamvr pointer's angle cannot be modified. And whenever I enable my overlays blocking input to the scene application, there's no way to hide the laser pointer showing up.

This would work along with how the new flag, VROverlayFlags_HideLaserIntersection, and VROverlayFlags_MakeOverlaysInteractiveIfVisible work together.

The new flag, VROverlayFlags_HideLaserPointer (default false), would either not show or show the laser pointer when an overlay has the "MakeOverlaysInteractiveIfVisible" flag set to true, and said overlay is visible. However, if any overlays that exist have interaction set to true and have HideLaserPointer false, it'll just show the pointer like normal.

Let me know if you have any questions about the logistics of how it would work, I would love to see this added in :)

[JoeLudwig]

Just to restate what I think you're asking for so I know I understand it.. There would be three flags:

• VROverlayFlags_MakeOverlaysInteractiveIfVisible - This causes SteamVR to go into laser mouse mode. It also causes the game to not get input.
• VROverlayFlags_HideLaserIntersection - This causes the laser blob to not draw when the laser is pointing at this overlay.
• VROverlayFlags_HideLaserPointer - If this were to be added, it would prevent the laser itself from drawing when the laser is pointed at this specific overlay while in laser mouse mode. The laser would still draw when pointed out into space, or at any other overlay.

Is that right?

Your overlay would keep getting mouse events, but it sounds like you would ignore the X/Y values in them and generate your own values based on some other rotation of the laser pointer?

since the default steamvr pointer's angle cannot be modified

This seems like the core issue here. Why do you want to change that angle from overlay to overlay?

[CircuitLord]

That's not exactly what I meant, the HideLaserPointer flag would basically just not render the laser pointer at all if all the interactive overlays that were visible had this tag. AKA it would appear as it did if no interactive overlays were open, but still block input to the current scene application.

Interaction events would still function in the same way they do right now, yes, and it would just allow the overlay application to render it's own pointer at a customizable angle.

The problem stems from the fact that I have an overlay application with a sort of wrist watch, which can be pointed at and interacted with. The problem that is on some controllers, the steamvr laser pointer is at a pretty sharp angle (aka the vive wand) which may work ok for big picture mode and stuff, but makes it really hard to click UI on your other hand's wrist.

A sort of alternative solution would just be to add a way for the user to modify the global laser pointer angle, but I know that's asking for a lot especially in terms of a configuration UI, so I figured being able to render my own pointer instead would be good enough.

[JoeLudwig]

What follows is more speculation than plan and I have no idea if/when such a thing would ship

I've been thinking about a similar problem to this from another angle. Often overlay apps want to take some of the input from the app, some of the time, but generally leave the app to do its own thing. Often "some of the time" is defined as "depending on where the user's hands and the virtual objects tracked by the overlay are", of which your wrist watch is a good example.

Apps have the ability to layer input from multiple action sets with priority, so a higher priority action set will suppress a given input source (e.g. the trigger) for any binding to an action in a lower priority action set. From the lower priority action set's point of view, the input is always false whenever the higher priority action set is active.

I've been thinking about how we could extend that concept to scene apps and overlay apps. If the scene app's priorities 1, 2, and 3 became 1.1, 1.2, and 1.3, and then the overlay app could put in its own priorities at 2.1, 2,2, and 2.3, any actions bound to inputs in active action sets in the overlay app would prevent the scene app from getting that specific input source whenever that app was active.

In your case, that means you could normally only activate an action set that gives you the poses of the two hands, and then when the right hand is near the left wrist, you could activate your own laser mouse action set and stop the game from getting trigger input. At the same time you would start drawing whatever other UI bits are relevant in that context. Then when the user moves their right hand away from their left wrist, your overlay app would deactivate its laser mouse action set and input would return to the game.

Does that sound like it would solve your problem?

[CircuitLord]

That's actually an even better solution than I thought of. Being able to use action sets AND block input to games would be a godsend.

I think the method you mentioned of higher priority action sets basically overriding lower priority sets would work really well for overlay apps. We'd be able to "block input" to the game, without actually completely blocking everything. (AKA: blocking your trigger with a gun being able to shoot, but still being able to walk around with your joystick, stuff like that.)

It does sound a bit ambitious, since I'm sure the action system would need quite a bit of reworking to make it work, but overall I think action sets are the best way to go forward, and being able to use them universally in OpenVR for all types of apps would allow for a lot of future functionality and remove the need for relying on the old event-based input for overlays that block input.

I'd absolutely love to see this added, just let me know if there's any other questions you have about how it could be better or anything, I'd be more than happy to help :D

[JoeLudwig]

Ok, there's a version of this in SteamVR 1.10.1. You can try it with these two constants:

static const int32_t k_nActionSetOverlayGlobalPriorityMin	= 0x01000000;
static const int32_t k_nActionSetOverlayGlobalPriorityMax	= 0x01FFFFFF;

You will also need to enable this experiemental feature in the Developer settings:

Any overlay app that provides an action set priority between the min and max values above will override scene app actions bound to the same inputs. This is currently button-by-button, but there's some talk of making it apply to the whole controller. This will prevent the app from getting input, so use care when activating action sets in this way.

We would love to have some feedback on this approach. Just leave more replies in this issue when you get a chance to try it out.

This may also be relevant to @matzman666 @ykeara @CrossVR @guygodin

[ykeara]

Def interesting for us, Will give it a shot this week.

Perhaps related, is there a possibility of using this system to also allow input to work while the dashboard is active.... as currently while dashboard is active no inputs are registered, so something say like PTT cannot work while dashboard is open. OR you need to be vary careful w/ things like HMD prox sensors as Dashboard will cause them to go to false.

Might not be feasible.... and might not be right spot... but having a continuous priority system including the dashboard option would be nice.

[JoeLudwig]

This doesn't currently change anything with the dashboard taking over input.

[JoeLudwig]

@JaredMonkey might be interested too.

[CircuitLord]

This looks awesome! Was on vacation so I haven't really had a chance to test any of this yet, but I'll give it a shot over the weekend.

[ykeara]

Okay did some initial testing

Setup

• Override APP: SteamVR overlay application all action set priorities set to 0x01000001, made sure no additional action sets had conflicting/duplicate uses for purposes of testing.

• Game App: (legacy input)

Results

• empty

  • overlay App: All inputs registered as normal for game.
  • game App: No reason to test.
  • Both: No reason to test.

• Regular (i.e. click)

  • overlay App: no reason to test
  • game App: no reason to test
  • Both: works as described, overlay App input works, but no game Input.

• Advanced (i.e. touch, double, long)

  • Touch:
    • Game App: not blocked
    • Overlay App: not blocked
    • Both: input's blocked
    • For all intents and purposes it appears that "touch" is effectively a separate button.
  • Double (and presumably others):
    • Overlay App: Blocks input from Game
    • Game App: input's blocked (overlay on simple)
    • Both: input's blocked

• Chords

  • Overlay App Chorded

    • Game App Simple: No Blocking
    • Game App Advanced: No Blocking
    • Game App Chorded (share 1 input): No Blocking
    • Game App Identical Chord: No Blocking

  • Game App Chorded

    • Overlay App Simple: Blocked
    • Overlay App Advanced: Blocked
    • Overlay App chorded (share 1 input): No Blocking
    • Overlay App Identical Chord: No Blocking

Notes

• Higher Priority also appears to disable dashboard input, i.e. if you rebind the trigger then you can no longer select with trigger in interfaces such as keybinding UI.

• Found a bug/Unintended? behavior with chords and advanced options, and advanced option such as double May disable a Chord using the same input. (SteamVR Input Chord conflict with advanced input options #1295)

• Did not try to adjust action sets on the fly or adjust priority.

• Just tested digital actions

• "Enable Global Input from Overlay's" is not really descriptive. "Allow Overlays to Override App Input" or similar is more indicative as by default overlay inputs are registered over apps.

Thoughts.

This will be an extremely powerful option for certain overlay applications.

Overlay Applications that are "always on" (like OVRAS) may see less benefit, and a small group of users may see it useful, but best implementation would require different settings for different games.

Overlay Applications that will benefit hugely are those that need a "temporary" state, such as overlay applications using it during an "edit" mode where any input going to a game/app will be undesirable.

Other applications could also be developed such as parental control/lockouts for arcades ETC. Or perhaps paging features etc.

It could also be used to limit a users ability to interact in certain ways.... perhaps good for Demo's etc. (though it is somewhat duplicate functionality to manually adjusting the key-binds in this instance)

With this system it adds additional design consideration to "what" your action sets are, and possibly needlessly inflates amount of action sets needed, while better than none a per "action" system might be better, but would need dynamic adjustment, otherwise it may inflate actions in a similar manner.

Conclusion

While not universally needed it should greatly help certain apps in providing a more intuitive and less "janky" experience.

If you need anything else or have specific questions feel free to ask.

edit: formatting.
edit2: added issue, a few more notes on testing
edit3: added note about global input
edit4: removed issue as I believe its intended behavior upon further investigation.

[Hokage3211]

@JoeLudwig @ykeara I'm having some trouble getting this to work, or how to tell if I'm doing it wrong, for testing purposes I'm trying out trying to override the grab functionality, and in my overlay's action set I have a binding in the grip button section, type "use as grab", and then an action assigned to that, which matches to how other VR games are using the index controller squeeze to grab functionality, such as steamVR home or VR-chat, which I am using as testing for now.

This is my code, trying to apply the action set to all controllers, to override it in the running applications.

            set = new VRActiveActionSet_t();
            ulong handle = 0;
            OpenVR.Input.GetActionSetHandle("FarmGrab", ref handle);
            set.ulActionSet = handle;
            set.ulRestrictedToDevice = 0;
            //smallest would be 16777217
            //biggest number is 33554430
            set.nPriority = 16777217;

            VRActiveActionSet_t[] ar = new VRActiveActionSet_t[1];
            ar[0] = set;

            var error = OpenVR.Input.UpdateActionState(ar, (uint)System.Runtime.InteropServices.Marshal.SizeOf<VRActiveActionSet_t>());
            Debug.Log("Update Action State Returned: " + error);

The error returned is "None" and I can see it returning that live to make sure, but I can still grab things in the scene game, I did see at one point when doing random changes (can't remember what exactly) that it had unbound the grab action in the grip button binding section of steamVR home, but apparently the home application uses both force sensor and grab to detect if the user wants to grab, so I tried adding an empty action "use as force sensor" to try and override both and fix it, but then it stopped working entirely again. My application is also initialized as an overlay application, so this should be working.

Any help would be very much appreciated!

[ykeara]

looks okay to me at a glance make sure you have k_pch_SteamVR_AllowGlobalActionSetPriority set to true, can do via SteamVR's Settings menu or pro-grammatically assuming you are on a version of the headers that has it.

I haven't tried it for analog actions, but

[Hokage3211]

I did make sure to enable that under developer settings, and I also made sure today I was on the latest master head of the repository.

[ykeara]

I'm not super comfortable w/ c# only thing I might see is your updateActionState has 2 params not 3.

[Hokage3211]

I think they removed the final argument for the c# version because you can query the length of the array automatically, as this one that I have only takes 2 arguments. Hm... so you're saying I should try it on simple ones like "button click" stuff?

For reference the method is here, but no documentation just listing things: https://valvesoftware.github.io/steamvr_unity_plugin/api/Valve.VR.CVRInput.html#Valve_VR_CVRInput_UpdateActionState_Valve_VR_VRActiveActionSet_t___System_UInt32_

[ykeara]

Yea Just implemented a version on my end and I don't really have issues, implemented much like yours though only digital actions.

[Hokage3211]

Hmm no dice, I tried assigning the B button (with "use as button") to an action inside the same set, in my overlay application, and then executing that code while running vr-chat, and I could still open up the quick menu with B button. So seems like something isn't working, despite there being no errors.

[Hokage3211]

Here is the image for my bindings

And here is how VR-chat binds their grip section

So I would expect my bindings to "overlap" with theirs, but it doesn't seem to unbind/override anything. Is this the kind of bindings you're using? Or what should I be testing on?

[Hokage3211]

Any help at all? I'd really like to get this working but no idea how, does setting certain flags break this? I'd really like if this were actually possible to get working on my end, and how to even tell if it's working at all.

[Toxic-Cookie]

Replying to #1236 (comment)

I'm also struggling to get this working and seem to be doing something wrong. Could you provide a working example code snippet or repo?