WebXR Hit Test Module

[mounirlamouri]

Hello TAG!

I'm requesting a TAG review of WebXR Hit Test Module.

Exposes hit-testing (raycasting) capability for WebXR. This API would allow a developer to cast a ray into the real world and return a list of intersection points for that ray against whatever world understanding the underlying system gathers.

• Explainer¹ (minimally containing user needs and example code): https://github.com/immersive-web/hit-test/blob/master/hit-testing-explainer.md
• Specification URL: https://immersive-web.github.io/hit-test/
• Tests: https://github.com/web-platform-tests/wpt/tree/master/webxr/hit-test
• Security and Privacy self-review²: https://github.com/immersive-web/hit-test/blob/master/security-privacy-questionnaire.md
• GitHub repo (if you prefer feedback filed there): https://github.com/immersive-web/hit-test
• Primary contacts (and their relationship to the specification):
  • Piotr Bialecki, @bialpio, Google, Editor
  • Trevor F. Smith, @TrevorFSmith, CG Chair
  • Ada Rose Cannon, @AdaRoseCannon, Samsung, WG Chair
  • Chris Wilson, @cwilso, Google, WG Chair
• Organization(s)/project(s) driving the specification: Immersive Web CG
• Key pieces of existing multi-stakeholder review or discussion of this specification: N/A
• External status/issue trackers for this specification (publicly visible, e.g. Chrome Status):
  • Chrome Status: https://chromestatus.com/feature/4755348300759040

Further details:

• I have reviewed the TAG's API Design Principles
• Relevant time constraints or deadlines: Chrome will have an Intent to Ship soon for this API.
• The group where the work on this specification is being done: Immersive Web CG.
• Major unresolved issues with or opposition to this specification: N/A
• This work is being funded by: N/A

You should also know that...

• N/A

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

[hober]

I'm worried that we're on track to spec hit testing for XR before we spec hit testing on the rest of the platform. If this happens, we risk speccing something for XR that doesn't match the (now implicit) hit testing model of the rest of the platform.

I wonder, would anyone working on XR hit testing be willing to simultaneously work on a hit testing spec for the rest of the platform, ideally with a common model shared between the two specs?

[mounirlamouri]

Hit Testing for XR is a very different problem than hit testing for a document. The obvious differences for example is that the XR environment is in 3D where hit tests are fairly known concepts (eg. video games). I don't know what are the concerns with the document hit testing but would they apply to a 3D DOM-less environment? I would be interested to know why you think we can make document hit tests and XR hit tests share the same concepts/models.

XR hit test can also be limited by the underlying platform. For example, with AR hit tests, the underlying ARCore/ARKit frameworks would actually be doing the checks and the UA would only package the response back to the website. Among others, the consequence is that the hit test API in XR has to be asynchronous while a document hit test wouldn't.

[hober]

Hit Testing for XR is a very different problem than hit testing for a document.

Given #470, it's not clear to me that this is the case.

[hober]

In mozilla/standards-positions#259, @larsbergstrom said:

We believe that it is premature to establish a Mozilla standards position or get TAG review of the WebXR Hit Test module. There are a significant number of security, privacy, naming, and functionality issues with the current proposal (https://github.com/immersive-web/hit-test/issues), and it has not been prototyped by any other party yet - and most importantly, never on a headworn AR device.

[a2sheppy]

One important issue here is that a better job needs to be done in WebXR specs and docs to clarify the difference between "hit testing" in the context of using ray casting with sensors to detect intersection with real-world objects vs. using virtual ray casting to determine intersection of a ray with virtual objects in the scene. These are very different concepts, and it actually took me quite a while to realize that the phrase "hit testing" in the WebXR specs was referring to real world, not virtual world, hit testing.

I am curious what the long term usage of virtual vs real-world hit-testing will be in terms of which will be most common; I suspect that once XR devices are commonplace and VR is the primary use case rather than AR, we will find that virtual hit testing is by far the more common scenario. Regardless, it's worth considering this issue when naming and describing this capability, and I would at least suggest renaming the spec to something more like "WebXR Physical Target Hit Testing" or something related to the real world.

[torgo]

Hi @mounirlamouri - any current status on this? Any reaction to the comments left since Jan here? Has anything substantively changed that we should take a look at?

[mounirlamouri]

Thanks for following up @torgo. I don't think there were any comment that required following up.

1. The API is for hit testing in mixed reality environment, not in documents.

2. Mozilla's feedback was somewhat strange given that they were implementing the API at the time and shipped it. I also don't see how it's relevant to the review given that the feedback had no technical basis.

3. The API is about hit testing and there were discussions about allowing that in VR but seemed not very useful given that the author is in full control of the VR experience and can do hit testing using regular Maths. If an implementer felt strongly about it, there is nothing that would forbid it.

[torgo]

Thanks @mounirlamouri. Thanks for the feedback on our feedback. We discussed this today in a TAG breakout today. We're gonna suggest to close this in our plenary this week.

[hober]

Hi @mounirlamouri,

@dbaron and I took a look at this during our vF2F today, and were wondering why this uses Float32Array to represent 4x4 matrices when DOMMatrix already exists on the platform for 4x4 matrices.

[dbaron]

I'm also curious if you have thoughts on @a2sheppy's comment.

[mounirlamouri]

@dbaron my comment above (point 3) answers @a2sheppy comment. To summarise, the spec is mostly intended for AR (real world) but doesn't forbid it to be used for VR (virtual world). Some members of the WG felt it could be useful. It is up to the implemeter.

@hober Float32Array is used by the base Web XR spec (https://www.w3.org/TR/webxr/). This spec stays consistent with it.

[atanassov]

Looking through the current version of the spec (a first read for me), I find it lacking use cases and example usage of the proposed API. Would be great to either take some of the explainer tests or add new ones to improve the quality of the spec.

[atanassov]

We did one final review during our Breakouts A and B today (Mar 15, 21) and agree with the overall direction of the proposal. Again, as pointed out in my previous comment, the spec can benefit from examples so that the dev community can have a better idea of the intended use. Thank you for working with us, good luck.