How can we integrate semantic labels into existing specs?

[cabanier]

WebXR Planes, Meshes and Hit Testing can return the semantic label of the real world objects.
How would we specify these?
OpenXR returns a list of labels but so far, only 1 item is ever returned.

/facetoface integrate semantic labels into existing specs

[De-Panther]

While currently most of the WebXR experiences are meant for room scale scenarios, in large scale scenarios, the developer would want to query for semantic data, so instead of going over all the Planes/Meshes in the scene and look for chair, an API that returns all the objects that are marked as chairs would work better.

[cabanier]

While currently most of the WebXR experiences are meant for room scale scenarios, in large scale scenarios, the developer would want to query for semantic data, so instead of going over all the Planes/Meshes in the scene and look for chair, an API that returns all the objects that are marked as chairs would work better.

Why can't a developer just iterate over the plane and mesh list to find the chairs?

[De-Panther]

If we are talking about the current wide implementations, yes, a list of 10 or 20 it's not a big deal to iterate over.
But I see the future with implementations of semantic data for larger scale, where you can easily have hundreds or thousands of objects on larger scenes. And there are already companies that handle those kinds of scenarios today. (e.g. handling a large malls or airports with lots of points to navigate to. Hotels and office buildings with lots of IOT devices to control)

So in general I'm not sure if it's the right decision to get the semantic data from the 3D objects. (maybe it is? maybe it won't affect future changes? I'm not sure)
But I do think that we'll need more data oriented approaches for more complicated scenarios.

Contrary to established APIs like head pose, controllers poses, etc... AR semantic data is still getting it's shape, it's hard to predict what future APIs would look like.

[cabanier]

We can always extend the API when the time comes that richer APIs are available. It's unclear if labels in the future will be user generated or if the system will be intelligent enough to label them automatically. I don't think we can wait until that time or come up with an API that will continue to work in an unknown future.

For now, the spec follows the path that the OpenXR group is setting up. I agree it's not great but at least authors will now have some idea what the planes and meshes represent.

[De-Panther]

Extending the API is great.
We do need to see if future possible changes would allow to only extend the API (which is great), or would require to deprecate it (which will make some experiences obsolete).
Looking at current less common but more advanced use cases, can help us with that.

The scenarios I mentioned of:

• Shopping centers with semantic data of stores or doors to stores, for indoor navigation.
• Airports with semantic data of stores, help and service desks, for indoor navigation.
• Hotels and Office buildings with semantic data of IOT devices to control.

Are all real world scenarios that are in operation on different stages.

On some cases, semantic label is too limiting, but I think that semantic label is something that can be extendable.
On other cases the API is what can be limiting.

Maybe we also need some kind of scheme/schema for the semantic data.
e.g. for now, the list of desk, couch, floor, ceiling, wall, door, window and other is enough, but maybe we'll have more data like the type of desk (writing desk, dining table) or the material of the door (wood, metal).
In that case, only label is limiting, and another structure is needed, like with the WebXR Input Profiles.