markerless tracking without tango (/ios equivalent)

[kylebakerio]

Do you want to request a feature or report a bug?
Request a feature

What is the current behavior?
No markerless tracking (?)

Please mention other relevant information such as the browser version, Operating System and Device Name

What is the expected behavior?
The equivalent markerless tracking on regular phones to this experience:
https://immersive-web.github.io/webxr-samples/immersive-ar-session.html?usePolyfill=0

If this is a feature request, what is motivation or use case for changing the behavior?
Obviously markerless AR would be pretty great, especially in making this a viable alternative to e.g. 8th wall.

I have a question... it seems like ar.js doesn't handle "markerless AR" unless one is running a special pixel (tango) or ios device (whatever their equivalent is). Is that right?

If that is right, I'm confused by how this demo seems to do it on my oneplus 6t and my girlfriend's oneplus 5, which don't have tango or anything: https://immersive-web.github.io/webxr-samples/immersive-ar-session.html?usePolyfill=0

That seems to do full markerless tracking, which blows me away--does ar.js support that kind of experience, or am I correct in understanding it doesn't and that some kind of marker (or gps coordinate) is needed?

In any case, is there an intro explanation for someone like me (a web dev who works with webxr vr stuff) on how this is possible without specialized hardware? They're just utilizing stuff in the browser, no special ML models or anything, right?

Unless I'm misunderstanding something, shouldn't it be possible to port this functionality into ar.js? What am I missing here?

[kylebakerio]

digging into this further... just going to put stuff as I read about it here.

• based on the discussion here (more specifically, this comment, I seem to gather that ARcore and ARkit are the proprietary frameworks that have to be utilized to get AR in iOS and Chrome, and are not real web standards, and thus not preferred to be utilized by AR.js. Is that right...? A bit confusing, since AR.js did support Tango and ARkit in the past, no? Would it not be ideal to support these when available?
• I did see mention of an open source SLAM option, and a comment by @Carnaux, who did the excellent NFT image tracking, about doing SLAM integration to AR.js in the future--is there / was there any work towards that?
• Haven't looked here, but this was also mentioned: https://github.com/XingMeansOK/slamjs_samples

[kylebakerio]

This article seems like it might be relevant: https://medium.com/naver-fe-platform/webar-with-webxr-api-part-2-dc76b20767fb

[nickw1]

@kylebakerio I am one of the contributors to the location-based component of AR.js. As you said above, WebXR uses ARCore on Android/Chrome, and it would indeed be nice to have a pure FOSS solution without relying on any proprietary services.

This is something I am interested in myself to enhance location-based AR.js, and @nicolocarpignoli pointed me in the direction of Thorsten Bux, who has done a lot of AR work. Thorsten gave me this pointer: TrackingJS. It sounds like TrackingJS needs updating to reflect some changes in web camera APIs, but also it looks like it could be used to implement real FOSS markerless AR within AR.js. Time permitting I'm aiming to look at this in the coming months.

[kylebakerio]

I see. So, the intention is to not utilize ARcore/ARkit stuff provided baked in by Google/iOS, because, for example, FF can't match those because they're proprietary? Is that right?

[kylebakerio]

Integrating something like the work of this guy: https://github.com/boehm-e/webxr_threejs_AR

Seems like it would be ideal. At one point AR.js supported Tango, which was a very early precursor to the modern ARcore. I know it would be better if it wasn't proprietary code there, but I'd say it's a critical part of the AR web. AR.js is going to lose relevance if an open source library (or fork) supports that tech and AR.js doesn't. It's just a matter of time.

By all means, let's support and encourage and push to integrate other solutions the reduce dependency and allow cross platform support, but... I don't think it makes sense to ignore ARcore/ARkit.

[nickw1]

@kylebakerio different contributors to AR.js have different opinions, but the general consensus - I think - is to aim to build something cross-platform and cross-browser. @nicolocarpignoli any comments as I know this has come up before?

[kylebakerio]

I fully agree we should aim to be cross browser, but as I mentioned before, the original aim (from the original mtaintainer, I recognize this project is now a collective project with a different maintainer status) included a 'tracking: best' that would use tango if available, and fall back to other options when not, for example. I still think this would clearly be the way to go. While I think it's fantastic to support and lean on fully cross-browser options where possible, it would be really a shame if AR.js did not include support for the best option on whatever platform it runs on (including ARcore when available, which as far as I understand itself will utilize ARkit when available).

Another example of hand-rolled 6dof markerless AR I just found today: https://github.com/ButzYung/SystemAnimatorOnline

[nickw1]

@kylebakerio I don't have a problem with ARCore/WebXR myself, but I am only a recent contributor to the project and have only added quite a small part (enhancements to the location functionality); I think we'd need to get consensus from more of us before adding in support @nicolocarpignoli @kalwalt

[kylebakerio]

I'm more of an a-frame dev and general JS dev. I recently built out a '3dof AR' proof of concept for a client product we'll be doing soon that's pretty crude. But I'm open to helping with this contribution as well, it's an interesting topic. I have a multiplayer webxr VR app that doesn't make a lot of sense on mobile, but I could make it make way more sense as an AR spectator mode, and this sounds like some fun cutting edge stuff to play with.

But I'm just making my case here, to be clear. Looking forward to whatever maintainers respond. Whether AR.js implements this or not, some open source solution needs to be out there that makes ARcore development more accessible.

[nicolocarpignoli]

Hi @kylebakerio

I'm Nicolò, the former AR.js maintainer (after Jerome Etienne). Thanks for your questions and your interest. I'm trying to give my opinions on this.

Talking about markerless Web AR, currently we have some proprietary-paid solutions (the most famous is 8th Wall) and some proprietary but free solutions, like Google's Model Viewer.
Google's Model Viewer is a web component that do exactly what you need: it gives you markerless Web AR on android and iPhones, and to the "fallback" trick: first, it tries to use the WebXR Device API - that is free, open source, but working on latest Chrome and FF only on androids, as for now- then fallback on QuickLook for iOS or on Scene Viewer on androids.

Those two systems (QuickLook and S.Viewer) are "fake" Web AR because they are native web app triggered by web link. So you can't customise much the experience. But until Apple does not adheres to WebXR Device API, markerless Web AR will remain on this state, probably. This is pretty much the state of the art.

What is AR.js now? It is the only Web AR solution that is free and open source and cross browser. But it gives you marker, image tracking and GeoAR. Not markerless AR (if you mean the "8th wall thing" where, after recognising the ground, you trigger a content and it feels like it is around you).

So, there are some scenarios here: first of all, we include Model Viewer on AR.js.
I think this does not make much sense, because Model Viewer is a great library, well maintained, and I do not see the point on wrapping it with AR.js. It is already easy to use, and if one needs both AR.js and Model Viewer, he/her can import both on a new WebAR App. It's how any web app works.

Another scenario: do a completely new, totally web-based markerless AR.
Why? Because WebXR Device API is not ready, not Cross Browser and will not be available on Apple for a lot, probably. And because Model Viewer is great but poorly customizable (talking about the AR part).
So, this indeed makes sense, and it makes sense to add it to AR.js. I'm not against this solution, but it requires a lot of work. So if anyone is willing to research and try to do this, I'm really happy about this.

My point is: when WebXR Device API will arrive (or a good polyfill for it) on Apple, with Markerless and other features, probably AR.js markerless solution will be deprecated, because WebXR Device API (for example, along with AFRAME support) will be more maintained and standard.

We are in between where there is no OSS markerless cross-browser Web AR, but we know there will be.
If someone wants to put some effort on filling this gap (that will last, at least, another couple of years I guess) I think It will be great.

[nickw1]

@nicolocarpignoli thanks for your comments - I know you've said the same to me before but thanks for letting @kylebakerio know your thoughts.
@kylebakerio I am keen to work on Nicolò's suggestion of adding open source web based markerless AR - and I have already briefly looked at trackingjs.com. Are you keen to work on this as a collaborative project? I have little time now but I will likely have more time in the summer, June to September time.

[kylebakerio]

@nicolocarpignoli I could very well be mistaken, but I've barely come across model-viewer recently when digging in to the state of AR, because it sounds like something too simply to be useful--just a tool to view single models. That to me is a relatively minimal subset of AR, as opposed to the '8th wall thing' which is just, to me, what I expect 'full AR' to be.

But I think your information / understanding of the state of the art might possibly be a bit out of date?

Last night I found, for example, aframe ar, through this blog post. I've also been in discussions with one of the few web-ar content creators I was able to find, who expressed that he just used raw WebXR JS to create his experience--no mention of model-viewer. Those examples (as well as one more I found from a french coder as a toy project), like the demo here (that example is surely not using model-viewer, we can agree, no?) work on my oneplus 6t (that last one also seems to work on my girlfriend's oneplus 5, didn't test the others), and I'm pretty sure from what I understand and have read that they would work in iOS 13 via the built-in ARkit that would be implemented seamlessly by calling the WebXR api via an immersive-ar session.

If I'm wrong, I apologize, this is a fast moving scene that's hard to find reliable information on, but that's the impression I've gotten--it seems different than what you've suggested here.

[kylebakerio]

Looking at Model Viewer's demo page, we can read,

"Finally, even complex DOM can easily function in both 3D and AR modes, including interactions via script. In this case a simple carousel of models is demonstrated. Unfortunately none of this DOM content can flow into SceneViewer or QuickLook, as these are native apps. Only through WebXR, now our default AR mode, can this be achieved, as the AR session is still inside of the browser. This also removes the need to redownload the model."

They're drawing a comparison between SceneViewer and WebXR, and mentioning they now default to WebXR, which has capabilities that SceneViewer does not have. It's also implied that using scene viewer was how it worked in the past, but not anymore--in other words, it looks like even Model Viewer no longer leans on SceneViewer, but on WebXR by default.

[nicolocarpignoli]

But I think your information / understanding of the state of the art might possibly be a bit out of date?

That's totally possible :)

I want to clarify two things:

• I'm not saying Model Viewer is the answer for the best Markerless Web AR around. It's limited, as I said on previous comment. You can for sure use WebXR Device API, but.. (and we go to the second point)
• Are you really sure that those examples you linked are working, using WebXR, on iOS? Have you tried them?

Because, from my perspective, if they do work on iOS, that will be great and I will agree to try to include that tech on AR.js.
But if not, then we go the the first problem: cross-browsability.
And about that I'm against introducing something that works only on androids on AR.js. That's my opinion, you can do that anyway, of course - now AR.js is owned by a free org, open to anyone.
But I do not see strong points on adding something that works on some devices only when we already have that "outside" AR.js. AR.js is cross-browser and that is its best feature, any new stuff added should adhere to that.
In such case, as I said, I'm agreeing with Nick to try to do a completely new solution, cross-browser and not using WebXR.

TLDR: let me know if you (or someone else) can test those WebXR examples on iOS. I was pretty sure they do not work, but I might be totally wrong. If they do work, my opinion is that they should be integrated on AR.js. Otherwise, I'm more for a new solutions that is cross-browser. I will not have time to work on that, so again this is my opinion and I do not want to stop anyone to do something. AR.js is open to any contribute and if you think a not-cross-browser solutions is good, you can add it anyway :)

[nicolocarpignoli]

Looking at Model Viewer's demo page, we can read,

"Finally, even complex DOM can easily function in both 3D and AR modes, including interactions via script. In this case a simple carousel of models is demonstrated. Unfortunately none of this DOM content can flow into SceneViewer or QuickLook, as these are native apps. Only through WebXR, now our default AR mode, can this be achieved, as the AR session is still inside of the browser. This also removes the need to redownload the model."

They're drawing a comparison between SceneViewer and WebXR, and mentioning they now default to WebXR, which has capabilities that SceneViewer does not have. It's also implied that using scene viewer was how it worked in the past, but not anymore--in other words, it looks like even Model Viewer no longer leans on SceneViewer, but on WebXR by default.

It is true. Model Viewer use WebXR by default and then fallback on Scene Viewer (for androids not supporting webxr) or QuickLook (for iOS). iOS, as for now, does not support WebXR.

[kylebakerio]

I see. You are correct. I was mistaken.

What about Apple devices? Apple has made a decision at this point not to allow mobile web applications to access the AR features of iOS devices directly. Apple’s strategy continues to be the use of QuickLook to enable these experiences. This is unfortunate, because interactive augmented reality experiences on IOS will require native apps, or will require time-consuming conversion of dynamic content into a static model that will be limited by the capabilities of the USDZ format.

We are looking forward to providing consumers with rich, dynamic, and fast AR experiences on Android devices and we hope Apple will enable similar experiences in the near future.

It's been confusing getting up to speed on the topic, but because everywhere that mentions both ARcore and ARkit juxtaposes them, and they're mentioned as parallel tech, and ARcore is used in browser behind WebXR... I just didn't see any conversation on the iOS side of things (which, admittedly, I didn't look for, as I kind of loathe iOS and don't focus on myself).

I would still push back, however, and say that it should be up to the developer whether they want to make sure they have cross-browser experience focus or cutting-edge-feature focus. To me, this is different than Model Viewer, this is about the de-facto open source AR library supporting the latest WebXR features or not. If iOS chooses not to be standards compliant and ignore it because they want a walled garden, I don't think we should be held back from access. On the contrary, the way to advance the state of AR on the web is to create content and make the ecosystem of available experiences out there compelling enough that Apple won't be able to ignore it.

[kylebakerio]

I personally would support having a pop-up that pushes iOS users to switch to webXR viewer, which is a way for them to view the standards compliant content. In practice, it's not much different from being prompted for play-vr services or xrcore download on first visit to such content on chrome mobile, no?

Would that not be the most viable cross browser path forward?

[kalwalt]

Hi to all here! Just few words on this... Ar.js was born because should became a developed interface to jsartoolkit5 the port to javascript (thanks to Emscripten) of the awesome ARToolKit5 project for Augmented Reality. At that time jsartoolkit5 support only Markers even if NFT marker tech was available in the C++ code. Tango was a google project that is disappeared ... i think it for just few years, it lasted as long as the beat of a butterfly. AR.js and jsartoolkit5 are still here. We need to improve them, because technology is moving fast. Slam would be awesome, there are guys that are trying testing different solutions (@Carnaux when you have time...). Adding support to (ARcore, ARKit...) would be nice but not a prority. I collaborated with @nicolocarpignoli to add the NFT (Natural Feature Tracking) to the AR.js core but this still needs some improves. Another super awesome feature is the ES6 support see this PR #116 This not to stopping you @kylebakerio but to give you some inputs to some aspect that AR.js need to be improved, before adding new features.

[jadams777]

Hey folks, can we please restart this conversation? The world needs an open source alternative to 8th Wall. They charge $3K per month, and have strict content moderation, so only vanilla content is allowed. Can I pay someone to add SLAM to AR.js?

[nicolocarpignoli]

Hi @jadams777 you can sure paid someone in order to have an OSS SLAM AR.js. The task is not easy at all, but this repo https://github.com/alanross/AlvaAR seems promising. try reaching the author

[holographica]

Hey, just wondered if anyone's had any progress with implementing SLAM for AR.js, or alternatives (eg hit testing)? Working on a simple indoor AR tour/navigation project for a webapp and having a lot of trouble with getting started - I'm just a student so a lot of it goes straight over my head at the minute. Did you have any luck @jadams777 ?

[HamzaSaboor]

I'm using ARFoundation for Vertical Plane detection in Unity but I came to find out that ARFoundation doesn't support AR in WebGL (Browsers). So, I'm now exploring different options to perform Vertical Plane detection in WebGL (Browsers), doing some research brought me here. So, can somebody guide me?