How to synchronize multiple D405 cameras

[willyum00]

---

Required Info
Camera Model	D405
Firmware Version	5.15.0.2
Operating System & Version	Win 11
Platform	PC
SDK Version	2.0
Language	opencv, python, unity
Segment	AR

Issue Description

For my purposes, I want to append multiple point clouds together to form a single 3D model / environment / image. Is this possible using a multi camera configuration (4-5 cameras) ? The cameras I am currently using for this project is the D405, which doesn't have the JST connector for synchronization. Is it still possible to synchronize them? And if so, would I be able to achieve my goal?

Thanks!

[MartyG-RealSense]

Hi @willyum00 If the cameras all have to be attached to the same computer then you may be able to adapt the ROS multicam pointcloud stitching tutorial below.

https://github.com/IntelRealSense/realsense-ros/wiki/Showcase-of-using-2-cameras

There is also a multiple computer version of the tutorial.

https://github.com/IntelRealSense/realsense-ros/wiki/showcase-of-using-3-cameras-in-2-machines

The above tutorials are for ROS1. RealSense's official ROS1 compatibility 'wrapper' does not have D405 support (the official ROS2 wrapper does support D405), so a custom 'forked' version of the wrapper created by a RealSense user would have to be built instead

https://github.com/rjwb1/realsense-ros/tree/development

---

Another multiple camera, mutiple computer pointcloud stitching tool can be found at the link below, though it has not been tested with D405:

https://github.com/conix-center/pointcloud_stitching

[MartyG-RealSense]

Regarding your question about multiple camera capture in Unity, you might find the resources listed at #2203 (comment) to be helpful.

[willyum00]

okay, I have looked at the extension by Ajingu as well as the two camera set up by user FrenchTouchX.

https://github.com/ajingu/RealSense/tree/master
#2203 (comment)

Frenchtouch needed to make changes to the case.modeRecord section of the code, but I needed to change the case.modeLive portion. made a change similar to him, by copy and pasting

                foreach (var p in Profiles)
                    cfg.EnableStream(p.Stream, p.StreamIndex, p.Width, p.Height, p.Format, p.Framerate);

under case Mode.Live

However, it didn't seem like this changed much for me. Is this still necessary or has this part of the SDK been updated since then?

I have not yet downloaded the extension by Ajingu, mainly because I couldn't figure out how to download it. Frenchtouch mentioned that the calibration from this extension would have been helpful to him, but not sure if he ever used it. Could you show me how to download it?

Here is a video of what unity looks like for me right now. As you can see, both cameras are recognized by unity, but only one of them is displaying a point cloud and the other one just distorts the other point cloud without generating its own point cloud. Not sure why this is happening.
Thanks!
https://github.com/IntelRealSense/librealsense/assets/81535097/4a1807c0-2e7e-4cde-a893-dee5b7e52a44

[MartyG-RealSense]

My understanding is that the use of the old Ajingu extension or the FrenchTouch method is now unnecessary and you should only need to create a separate RsDevice object for each camera like in the JMcC16 guide and set an individual camera serial number for each RsDevice profile.

https://support.intelrealsense.com/hc/en-us/community/posts/360047049173-Community-contribution-for-using-multiple-cameras-in-RealSense-Unity-wrapper

[willyum00]

Hi Marty,

So I attempted to follow this guide by JMcCl6 but I am not achieving the desired results. I believe it has something to do with steps 2 and 4, which I was confused about.
For example, in the current version of Unity, I am unable to duplicate shaders. When I copy and paste the point cloud shader and try to assign it to the duplicated device, both the original and duplicated shader are selected, as if they are connected.

I changed the serial numbers for both cameras:

At first, both point clouds seemed to be "invisible."

I believe the reason the point clouds are like this is because I am unable to assign a shader to each rsdevice, because as preivously mentioned, duplicating the shader just does this during selection:

Which causes issues.

Only one of the cameras seems to have a feed. Even though there are two point clouds created, both point clouds are created by the same camera. Please let me know if you believe I performed any of the steps incorrectly.

If I can't figure out how to get it working in Unity quickly, I am considering moving away from Unity and doing a code based solution, similar to #11941. Can what this user did be applied to creating a real time 3D model? I was wondering, how can I access opencv_pointcloud_viewer.py? I cannot find it in the SDK I've downloaded, just in the librealsense GitHub.

Thanks!

[MartyG-RealSense]

Your steps look fine to me.

I think that it may be selecting both the original and the duplicate because despite the duplicate likely having a different name (such as PointCloudGeom [1]), the duplicated shader's internal contents still have the name PointCloudGeom. So the only way to change that would probably be to open the duplicated shader in Visual Studio and edit its name within the shader code.

With the Python example programs you should be able to just create a new pyrealsense2 script file and paste the example code into it, then launch the script.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/opencv_pointcloud_viewer.py

A RealSense user created a multiple camera Python program at the link below, though the streams from the different cameras are not aligned.

https://github.com/ivomarvan/samples_and_experiments/tree/master/Multiple_realsense_cameras

In the box_dimensioner_multicam.py tool shown in #11941 my understanding from its code is that it does generate a merged pointcloud from all attached individual cameras but does not show the pointcloud onscreen visually, instead using the data to generate a bounding box around a box on the chessboard.

[MartyG-RealSense]

Hi @willyum00 Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[Gowthers]

@MartyG-RealSense, sadly I couldn't find an answere on how to (hardware) synchronize frames between different D405 in this thread. Could you help me with this? Thanks in advance

[MartyG-RealSense]

Hi @Gowthers The D405 model is not equipped with the set of hardware sync pins necessary for hardware synchronization between cameras, unfortunately.

[LLLYLong]

@MartyG-RealSense Hi，I would like to ask which model of camera is equipped with a time synchronization pin, and I need to connect 3-4 of the same model in parallel to get the depth information as well as the point cloud information.

[MartyG-RealSense]

Hi @LLLYLong All RealSense 400 Series camera models except D405 have a set of hardware sync pins. That includes D415, the D435-type and D455-type cameras, D456 and D457.