extracting pointclouds of multiple cameras

[Nimaro76]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	{ R200 / F200 / SR300 / ZR300 / D400 }
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	{Win (8.1/10) / Linux (Ubuntu 14/16/17) / MacOS
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC/Raspberry Pi/ NVIDIA Jetson / etc..
SDK Version	{ legacy / 2.. }
Language	{C/C#/labview/nodejs/opencv/pcl/python/unity }
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

Hello,

I am currently working on synchronizing multiple cameras to obtain point clouds from each of them when working simultaneously. To achieve this, I require both depth and color images from each camera. My question is whether hardware synchronization is necessary for this task or if it can be accomplished through software synchronization.

I came across the multicam example, but it rendered a frameset where depth and color frames were not separate variables. I plan to try separating them soon to assess the impact. I have also reviewed previous discussions, but they did not alleviate my uncertainty about which synchronization method to choose.

I would greatly appreciate your assistance and advice on this matter.

Thank you for your time and consideration.

Kind regards,
Nima Roshandel

[MartyG-RealSense]

Hi @Nimaro76 RGB color cannot be synced by hardware synchronization, but the RealSense SDK has mechanisms for synching depth and color. When depth and color FPS speed is the same then sync between them should automatically take place. It is only synching the depth and color on that particular individual camera though.

You can help to ensure that depth and RGB FPS remain the same by having auto-exposure = true and setting to false an RGB option called Auto-Exposure Priority. When auto-exposure is true and auto-exposure priority is false then the SDK will try to enforce a constant FPS for both.

It is not a necessity to use hardware sync when generating point clouds from multiple cameras though. For example, at the link below is a guide for doing so in ROS without sync and combining the separate clouds from each individual camera on the same computer into a single merged cloud.

https://github.com/IntelRealSense/realsense-ros/wiki/Showcase-of-using-2-cameras

This merging of pointclouds can also be done with cameras that are attached to different computers instead of all being attached to the same computer.

https://github.com/IntelRealSense/realsense-ros/wiki/showcase-of-using-3-cameras-in-2-machines

[Nimaro76]

thank you for your reply,
for my project, I simply don't want to use ROS. the preferred approach would be using a single c++ algorithm for this purpose. for example, expanding the current multicam example to be able to get the point clouds and merge them. I was wondering if this expansion is possible. and I would like to use a single computer for running the algorithm.

[MartyG-RealSense]

If you have a Windows computer and are able to make use of commercial software then the RealSense-compatible scanning tool RecFusion Pro could do that as the Pro version supports multicam.

https://www.recfusion.net/

An open-source C++ alternative would be to use the RealSense SDK's rs-kinfu example. Instead of combining together pointclouds from multiple cameras, you can move a single camera around the scene and progressively build up a pointcloud with 'frame fusion'.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/kinfu

[Nimaro76]

Unfortunately, it is not in the nature of my project to use softwares in order to perform this task. there are two necessary features that should be in the algorithm:

1. it has to be multi camera because the idea is segmenting the frames of each camera to do objection detection and further tasks
2. the algorithm has to be running on one computer . so we want to have like 4 cameras connected to Xilinx board and then run that C++ algorithm to get the point clouds of each of theses cameras that are running simultaneously.

with theses two key features that are required, do we need to do hardware synchronization? or software synchronization is still a possible choice
thank you for your reply

[MartyG-RealSense]

Hardware sync is not a necessity for pointcloud merging, as demonstrated by the ROS example and by RecFusion Pro.

If 4 cameras are going to be active simultaneously though then you will need a computer specification that can cope with the amount of computing resources that 4 cameras attached to the same machine will consume. For that number of cameras, a computing device with a CPU equivalent to an Intel i7 would be recommended.

If Global Time is true (it is true by default on 400 Series cameras) then it will attempt to match up multiple cameras to a common timestamp, as described at #3909

[MartyG-RealSense]

Hi @Nimaro76 Do you require further assistance with this case, please? Thanks!

[Nimaro76]

thank you, I will contact further if I had an issue during implementation.

kind regards
Nima

[MartyG-RealSense]

You are very welcome, @Nimaro76 - thanks very much for the update. Please do feel free to contact us if you have further issues. Good luck!