How to obtain unsynchronised colour and depth frames?

[LinasKo]

---

Required Info
Camera Model	D400
Firmware Version	05.12.05.00
Operating System & Version	Linux (Ubuntu 18.04 LTS)
Kernel Version (Linux Only)	5.4.0-42-generic
Platform	PC
SDK Version	2.36.0-0~realsense0.3169
Language	C++
Segment

Issue Description

TL;DR: How to obtain colour and depth frames separately, ideally via the pipeline API, instead of waiting for RealSense to time-sync them?

I am building a multi-camera system (not just RealSense) and am currently analysing the time it takes to obtain images from D435i and trying to synchronise the clocks. I am using the pipeline API, and am opening the colour (reduced resolution), depth and some IMU streams, receiving the frame / frameset through a callback function. Here, I take the system time and compare to the frame timestamp.

I found that on average, the depth frames arrive in the callback ~35ms after the RS timestamp. Next, I tried only opening one of the streams at a time, and this time, the time difference was only ~15ms. Colour stream behaved in a similar manner. I believe it could be because of the synchronisation.

So far I found the frame-management page that suggested rs2::deveice::start, but start did not exist when I tried using it. Indeed, I don't see it in the API reference - rs2::device.

So, is there a way to disable the automatic colour and depth synchronisation?

[dorodnic]

Hi @LinasKo
Using pipeline, you could create two separate pipeline objects. This should work in most cases, but its not portable solution (some platforms / devices lock the device exclusively once you create one pipeline)
Open and start methods moved from device into sensor object, but otherwise its a viable alternative.

[MartyG-RealSense]

Hi @LinasKo I support what @dorodnic advised above, especially if you foresee a situation where you will be using depth, color and IMU streams simultaneously. There have been cases where having any two of these three stream types active simultaneously has been fine (e.g RGB and IMU or depth and IMU) but there would be poor performance when depth, color and IMU were all active at the same time. Putting the IMU stream on a separate pipeline (e.g IMU on one pipeline and depth and color on the other) can significantly improve a program's operation.

This principle of setting up multiple pipelines when using a depth-color-IMU setup was discussed in a July 2020 case:

#6773

[LinasKo]

Thank you very much - that's really useful

[dorodnic]

Also I forgot to mention - https://github.com/IntelRealSense/librealsense/tree/master/examples/sensor-control can be helpful

[dorodnic]

Hi @LinasKo
Can I close the issue?

[LinasKo]

Hi @dorodnic,

I'm still working on it, but the results are somewhat promising. Instead of both color and depth lagging behind by 35ms, if run on separate pipelines, the depth latency decreased to 10ms. I've noticed that not including Accel / Gyro also changed the delay, but I didn't test it very thoroughly so far.

And yes, I think you can close the issue. Thanks for your help!

[dorodnic]

Great, feel free to open additional issues if you encounter any problem.