Splitting multicam rs2::syncer frameset into one frameset per device

[roelofvandijk]

---

Required Info
Camera Model	D400
Firmware Version	05.12.10.00
Operating System & Version	Win 10
Platform	PC
SDK Version	2.38.1
Language	C++
Segment	Robot

Issue Description

I am using an rs2::syncer object to receive synchronized frames from two D415s (similar to this comment #4158 (comment)).

However, this gives me a single frameset for all images. I would like to split this frameset into two framesets, one per sensor, to be able to align the color frame to the depth frame per sensor (rs2::align::process requires a frameset).

Splitting frames by serial into e.g. arrays is no problem, however, re-creating two framesets seems difficult. Another theoretical option would be to clone the frameset, and remove (e.g. swap out) the frames from one sensor from each of the framesets.

Any suggestions how I could split the frameset into two, create new framesets from rs2::frame objects or align frames without a frameset?

A cumbersome solution seems to temporarily swap out the double depth/color frames (using rs2::frame::swap), so the alignment works, but I hope there is a better way.

[MartyG-RealSense]

Hi @roelofvandijk In a case in the link below, advice was given that if you need to listen to multiple sensors then you should use one of the following methods:

• Create an rs2::syncer object (as you have already done) and pass it as a callback to multiple sensors; or

• Create a frame queue per sensor and poll each one.

#1238 (comment)

A RealSense user who was having problems with syncer instead created a C++ script with a technique that they described as using a frame queue with a custom bundler as a processing block.

https://support.intelrealsense.com/hc/en-us/community/posts/360037090133/comments/1500000449662

//Prepare frame-set getter (the frame queue!)
 rs2::frame_queue q(2);
 std::vector<rs2::frame> bundle;

  rs2::processing_block bundler([&](rs2::frame f, rs2::frame_source& src) {
    bundle.push_back(f);     //add frame to vector
    if (bundle.size() == 2){ //or however many when full
        auto fs = src.allocate_composite_frame(bundle);
        src.frame_ready(fs);
        bundle.clear();
     }
  });
  bundler.start(q); //when enough frames are loaded, the queue is ready

  depth_sensor.start([&](rs2::frame f){
    bundler.invoke(f);  //add frame to bundler
  });
  color_sensor.start([&](rs2::frame f){
    bundler.invoke(f);
  });

The subject of custom framesets is also discussed here:

#5847 (comment)

---

If the above information is not helpful to your particular goals then please let me know and I will do my best to suggest other potential solutions.

[roelofvandijk]

Hello @MartyG-RealSense, thank you for the quick reply, it is much appreciated! The links you shared were spot on, and I can split the frameset into two frameset per device using the custom processing block, thank you!

For posterity, the route is:

1. Split single syncer frameset with synchronized frames from two sensors into two frame vectors by serial
2. Convert the frame vectors to framesets using the processing block from here: Manual frameset creation #5847 (comment)
3. Align each individual frameset.

[MartyG-RealSense]

That's great to hear @roelofvandijk - thanks so much for sharing information about your method with the RealSense community! :)

[MartyG-RealSense]

Hi @roelofvandijk Do you require further assistance with this case, please? Thanks!

[roelofvandijk]

Hello @MartyG-RealSense, no thank you.

[MartyG-RealSense]

Thanks very much @roelofvandijk for the update!