D435i: Getting inconsistent frames for depth and color when aligned to depth

[adharma1]

---

Required Info
Camera Model	D435i
Firmware Version	5.12.11
Operating System & Version	Linux (Ubuntu 18)
Kernel Version (Linux Only)	5.4.70
Platform	VM
SDK Version	2.42.0
Language	C++
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

I am using the sample code attached to get the depth and color messages and saving them into a rosbag. # of frames received for color and depth are not matching.

here is the sample code

#include <librealsense2/rs.hpp>
#include <sstream>
#include <iostream>
#include <iomanip>


using namespace std;

class RealSense
{
  private:
      int           width;
      int           height;
      int           fps;
      int           gyroFmt;
      int           acclFmt;
      rs2::pipeline pipe;
      rs2::config   rconfig;

      bool          enableGyro;
      bool          enableAccl;
      bool          enableDepth;
      bool          enableColor;

  public:
      int initialize()
      {

          enableGyro = true;
          enableAccl = true;
          enableDepth = true;
          enableColor = true;

          width  = 1280;
          height = 720;
          fps    = 15;
          acclFmt = RS2_FORMAT_MOTION_XYZ32F;
          gyroFmt = RS2_FORMAT_MOTION_XYZ32F;

      }

      /*
       * Getter functions for RealSense recording settings
       */
      const int getFps()     { return fps; };
      const int getWidth()   { return width; };
      const int getHeight()  { return height; };
      const int getGyroFmt() { return gyroFmt; };
      const int getAcclFmt() { return acclFmt; };

      /*
       * Setter functions for RealSense recording settings
       */
      void getFps(const int val)     { fps = val; };
      void getWidth(const int val)   { width = val; };
      void getHeight(const int val)  { height = val; };
      void getGyroFmt(const int val) { gyroFmt = val; };
      void getAcclFmt(const int val) { acclFmt = val; };

      /*
       * Start recording
       */
      int startRecording(std::string rbagname)
      {
          int err = 0;
          rconfig.enable_record_to_file(rbagname);
          if (enableColor)
            //rconfig.enable_stream(RS2_STREAM_COLOR, width, height, RS2_FORMAT_RGB8, fps);
            rconfig.enable_stream(RS2_STREAM_COLOR, width, height, RS2_FORMAT_RGB8, fps);

          if (enableDepth)
            rconfig.enable_stream(RS2_STREAM_DEPTH, width, height, RS2_FORMAT_Z16H, fps);

          if (enableGyro)
            rconfig.enable_stream(RS2_STREAM_GYRO, RS2_FORMAT_MOTION_XYZ32F);

          if (enableAccl)
            rconfig.enable_stream(RS2_STREAM_ACCEL, RS2_FORMAT_MOTION_XYZ32F);

          try {
            pipe.start(rconfig);
            rs2::align align_to_depth(RS2_STREAM_DEPTH);
            rs2::frameset frameset = pipe.wait_for_frames();
          }
          catch (const rs2::error& e) {
            cerr << e.what() << endl;
            err = -1;
          }
          return err; 
      }

      /*
       *Stop recording
       */
      int stopRecording()
      {
          pipe.stop();
      }
};

int main()
{
    RealSense rSense;
    int err = 0;

    if(!rSense.initialize()) {
        printf("Unable to initialize realSense Camera");
        err = -1;
        goto out;
    }

    if((err = rSense.startRecording("testbag.bag")) < 0) {
        cerr << "Failed to start recording" << endl;
        goto out;
    }

    // Wait for sometime
    for(int i = 0; i< 1000; i++);

    rSense.stopRecording();

out:
    return err;
}

[MartyG-RealSense]

Hi @adharma1 If you have depth, RGB and IMU (gyro / accel) enabled simultaneously then that would be the first aspect of the program that I would recommend investigating if you are experiencing performance problems.

Performance can be improved if you use two pipelines and put the IMU on one pipeline on its own and depth & color on a second pipeline. The subject is discussed on the link below in regard to C++.

#7016

Alternatively, you can use callbacks instead.

#7391 (comment)

---

Edit: if you are attempting to record aligned depth and color frames to the bag then this will not work because bags recorded in the RealSense SDK do not save aligned frames, only the individual streams. Alignment is performed by loading the pre-recorded bag file into memory and then applying alignment to the depth and color frames in real-time.

[adharma1]

@MartyG-RealSense, I am also seeing that number of frames received for depth and color are also different, is there a way to ensure I get same # of frames?

[MartyG-RealSense]

The comments linked to below provide good introductory information about the subject of RGB-Depth sync and how to enforce a constant FPS rate.

#3689 (comment)

#2224 (comment)

#4551 (comment)

#1548 (comment)

---

Please do feel free to ask further questions after reading the above links.

[adharma1]

Hi @adharma1 If you have depth, RGB and IMU (gyro / accel) enabled simultaneously then that would be the first aspect of the program that I would recommend investigating if you are experiencing performance problems.

Performance can be improved if you use two pipelines and put the IMU on one pipeline on its own and depth & color on a second pipeline. The subject is discussed on the link below in regard to C++.

#7016

Alternatively, you can use callbacks instead.

#7391 (comment)

Edit: if you are attempting to record aligned depth and color frames to the bag then this will not work because bags recorded in the RealSense SDK do not save aligned frames, only the individual streams. Alignment is performed by loading the pre-recorded bag file into memory and then applying alignment to the depth and color frames in real-time.

Is there any sample code for "Alignment is performed by loading the pre-recorded bag file into memory and then applying alignment to the depth and color frames in real-time."?

[MartyG-RealSense]

The RealSense SDK has an example C++ depth-color alignment program called align-advanced:

https://github.com/IntelRealSense/librealsense/tree/master/examples/align-advanced

Although it uses live camera data by default, the SDK examples can be modified to use a recorded bag as their data source instead of a live stream, as described on the SDK's Sample Data archive of bag test-files.

https://github.com/IntelRealSense/librealsense/blob/master/doc/sample-data.md

[MartyG-RealSense]

Hi @adharma1 Do you require further assistance with this case, please? Thanks!

[adharma1]

@MartyG-RealSense I am good for now. Thanks! We can close the issue

[MartyG-RealSense]

Great, thanks very much @adharma1 for the update. As suggested, I will close the case now. Thanks again!