Depth and IR frame counters are not consistent

[cvasfi]

Required Info
Camera Model	{ D435}
Firmware Version	(05.12.13.50)
Operating System & Version	{Ubuntu 20.04
Kernel Version (Linux Only)	(5.10.104-tegra)
Platform	Jetson
SDK Version	{ 2.53.1 }
Language	{C++ }
Segment	{Robot/ }

Issue Description

<Describe your issue / question / feature request / etc..>

I have realized the frame counter metadata (RS2_FRAME_METADATA_FRAME_COUNTER) of the IR frame is off by 2 compared to the depth frame. (i.e. first depth frame_counter is 2 whereas first IR frame counter is 0).

I don't mind this since the offset is consistent during runtime but this causes the hdr-merge algorithm to ignore the IR data and use the depth information only(See hdr-merge).

Is there a way to fix this issue for the sake of bypassing the hdr-merge check?

This is how I initalize both streams:

    rs2::config cfg;
    cfg.enable_stream(RS2_STREAM_DEPTH, width, height, RS2_FORMAT_Z16,
                      30);
    cfg.enable_stream(RS2_STREAM_INFRARED, 1, width, height, RS2_FORMAT_Y8,
                        30);

[MartyG-RealSense]

Hi @cvasfi May I first ask whether you are using hardware sync to synchronize the depth to the timestamps of another camera or to an external trigger pulse? The reason that I ask this is that HDR and hardware sync cannot be used together because when running in HDR mode with multiple cameras, the different cameras may run at different exposure times at the same time. Hardware sync may be interrupted and so this usage is not recommended.

Infrared and depth frames should always be automatically synchronized whether hardware sync is active or not though. This is because the depth is derived from infrared frames and so there is no way for them to be out of sync with each other.

[cvasfi]

@MartyG-RealSense

Thanks, I will check whether we use hardware-sync or not. They are not synced together.

Infrared and depth frames should always be automatically synchronized whether hardware sync is active or not though.

Then I understand that it is still unknow what might be the cause of this phenomenon of depth/IR frames with different RS2_FRAME_METADATA_FRAME_COUNTER values being bundled in the same rs2::frame, right?

[MartyG-RealSense]

There was a case at #7807 where different frame numbers were being reported for depth and infrared and it was resolved in that particular case by building the librealsense SDK from source code with the RSUSB Backend using the CMake build flag -DFORCE_RSUSB_BACKEND=true

An RSUSB build of the SDK bypasses the Linux kernel and so is not dependent on Linux versions or kernel versions and does not require kernel patching.

[MartyG-RealSense]

Hi @cvasfi Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.