Problem with image flickering when changing exposure time in HDR mode

[TakahiroAoyagi]

Required Info
Camera Model	D455
Firmware Version	05.13.00.50
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	5.4.0-110-generic
Platform	PC
SDK Version	2.50.0-0
Language	C++/ROS/opencv
Segment	others

Issue Description

Hello, I am having a problem with flickering images when changing the exposure time in HDR mode.
I am currently trying to add automatic exposure to HDR mode.
I have implemented this functionality as an external ROS node and am changing the exposure time and gain every few frames through the ROS functionality.
However, it does not work well in HDR mode because when I change the exposure time and gain, I get black frames and what seems to be contamination with other sequence IDs.
Is there any way to prevent this?
Since I am using ROS, I was wondering whether the ROS Wrapper repository or this repository would be more appropriate, but I have confirmed that this problem occurs on Realsense Viewer as well, so I have posted this in this repository.
Below is a video of the exposure controlled by the node I created and when the exposure is changed in Realsense Viewer.
Both are filtered to show only Sequence Id=1 with HDR mode enabled.
You can see the flickering.

hdr_ae_trial_small.mp4

realsense_viewer_small.mp4

[MartyG-RealSense]

Hi @TakahiroAoyagi Intel has a white-paper guide about High-Dynamic Range (HDR) at the link below.

https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras

The image is flickering because it is alternating between two pre-specified exposures.

To deal with flicker, you can enable the HDR Merge option to replace the depth stream with synthetic HDR depth (though the left infrared image will still flicker) or split the flickering stream into its individual sequence components.

Section 3.1 of the paper describes how to change the options in the RealSense Viewer, whilst Section 3.2 shows how to do so with Python script code. I note that you are already influencing from ROS with a node script..

The camera already streams in auto-exposure mode by default. Section 2.4 of the paper describes how exposure and gain can be indirectly influenced when auto-exposure is enabled by defining an auto-exposure setpoint value.

[TakahiroAoyagi]

Thank you for your reply.

I'm not sure you are understanding my situation correctly, so let me elaborate.
First, I understand that when I activate the HDR mode, the two exposure times alternate, causing the "intended" flickering.
My problem is the "unintended" flickering that is inserted when changing one of the two exposure times.
To check this, I have enabled HDR mode and set the "Filter By Sequence id" to 1 in "Post-Processing" so that only one of the two exposure times is visible.
However, when I manually change the exposure time, I get flickering as in the previous video.

Regarding the built-in AE, I think it is not available when HDR mode is enabled, am I wrong?
Please let me know if there is a way to enable the built-in AE and set separate target values for the two exposure times that are automatically switched in HDR mode.

Thank you in advance for your help.

[MartyG-RealSense]

You are correct about auto-exposure not being available when HDR is enabled. I checked the HDR white-paper and although auto-exposure is discussed in the paper, a note at the bottom of it in its Limitations section states that "We also at this time do not provide an option to combine HDR with auto-exposure as one of the steps". So when HDR is enabled and disabled in the RealSense Viewer, auto-exposure automatically switches its status off when HDR is enabled and on when HDR is disabled.

Like you, I found that when HDR is enabled and the left infrared image is flickering, it stops flickering if the post-processing filter Filter By Sequence Id is set to '1' and enabled.

Once 'Filter By Sequence Id' was set to '1' then I found that I could adjust exposure manually with the slider if the HDR Enabled controls had their values set to 1, 2, 1.

On my own computer, although Infrared flickered when the slider was being moved, it ceased flickering as soon as I stopped moving the exposure slider. So that implies that if Infrared was flickering on your computer then something was constantly updating the manual exposure value that is set,

[TakahiroAoyagi]

I am glad that my understanding seems to be correct.
You are right, the flickering when changing the exposure time does subside quickly.
However, I am trying to implement AE as an external feature in HDR mode.
Since external AE requires frequent exposure time changes, this momentary flickering is a problem.
So I would like to know how to solve this.

[MartyG-RealSense]

I ran extensive tests with the RealSense Viewer and did not find a simple way to stop the flickering when changing the value. I did observe though that if exposure is being set at a value between 33000 and 64000 then the flicker disappears when changing exposure until its value goes above 65000, above which the flicker returns.

[TakahiroAoyagi]

Thank you for your cooperation.
It is very disappointing that there is no easy fix.
Hopefully this issue will be fixed in a future update.

[MartyG-RealSense]

Hi @TakahiroAoyagi Were you able to make any progress with improving the flicker, please? Thanks!

[TakahiroAoyagi]

Unfortunately, there has been no improvement.
However, I have noticed a few things.

It appears that the flickering is caused by mixing in frames with incorrect sequence IDs.
I set ID=2 for an over exposure so that a white image would be captured.
When ID=1 was observed with this setting in the same way as before, the frame causing the flicker also turned white in conjunction.
So it does not seem that an invalid black frame is inserted.

I also found that the "Sequence Size" in the metadata attached to this wrong frame is 0.
*The Sequence Size is usually 2 in HDR mode.

From this I suspect that the HDR mode is momentarily deactivated when the exposure time is changed.
Thank you.

[MartyG-RealSense]

I ran further tests with the above procedure (HDR enabled, sequence ID filter enabled, with 1, 2, 1 configuration. At first it continued to flicker when the exposure value was updated.

I discovered an unusual effect though. If the 'DS Second Peak Threshold' setting in Stereo Module > Advanced Controls > Depth Control had its value changed from the default of '325' to '0' then the infrared image stopped flickering when exposure was changed. This effect persisted when the Viewer was closed and re-opened and even when the computer was completely rebooted.

I also tested the HDR configuration with the same camera on a different PC without changing Second Peak Threshold and found that the flicker did not occur at all. So it may be an issue that is specific to a particular computer.

[MartyG-RealSense]

Hi @TakahiroAoyagi Do you have an update about this case that you can provide, please? Thanks!

[TakahiroAoyagi]

The 'DS Second Peak Threshold' change had no effect in my environment.
I don't think this is a problem specific to any particular computer.
Because I have tried a total of 5 environments: Ubuntu 20.04 and Windows 10 on my personal laptop, Ubuntu 18.04 and Windows 10 on my desktop PC, and Ubuntu 18.04 on my work desktop PC.
But commonly this problem occurs.
If possible, could you provide details of the PCs where the problem did not occur?
Also, if the problem is specific to a particular environment, what do you think the problem depends on in the environment?

[MartyG-RealSense]

Here are the specifications of the two Windows 10 PCs I have where the flicker issue was fixable by changing DS Second Peak Threshold.

Linux builds of the SDK are based on the native V4L backend by default, whilst Windows builds are based on the Windows Media Foundation backend by default but can be switched to use a WinUSB rewrite of the UVC protocol as the backend instead of Media Foundation when building the RealSense SDK on Windows from source code with CMake and setting the flag FORCE_RSUSB_BACKEND=true

Linux source-code builds can also use FORCE_RSUSB_BACKEND=true to bypass the Linux kernel so that the SDK is not dependent on Linux versions or kernel versions and does not require patching.

[MartyG-RealSense]

Hi @TakahiroAoyagi Do you require further assistance with this case, please? Thanks!

[TakahiroAoyagi]

I have not obtained a solution to my problem through this discussion, nor am I satisfied with it.
I use Realsense for my work and would very much like to see the cause of this problem identified and resolved.
However, there is no further information I can provide.

[MartyG-RealSense]

You stated earlier in this discussion at #10505 (comment) that the flickering results from you trying to implement frequent auto-exposure changing as an external feature. The flickering could therefore be said to be a consequence of using HDR mode in a way for which it was not designed to operate.