Infrared image goes dark randomly

[fiorano10]

---

Required Info
Camera Model	D400
Firmware Version	5.12.7.100
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	5.3.64
Platform	PC
SDK Version	2.38
Language	C++
Segment	Robot

Issue Description

We have a 2 camera setup mounted horizontally to get a 30° overlap. Running librealsense 2.38/2.33, fw 5.12.7.100 on Ubuntu 18.04, kernel version 5.3. The infrared stream randomly goes dark for a couple of frames causing depth issues. It appears that the image goes darker than the lowest exposure setting possible. Is there a way to get exposure value when auto exposure is enabled? Could this caused by turning on the emitter?

[MartyG-RealSense]

Hi @fiorano10 When depth auto-exposure is enabled, the exposure value is not updated. To get the real exposure value, you can access exposure metadata, as described in the discussion in the link below

#2549

The darkening effect in your infrared image could indeed be produced by the exposure value reducing.

There was a past case where the camera exposure value was reported to be dropping to minimum when exposed directly to sunlight. The auto-exposure should recover the image once the camera is no longer directly observing the sunlight. Is there the possibility that your camera is observing such flashes of direct sunlight in the outdoor environment?

#4928

[agrunnet]

It looks like you are using two d435. By default their emitters are set to only pulse when they expose. If you have two cameras that are not HW synchronized it is possible that sometimes their pulses will be on at same time and sometimes not.

You could try to turn on the feature that forces the projectors to be on all the time (like on the D415). Alternatively you can consider HW syncing. The former is a simple SW test. The latter requires wiring cables and making one master and the other slave.

[fiorano10]

@MartyG-RealSense this does not seem be caused only in scenarios with direct sunlight. You can see in the image that is darkens randomly for a few frames. I was curious if this could be caused by the emitter turning on? Also, I have a ROI for autoexposure looking for bottom 2/3 of the image so direct sunlight could have caused this. The other camera was looking at that region couple of frames before and was fine.

@agrunnet since we operate outside in bright sunlight we always have the emitter turned off.

[agrunnet]

Well then we can rule out my theory. (I thought you mentioned emitter was on).
This is indeed mysterious.
Can you confirm that you don’t see this with manual exposure?

[fiorano10]

@agrunnet I can't confirm, yet.

@MartyG-RealSense @agrunnet could this be caused by an increased ASIC temperature?

[MartyG-RealSense]

I would not think it was related to increased ASIC temperature.

I note in your opening message that you said "Could this caused by turning on the emitter?". Does that mean that you normally have the IR emitter turned off, please?

[fiorano10]

The emitter is normally turned off but we have an issue where is randomly turns on. Was curious if it could be the cause?

[MartyG-RealSense]

I performed thorough further testing. Another factor that could cause a very dark infrared image is if the auto-exposure was momentarily turned off. The scene that the infrared stream of the camera was observing had to be quite dimly lit in order for there to be this kind of significant difference in lighting level when auto-exposure was toggled though.

[fiorano10]

I don't think the auto-exposure was toggled since it happens even if the scene has no bright/dark spots

[MartyG-RealSense]

@agrunnet may be able to offer a theory based on the new information discussed above today.

[MartyG-RealSense]

Hi @fiorano10 Do you still require assistance with this case, please? Thanks!

[fiorano10]

Yes, there has been no assistance that has worked for us yet.

[MartyG-RealSense]

@fiorano10 I read through the case from the beginning again. Does the issue occur if you do not set an auto-exposure ROI?

[fiorano10]

@MartyG-RealSense All of the cameras have an ROI set, but not all cameras have the issue. I'm not sure if the issue would occur if the ROI is not set.

[MartyG-RealSense]

You mentioned earlier that you had an issue where the emitter was off but randomly turned on. How do you know when the emitter has turned itself on, please? Because dots appear on the infrared image?

[fiorano10]

Yes, dots appear on the infrared image. We also have a warning that monitors the state of the projector

[MartyG-RealSense]

And do the dots remain on the image once enabled, or does the emitter turn off again by itself?

Thanks very much for your patience during this investigation!

[fiorano10]

On some machines we are able to turn the emitter off automatically, on others if we see the dots, we turn it off manually. That said, there are cases where the turned on emitter goes unnoticed.
There was an issue when trying to get the emitter state on some linux kernels.

[MartyG-RealSense]

This case reminds me of an old issue from 2019 where the IR emitter value was apparently being corrupted. Having considered a large range of different possibilities for the cause of your emitter problem and eliminated them, I am more open to considering that possibility.

#3645

I have not encountered any other cases that display the same symptoms as yours though.

[candersen10]

@MartyG-RealSense what would you suggest we look at further?
The IR emitter dos appear to randomly turn on on some units, even with it turned off. Do you believe that is a likely cause of the sudden blackouts? What more should we review? What more can we supply to debug this issue?

[MartyG-RealSense]

Could you confirm to me which RealSense camera model you are using? @agrunnet thought it was D435. Is this true, please?

[candersen10]

D435i.

[MartyG-RealSense]

Okay, thank you very much. I will seek advice from Intel about your case. Thanks for your patience!

[candersen10]

Marty, thank you, please keep us posted. We will continue to escalate this until we hear some answer on it that enables a solution . It renders a realsense completely blind at a high degree of frequency. We've seen it consistently over thousands of autonomous miles at this point. It's not unit specific, it's not environment specific (other than being outdoors). I'm very surprised that we are the only ones who see it per your suggestion.

[candersen10]

Should we be mechanically removing the IR projectors to make them stop turning on randomly also?

[MartyG-RealSense]

It would be preferable to keep the cameras intact so that they can be tested.

[agrunnet]

1. Try to catch error, by looking at metadata for the frame exposure. Use the command "GetFrameMetadata" and select "RS2_FRAME_METADATA_ACTUAL_EXPOSURE" and "RS2_FRAME_METADATA_GAIN_LEVEL". You can also read the laser power mode: "RS2_FRAME_METADATA_FRAME_LASER_POWER_MODE". This should always be off.
2. The fact that the laser turns on intermittently is concerning. Try setting laser power to low, as well as turning it off. I am trying to make sure that when it turns on it does not affect image, so power would be low. In particular, if you have glass in front of the camera, I can imagine that a burst of the laser may cause back-reflections into the sensor, which will activate the autoexposure to try to compensate. To rule out the projector you can also try pasting black tape over it.
3. ROI: There seems to be some dependence on having turned on the ROI.
4. ROS or Linux: I cannot rule this out, but I dont think we have enough data to point at this.
5. Multi-camera: You are using multiple cameras. Are they HW synced? I want to rule out some electrical noise.
6. Sunlight Glare: If sunlight hits the sensor or glass (any ND filters) directly it is possible this could cause some glare. I would not expect just a couple of frames to go bad though. But having a baffle or light shield may help if this is the case.

Of all these, there seems to be some correlation with laser, so perhaps we focus on that?

[MartyG-RealSense]

Thanks so much @agrunnet for the new guidance!

@candersen10 I have also received additional advice from Intel to try updating your librealsense version to 2.41.0 and upgrade the firmware of both of the cameras to firmware version 5.12.9.0 or newer.

[fiorano10]

We've been publishing/recording all of the info suggested (after applying the librealsense patch to get the metadata) and it seems there is an issue of inconsistent exposure between cameras. The emitter is definitely turned off, and the camera gains are at 16 for all.
PTAL: Issue #8375

[MartyG-RealSense]

Thanks @fiorano10 for the updated information. I have referred your case to Intel again for further advice.

As the librealsense SDK version 2.42.0 and firmware version 5.12.11.0 that have just been released provide IR enhancements, would it be possible to test that SDK and firmware combination please to see if it results in improvements for you?

Auto-Exposure and Gain max limits for IR sensors. The new APIs set the effective upper bound for Exposure and Gain values for Depth Auto-Exposure. This allows to control and limit the changes in FPS rate while enabling for AE auto-adjustments.
Requires FW5.12.11.0 or later.

[MartyG-RealSense]

Hi again @fiorano10 I received feedback from Intel.

1. Are you using hardware sync with your multiple cameras? If you are, do you have genlock mode enabled? And which Inter Cam Sync Mode value are you using (the value will tell us for sure whether you are using genlock mode or not).

2. Could you provide a video of the exposure issue, please?

[MartyG-RealSense]

Hi @fiorano10 Do you have an update that you can provide please regarding the earlier information provided in the comments above? Thanks!

[MartyG-RealSense]

Hi @fiorano10 Do you have an update that you can provide about Intel's questions to you in the link below please? Thanks!

#8129 (comment)

[MartyG-RealSense]

This case will be closed in 7 days from the time of writing this if we do not hear from you. Thank you.

[MartyG-RealSense]

Case closed due to no further comments received.

[candersen10]

This issue was not at all fixed and remains a problem. No solution has been offered that fixes it. @MartyG-RealSense do you have anything you could suggest?

[isaiahburro]

Hi, @candersen10 is it possible to continue this thread, or is it best to re-open a new one? Since this issue is still occurring our requirements spec might have changed since this thread has opened. To answer the questions in this post:

1. No we are not using hardware sync
2. I attached a video of what we're seeing. it starts off with pitch whiteness (if this isn't fixed on its own we can kill it with a rosnode) then the exposure goes to darkness. Still unsure about solutions to this.

example-video.mp4

Hi again @fiorano10 I received feedback from Intel.

1. Are you using hardware sync with your multiple cameras? If you are, do you have genlock mode enabled? And which Inter Cam Sync Mode value are you using (the value will tell us for sure whether you are using genlock mode or not).
2. Could you provide a video of the exposure issue, please?

[MartyG-RealSense]

Hi @candersen10 and @isaiahburro Which firmware driver version is your camera using, please? If your firmware has not been updated since you purchased the camera and has a very old firmware then it may not contain fixes in the firmware that were later made to address problems like this.

[isaiahburro]

Hi @MartyG-RealSense we are using real-sense firmware version: 05.12.11.00 across all of our units. Is this the latest version?

[MartyG-RealSense]

5.12.11.0 is not the latest version as it dates from February 2021 and there have been 7 firmware releases since then. The fixes that I referred to were incorporated into the firmware long before then. The exposure fixes were designed for auto-exposure rather than manual exposure though, so the image would likely not be auto-corrected when an exposure problem occurs in manual exposure mode.

An example of such an exposure problem would be the infrared sensor being saturated by a strong light source such as sunlight directly in the camera's view. With auto-exposure enabled, the camera could auto-correct by putting a hand over the camera lenses for several seconds or turning the camera away from directly facing the light source.

[candersen10]

@MartyG-RealSense "With auto-exposure enabled, the camera could auto-correct by putting a hand over the camera lenses for several seconds or turning the camera away from directly facing the light source."

You mean to say that the camera does this automatically with the new firmware?

I can assure you that on 05.12.11.00 the camera does not auto correct when it is turned to face away from the light source, and we have thousands of examples of this.

[MartyG-RealSense]

@candersen10 These changes were a long time ago, but my understanding is that the auto-correction when turning the camera away from the sun or covering the lenses is meant to be done by the auto-exposure algorithm. There have been past cases where exposure dropped to minimum value suddenly and the image became stuck and did not auto-recover.

An attempt to address this scenario was adding a feature called fractional exposure to the auto-exposure algorithm to help the camera to cope better when exposure was minimized, as described by a RealSense team member at #2875 (comment)

The link below has an example of such a case, where a RealSense user said that covering all lenses "sometimes" auto-corrected the image.

https://support.intelrealsense.com/hc/en-us/community/posts/360033445713-D435-Sunlight-saturation-problem

Intel recommend dealing with sunlight-related problems by setting an auto-exposure Region of Interest (ROI) in the lower half of the image (as described in Intel's sunlight guidance in the link below) or purchasing and applying a physical ND filter product over the top of the camera lenses.

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#section-use-sunlight-but-avoid-glare

[candersen10]

@MartyG-RealSense our team has ND filters on all cameras (we have roughly 1000 running). Our ROI is the lower half or third of the image.

Do you have other suggestions on things we might try? Again, the problem: see a bright bit of light for a brief period of time. Thereafter, camera locks down to lowest exposure level and won't recover. It's a pretty weird auto exposure glitch in the first place - haven't seen it on any other device.

Nothing you have suggested thus far has mitigated the issue.

[MartyG-RealSense]

@candersen10 In a past case involving reflectivity, a RealSense team member also advised: "In extreme case where you have brightly illuminated surfaces and really dark surfaces, it is impossible to find one value of sensor exposure without having either completely black or completely white regions, but for this case, the camera does offer a control to rapidly switch between two values of exposure".

I believe that they were referring to a mode compatible with D435 - described at #3066 - where the laser emitter is very rapidly alternated on / off. The link has an animated image with a fast-strobing light.

You can test in the RealSense Viewer whether alternating the emitter makes a positive difference by enabling the Emitter On Off option under Stereo Module > Controls

---

Another means of controlling exposure and gain in scenes with both light and dark areas is to use the SDK's High-Dynamic Range (HDR) mode. Intel have published a white-paper guide about this feature at the link below.

https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras

If you try this mode and experience flickering on the infrared image, #10505 (comment) describes how to correct this.