Auto-exposure fails to readjust after reaching min exposure unless sensor is covered

[kylevedder]

---

Required Info
Camera Model	D435
Firmware Version	05.11.11.100
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	4.14.13 + RealSense Video Driver Kernel Patch
Platform	PC
SDK Version	2.28.0
Language	C++
Segment	Robot

Solved: make sure you're running the RealSense in USB 3 mode.

Issue Description

When I expose the D435 RealSense to direct sunlight with autoexposure enabled, it correctly drops the gain and exposure of the camera to min settings; however, if the sunlight is removed, the camera does not adjust away from min settings unless it is placed in total darkness. If placed in reduced light, such as a closet away from any direct sun exposure, it is unable to update the exposure and gain to proper settings, but if I place my hand over the sensor for several seconds, it is able to adjust back to normal behavior and produces reasonable gain and exposure settings for the current environment.

Here are images of the sequence; see left terminal and ROS image:

Before min exposure

Min exposure from sun

Still at min exposure though away from sun

Adjusting from min exposure to max exposure due to hand covering (note changes in left terminal)

Proper exposure after hand covered sensor

[MartyG-RealSense]

I wonder if you may get better results if you use a feature called fractional exposure.

#2875 (comment)

[kylevedder]

Thanks, do you have more documentation on how to enable that feature?

No worries on timing, it's dark here anyway and the sun won't be up for another 12 hours.

[MartyG-RealSense]

My reading of the information provided by agrunnet is that there are two approaches to using fractional exposure.

1. Turn off auto-exposure and manually set the exposure to a lower value, which can go as low as '1'.

Or alternatively, 2. Have auto-exposure enabled and set a Region of Interest (ROI). In the RealSense Viewer it can be done by starting the stream, clicking the 'Set ROI' button next to the "Enable Auto Exposure" option and dragging on the image to set a ROI.

[kylevedder]

None of those options lead to desirable behavior. I do want auto-exposure enabled, as I will be running in both low light and saturated light environments without prior knowledge of the transitions, and I do want the ROI to be the entire image. I am trying to use the IR image + its gain and auto-exposure from the RealSense to predict when the background IR will saturate other IR based sensors, as well as use the RealSense to produce a depth image.

To be blunt, I'm pretty sure the fact that the auto-exposure doesn't readjust is either a bug in the auto-exposure firmware code (I suspect this is the case) or there's something in hardware that puts the RealSense in a weird mode to protect the sensors which needs to be better documented so I can figure out how to avoid it.

[MartyG-RealSense]

There was a known issue earlier this year where the IR could freeze if the camera was directed at outdoor bright light such as the sun. It was more likely to occur when USB 2.0 was being used.

#4140

The most recent firmware release notes at the time of writing this (August 2019) say that a glitch referred to as "Auto-exposure (AE) for Color is not optimized in bright sunlight" has the status of "no fix" The issue's code name is DSO-10011, The firmware release notes say that 'no fix' means that there are no plans to fix that issue.

[kylevedder]

Are auto-exposure for color and auto-exposure for IR controlled by the same system? If so, I can see that being the root cause.

[kylevedder]

Regarding #4140, the IR stream isn't freezing; it's just not updating its auto-exposure or gain when placed in a lower light environment.

[MartyG-RealSense]

I was sure I had seen a case similar to yours where covering the camera up resets it after it stops updating. I just managed to finally track the link down.

https://support.intelrealsense.com/hc/en-us/community/posts/360033445713-D435-Sunlight-saturation-problem

[kylevedder]

Thank you very much for tracking that link down. Sounds like that the new firmware version will fix this issue. Do you know when that version of the firmware will be released or, if it's already released, how to get it?

[MartyG-RealSense]

The firmware version is a point of minor confusion. When an Intel support member said it had been fixed in a firmware (5.11.12.0), this was apparently an internal firmware at Intel and a final firmware release had not been made to the public at the time that the statement was made in July 2019.

There has not been a firmware 5.11.12 yet though. The most recent release was 5.11.11.100 in August 2019, the one that you are already using. According to the developer notes for the next release of the SDK, the recommended firmware version will become 5.11.15, which is not on the public firmware download page yet.

c3102f5

[agrunnet]

Here are some additional inputs that may help. From the screenshot, it looks like you may be running in USB 2.1 mode. There was an issue with fractional row exposure control settings. We updated the settings from OVT in firmware 5.11.12.0 to correct the issue. More details: #4140. Or make sure to use USB3.
In any event, if you update to firmware 5.11.12.0 or later versions, like @MartyG-RealSense says, the AE should work properly in bright sunlight.

[kylevedder]

@agrunnet I had the device plugged into a port on my computer that supports USB3

However, I have noticed on several occasions that the realsense-viewer will report being connected to a USB2 port despite being connected to the exact same physical port with USB3 support; unplugging and replugging the cable from the RealSense and the computer seems to succesfully make the realsense-viewer report it is once again connected to a USB3 port. Do you have any insight into what is happening in that case?

Also, why are there significant differences in the behavior of the D435 by being plugged into a USB2 vs USB3 port? My understanding of the differences between USB2 and USB3 were with respect to throughput data rate which to my mind shouldn't impact onboard auto-exposure; is there differences in power modes as well that's causing varying performance of the sensor? This is important to us as we would like to deploy these on mobile robots and if something non-transparently changes sensor performance (e.g. it decided to use USB2 instead of USB3 on a USB3 port), it's something we need to account for.

[kylevedder]

As a follow up to my above comment where I took the USB3.2 screenshot, I took the same setup (literally without unplugging anything) and moved the cable around while ensuring both ends remained plugged into the computer. I then reopened realsense-viewer and it stated that I was connected to a USB2.1 port.

I then ensured that both cable ends were all the way in on both ends by pressing on them and then reopened realsense-viewer, but once again it stated I was connected to a USB2.1 port. Unplugging the cable from the computer port and plugging it back in makes it correctly report USB3.2 again. I suspect this may be caused by flexing in the cable causing momentary drop in connection or something, causing the RealSense device to fall back from USB3 to USB2 speeds, but it does not again try to re-negotiate a USB3 connection until it is power cycled.

[MartyG-RealSense]

Intel give some technical insights into operation in USB2 mode in the blog post linked below.

https://www.intelrealsense.com/usb2-support-for-intel-realsense-technology/

Though I believe that the advice about not being able to update firmware on USB2 is outdated under Intel's new cross-platform firmware system, as I was able to update my D435i firmware in the RealSense Viewer via USB2 this morning.

[MartyG-RealSense]

In regard to the USB2 mis-detection problem, the camera may mis-detect due to the way it is inserted, with slow insertion increasing the chance of mis-detection and a quick and firm insertion reducing it. I have also heard that if the cable starts to withdraw from the USB port (e.g accidentally jostling out), that may also trigger a USB2 mis-detection.

If you can confirm that inserting the cable quickly and firmly increases the chances of the camera being detected correctly, this may justify using a USB cable with screw-locks on the connector to keep the connector firmly in the USB port. I once handled a case where a RealSense user with a mobile robot was having the camera repeatedly disconnected due to the robot's motion, and I recommended a screw-lock cable in that case.

[kylevedder]

@MartyG-RealSense thanks for the screwlocks tip. That will hopefully solve the USB3 => USB2 problem. What is the best way to detect what mode (USB2 vs USB3) the RealSense is running in via the librealsense API? I failed to find anything but I might just be missing something.

[MartyG-RealSense]

rs-enumate-devices is a program that can retrieve information about connected devices.

https://github.com/IntelRealSense/librealsense/tree/master/tools/enumerate-devices

The sensor-control example can similarly get information about connected devices.

https://github.com/IntelRealSense/librealsense/tree/master/examples/sensor-control

In a simpler example, a RealSense user got the current USB detection state of a connected device by searching for it via the model 'PID' ID number of that particular model (a T265 in this example's case).

#4331

A list of model PIDs can be found here:

https://github.com/IntelRealSense/librealsense/blob/master/src/ds5/ds5-private.h#L19

To pre-empt a possible question, the reason there are a number of unfamiliar model numbers on the list is that because Intel support more RealSense model configurations than are available for public purchase.

[kylevedder]

As a followup to @agrunnet and @MartyG-RealSense, I tested the RealSense again in direct sunlight while in USB3 mode by being very careful with the wires (I also confirmed the device was in USB3 with realsense-viewer before and after this experiment took place). It correctly dropped to 1 exposure and 16 gain when I pointed the RealSense at the sun, and it then correctly increased the exposure when I turned it to face slightly away from direct sunlight.

Direct sun:

Slightly away from direct sun:

The critical piece here seems to be running in USB3 mode, so thank you so very much for that insight, I deeply appreciate it.

@MartyG-RealSense Thank you for the links on USB3 vs USB2 detection. Do you know if there's a way to go so far as to disable USB2 mode and have the RealSense only try to maintain a USB3 connection? We're planning on deploying these sensors on robots for many hours at a time and I am worried we might have a problem of them constantly dropping to USB2 which seems to require a power cycle to restart; it would be ideal if the RealSense would just retry for a USB3 connection.

[MartyG-RealSense]

I haven't heard of a specific mode to disable USB2 in Librealsense. I wonder if you could do a check for whether USB2 is detected and if it is, initiate a hardware_reset instruction to automatically reset the camera from within the script until USB3 is detected and the reset loop is halted until the next time that USB2 is detected. Here's an example code snippet for hardware_reset:

#2161 (comment)

[kylevedder]

@MartyG-RealSense thank you for that snippit. I was worried I was going to have to figure out how to kill power to the USB port then power it again.

If you find a way to disable USB2 mode, please let me know, but for now I think cable screws + power cycling to keep the camera in USB3 mode is a hack to get me off the ground.

Finally, @MartyG-RealSense I want to thank you for all your help in this issue, you've been extremely responsive and I am very impressed with your quality of support.

TL;DR to anyone who stumbles upon this issue: use USB3 mode.

[MartyG-RealSense]

@kylevedder A new SDK version is available now.

https://support.intelrealsense.com/hc/en-us/community/posts/360035827554

[xerxesb]

Hi @MartyG-RealSense - I'm tracking the same AE problem raised here and in #4140 with the limitation that I'm constrained to USB 2.0 only.

The internal support rep identified 5.11.12 as the fix version and now 5.11.15 has come out, but I can't see any mention of this ticket nor #4140 mentioned anywhere in the release notes.

Do you know if this issue has been addressed in the latest 5.11.12 firmware?

Thanks

[MartyG-RealSense]

The firmware release notes for 5.11.15.0 have been published but do not include auto exposure.

https://support.intelrealsense.com/hc/en-us/community/posts/360036665993-Real-sense-firmware-release-notes-updated-to-5-11-15-0

[xerxesb]

Thanks @MartyG-RealSense & @RealSenseCustomerSupport

We would still like to see this issue get fixed for USB 2.0 mode because our hardware constraints prevent us from being able to use USB 3.0. If the fix was indeed made in firmware version 5.11.12.0 then it wasn't pulled into the .15.0 release, so it should be a relatively simple change, no?

How would you suggest we proceed? Should we have this ticket re-opened or request to re-open #4140, or something else?

Thank you.

[MartyG-RealSense]

Just for reference, I pulled up the link to the mention of the auto exposure fix in 5.11.12.0.

#4140 (comment)

As you didn't create this discussion thread and wouldn't be able to open it again, you could create a new issue and link to this one.