Cannot set emitter_on_off and enable_auto_exposure at the same time through the SDK

[jakemclaughlin-luxolis]

---

Required Info
Camera Model	D455
Firmware Version	05.15.00.02
Operating System & Version	Ubuntu 20.04
Kernel Version (Linux Only)	5.15.0-73-generic
Platform	PC
SDK Version	2.53.1
Language	c++
Segment	AR/VR

Issue Description

I am attempting to set auto exposure and use emitter_on_off at the same time. Typically setting one of the values effects the other, i.e. when i turn the emitter_on_off to on in realsense viewer while auto exposure is on, the exposure stays constant. If I set auto exposure to on while emitter on off is on it won't turn on at all. I have tried lowering the laser power and that doesn't seem to do anything.

  int index = 0;
  for (rs2::sensor sensor : sensors) {
    if (sensor.supports(RS2_CAMERA_INFO_NAME)) {
      ++index;
      if (index == 1) {
        sensor.set_option(RS2_OPTION_EMITTER_ALWAYS_ON, 0.0);
        sensor.set_option(RS2_OPTION_EMITTER_ENABLED, 1.0);
        sensor.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, 1);
        sensor.set_option(RS2_OPTION_EMITTER_ON_OFF, 1.0);
        sensor.set_option(RS2_OPTION_LASER_POWER, 360);
        sensor.set_option(RS2_OPTION_VISUAL_PRESET, rs2_rs400_visual_preset::RS2_RS400_VISUAL_PRESET_HIGH_ACCURACY);
      }
      if (index == 2) { sensor.set_option(RS2_OPTION_EXPOSURE, 100.f); }
    }
  }

[MartyG-RealSense]

Hi @jakemclaughlin-luxolis You are correct that when auto-exposure is enabled, it should stay enabled when the emitter on-off function is enabled.

The auto-exposure and emitter on-off can both be enabled at the same time.

If auto-exposure and emitter on-off are both disabled and then emitter on-off is enabled first followed by auto-exposure secondly, auto-exposure should enable.

However, if auto-exposure and emitter on-off are both disabled then neither will be able to enable if the HDR Enabled option is currently enabled.

In regard to your script, if the index is 2 and a manual exposure value is set then auto-exposure automatically disables when a manual exposure value is applied.

[jakemclaughlin-luxolis]

@MartyG-RealSense I am able to enable the settings simultaneously yes, but the behaviour when both are enabled is not as expected. If i have auto exposure on and turn emitter_on_off on, whatever the exposure was at last is set constant (i see this behaviour in realsense viewer as well), and when i turn emitter_on_off back to off the auto exposure returns to acting as expected

[MartyG-RealSense]

The behaviour of the auto-exposure mechanism is that when auto-exposure is enabled, the exposure value is not updated, though the real depth exposure value can be read from camera metadata whilst auto-exposure is enabled. When auto-exposure is disabled, exposure returns to the last manual value that it was at.

#3066 has further information about the emitter on-off mode, which alternates the emitter on a per-frame basis. The link has an animated image with a fast-strobing light, so beware if you have epileptic sensitivity.

[jakemclaughlin-luxolis]

@MartyG-RealSense I have taken a video of the behaviour I am talking about

out.mp4

[MartyG-RealSense]

I noted down from the video the exact test procedure that you used and then tested it myself. It worked fine up until the point where you disabled auto-exposure and the dot pattern disappeared. This effect did not happen for me and the dot pattern remained visible.

I compared my SDK version and firmware version to yours. We both used SDK 2.53.1, but my firmware is 5.14.0.0 (the recommended firmware for 2.53.1) and you are using the newer firmware 5.15.0.2 (the recommended firmware for SDK 2.54.1).

With recent firmware versions it has been necessary to use the exact recommended firmware for a specific SDK version in order to avoid problems occurring. So as 5.15.0.2 is meant for SDK 2.54.1, please try downgrading your firmware to 5.14.0.0 to see whether it resolves your issue.

[jakemclaughlin-luxolis]

SOrry the dot pattern didnt dissapear, the compression of the video didn't capture that it was flickering

[MartyG-RealSense]

The original report problem was "If I set auto exposure to on while emitter on off is on it won't turn on at all".

I disabled auto-exposure, enabled emitter on-off and then re-enabled auto-exposure and it enabled with no problem whilst emitter on-off remained active at the same time. As mentioned above, a difference between our test setups is that I was using firmware 5.14.0.0 with SDK 2.53.1 instead of 5.15.0.2.

[jakemclaughlin-luxolis]

In the code it would not turn on, in the viewer it could. Also I know that both the settings can be turned on at the same time, but the behaviour was not expected. When auto exposure is on and the emitter_on_off is on, the image does not get automatically exposed based on the lighting in the scene. Could you send me a video of this working on your end? Where the emitter is switching on/off and auto exposure is correctly working (i.e. when pointing at a bright light the image will lower exposure and vice vers)

[MartyG-RealSense]

I uploaded a bag file to my Google Drive, where it can be downloaded from at the link below.

https://drive.google.com/file/d/1YpZkxwhTTmKVoIOAYuzU_uGPZwG9Jt7J/view?usp=drive_link

it can be played by launching the Viewer without the camera plugged in and then drag and dropping the file into the center panel of the Viewer.

It starts with emitter on-off enabled and auto-exposure disabled. At the 4 second point in playback, auto-exposure enables so both it and emitter on-off are simultaneously active.

[jakemclaughlin-luxolis]

I just upgraded my sdk, and on the same firmware the issue is still occuring. I'm very positive that if you were to move the camera around while auto exposure is on, with the emitter switching, the auto exposure will not be working, despite it beingproperly set to be on. I know they can both be set to be on, but the auto exposure isn't actually working, if you were to point the camera at a bright area it will be blown out because it isn't being automatically exposed.

[MartyG-RealSense]

The auto-exposure effect that you describe is not related to emitter on-off. It is a known behaviour that if the camera's viewpoint is directly facing a bright light source such as the sun then the IR sensor can become saturated with light and the images go wrong. Exposure may also drop to minimum.

You can test this by exposing the camera to the bright light to trigger the effect and then either turn the camera away from the light or cover your hand over the front of the camera for several seconds. The auto-exposure should then attempt to adjust the exposure to correct the image.

Ways to deal with the effect of a direct light source include (a) applying an ND filter product over the lenses, (b) defining an auto-exposure Region of Interest (ROI) in the lower half of the image, or (c) using the D435f or D435if camera models that have a built-in ND filter to absorb light.

You can also have greater control over exposure by alternating between two exposure values using the High-Dynamic Range (HDR) described in the guide at the link below, though auto-exposure cannot be used when HDR is enabled.

https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras

[MartyG-RealSense]

Hi @jakemclaughlin-luxolis Do you require further assistance with this case, please? Thanks!

[jakemclaughlin-luxolis]

Yes the problem still persists, auto exposure does not perform as expected when it is one while the emitter is in on/off mode

[MartyG-RealSense]

I performed testing again but was again unable to replicate the issues that you reported, such as auto-exposure not enabling when emitter on-off is already on.

If there is disruption to the exposure value when the camera is pointed directly at a bright light then this is an expected effect. The recent RealSense camera models D435f and D435if are equipped with filters on their sensors that absorb light.

You originally reported that you were using SDK 2.53.1 with firmware 5.15.0.2 (which is designed for use with SDK 2.54.1). Can you confirm please if you tested the advice at #11908 (comment) to either update your SDK to 2.54.1 or downgrade your firmware to 5.14.0.0.

May I also ask the purpose why emitter on-off is being used in your project.

[nikola-j]

Hi, I also have the same issue, the autoexposure doesn't work when turning on emitter_on_off.

I've recorded a video demonstrating the issue.

In the first part the autoexposure is on and it works, but when I toggle emitter on off, it stops working.
Also, when the emitter on off is toggled I cant even manually adjust the exposure.

output.mp4

[MartyG-RealSense]

Hi @nikola-j I note from the video that RealSense Viewer version 2.55.0 is being used. This version is still on the development branch of librealsense and not yet released as a public master version at the time of writing this.

It also looks as though you are using the Windows version of the Viewer. If you are then please try downloading and running version 2.54.2 of the Intel.RealSense'Viewer.exe standalone Viewer from the SDK Releases page's 'Assets' file list and see whether the reported auto-exposure problem also occurs for you in that version.

https://github.com/IntelRealSense/librealsense/releases/tag/v2.54.2

[nikola-j]

Hi @MartyG-RealSense,

Thanks for the prompt reply.
I’m on Ubuntu 22.04. I’ve tried the apt version and have tried with the master and development version. The camera is on the latest firmware.

[MartyG-RealSense]

Using 2.54.2, I didn't experience a problem with auto-exposure ceasing to work when using emitter on-off. I was though able to replicate the error pop-up when trying to change manual exposure whilst emitter on-off is enabled.

[nikola-j]

Using 2.54.2 the issue persists. It just stops adjusting the exposure when emitter on-off is enabled. How did you test it your end?
In this version I also get a periodic warning wherever autoexposure and emitter on-off are enabled:

 15/03 11:14:25,698 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 15/03 11:14:28,710 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 15/03 11:14:41,726 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 15/03 11:14:55,742 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 15/03 11:14:57,005 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 15/03 11:15:03,756 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 15/03 11:15:09,021 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 15/03 11:15:12,771 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 15/03 11:15:15,783 WARNING [140184465155648] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11

[MartyG-RealSense]

I tested in realsense-viewer like you did.

I will highlight your issue to my Intel RealSense colleagues. Thanks very much for your patience!

[MartyG-RealSense]

Hi @nikola-j I consulted with my Intel RealSense colleagues, who confirmed that it is not a bug despite the pop-up error box. They advised "It makes sense that when emitter on-off enabled, exposure is fixed, as auto exposure adjustment needs a couple of frames to stabilize".

[nikola-j]

Thank you. That makes sense. But would it be possible to set the exposure manually when emitter on-off is enabled? Or is it just a limitation of the viewer? Or would the process have to be, disable emitter on off, set exposure, enable it?

[MartyG-RealSense]

When manual exposure is used it does not require time to settle down when its value is changed.

The error pop-up box in the Viewer when changing manual exposure does not occur when doing so in the similar Depth Quality Tool, suggesting that the error box is a Viewer-specific glitch.

[MartyG-RealSense]

Hi @nikola-j Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.