Actual exposure of frames

[fwindolf]

---

Required Info
Camera Model	D400 (D435, D415)
Firmware Version	05.10.06.00
Operating System & Version	Ubuntu 16
Kernel Version (Linux Only)	4.15.0-36.39~16.04.1-generic 4.15.18
Platform	PC
SDK Version	2.16.1
Language	C

Issue Description

I try to get the exposure for a frame that was captured with auto-exposure, but the extracted value stays constant. Even when using different sensors, the values are the same.

int exposure = 0;
try {
    if (f.supports_frame_metadata(RS2_FRAME_METADATA_ACTUAL_EXPOSURE))
        exposure = f.get_frame_metadata(RS2_FRAME_METADATA_ACTUAL_EXPOSURE);
} catch (rs2::error&) {
}

Even when the exposure is visibly corrected, the value extacted from frames stays at a constant value. Is it just not possible to extract the value when auto-exposure is activated?

[MartyG-RealSense]

An answer to a previous query about the exposure value says "When Autoexposure is on, the manual exposure control (and therefore UI) is not updated. In order to get the actual exposure value the user have to read it from metadata. (frame metadata API)".

A script for getting exposure from the metadata is in the comment linked to below.

#1624 (comment)

[fwindolf]

In order to get the actual exposure value the user have to read it from metadata. (frame metadata API)"

That is exactly what I am doing in the snippet I posted, right? So it should return the exposure for the very specific frame (that was calculated during auto exposure), which it does not.

[MartyG-RealSense]

A user who posted on that discussion after the case had been closed also had the same problem of the value not changing despite using metadata. Hopefully one of the Intel guys on this forum can provide some suggestions about the next step.

[dorodnic]

I think this should be working, but please let me check. Is f the depth frame or the RGB frame?

[fwindolf]

I adapted my application to dump the exposure and mean intensity (over all channels) of the images. I am using color frames of 3 devices here.

First, I just let the autoexposure work, it converges quite fast...

OpenGL info: version 4.6.0 NVIDIA 410.48, GPU GeForce GTX 1060 6GB/PCIe/SSE2 (NVIDIA Corporation)

Created 1 RealSense sensors
Initializing RealSense_0
Sensor intrinsics: size=424x240, fx=305.237, fy=305.1, cx=206.767, cy=117.156
RealSense_0: new color frame 424x240 count: 0 expos: ? mean intensity: 0.00473664
RealSense_0: new color frame 424x240 count: 1 expos: ? mean intensity: 0.0045663
RealSense_0: new color frame 424x240 count: 2 expos: ? mean intensity: 0.00450079
RealSense_0: new color frame 424x240 count: 3 expos: ? mean intensity: 0.00613863
RealSense_0: new color frame 424x240 count: 4 expos: ? mean intensity: 4.87781
RealSense_0: new color frame 424x240 count: 5 expos: 166 mean intensity: 5.39488
RealSense_0: new color frame 424x240 count: 6 expos: 166 mean intensity: 5.44061
RealSense_0: new color frame 424x240 count: 7 expos: 166 mean intensity: 4.89282
RealSense_0: new color frame 424x240 count: 8 expos: 166 mean intensity: 26.8649
RealSense_0: new color frame 424x240 count: 9 expos: 166 mean intensity: 28.0421
RealSense_0: new color frame 424x240 count: 10 expos: 166 mean intensity: 26.8127
...
RealSense_0: new color frame 424x240 count: 20 expos: 166 mean intensity: 36.1723
...
RealSense_0: new color frame 424x240 count: 30 expos: 166 mean intensity: 35.1062
...
RealSense_0: new color frame 424x240 count: 40 expos: 166 mean intensity: 34.439
...
RealSense_0: new color frame 424x240 count: 50 expos: 166 mean intensity: 33.86
...

The next thing I did was to hover my hand directly above the camera, making the intensity increase. Auto-exposure is still active and only shows the same value, while it is actively decreasing the intensity from 120 back to 90, where it converges again.

RealSense_0: new color frame 424x240 count: 97 expos: 166 mean intensity: 33.5748
RealSense_0: new color frame 424x240 count: 98 expos: 166 mean intensity: 33.5634
RealSense_0: new color frame 424x240 count: 99 expos: 166 mean intensity: 33.792
RealSense_0: new color frame 424x240 count: 100 expos: 166 mean intensity: 37.7283
RealSense_0: new color frame 424x240 count: 101 expos: 166 mean intensity: 66.409
RealSense_0: new color frame 424x240 count: 102 expos: 166 mean intensity: 109.561
RealSense_0: new color frame 424x240 count: 103 expos: 166 mean intensity: 125.945
RealSense_0: new color frame 424x240 count: 104 expos: 166 mean intensity: 121.975
RealSense_0: new color frame 424x240 count: 105 expos: 166 mean intensity: 118.48
RealSense_0: new color frame 424x240 count: 106 expos: 166 mean intensity: 88.8587
RealSense_0: new color frame 424x240 count: 107 expos: 166 mean intensity: 87.6908
RealSense_0: new color frame 424x240 count: 108 expos: 166 mean intensity: 86.9825
RealSense_0: new color frame 424x240 count: 109 expos: 166 mean intensity: 87.0125
RealSense_0: new color frame 424x240 count: 110 expos: 166 mean intensity: 88.323
...
RealSense_0: new color frame 424x240 count: 120 expos: 166 mean intensity: 91.1366
...
RealSense_0: new color frame 424x240 count: 130 expos: 166 mean intensity: 89.2045
...
RealSense_0: new color frame 424x240 count: 140 expos: 166 mean intensity: 89.8442

The next thing I did was to disable auto-exposure, which the sensor correctly feedbacks (my conversion might be off, not quite sure). Without auto-exposure, the image is almost black.

RealSense_0: new color frame 424x240 count: 216 expos: 166 mean intensity: 34.4158
RealSense_0: new color frame 424x240 count: 217 expos: 166 mean intensity: 34.4269
RealSense_0: set Enable Auto Exposure to Manual
Setting sensor exposure to 333.3ms
RealSense_0: set Exposure to 3333
RealSense_0: new color frame 424x240 count: 218 expos: 166 mean intensity: 33.5712
RealSense_0: new color frame 424x240 count: 219 expos: 166 mean intensity: 32.7576
RealSense_0: new color frame 424x240 count: 220 expos: 3333 mean intensity: 7.90701
RealSense_0: new color frame 424x240 count: 221 expos: 3333 mean intensity: 7.87343
RealSense_0: new color frame 424x240 count: 222 expos: 3333 mean intensity: 7.87977

After I enable auto-exposure again, the sensor reacts and increases brightness, but still feedbacks the manually set auto-exposure.

RealSense_0: set Enable Auto Exposure to Auto
RealSense_0: new color frame 424x240 count: 241 expos: 3333 mean intensity: 7.94079
RealSense_0: new color frame 424x240 count: 242 expos: 3333 mean intensity: 8.9562
RealSense_0: new color frame 424x240 count: 243 expos: 3333 mean intensity: 35.3856
RealSense_0: new color frame 424x240 count: 244 expos: 3333 mean intensity: 36.7905
RealSense_0: new color frame 424x240 count: 245 expos: 3333 mean intensity: 35.5866
RealSense_0: new color frame 424x240 count: 246 expos: 3333 mean intensity: 35.5659
RealSense_0: new color frame 424x240 count: 247 expos: 3333 mean intensity: 35.5873
RealSense_0: new color frame 424x240 count: 248 expos: 3333 mean intensity: 35.4221
RealSense_0: new color frame 424x240 count: 249 expos: 3333 mean intensity: 35.3923
RealSense_0: new color frame 424x240 count: 250 expos: 3333 mean intensity: 35.3569

After disabling auto-exposure, the brightness decreases while the exposure stays the same.

RealSense_0: set Enable Auto Exposure to Manual
Setting sensor exposure to 333.3ms
RealSense_0: set Exposure to 3333
RealSense_0: new color frame 424x240 count: 335 expos: 3333 mean intensity: 34.9636
RealSense_0: new color frame 424x240 count: 336 expos: 3333 mean intensity: 34.0407
RealSense_0: new color frame 424x240 count: 336 expos: 3333 mean intensity: 34.0407
RealSense_0: new color frame 424x240 count: 337 expos: 3333 mean intensity: 7.89682
RealSense_0: new color frame 424x240 count: 338 expos: 3333 mean intensity: 7.87842
RealSense_0: new color frame 424x240 count: 339 expos: 3333 mean intensity: 7.86916
RealSense_0: new color frame 424x240 count: 340 expos: 3333 mean intensity: 7.87771

I extract the metadata like this:

try
{
  data_ = pipeline_.wait_for_frames();
}
catch (rs2::error &e)
{
  std::cerr << "Error: " << e.what() << std::endl;
  return false;
}

rs2::frame cframe = data_.get_color_frame();

// Copy data to cv::Mat
int colorBytes = height_ * width_ * 3; // 8 bit for 3 colors = 2 bytes
std::memcpy(color.data, (void*)cframe.get_data(), (size_t)colorBytes);

// Calculate mean intensities per channel
cv::Scalar cMean = mean(color);        

auto width = cframe.get_profile().as<rs2::video_stream_profile>().width();
auto height = cframe.get_profile().as<rs2::video_stream_profile>().height();
std::cout << name() << ": new color frame "
               << width << "x" << height
               << " count: " << getFrameMetaData(cframe, RS2_FRAME_METADATA_FRAME_COUNTER)
               << " expos: " << getFrameMetaData(cframe, RS2_FRAME_METADATA_ACTUAL_EXPOSURE)
               << " mean intensity: " << (cMean[0] + cMean[1] + cMean[2])/ 3
               << std::endl;

std::string SensorRealSense::getFrameMetaData(rs2::frame& f, const rs2_frame_metadata_value& option)
{
  try {
    if (f.supports_frame_metadata(option))
      return std::to_string(f.get_frame_metadata(option));
    } catch (rs2::error&) {
    }
  return "?";
}

[ev-mp]

Hello @fwindolf ,
Unfortunately the actual exposure metadata attribute is updated for depth sensor only. The current RGB sensor doesn't have this capacity, so you just cannot retrieve the actual exposure values when auto-exposure is "on".
Note that using rs2_get_option(EXPOSURE) will provide correct values only when the exposure control is set to "manual". This applies both for color and depth sensors.

[fwindolf]

Thanks for the quick reply.
You say it doesn't have the capability, does that mean that feature will also not be added in the future?

Can you share about what parameters the auto-exposure controls? I was not able to reproduce how images looked during auto-exposure when manually controlling the exposure and gain (maybe i also just didnt find the right settings).

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hello @fwindolf,
Do you still need assistance for this issue?

​I was not able to reproduce how images looked during auto-exposure when manually controlling the exposure and gain

Can you please elaborate?

[fwindolf]

If I have an image with auto exposure enabled and a fixed white balance it looks quite different from anything I can get when manually controlling gain and exposure (or I need to set the exposure really long, so it interferes with getting 30 fps, or I get motion blur).
Are there more parameters that are controlled with auto exposure than exposure and gain?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi

I remembered that the FW5.10.7 or newer ones fixed some RGB gain issue. Please try the latest firmware and let me know if you still see same issue.

Thanks

[RealSenseCustomerSupport]

Hi
Is this issue resolved after firmware update?

[fwindolf]

Sorry, I cannot flash a new FW right now, I will reopen if the issue persists after an FW update. Thanks!

[piotrpolatowski]

@fwindolf Did you manage to read RS2_FRAME_METADATA_ACTUAL_EXPOSURE for RGB sensor? With the latest firmware 5.12.3, depth works well, but RGB always return same value.

[fwindolf]

I sadly dont have access to the sensor anymore. But as far as I remember I never managed to get the correct value, as well as set it. But that was with a way older firmware.