RSUSB backend fails to set options

[SirDifferential]

Required Info
Camera Model	D415/D435
Firmware Version	Signed_Image_UVC_5_12_6_0.bin
Operating System & Version	Debian 10 & Raspbian 10
Kernel Version (Linux Only)	4.19.59 & 4.19.118-v7l+
Platform	PC x86_64 and Raspberry Pi 4
SDK Version	2.36.0, latest master f7cdf6e
Language	C++
Segment	others

Issue Description

When building the SDK with FORCE_RSUSB_BACKEND=ON, the emitter, auto exposure and region of interest requests don't always work on a D415/D435. Here's what the errors are:

Set region of interest
RealSense error calling rs2_set_region_of_interest(sensor:0x607000131d90, min_x:256, min_y:192, max_x:384, max_y:288):
 failed to set power state when setting auto exposure region of interest.
Setting auto exposure off
RealSense error calling rs2_set_option(options:0x607000131d90, option:Enable Auto Exposure, value:0):
 set_xu(id=11) failed! Last Error: Resource temporarily unavailable when toggling device auto exposure settings.
Enabling emitter
re-enable auto exposure
Set region of interest
RealSense error calling rs2_set_region_of_interest(sensor:0x607000131e00, min_x:256, min_y:192, max_x:384, max_y:288):                                                                      
 hwmon command 0x75( c0 120 100 180 ) failed (response 0= Success) when setting auto exposure region of interest.                                                                           
Setting auto exposure off
RealSense error calling rs2_set_option(options:0x607000131e00, option:Enable Auto Exposure, value:0):                                                                                       
 failed to set power state when toggling device auto exposure settings.
Enabling emitter
Sensor does not support RS2_OPTION_EMITTER_ENABLED
re-enable auto exposure
RealSense error calling rs2_set_option(options:0x607000131e00, option:Enable Auto Exposure, value:1):                                                                                       
 insufficient data writen to USB when toggling device auto exposure settings.

Two things of note:

• Region of interest fails to set with two different errors: first with failed to set power state, then with response 0= Success. The second one is an oversight in the RealSense SDK that confused me for a while, but happens because of this part of hw-monitor.cpp where the firmware is supposed to respond with newCommand.cmd, in this case 0x44, but it responds with zero, which is then parsed as hwmon_response where 0 means hwm_Success.
• The operations sometimes fail and sometimes succeed. In the above example, ROI fails to set followed by auto exposure failing to set, then the emitter is toggled successfully, followed by auto exposure toggling successfully, after which region of interest fails to be set followed by the sensor suddenly not supporting RS2_OPTION_EMITTER_ENABLED at all.

Other errors the requests sometimes produce are:

RealSense error calling rs2_set_region_of_interest(sensor:0x12f65c8, min_x:512, min_y:288, max_x:768, max_y:432):                                                                           
 set_xu(ctrl=1) failed! Last Error: No such file or directory when toggling device auto exposure settings.                                                                                  
RealSense error calling rs2_set_option(options:0x12f65c8, option:Enable Auto Exposure, value:1):                                                                                            
 map::at when toggling device auto exposure settings.

Here's a minimal program demonstrating this.

[MartyG-RealSense]

Apologies for the delay in responding. I was going through your case very carefully, looking at it from many different angles.

Given that you stated that operations can fail and then succeed, a first action to try could be to create a short sleep period inbetween tries. This would give librealsense more opportunity to pause and catch its breath before re-trying. I see that you are already using sleep in your minimal script, but it was not clear whether it was sleeping after each try or only after more than 5 tries had been attempted.

If it is per try, you could also see whether setting a longer sleep period makes a positive difference.

It may also be worth simplifying the script by removing some of the steps temporarily (e.g taking out the Laser Power check) and see if stability improves. This trial and error process of taking a section out, doing a test run and putting it back if performance does not improve, and then repeating the process for a different code block, may indicate whether a particular block of code is introducing instability.

[ev-mp]

@SirDifferential , I've looked up the mentioned code and there are definitely things you can improve:

1. https://github.com/SirDifferential/minimal_rs_advancedmode/blob/master/main.cpp#L397-L406
   The ROI functionality is not available for all sensors. You have to check whether the sensor supports it before making the call (to avoid receiving errors)

    if (sensor->is<roi_sensor>())
    {
        sensor->as<roi_sensor>().set_region_of_interest(<TBD>);
    }

2. In case the sensor supports "ROI" interface you need to ensure Auto-Exposure (AE) is ON before making set_roi call - it can't work with Manual Exposure. And currently this check is omitted in the code.
   This is consistent with

The operations sometimes fail and sometimes succeed

as when you reach that code segment the AE state is unverified.

after which region of interest fails to be set followed by the sensor suddenly not supporting RS2_OPTION_EMITTER_ENABLED at all.

When iterating over all the sensors available you have use use heuristics to differentiate between sensors as the controls that you've selected do not apply to all sensors - set/get RS2_OPTION_EMITTER_ENABLED will fail for RGB.
At the minimum you apply the check->use guard aproach:

    if (sensor.supports(ABC))
    {
        sensor.set_option(ABC);
        ....
    }

4. The above applies to both RSUSB and v4L drivers implementation so it is strange that you encounter this only with RSUSB. Can you confirm that this code work correctly with v4l backend?.

[SirDifferential]

Thank you for your replies.

I simplified the test program to do less things. Now I load my current settings from a JSON to reduce the potential for errors.

The ROI functionality is not available for all sensors. You have to check whether the sensor supports it before making the call (to avoid receiving errors)

I've adjusted this in.

In case the sensor supports "ROI" interface you need to ensure Auto-Exposure (AE) is ON before making set_roi call

Ah - I wasn't aware of this. I've adjusted the code to do this.

When iterating over all the sensors available you have use use heuristics to differentiate between sensors as the controls that you've selected do not apply to all sensors

Good point - the error about the option not being supported was likely for the RGB sensor.

Can you confirm that this code work correctly with v4l backend?

I set up a new Ubuntu 18.04 4.15.0-37-generic Upboard for testing today, with the kernel patches and v4l backend. I first applied your recommended changes and repeated my test on Debian 10 / Raspbian10 without the kernel patches on RSUSB. The results:

Debian 10 with RSUSB

• Testing if the sensor supports the required options works without errors and the correct sensor is selected
• Querying the auto exposure state with s.get_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE) fails with:

RealSense error calling rs2_get_option(options:0x607000131620, option:Enable Auto Exposure): failed to set power state when disabling auto exposure.

• Attempting to then set the auto exposure to 1 with s.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, 1.0f) gives the error:

RealSense error calling rs2_set_option(options:0x607000131620, option:Enable Auto Exposure, value:1):
set_xu(id=11) failed! Last Error: Resource temporarily unavailable when toggling device auto exposure settings.

• Further on the ROI setting is not set due to the auto exposure state being unknown. Trying to set it regardless gives the same errors as the original message in this issue.

Raspbian 10 with RSUSB

• The same power state errors are printed when calling set/get option for RS2_OPTION_ENABLE_AUTO_EXPOSURE, although sometimes this is printed:

get_xu(id=11) failed! Last Error: Resource temporarily unavailable when setting auto exposure region of interest.

• Sometimes the get request goes through without exceptions, but the returned value for the auto exposure state is not 0 or 1, but 255, which to me sounds like uint8 being assigned -1. If I move on and then set the ROI settings, I (as expected) get an error:

RealSense error calling rs2_set_region_of_interest(sensor:0x1c1acd0, min_x:256, min_y:192, max_x:384, max_y:288):
 set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable when setting auto exposure region of interest.

Ubuntu 18.04 4.15.0-37-generic with kernel patches and V4L backend

Found a depth sensor
Getting auto exposure state (1)
Setting auto exposure off
Setting auto exposure on
Getting auto exposure state (2)
auto exposure value before ROI request: 1
Set region of interest
Enabling emitter

All requests completed without error and the modes were toggled properly.

As such, it looks like this is an issue with RSUSB.

[SirDifferential]

And as per Marty's recommendation to add a few sleeps and retry the requests, I could try adding that in. At some point I noticed that with RSUSB backend some set/get requests don't always succeed on their first try, but if you repeat them a few times with sleeps between, they sometimes do succeed. This is far from ideal, but it's worth a try, although I'd like to understand better what causes in this sort of behavior.

[MartyG-RealSense]

Hi @SirDifferential Do you require further assistance with this case, please? Thanks!

[SirDifferential]

Yes please. Even with the fixes and suggestions, the options fail to toggle. Especially the 255 return when getting RS2_OPTION_ENABLE_AUTO_EXPOSURE sounds like something is off.

I'm going to be on a vacation for the next 3 weeks, but will look back at this when I get back in the office.

[MartyG-RealSense]

In a recent discussion about how events / callbacks can be missed when using RSUSB, advice was provided about changing the timing of timeout events:

#6921 (comment)

[MartyG-RealSense]

Adding a note to keep this case open a further period whilst @SirDifferential is on vacation.

[MartyG-RealSense]

Hi @SirDifferential Do you have an update on this case, please? Thanks!

[SirDifferential]

Sorry about the delay. Things were pretty busy here for a while.

I tried applying pull request 7085 and played around with various BACKEND_POLLING_INTERVAL values (even 1, which on pretty much all implementations means sleep and return as soon as the scheduler has time for it), but this did not seem to change anything. I also tried adding several retries for setting these options, but the errors listed above keep happening. The non-RSUSB backend with the kernel patches works with the same code and requires no retries.

As such, it looks like this is still a valid issue, and I'd appreciate hearing on what could cause it / how we could get past it. Considering Realsense Viewer seems to be able to control these values when built with RSUSB, I'm guessing there is some way to send these requests in without them failing. However, in the Realsense Viewer log, there are errors being printed when I toggle auto exposure or specify its region of interest, although the live feed clearly shows auto exposure toggling and adjusting correctly.

 15/09 10:13:45,974 WARNING [140566662723328] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61

EDIT: To sum up what my code does:

• Find the depth sensor out of all current sensor (works)
• Read the current value of auto exposure - always returns 0, off, even if the auto exposure is on
• Disable auto exposure (works)
• Re-enable auto exposure (value 1) (works after one or two retries)
• Read the current value of auto exposure - does not work, always returns 0
• Ignore the fact that the SDK reports auto exposure as being off and try setting region of interest. Fails, as expected due to auto exposure being off

[MartyG-RealSense]

Thanks very much for the detailed feedback @SirDifferential I re-read your case from the start to refresh my memory. The main thing that stood out was that you tested successfully with kernel patches and V4L backend and didn't get the problems that you do with RSUSB. Yet I did not see any reasons arguing against use of the kernel-patching approach.

If kernel-patching works fine for you, is there a reason why the RSUSB method is preferred please?

[SirDifferential]

We use a Raspberry Pi 4 and the kernel patches aren't currently available for kernel 4.19.118-v7l+ (EDIT: or 5.4.51-v7l+) which is what Raspbian uses. There was some talk long ago about porting the patches in one of the issues on this repo, but no one involved seemed to have the technical skill to work with the kernel, myself included.

[MartyG-RealSense]

Would switching to Ubuntu 18.04 LTS on Pi 4 instead of using Raspbian be an option for your project in order to make kernel patching more feasible?

https://maker.pro/raspberry-pi/projects/install-ubuntu-18044-lts-on-your-raspberry-pi-board

[SirDifferential]

We've tried running Ubuntu 18.04, and currently do with our old Upboard based model, but on raspberry it had other issues, namely with the communications to the hardware via vcgencmd as well as access to the GPU, although that's likely going to get fixed eventually. Generally speaking Ubuntu hasn't been the distro we've found the easiest to work with on embedded devices. So if this bug (if it is one) can be fixed in the SDK, I'd obviously prefer that. If it won't get fixed, then we'll have to start looking at using a supported kernel instead.

[MartyG-RealSense]

As @ev-mp kindly provided advice earlier in this discussion, I will tag @ev-mp to see if they have further advice about your problem of auto-exposure and ROI requests not always working on RSUSB.

[MartyG-RealSense]

Tagging a reminder to @ev-mp to kindly take a further look at this case if possible. Thanks!

[MartyG-RealSense]

Sorry that you haven't received responses @SirDifferential - can you confirm that the problem in this case remains an ongoing issue?

[SirDifferential]

Thank you for your attention on this issue. I tried updating to the new SDK version and applied the latest development firmware, but it didn't seem to change anything.

[MartyG-RealSense]

I have re-reviewed the entire case from the beginning but cannot see many other options to try. The need to use Pi and the Raspbian OS particularly are restricting factors.

I know that you earlier ruled out compiling from source with kernel patching instead of using RSUSB. The dependency on Raspbian instead of Ubuntu may also make DKMS package distribution unsuitable.

#6952 (comment)

Before the RSUSB method was introduced, there was another patchless installation method that was called libuvc-backend: As far as I know, libuvc-backend is still a valid method of installation even after RSUSB was introduced.

https://github.com/IntelRealSense/librealsense/blob/master/doc/libuvc_installation.md

[SirDifferential]

Correct me if I'm wrong, but isn't libuvc the old name for RSUSB? CMakeLists.txt seems to say so, and flips RSUSB on if you specify the flag:

#Deprecation message, should be removed in future releases
if(${FORCE_LIBUVC} OR ${FORCE_WINUSB_UVC} OR ${ANDROID_USB_HOST_UVC})
    MESSAGE(DEPRECATION "FORCE_LIBUVC, FORCE_WINUSB_UVC and ANDROID_USB_HOST_UVC are deprecated, use FORCE_RSUSB_BACKEND instead")
    set(FORCE_RSUSB_BACKEND ON)
endif()

Without FORCE_RSUSB_BACKEND enabled, CMake/unix_config.cmake specifies:

    if(FORCE_RSUSB_BACKEND)
        set(BACKEND RS2_USE_LIBUVC_BACKEND)
    else()
        set(BACKEND RS2_USE_V4L2_BACKEND)
    endif()

Which translates to using the direct v4l backend with patched kernel modules.

[MartyG-RealSense]

RSUSB did replace the term libuvc in the CMake build flag instructions, yes. There were several different libuvc commands that were simplified to just a single FORCE_RSUSB_BACKEND command. Having looked at the instructions in the libuvc_installation.sh script, it does seem to essentially perform the same CMake commands as the RSUSB instructions. My apologies.

[MartyG-RealSense]

Hi @SirDifferential Do you wish to continue this discussion please? Thanks!

[SirDifferential]

Yes, I would prefer this to be kept open until it's verified as a bug and fixed, or until there's verification that this is simply invalid usage of the API from my part.

[MartyG-RealSense]

I have been over the case thoroughly again and there does not seem to many options to pursue.

• It is necessary to use RSUSB, because patches have problems with Raspbian on your Pi.
• Ubuntu cannot be used.
• Changing librealsense settings on RSUSB results in errors.

I suspect that a solution is going to involve some kind of progress on the issue of RSUSB timeout periods that was previously discussed.

#6921 (comment)

[MartyG-RealSense]

Hi @SirDifferential You mentioned earlier that "I would prefer this to be kept open until it's verified as a bug and fixed, or until there's verification that this is simply invalid usage of the API from my part."

I will ask about this case within Intel in order to bring the case to a resolution, one way or the other.

[MartyG-RealSense]

After discussion with Intel, I have filed an offcial bug report. This is not confirmation that it definitely is a bug, but rather that it should be investigated to determine whether or not it is one.

[SirDifferential]

@MartyG-RealSense Has there been any kind of reaction from Intel yet? We're still experiencing issues toggling the emitter exposure with the latest firmware + SDK version.

[MartyG-RealSense]

The official bug report I sumitted on October 23 2020 is still active but there is no further information that I can provide at this time.

[puzzlepaint]

I also experienced "failed to set power state" errors with the RSUSB backend for code which worked fine with the V4L2 backend. I found a solution to this problem which I described here:
#9157 (comment)
Perhaps it applies to this case as well.

[SirDifferential]

That's an interesting find! I'll give it a try when I'm back in the offices after my vacation.

[MartyG-RealSense]

Hi @SirDifferential Were you able to test the suggestion of @puzzlepaint please? Thanks!

[MartyG-RealSense]

Hi @SirDifferential Do you have an update about this case that you can provide regarding testing the suggestion of @puzzlepaint please? Thanks!

[SirDifferential]

Sorry about the delay. I just got back in the office and will check this during this week.

[MartyG-RealSense]

Thanks very much for the update @SirDifferential

[SirDifferential]

I applied the suggestion by @puzzlepaint into minimal_rs_advancedmode, and the result is the auto exposure and related settings toggling properly with RSUSB. Thanks for suggestion!

As to librealsense itself, I tried digging the sources for how this issue could be fixed, but it didn't seem that easy to do. I recommend you at least document this bug in the pipeline / sensor classes in some way. Do you want to keep this issue, and perhaps #9157 as well, open until a proper solution is introduced into the library?

[MartyG-RealSense]

Hi @SirDifferential It's excellent to hear that you were able to implement a successful toggle with the advice of @puzzlepaint :)

As this case has an active Intel bug report associated with it, the case should be kept open until that bug report has an outcome.

[MartyG-RealSense]

The Intel bug report investigation was concluded and closed as it was decided that the problem was in the script code being used rather than an SDK bug.

@SirDifferential and @puzzlepaint Is there any further comment that you would like to make about this issue, please or can it be closed? Thanks!

[puzzlepaint]

Given that the requirement for users of librealsense to re-query the camera objects after starting streaming is easily overlooked (since that works fine with the V4L2 backend and only breaks with RSUSB), from a library design standpoint, I believe that it could be worthwhile for Intel to look into removing this requirement on the library's side (making it work with RSUSB too), or at least detect the wrong usage and warn about it. Otherwise, it seems very likely to me that more users of librealsense will run into this issue in the future. But if there is no interest in doing that on Intel's side, then I think that this can be closed.

[MartyG-RealSense]

Thanks very much @puzzlepaint for your detailed feedback! Intel made some changes to RSUSB in SDK 2.50.0. Quoting from the 2.50.0 release notes:

---

[RSUSB] Backend - reattach kernel driver.
Fixes RSUSB/V4L interoperability:
RSUSB backend detaches the device from the V4L2 kernel driver and never reattaches it. Consequently, after using RSUSB backend, applications using the v4l backend can't find the device

[puzzlepaint]

This sounds to me like it relates to a different issue. In any case, from my point of view, this issue here could be closed unless @SirDifferential wants to keep it open.

[MartyG-RealSense]

Thanks @puzzlepaint - I just wanted to highlight that there have been changes to the RSUSB system since this case was began.

[SirDifferential]

I've changed companies ever since this bug was started, so I no longer have a way to debug or test this. But I agree with everything @puzzlepaint said. Close this, but some people may be experiencing this in the future.