Heavy memory and thread usage

[ek9852]

librealsense open too many threads and have large memory usage.
Tested on D435 camera , but this problem is not specific to the camera module.
My app does not create any thread (just main thread) and use librealsense.
System shows librealsense create 16 threads. and consuming > 100M bytes of memory.
We only want to get the depth frame using following codes:

rs2::frameset data = pipe.wait_for_frames();

Why it needs 16 threads in the background. In Linux each thread by default will use 8Mbytes of stack. which consume 8x16 = 128 M bytes memory already.

The tools is nice for prototype. Do we have any lite version of this library to get the depth frame only, we don't need any other heavy weight tool for running it in real production device.

[dorodnic]

Hi @ek9852
Are you running on memory constrained platform?
Most of the threads are waiting, not consuming CPU, only memory as you pointed out.
Ultimately, D435 is a UVC camera, so you could use V4L2 or Microsoft Media Foundation directly. However, all threads serve some functional purpose, so unless you are willing to sacrifice functionality, you need at least some of them.

[ek9852]

Hi @dorodnic
We don't want to modify/patch the kernel to add the custom UVC color format.
So we were using librealsense via ibusb.
Why it create so many threads even those features are not enabled.
Those should create on demand.
And might I know why there are 16 threads ? What features need 16 threads to handle.
It should be at most 1-2 threads in a decent design.

[dorodnic]

From memory - 2 threads per streaming endpoint (3 for D400, 4 for L500). One to memcpy frame data from USB requests, second to perform internal format conversions. 1 thread per control endpoint (2 on D400 and L500) - the camera will lock up if someone doesn't poll on interrupt endpoints. Usually the driver is handling this, but not with libusb. 1 thread for device discovery, it is created only when using pipeline or device_hub, so in a way, on demand. 1 thread for camera error monitoring. Accelerometer and Gyro each have a thread (with D435i, L515 and D455). In addition, camera specific logic is running depending on the model (things like thermal loops).
You could do that in 1-2 threads, and you would get high percentage of frame drops due to USB buffer overflows.
The library is open-source, so you are welcomed to take any part of it, if 100-200 mb RAM is prohibitive for your project.

[ek9852]

Hi @dorodnic
Thanks for the clarification.

Are we using memory map in the v4l2 backend ? I notice that the memory map (use_memory_map is default to false). and libusb libusb_dev_mem_alloc is not used for mmap in the project. We notice a high system cpu usage (sys, not user space) on embedded device (7-8%) due to memory map is not used.

Might I know that what streaming format to use for depth 16 bits only ?
I grep V4L2_PIX_FMT, and cannot find the project is using/setting that. Only can grep it inside the kernel patches.

[dorodnic]

V4L2_PIX_FMT_Z16 - it is available starting from kernel 4.4.
Ultimately, you don't need to patch the kernel if you only need depth and either:
a. Don't care about hardware timestamp
b. Running 4.18+

The SDK is using V4L2_MEMORY_USERPTR and not V4L2_MEMORY_MMAP. The reason, I believe, was incompatibility of memory map with the metadata patch. Regarding libusb implementation - I'm not sure what was the reason not to use it, but I do believe it was evaluated. RSUSB code is shared between Windows 7, Mac, Linux and Android and as a result may not be most optimised for each.

No doubt, many things can and should be improved. However, if you are looking for tightly optimised solution for your specific platform and use-case it may be possible to get better results with custom code. The SDK is trying to address everything at once, this often creates implicit constraints that are not relevant for you. I will be happy to provide any technical information and also, we will take your feedback with the team, with regards to reducing the number of threads.

[ek9852]

Hi @dorodnic
Many thanks, The information you provided is very helpful and very positive.
I will try to implement use v4l2 api directly and use mmap to save system cpu loading. (This help to reduce the thermal of the system).

One quick question regarding how to enable/auto or disable the IR projector on D435.
How can I control that via v4l2 api. Is it control via the v4l2 control interface api, just like changing the brightness etc. I cannot find the corresponding code in librealsense. Seems librealsense just query the control interface descriptions from the camera and present the corresponding description directly to the user in the U.I.

[dorodnic]

Laser Enabled and Laser Power are implemented as extension-unit controls.
You should be able to do:

uint16_t value = ...;
uvc_xu_control_query q = { 3, 2, UVC_SET_CUR, sizeof(uint16_t), const_cast<uint8_t *>(&value)};
xioctl(_fd, UVCIOC_CTRL_QUERY, &q);

Reference:
https://github.com/IntelRealSense/librealsense/blob/master/src/linux/backend-v4l2.cpp#L1018-L1045
https://github.com/IntelRealSense/librealsense/blob/master/src/ds5/ds5-device.cpp#L1016-L1024

[ek9852]

Thanks.

[RealSenseCustomerSupport]

---

Hi @ek9852 Do you need further help on this? If we don't hear from you in 7 days, this issue will be closed.