Issue with Hard Crashes on Realtime Linux Kernel + ROS Melodic

[jwatson-CO-edu]

Required Info
Camera Model	D405
Firmware Version	5.15.0.2
Operating System & Version	Ubuntu 18.04
Kernel Version	4.19.50-rt22, Realtime Kernel Patch
Platform	PC, Xeon E3-1200
SDK Version	v2.54.1, Commit # 8ffb17b
Language	C++, ROS Melodic
Segment	Robot

I am having an issue with a node on ROS Melodic based on the librealsense SDK on Ubuntu 18.04 with 4.19.50-rt22 realtime kernel patch. The node itself is a light wrapper that packages RealSense data as ROS messages.
After anywhere between 5 and 50 minutes of starting the node, my computer will crash; the UI will freeze and a few seconds later the OS will restart entirely. (I tracked it down to this node by process of elimination.)
Neither dmesg nor journalctl reveal anything particularly useful about the crash. Data coming from the D405 appears correct and arrives in a timely manner.

I built librealsense from source with RSUSB based on the idea that this might support older kernels, but the problem persists.

I used the following steps to build librealsense:

1. sudo apt install libssl-dev libusb-1.0-0-dev libudev-dev pkg-config libgtk-3-dev cmake
2. sudo apt install libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev at
3. cd /tmp/
4. git clone https://github.com/IntelRealSense/librealsense.git
5. cd librealsense/
6. Unplug any attached RealSense cameras
7. ./scripts/setup_udev_rules.sh
8. ./scripts/patch-realsense-ubuntu-lts.sh
9. Add the following near the top of CMakeLists.txt: set( FORCE_RSUSB_BACKEND ON )
10. mkdir build && cd $_
11. cmake ../ -DCMAKE_BUILD_TYPE=Release -DBUILD_EXAMPLES=true
12. sudo make uninstall && make clean && make && sudo make install

Can you think of anything in particular about librealsense that may interfere/interact with realtime scheduling?

[MartyG-RealSense]

Hi @jwatson-CO-edu The RealSense ROS1 wrapper does not have official support for the D405 camera model. The ROS2 wrapper does officially support it.

It is possible to add D405 support to the ROS1 wrapper by editing its code or by using a pre-made custom 'fork' version of the wrapper created by a RealSense community member. Please see IntelRealSense/realsense-ros#2582 (comment) and the link below.

https://github.com/rjwb1/realsense-ros/tree/development

Below are some other factors to consider.

---

FORCE_RSUSB_BACKEND should be set in the CMake build instruction rather than by adding a line to CMakeLIsts.txt.

cmake ../ -DCMAKE_BUILD_TYPE=Release -DBUILD_EXAMPLES=true -DFORCE_RSUSB_BACKEND=ON

If RSUSB Backend is successfully enabled in the SDK build then the kernel that is being used should not matter, as an RSUSB build bypasses the kernel.

When using RSUSB = On, you also do not need to apply a librealsense kernel patch script such as ./scripts/patch-realsense-ubuntu-lts.sh. Kernel 4.19 is not supported by the patch script anyhow.

[jwatson-CO-edu]

1. I use the ROS1 wrapper that you mentioned, but thank you for the suggestion, @MartyG-RealSense.
2. I made the changes to the build process that you suggested, and librealsense built successfully, but the problem persists.

Does librealsense perform any logging that would remain after a restart that I can use for troubleshooting?

This bug is currently hampering my entire ROS stack.

[MartyG-RealSense]

Linux writes all OS logs to the /var/log folder. You can review the entire kernel log file with the command below.

less /var/log/kern.log

If your ROS log has a process has finished cleanly message after the crash then there should be a log location displayed on the next line of the log. For example:

log file: /home/ubuntu/.ros/log/bb60da40-105c-11ec-bdf8-27733b72faf1/camera-realsense2_camera-3*.log

[jwatson-CO-edu]

This was an unknown hardware issue that resolved itself when I replaced the computer I was using with an identical machine.
The problem is not present on the new computer and I was able to run programs using librealsense both inside and outside of ROS for hours at a time with no issue.
Troubleshooting did not reveal the problem on the original computer, but evidence does not point to a problem with this library.
Issue closed.

[MartyG-RealSense]

Thanks very much @jwatson-CO-edu for the update. It's great to hear that a solution was achieved.