D455 IMU on Jetson Nano with librealsense v2.50.0 or newer

[Brac24]

Current Configuration:
Jetson Nano with Jetpack 4.6.1 which is L4T 32.7.1
Docker running Ubuntu 20.04 with ROS2 Foxy
Realsense SDK v2.50.0 built from source with CUDA support
Most recent realsense-ros built from ros2 branch

I have a main ROS 2 underlay which uses sdk v2.48.0 and uses an older version of realsense-ros
and I've built my own overlay which is the newest realsense-ros and was built with sdk v2.50.0.

I can test each of these independently by sourcing the overlay it will use the newest version. If I don't source the overlay then it will use the older version with sdk v2.48.0.

The IMU data does publish when I use the older version but does not work when using the newer realsense-ros with sdk v.2.50.0.

Not sure if this is a librealsense or realsense-ros issue. I have seen that it is likely a librealsense issue.
But why would the newer version of librealsense break access to the IMU?

Where can I start looking to try to get this working again with newer versions of librealsense?
I have not tried the newer sdk v2.51.0 but I am in the process of downgrading to sdk v2.48.0 and testing if that fixes the issue.

Thank you

[MartyG-RealSense]

Hi @Brac24 When you mention 'underlay' and 'overlay', are you referring to running different librealsense + wrapper configurations on a 'host' computer and within a Docker container, please?

Each time that the librealsense version is changed, the ROS wrapper must be built again afterwards. Can you confirm that you have rebuilt the wrapper after an SDK version change, please?

The wrapper version should also be paired with a specific SDK version. In the older ros2 wrapper, 2.48.0 should be matched with wrapper 3.2.2 and 2.50.0 should be matched with 3.2.3

[Brac24]

I see so I need to use the tags instead of the ros2 branch? I just pulled the most recent ros2 branch and built that against librealsense v2.50.0.
If I build ros wrapper 3.2.3 does that also have support for ros2?

[MartyG-RealSense]

3.2.3 is a ROS2 wrapper that is compatible with Rolling, Galactic, Foxy, Eloquent and Dashing. Version 2.50.0 of librealsense is the correct SDK to use with it.

[MartyG-RealSense]

Hi @Brac24 Do you require further assistance with this case, please? Thanks!

[Brac24]

Yes sorry I will try to update it this weekend and test it.

[MartyG-RealSense]

Thanks very much for the update. Good luck!

[Brac24]

So apparently I was already using realsense-ros v3.2.3 as noted in the log line specifying RealSense ROS v3.2.3 and using LibRealSense v2.50.0

I am using the following command to run the camera:
ros2 launch realsense2_camera rs_launch.py align_depth:=true linear_accel_cov:=1.0 enable_accel:=true enable_gyro:=true unite_imu_method:=linear_interpolation color_width:=640 color_height:=360 color_fps:=30.0 depth_width:=640 depth_height:=360 depth_fps:=30.0
And again the above command works when running RealSense ROS v3.2.2 using LibRealSense v2.48.0.

When running v3.2.3 I get a Motion Module failure warning. Not sure if that tells me something. I include the log that has that warning below.

And actually I did some more testing and I also get the Motion Module failure when using RealSense ROS v3.2.2 with librealsense v2.48.0 but the IMU still works in this version. The following image is the logs for imu_filter_madwick node which is the node that ingests these IMU messages. As you can see the first few logs show that it is still waiting for data on topic imu which was me using RealSense ROS v3.2.3. But at the end of that you can see it says first IMU message received. That happened when I used RealSense ROS v3.2.2 with librealsense v2.48.0

[Brac24]

I also just looked over an old video I recorded back in the beginning of August where I used RealSense ROS v3.2.3 with librealsense 2.50.0 and it worked. I remember thinking I had no idea why it had worked that time but it did for some weird reason. After that one time it has never worked again

[MartyG-RealSense]

Does performance improve if you remove align_depth:=true from the launch instruction? It is a known issue that the RealSense ROS wrapper can experience high FPS lag when enabling align_depth on Nvidia Jetson boards specifically (non-Jetson computers such as desktop / laptop PCs do not have this issue).

IMU performance may also become more stable if you use unite_imu_method:="copy" instead of linear_interpolation.

When the IMU continues to work after a Motion Module Failure, it can indicate that there was a momentary hiccup that was automatically recovered from.

I recall a case in which a RealSense ROS2 user with a Jetson had problems with align_depth and pointcloud (which also has Nvidia-specific problems) that did not occur with SDK 2.48.0 and wrapper 3.2.2. That is consistent with the success that you reported having with this SDK / wrapper combination.

[Brac24]

So I currently had librealsense v2.50.0 natively installed on the nano. Using realsense-viewer I was still not able to get IMU data. I uninstalled this version and have now installed v2.51.1. Using realsense-viewer still showed no accelerometer data but gyro data would sometimes work so it was not very consistent when it would work. This was all using two D455's I have and would just switch between them to see if it was a device issue. Then I tried using the D435i that I have. The first time I used the D435i camera I got acceleromter data but no gyro data. Since then I no longer get accelerometer data but now I do get gyro data sometimes so it is still not very consistent. At this point it seems like a librealsense issue than a realsense-ros issue so I apologize.

[Brac24]

I just tested with a different jetson nano running JetPack 4.6.1 L4T 32.7.1 with a librealsense version of v2.50.0. I ran the rs-motion application with the D455 and it was not working at first. I switched to the D435i and it seemed to render maybe the first frame because the camera would move once and then no longer move. After a few more tries with the D435i it would still only render one frame. Then I swtiched to the D455 and it was working perfectly.
How does that happen?
Now it won't work anymore. I tried re-building liberalsense v2.50.0 on this nano and made sure to patch the kernel with the script provided by librealsense but still did not work. The most unstable seems to be the accelerometer but any ideas as to why?

[Brac24]

I restarted the jetson nano after the above installation and on the first run of rs-motion it seems to partially work. It only seems to rotate around the blue axis which I believe is the Z-axis. Then I re-ran the rs-motion application and this time it seemed to rotate about the X and Z axes but not Y. Still seems to be very unreliable but it seems to be partially working more often now than it was before. Sometimes it will work but never about all three axes.

[MartyG-RealSense]

There was a Jetson case at IntelRealSense/librealsense#10394 where a RealSense user with JetPack 4.6 had problems with not receiving frames from the accelerometer. In their particular case they resolved the problem by building librealsense from source code with the RSUSB backend installation method so that librealsense bypassed the kernel.

With the RSUSB method, librealsense is not dependent on Linux versions or kernel versions and does not require patching. An example of a guide for installating on Jetson with this method can be found at IntelRealSense/librealsense#6964 (comment)

In a RealSense ROS at #2212 another user reported no IMU data when using JetPack 4.6. Someone else on the same discussion who had the same issue resolved it by uninstalling the librealsense / wrapper combination installed from packages with the wrapper's 'Method 1' instructions and installing librealsense on its own.

On the same day as that case, another RealSense ROS user with a Jetson at #2213 also had IMU problems with Jetson 4.6.

---

As you are using 4.6 with Ubuntu 20.04, it could also be an issue with the Ubuntu version as Nvidia did not introduce official JetPack compatibility with 20.04 until JetPack 5 onwards. Prior to that, Jetson users had to develop workarounds to use 20.04, as described at #2213 (comment)

[Brac24]

Ok well I kind of wanted the ability for multi-camera so I think RSUSB would not be an option. I just went ahead and installed 2.48.0 for now and I think I'll go with that. Although I was getting a weird issue using the rs-motion application where the second time I would run the application the the camera looked like it was moving all over the place sporadically but the actual camera was not moving at all.

Also, do you have any suggestions or a possible direction on getting the IMU working on the Nano for newer versions of librealsense? I know it's probably not a simple task but I might be interested in diving deep into it just for my knowledge. Thank you

[MartyG-RealSense]

The Jetson user at #2213 (comment) with IMU problems with Jetson 4.6 who was using SDK 2.50.0 reported that downgrading their JetPack version from 4.6 to 4.5.1 instead of downgrading their librealsense version resolved their particular problem with no IMU data provided in the ROS wrapper.

My understanding of the SDK release notes is that the support for JetPack 4.6 / L4T 32.6.1 in 2.50.0 was contributed by a RealSense user at IntelRealSense/librealsense#9855 and the most recent official JetPack support in the SDK was for 4.5, added in SDK 2.43.0.

[Brac24]

So I'm trying JetPack 4.6 and I applied the patch but I noticed when the script finishes it doesn't apply patches for the hid sensors as in the instructions for the nano https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

[Brac24]

Also just tried it with JetPack 4.5.1 and running the patch also doesn't seem to apply it to hid sensors either. Only uvcvideo

[Brac24]

Never mind I read the comment in the script that the hid patches only apply to JetPack 4.4 and older

[Brac24]

So building SDK version 2.50.0 from source did not work on any JetPack version from 4.5.1 up to 4.6.1. Installing the sdk from the repository works fine but I also need CUDA. Is there a way to install from the repo that includes CUDA? Or is building from source the only way to get the CUDA capability?

[MartyG-RealSense]

Jetson packages installed from the instructions on the installation_jetson.md page should include CUDA in them. The librealsense installed from packages by 'Method 1' of the RealSense ROS wrapper instructions does not include CUDA though.

Given that the installation_jetson.md instructions would install the latest version of the librealsense packages though, it is probable that they installed 2.51.1 instead of 2.50.0. A RealSense user recently reported that they had CUDA support included when building 2.51.1 from source code but not with packages. A RealSense team member responded to this at IntelRealSense/librealsense#10911 (comment)

If 2.51.1 is being installed from packages and you are experiencing missing CUDA support then you could try installing 2.50.0 packages using guidance provided at IntelRealSense/librealsense#5773 (comment) for selecting a specific version.

[MartyG-RealSense]

Hi @Brac24 Do you have an update about this case that you can provide, please? Thanks!

[Brac24]

Hi @MartyG-RealSense I think I will close the issue for now and maybe create a librealsense issue in the future since I believe the underlying issue is with the SDK and not the ros wrapper. Thank you very much for your help.

[MartyG-RealSense]

You are very welcome, @Brac24 - thanks very much for the update!

[tanmayyb]

hey @Brac24, have you tried rs-motion command in your terminal? I am using D455 with librealsense 2.50.1(ubuntu 20.04) and when I use rs-motion I can see the movement of the camera. But when I use rqt-reconfigure and change the parameters of the ROS2 camera node to stream gyro and accel data alongside pointcloud, I don't get any publishings.

[MartyG-RealSense]

Hi @tanmayyb The gyro and accel topics are disabled by default in the RealSense ROS wrapper. You can either edit enable_gyro and enable_accel in ros2_beta's rs_launch.py launch file to set them to true instead of false or enable them from the launch instruction (ros2 launch or ros2 run).

ros2 launch
ros2 launch realsense2_camera rs_launch.py enable_gyro:=true enable_accel:=true pointcloud.enable:=true

ro2 run
ros2 run realsense2_camera realsense2_camera_node --ros-args -p enable_gyro:=true -p enable_accel:=true -p pointcloud.enable:=true