D435i IMU calibration issue

[feri113]

Camera Model D435i
Firmware Version 05.12.11.00
Operating System & Version Ubuntu 18.04 / 20.04
Kernel Version (Linux Only) 4.9.140-tegra / 5.8.0-44-generic
Platform PC / NVIDIA Jetson Tx2
SDK Version v2.42.0

Issue Description

I am not able to perform the IMU calibration (using the rs-imu-calibration.py) script. The program starts, the IMU sensor feed seems to be normal, but I can not go past Position 1 (mounting screw facing down as per this document).

See blow screenshots showing IMU values for position 1-6.
I am using the camera for the first time, is it possible that it has a HW deffect?

Position 1

Position 2

Position 3

Position 4

Position 5

Position 6

Also rs-motion seems to be working fine and reasonably tracking the the camera rotation. When starting realsense-viewer, I get a warning stating that "IMU Calibration is not available, default intrinsic and extrinsic will be used."

Thanks a lot in advnace.

[MartyG-RealSense]

Hi @feri113 It is highly unlikely that there is a hardware defect. The IMU calibration tool has been known to be over-sensitive in a small number of past cases though.

As you observed, an IMU calibration is not compulsory and the IMU can function in programs such as the RealSense Viewer and rs-motion without one.

The IMU calibration tool requires that the camera be rotated to the Align to direction set of coordinates specified by the tool and then the camera be left untouched. When the camera's current coordinates are confirmed by the tool to match the requested ones, an on-screen timer in the form of a row of dots should activate.

When the camera has been left at the requested coordinates long enough for the dot sequence to complete its full length, the tool should automatically move on to the next stage of calibration and ask you to rotate the camera to the next set of coordinates.

Except for the Position 2 and Position 4 orientations where the camera is rotated side-on, it is important that the camera is placed on a surface that is as flat and level as possible, such as a table, to make it easy to match the camera's current coordinates to the requested ones so that the timer can begin.

If you have placed the camera on a flat and level surface, rotated it to the requested coordinates and then left it untouched, are the coordinates relatively stable or are they fluctuating strongly and preventing a match, please?

[feri113]

Hi @MartyG-RealSense, thank you very much for the quick response.
As you are suggesting I am using flat and level surface and the displayed IMU values are fairly stable.
The issue I see is that in Position 1 there is a +1.0 offset for axis Y compared to the expected value and all 3 IMU values are near 0. It's funny because it would mean that I am in weightless environment, but I can assure you I am not :)

[MartyG-RealSense]

Could you explain please about your labelling of the results as Position 1 to 6 in the opening comment, please? All of the listed results refer to Position #1 – Mounting screw pointing down, device facing out. The testing process never seemingly goes beyond the first Position 1 orientation of the camera.

In your first set of test results, it does seem as though the conditions for successful detection of the requested coordinates were met (indicated by True, True, True).

To be clear, when the tool accepts the coordinates as correct, you should not move the camera yet. The program should be left to automatically proceed to the next step of generating an increasing row of dots until the dot counter is completed, causing the tool to then ask you to rotate the camera to the next orientation (Position 2 below).

[feri113]

@MartyG-RealSense thanks again for the quick response.

I think I figured out the source of the misunderstanding.

When looking at the Status.rotate values I was assuming that it corespodents to the actual g force values. With this in mind, for Position 1, I was trying to reach approx. [0 -1 0].

Now I understand that Status.rotate is not the actual g force value measured by the IMU but the delta to the target value/position (for Position 1 it is the delta to [0 -1 0]).

Knowing this I was able to succesfully perform the calibration.

As an imporvment I would suggest to change Status.rotate label to something that would better reflect this.

[MartyG-RealSense]

Thanks very much @feri113 for the update about your success in calibrating the IMU!

[phu251099]

@MartyG-RealSense Hi, I also have problem when I calibrate IMU from D435i camera, when I calibrate my 3 values ​​in position 1 all changed to true but the program doesn't automatically go to the next step. Do you have any way to solve this problem??? Looking forward to hearing from you soon. I have done the calibration in both Ubuntu and Windows but have the same problem.

[MartyG-RealSense]

Hi @phu251099 Once you have got the live-updating coordinates as close as possible to the requested target coordinates (indicated by True, True, True) then let go of the camera.

After three to four seconds it should then start displaying a slowly increasing row of dots on the screen. When the row reaches about 20 dots in length, the program should move on to the next stage and request the next orientation to rotate the camera into.

[phu251099]

Hi @MartyG-RealSense , Thank you for the quick response. I still don't understand what your saying "then let go of the camera" mean here?

[MartyG-RealSense]

Once you have finished moving the camera into position then take your hand away from the camera to give the tool time to think about the position and then start the dot count-up.

[phu251099]

I have placed the camera on the table plane and let go when performing the calibration in python, but for some reason my calibration is still at position 1 (with 3 true values) and not moving to the next steps.

[phu251099]

@MartyG-RealSense
This is the process by which I calibrate the IMU.

4364323683342167702.mp4

[MartyG-RealSense]

This does sometimes happen with the IMU calibration tool. The camera should be on a completely flat surface If it is completely flat and the program is still not moving on to the next orientation stage then it is likely best to stop trying.

I say this because it used to be necessary to complete at least one calibration and save it to the camera hardware before the RealSense SDK's IMU data correction feature unlocked in the RealSense Viewer. In the present day though, the option to apply 'correction' to the IMU data to make it more accurate is unlocked by default in the Motion Module section of the Viewer's options side-panel, so it is not necessary to go through the calibration process.

[phu251099]

So can you please guide me to calibrate the IMU through the camera viewer.

[MartyG-RealSense]

In the side-panel of the RealSense Viewer there is an option called Motion Module. Click on the arrow next to this option to view its settings.

In this list of settings, there is an option called Controls that also has an arrow beside it. Click on it to view the controls under the Controls category. The data correction option should be in this section. Click on it to enable it so that the Y-accel value of the IMU is adjusted to be closer to the ideal value of 9.80. Without correction, Y-Accel might have a value of between 9.4 and 9.6.

This data correction procedure with software is necessary because the RealSense IMU hardware component does not have internal calibration.

[phu251099]

This is what I see in my realsense viewer, but I still don't know how to correct it. Currently my Y-accel value is only 8.9. Hope you can help.

[MartyG-RealSense]

'Enable Motion Correction' has a blue box next to it, which means that it is On and is already automatically correcting the IMU data displayed in the main window.

Can you move the mouse cursor over the Accel ring so that I can see the values for Accel, please? Also place the camera motionless on a flat surface before capturing the screen to see if Y-Accel is still as low as 8.9 when the camera is not moving.

[phu251099]

@MartyG-RealSense here you are.

5084116711326350283.mp4

[MartyG-RealSense]

Thanks very much for the video. I have never seen a Y-Accel value of less than -9.4 on a stationary camera (even an uncalibrated one), so that is very unusual. I see that your Accel speed is set to 63. This indicates that your camera is older than April 2022, as around this time the D435i model changed to a different IMU component that supports a speed of 100 instead of 63. So it is possible that there is a hardware flaw in your camera's IMU component.

When did you purchase the camera?

It is probable that the camera's IMU can still work okay with a less than ideal Y-Accel value.

[phu251099]

Currently I am trying to use the IMU from the camera to assist in positioning the robot, but as you can see in the video, the IMU automatically drifts along the y-axis while the camera remains stationary. I tried to calibrate the IMU to improve this problem, but encountered the same problem as above. So bad!

3210334925624585246.mp4

768268569736651817.mp4

[phu251099]

I bought this camera in 2022 through a 3rd party supplier, and I don't remember the specific time.

[MartyG-RealSense]

Okay, so the camera would likely be outside of its one year return and replace warranty now if bought earlier than August 2022, and as it was a third party supplier rather than the official Intel RealSense store then the return claim would have to be made with that supplier.

Going back to the Python calibration tool and looking at your images of the orientation values, your Y rotation value seems to be quite far off the requested target value (target -1, actual value -0.0003 or -0.0004). Could you repeat the process and try to get Y angle close to -1 please to see if that is what is preventing it from moving to the next stage?

[phu251099]

i tried to do things like you said, but there is no way to rotate camera to get value [0, -1, 0].

[MartyG-RealSense]

0, -1, 0 should be facing straight ahead with the bottom of the camera flat on the surface.

[phu251099]

i tried many times but the result is still the same, all 3 values ​​are true but the program still does not execute the next step.

6990369636029249488.mp4

[MartyG-RealSense]

It appears that once the camera is left alone then the three live-updating values are continuing to fluctuate as though the camera is still being moved. As the calibration program needs about 3 seconds of being stationary before the dot count begins, the continuous fluctuations are likely misleading the program into thinking that the camera is still being moved.

I went through the video frame by frame in pause mode and observed that the status frequently changes to 'wait_to_stable' and then immediately goes back to the fluctuating values. This happens faster than the eye can see when the video is playing at normal speed. It suggests that it was trying to start the 3 second wait period before the dot count begins but the value fluctuations were immediately cancelling it.

Which RealSense SDK version are you using, please? As you have camera firmware driver 5.15.0.2, SDK 2.54.1 should be used with that firmware.

As you have tried the calibration on both Windows and Linux, it suggests that the problem isn't because of Windows (which uses a different back-end to the Linux version of the SDK and so performs some calculations differently).

You could try putting the camera on top of another object to see if it makes a difference to the fluctuations, such as putting a cardboard box or book on top of the table and then putting the camera atop that object.

[phu251099]

@MartyG-RealSense
I changed the camera and it worked fine. Currently I'm trying to read IMU data from the camera for navigation but I realized that the imu coordinate axes don't match the standard coordinate axes in ros. Do you have a way to fix this problem?

[MartyG-RealSense]

If the RealSense ROS wrapper is used then it converts the RealSense coordinate axes (Z is forward and Y is up) to the standard ROS coordinate system where Y is forward and Z is up). The Intel RealSense team member who created the ROS1 wrapper provides further information about this conversion process at IntelRealSense/realsense-ros#1099 (comment)

[phu251099]

@MartyG-RealSense I read the discussion as you said, and changed edit line of code 2031 to " float3 trans{ex.translation[2], -ex.translation[0], -ex.translation[1]}" but the result still nothing changed, could you please guide me in detail about this issue?

[phu251099]

Currently, the IMU data is read from the camera I put into the imu_filter_madgwick filter to be able to get more about the "orientation" value of the imu, and the problem is still the axis of the imu...

[MartyG-RealSense]

The most recent previous case about the RealSense axes in ROS is at IntelRealSense/realsense-ros#2741

In that case at IntelRealSense/realsense-ros#2741 (comment) I suggested using a static transform to change the axes or to change the camera position / orientation in RViz.

I do not have programming advice that I can offer unfortunately and development of the ROS1 wrapper has ceased as development of the RealSense ROS wrapper has moved to ROS2.

[phu251099]

I wrote 1 more node to subscriber to camera/imu topic and do the conversion, but when I go through imu_filter_madgwick, the coordinates displayed on rviz still have the phenomenon of deviation from the coordinate system in ROS. Below is my python node.

[MartyG-RealSense]

The only other potentially helpful reference that I have is Intel's D435i SLAM guide that makes use of imu_filter_madgwick, robot_localization and rtabmap_ros.

https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i

[phu251099]

@MartyG-RealSense It doesn't seem to help me, thanks for the help!

[MartyG-RealSense]

@phu251099 I researched your problem again but did not find any further new references that would be helpful, unfortunately. I do apologize.

[phu251099]

@MartyG-RealSense
Now I'm wondering what the covariance matrix of the IMU from the D435i camera is determined based on? I wonder if you can help me with this problem. Thank you very much!

[MartyG-RealSense]

@phu251099 The subject of a covariance matrix for tracking confidence has usually only arisen regarding the RealSense T265 Tracking Camera - now a retired camera model - which used the same IMU component as D435i. IntelRealSense/realsense-ros#770 is an example of a discussion about covariance matrix on T265 and its use in the camera's pose stream.