Running SLAM system on AirSim

[xuhao1]

Hi Guys,
I have successfully run a well-known visual-inertial system VINS(https://github.com/HKUST-Aerial-Robotics/VINS-Fusion) on AirSim (which is amazingly good!) based on the ros package which verified the possibility of simulating my own aerial swarm in Airsim.

However, I found a few issues about the ros package;

1. The Unit of the angular velocity is wrong. The data from airsim is already rads
2. The coordinate system seems is confused
3. The unnecessary process lock lower the whole performance, with the useless lock we can only get IMU for ~20hz(because IMU always wait for images....For slam system we need high frequency IMU and low frequency images and depth), without we can get 300hz!
4. The ros package used simGetImages with single request a few times, which will be slow when we use a stereo camera. Change simGetImages with multiple requests will be much faster on mutliple cameras!
5. The ros package use soft time stamp (the receive ros::Time::now) instead simulation time
6. Nowhere to set host IP

For problem 1. 5. 6. I have submit a pull request #2368,
For problem 234, my modification loses the generic but it solves the problem, gives me 20HZ stereo images and 300hz IMU on a remote notebook and successfully running VINS-Fusion in AirSim smoothly, here is my repo https://github.com/xuhao1/airsim_ros_pkgs.
If you are interested in running VINS or other slam methods, you can have a try. The configs file for VINS and settings.json for Airsim have been placed in vins_airsim/ folder

[simengangstad]

We've worked some with VINS in AirSim as well, but haven't gotten any good results. Do you have any numbers on how much the position drifts compared to ground truth from your current setup?

[xuhao1]

@simengangstad , there is drift. However, drift is normal for VINS-Fusion without Loop Closure on. If you are interested in it, you can have a try with my package.

[Beniko95J]

Hi, may I ask how did you achieve 20Hz stereo images? I am also trying to run VINS-Mono on AirSim, and when I set the image resolution to 288x512 (original 144x256) and set the timer to save images every 0.05 second, I can only obtain 3Hz 288x512 images through its image apis. Do you have any idea about this?

[kminoda]

I am having similar issue in similar situation, which is to run VIO algorithm on AirSim ros pkg. I want to change frame rate of camera and IMU.

I assume that its "update_airsim_img_response_every_n_sec" in here that corresponds to camera frame rate. But it doesn't make any difference. Camera frame rate is still 3Hz even though I set the value to 0.1 (so it should be 10Hz).

I modified airsim_ros_wrapper.cpp according to this repos, but it doesn't solve the problem either.

Does anyone know how to solve this problem?

[dssdyx]

@xuhao1 Which coordinate did you transform imu link to? NED to ENU or other? I see your body_T_cam in vins yaml, is corresponding to ENU. The linear vel from NED to ENU should be (x,y,z)->(x,-y,-z),but your code is (x,y,z)->(-x,y,-z). I feel really confused.

[manthan99]

I am also having a similar issue with the published rate. I tried modifying the code according to this repo but I am only getting a publish rate of around 5 Hz for the camera and 10 Hz from IMU. Changing the "update_airsim_img_respone_every_n_sec" doesn't seem to increase the rate.
Any solution to increase the rate?

I am having i7-6700HQ, 16 Gb ram and Gtx 1070

[rajat2004]

Could you check the parameter DrawDebugPoints in Lidar in the settings.json? If true, that causes a huge slowdown in the current latest binaries. Fix #2614 has been merged into master, but the problem is there in the binaries. Just change that to false and should improve the rates
There's also another WIP PR #2472 to increase the camera rate

[manthan99]

@rajat2004 I am not using a Lidar. Here is the settings file which I am using [Settings.json]

Though the rate significantly improved to 20 Hz upon using the "NoDisplay" viewmode

[rajat2004]

Mentioned about the Lidar since that was causing problems with people trying out the ROS package tutorial and all. Using NoDisplay will definitely improve the rate since rendering is less
@xuhao1 It would be great to have a PR with your changes if you're interested

[manthan99]

There seems to be a problem with the sync of images from left and right cameras ( ~0.05 sec)
@xuhao1 did you face this issue?

[kminoda]

@manthan99 It’s nice to hear that you managed to get images on 20 Hz by using NoDisplay mode.
Did you just change “ViewMode” to NoDisplay in the settings.json you posted?

[manthan99]

@kminoda Yes I just changed the "ViewMode" to NoDisplay in the settings.json file. In addition, I also replaced the airsim_node.cpp file that is given in this repo though I do not know if that made a difference. But still there is a problem with the syncing of both the images. The difference is pretty high (~0.05s)

[kminoda]

With an advice from a researcher who previously worked on running VIO in AirSim, I managed to run VINS-Mono on AirSim by changing "ClockSpeed" in settings.json.
For me, since I could get 3Hz at maximum, I made ClockSpeed to 0.033 so that I can get 100Hz (in simulation time, not in actual time) for IMU.
OFC it take much longer time to collect data even for short ones, but I can now estimate the pose of UAV without any drift.

[manthan99]

@kminoda Did you change the coordinate system for IMU ? And which parameters file did you use ?

[kminoda]

@manthan99 I didn’t change IMU frame (but maybe it’s better to do so).
I verified that VINS-Mono is working by comparing the odometry output from vins and odom_local_ned. I changed the frame of odom_local_ned in comparing them.
For parameters file (you mean the config for VINS-Mono/ Fusion, right?), I used this config file

[GimpelZhang]

@kminoda Yes I just changed the "ViewMode" to NoDisplay in the settings.json file. In addition, I also replaced the airsim_node.cpp file that is given in this repo though I do not know if that made a difference. But still there is a problem with the syncing of both the images. The difference is pretty high (~0.05s)

I wonder if you are still working on it.

The link posted by @xuhao1 : https://github.com/xuhao1/airsim_ros_pkgs may be able to solve the timestamps difference problem between the left and right images.

Main changes are in airsim_ros_wrapper.cpp and airsim_ros_wrapper.h. You may need to comment dji_sdk related parts which xuhao1 was using to connect with some particular hardwares.

Here is my modified version that might be just for reference.
https://github.com/GimpelZhang/airsim_ros/tree/slam

The timestamps between left and right has a max ~0.0003s difference, which I think is good enough for stereo vslam algorithms. (I set the clockspeed to 0.1 when I tried)

P.S.
I only checked the timestamps. I haven't used the data on any vslam algorithms yet.

[kminoda]

I also made it work by collecting IMU data with ROS while using computer vision mode for camera images (This way you can get completely synchronized images).

Here is a procedure.

• Use airsim_ros_pkg to record IMU and ground truth odometry msgs to generate a rosbag.
• Set AirSim to computer vision mode.
• Capture images from the point which you can get from gt odometry in the rosbag, and write these into it as rosmsg.

Hope it will help someone :)

[rajat2004]

If anyone wants to try the Faster Image Capture PR, they can try the binaries in #2713, On Linux need to run with -opengl
Also any PRs to fix problems in ROS wrapper would be great :)

[dvdmc]

Since airsim_ros_wrapper changed quite a lot since this post, the referenced repos are old for being used as reference. I have added a timer to publish the IMU at a larger frequency in the main thread (as opposed to images and Lidar) link. I don't know if this is a good solution but still I'm not able to use VINS-Fusion. I think that now the problem is related to synchronization or the movement I'm using. Main complaints from VINS:

[ INFO] [1624356525.747085129]: misalign visual structure with IMU
[ INFO] [1624356525.863072756]: Not enough features or parallax; Move device around

I am still having the problems of stereo pair synchronization and it is not clear for me if the coordinate system confusion is fixed in later versions of AirSim. Maybe the users who previously answered could provide updated answers or hints about the specific changes in the ROS wrapper code. Also, is there any intention of considering this issue in future versions of the official repo? I am willing to contribute :)

[phil333]

@davdmc
That sounds like you having trouble configuring Vins-fusion? i needs a correct transformation matrix between the imu and each camera. If thats not properly set. Secondly, what environment are you using? is empty or are there any things that can be tracked
I stopped going down this road because the left and right image are not synchronized. As long as this is not sorted, testing VO or VIO objectively is difficult, especially when producing larger images like for example the ZED camera. Essentially the images should be captured in parallel, but this might require freezing the sim, capturing and then unfreezing (not an expert on this).

[xxEoD2242]

@davdmc

I'll give you some general tips on how to utilize this:

1. You need to rewrite the image request structure of the ROS wrapper. Currently, the ROS wrapper is not written efficiently. I would recommend building your own thread that only gets the stereo images as a pair in the image request.
2. You need to increase the tolerance of VINS Fusion. The value is currently 0.03 but you need to increase that to 0.07.
3. Most likely your timestamps are not working properly for the images. I would look at those stamps and make sure they are in ROS time.

Just general pointers for working with VIO systems in AirSim. Happy to help with more specific issues.

[phil333]

@xxEoD2242
when you say tolerance, which value in vins fusion are you talking about?
https://github.com/HKUST-Aerial-Robotics/VINS-Fusion/blob/master/config/euroc/euroc_stereo_imu_config.yaml

[GimpelZhang]

@xxEoD2242
when you say tolerance, which value in vins fusion are you talking about?
https://github.com/HKUST-Aerial-Robotics/VINS-Fusion/blob/master/config/euroc/euroc_stereo_imu_config.yaml

I think @xxEoD2242 means the sync tolerance in the code processing stereo images: rosNodeTest.cpp

[phil333]

ok. There is a good reason for this tolerance to be so low, and that it is hard coded. But i guess you can test if there are any other issues. In that case i would use the car model for testing, as it can move slower.

[xxEoD2242]

@phil333 @GimpelZhang Yup, that was what I was talking about.

From my experience with AirSim, even with every optimization, you aren't going to get that time difference to be within Fusion tolerance. If AirSim is updated with faster image capture internally, then that is possible. But from how AirSim works under the hood, getting UE to actually generate the images is just not fast enough for 3 milliseconds of latency.

[phil333]

:(

[catproof]

I understand there is something fundamentally wrong with how the images are captured. But I don't see how setting the clock speed very low in simulation doesn't give an acceptable delay between when left images are taken and right images are taken. doesn't AirSim also have a RGBD camera sensor? That is ultimately quite similar to what stereo cameras have to offer.

[xxEoD2242]

@NickPerezCarletonUniversity if that works, then great. However, when working with Fusion, we raise the tolerance of the image difference. For our purposes, lowering the clock speed isn't an option due to processing requirements and time calculations.

Depth is great but when doing VIO, you need both camera images. Doing VIO from depth is possible but VINS Fusion doesn't operate off of that type of image.

[catproof]

"For our purposes, lowering the clock speed isn't an option due to processing requirements and time calculations." what do you mean by this? All I am saying is you can reduce how fast the simulation environment runs. https://microsoft.github.io/AirSim/settings.html#clockspeed

[xxEoD2242]

Reducing the clock speed means that the environment takes longer to update. We do real time trajectory generation that requires feedback that is close to what we do in the real world. Great for developing VIO but not for implementing on real world systems.

[catproof]

@phil333 "Essentially the images should be captured in parallel, but this might require freezing the sim, capturing and then unfreezing (not an expert on this)."

you can freeze and unfreeze the simulation using simPause https://microsoft.github.io/AirSim/image_apis.html#getting-images-with-more-flexibility

I emailed the creators of the TartanAir dataset https://theairlab.org/tartanair-dataset/ and they said this is what they did to get around the left and right camera synchronization issue. They pause the simulation, get the camera views, then unpause the simulation, etc...

You could perhaps still do SLAM in real-time by quickly pausing and unpausing the simulation. It might require some modifications to your SLAM algorithm to handle this slight delay... but maybe not. I'll likely be exploring this avenue in the near/distant future and will be providing updates on progress.

[phil333]

@NickPerezCarletonUniversity thx for the input. I also assume this is what needs to be done with the current airsim-rosbridge to get this working. I currently dont have the time to look into this. For a SLAM system, the delay is not necessarily a problem. It is also possible to record the data and build a rosbag with the correct timestamp instead. As we are talking about VIO data, i would also still want the IMU data generated. Is this something that can be retrieved with the API?

[catproof]

@phil333 yes, you can definitely record IMU data! I'm not sure what the exact API call is.

[phil333]

@NickPerezCarletonUniversity
that sounds like a good approach then. It would be great to make an offline rosbag-bridge then. Is that something that would be useful to you? At this stage I am a bit on the fence to switch over to isaac-sim, which already has synchronized images but is lacking the imu support.

[catproof]

@phil333 yes an offline rosbag-bridge would be useful. if you get around to it before I do, do you mind sharing? I think Gazebo is a good simulator as well (https://www.youtube.com/watch?v=UtJFUeHXH6I&lc=UgzVgecehqw-QEwCXj94AaABAg.9Pa2S1Iossx9Pa2nEbmfCG). Not sure if it offers IMU support. you might still be able to simulate real-time SLAM while pausing and unpausing AirSim. I sent an email to the creators of AirSim (authors listed in the AirSim paper: https://arxiv.org/abs/1705.05065) politely enquiring about this underlying (and pretty big) issue. Their paper is misleading, and makes it seem as if AirSim is polished off and perfectly ready-to-go for real-time simulation. this is obviously not the case... they ignored my email. Probably don't want to admit to it, makes their paper look a bit like propaganda. LOL.

[phil333]

@NickPerezCarletonUniversity i might look into this after i finish my thesis, but not a priority right now. Gazebo is a good simulator and well integrated into ROS, but for computer vision tasks, it is just not comparable to real data. If that works for you, then you should avoid Airsim and stick to gazebo, no question. Airsim was designed for drones, which generally don't have a stereo camera setup. For such a configuration, the simulator is perfectly fine, but i agree, this issue is a major issue.

[xxEoD2242]

@phil333 @NickPerezCarletonUniversity

For stereo vision, if you are proficient in threading in C++, it is entirely possible to use ROS to generate stereo images as a pair. I would avoid the provided AirSim ROS wrapper, however. That system is not built with performance in mind, generally because it processes each camera frame as a separate image request. You can, however, use the C++ API provided to build your own ROS wrapper that has minimal features.

From that standpoint, you can then write 2 threads, one for IMU and one for the stereo images. From our work, I would highly recommend looking at synchronization policies to ensure that you have a fixed time difference between the IMU data and the images. Because of variable performance from AirSim this will destroy performance of a VIO algorithm, as the time differences shift constantly. You can use adaptive algorithms (which are in the sync policies in ROS) to mitigate some of this but everything has tradeoffs.

Sorry that this is really a "You have to do a bunch of work yourself" answer but that's what we have done to get our systems to work.

[catproof]

@xxEoD2242 can you share the code to your solution? would be really helpful. there are many people who would benefit from an improved ROS wrapper.

[hipforth]

@xuhao1 Which coordinate did you transform imu link to? NED to ENU or other? I see your body_T_cam in vins yaml, is corresponding to ENU. The linear vel from NED to ENU should be (x,y,z)->(x,-y,-z),but your code is (x,y,z)->(-x,y,-z). I feel really confused.

NED to ENU : I think it should be (x, y, z) -> (y, x, -z), why (x,y,z)->(x,-y,-z)?

[catproof]

@xxEoD2242 any progress on sharing your ROS wrapper code?

[xxEoD2242]

@NickPerezCarletonUniversity

I have been swamped with other projects. If you want to a swipe at it, check out my ROS wrapper:

AirSim ROS Wrapper

This assumes that you have the following in your settings.json file:

Drone Name is PX4

You have 1 camera called `front_center_custom` that is a DepthPerspective Camera (option 2).
You have 1 camera called 'front_center' that is a scene camera.

The way that this works is that it builds a thread for each type of image. It does not do this automatically! So, jf you wanted stereo images, you will need to modify the scene image to be two images. Then, on line 1496, you need to modify the AirSim request to be two scene images taken at once. I have had issues with this for VINS Fusion so you may even want to try two different threads, one for each image. Syncing those will be hard but there are algorithms out there that can do this.

I'll get around to refactoring this ROS wrapper to remove all of the other stuff and slim down state updates as well. Ask any questions that you may have about how to set this up.

[lnexenl]

I also made it work by collecting IMU data with ROS while using computer vision mode for camera images (This way you can get completely synchronized images).

Here is a procedure.

• Use airsim_ros_pkg to record IMU and ground truth odometry msgs to generate a rosbag.
• Set AirSim to computer vision mode.
• Capture images from the point which you can get from gt odometry in the rosbag, and write these into it as rosmsg.

Hope it will help someone :)

@kminoda Thanks! This definitely works well! I recorded a stereo camera rosbag, and run VINS-Fusion with it. Finally I get a perfect result.

And I recommend everyone who wants to get synchorized image pairs to use this method:

1. Use airsim_ros_pkgs to record vehicle groundtruth pose and IMU data in any mode you want.
2. Switch to ComputerVision mode, add cameras you want to record.
3. Read rosbag, get recorded vehicle pose from it. Use AirSim API to set vehicle pose, and use AirSim C++ API to get images and write images back to rosbag. (I found that Python API is much more slower than C++ API)

[hoangvietdo]

I also made it work by collecting IMU data with ROS while using computer vision mode for camera images (This way you can get completely synchronized images).
Here is a procedure.

• Use airsim_ros_pkg to record IMU and ground truth odometry msgs to generate a rosbag.
• Set AirSim to computer vision mode.
• Capture images from the point which you can get from gt odometry in the rosbag, and write these into it as rosmsg.

Hope it will help someone :)

@kminoda Thanks! This definitely works well! I recorded a stereo camera rosbag, and run VINS-Fusion with it. Finally I get a perfect result.

And I recommend everyone who wants to get synchorized image pairs to use this method:

1. Use airsim_ros_pkgs to record vehicle groundtruth pose and IMU data in any mode you want.
2. Switch to ComputerVision mode, add cameras you want to record.
3. Read rosbag, get recorded vehicle pose from it. Use AirSim API to set vehicle pose, and use AirSim C++ API to get images and write images back to rosbag. (I found that Python API is much more slower than C++ API)

Could you provide more detailed steps about this procedure?

[lnexenl]

I also made it work by collecting IMU data with ROS while using computer vision mode for camera images (This way you can get completely synchronized images).
Here is a procedure.

• Use airsim_ros_pkg to record IMU and ground truth odometry msgs to generate a rosbag.
• Set AirSim to computer vision mode.
• Capture images from the point which you can get from gt odometry in the rosbag, and write these into it as rosmsg.

Hope it will help someone :)

@kminoda Thanks! This definitely works well! I recorded a stereo camera rosbag, and run VINS-Fusion with it. Finally I get a perfect result.
And I recommend everyone who wants to get synchorized image pairs to use this method:

1. Use airsim_ros_pkgs to record vehicle groundtruth pose and IMU data in any mode you want.
2. Switch to ComputerVision mode, add cameras you want to record.
3. Read rosbag, get recorded vehicle pose from it. Use AirSim API to set vehicle pose, and use AirSim C++ API to get images and write images back to rosbag. (I found that Python API is much more slower than C++ API)

Could you provide more detailed steps about this procedure?

sorry for a late reply, I have written a blog post and maybe you can refer to that: Record Synchronized Image Pair with AirSim