Advanced Time Synchronisation

[mhkabir]

This brings in an advanced version of the current mavlink time synchronisation algorithm. It performs much better at lower synchronisation rates (< 10Hz) and on high jitter serial links. It is also much more robust to outliers and provides an accurate (sub-millisecond) clock offset estimate after only 30 seconds of synchronisation @ 10Hz rate. RMS error after a minute of synchronisation is under 50 microseconds.

The filtering algorithm itself is based on a gain-scheduled double exponential filter. The main improvement over the current lowpass filter is that the derivative of the estimate is also estimated in order to produce an estimate without steady state lag. Further reading : https://en.wikipedia.org/wiki/Exponential_smoothing#Double_exponential_smoothing

Performance :

I have tested this extensively on "typical" onboard setups, and should be good to merge. It would still be good to get some community testing with a bunch of different setups. I will also open a pull request to Flight Review so that we can view timesync performance for logs which have it. This will allow us to later improve performance based on crowdsourced data.

Note for testers (should also go into PX4 Devguide) :

1. To test, just make sure mavros is running on the other end of a companion computer UART link, and start the autopilot. You should set SDLOG_MODE to log from boot. After collecting data for ~10mins, please send upload logs to Flight Review and post a link here.
2. You need to disable NTP or similar network time synchronisation on the companion computer side for good performance. The companion timestamps will drift otherwise. For example, on Ubuntu 16.04 you can use : sudo timedatectl set-ntp no
3. Your link round-trip-time must be under 10ms for good performance. This should help : libmavconn : enable low-latency mode on Linux mavlink/mavros#975.

[RomanBapst]

Looks pretty good! Is the gain scheduling something you tried and found to work well or is there a reference somewhere?

[mhkabir]

@RomanBapst The gain scheduling is something which worked well during testing.

I found that in the initial phase of the synchronisation (without enough data to build up a good hypothesis), the derivative estimate would diverge if the gains were too tight. In the later stages, the loose gains which worked well for the initial convergence would cause the estimate to oscillate.

[mhkabir]

@Changliu52 Could you please test this with your VIO setup? I want to make sure that everything still behaves correctly.

[bkueng]

Can you document these a bit? I.e. in which cases you need to adjust them and how.

[mhkabir]

Will do, thanks for pointing it out!

[Changliu52]

@mhkabir Yeah. Works pretty nice. I had two test in this log here: https://review.px4.io/plot_app?log=07e16bc5-7794-40e0-a716-690c2f249185. I used software sync in the first test and hardware sync (images were tagged at trigger time, and no exposure compensation) in the second test. Both tests start with slower motion, then faster, and then added more rotations.
Here is the quick image

The new software sync gives lest drift in slower continuous motion since the clock offsets are completely eliminated, but it seems that the hardware sync still has slightly better robustness in fast motions. Also sometimes software sync gives out of order imu measurements if I start the SLAM too quickly. I guess it was because of the small oscillation in the beginning in your curve.

Let me know your thoughts.

Chang

[mhkabir]

@Changliu52 Looks good! Timesync performance is nominal in your logs inspite of some patches of bad latency. Maybe from killing and restarting mavros? You might want to check where that comes from.

Out of order imu measurements if I start the SLAM too quickly.

Yes this is because of the convergence of the sync takes some time, and the derivative estimate is not good until then. I will open a PR to mavros to prevent using the synchronised timestamp until convergence is complete.

TODOs based on your logs :

• Potentially reduce the convergence window to speed things up a bit.
• Timesync should reset when receiving a sample after a long time, like in your logs.
• Mavros needs a PR with this updated synchronisation method.

[Changliu52]

@mhkabir

inspite of some patches of bad latency. Maybe from killing and restarting mavros?

Yes I did relaunch everything including the mavros. The glitch happened at the time when killing everything. I did experience any issue caused by that.

Thank you for the very nice improvement.

Chang

[mhkabir]

@bkueng @dagar Can you review again? We should be generally good to merge.

[bkueng]

Thanks!