EKF errors with Fixed Wing

[Kjkinney]

This was tested on a slightly modified version of PX4 Master

After letting the vehicle sit for an extended period of time we are experiencing two errors "PREFLIGHT FAIL: EKF INTERNAL CHECKS" and "PREFLIGHT FAIL: HIGH IMU ACCEL BIAS". We recalibrated the accelerometers but still experienced these issues. If the plane is armed moderately quickly after booting up and there are no errors thrown, and the aircraft flies normally.

Here are some log examples of the preflight fails on ground:

https://review.px4.io/plot_app?log=a0fb96a2-de2f-4e2d-a4f6-0f930ea53248

https://review.px4.io/plot_app?log=42eda972-2802-4d1e-b236-bed93244427c

We also had a case where the aircraft threw a "PREFLIGHT FAIL: HIGH IMU ACCEL BIAS" error but then still allowed us to arm. On takeoff the aircraft flew erratically and had to be taken over with manual control. Log is posted below.

https://review.px4.io/plot_app?log=e29821e7-9539-46ec-b563-79be1b655cf4

[dagar]

EKF INTERNAL CHECKS refers to a set of additional checks in the EKF frontend for horizontal and vertical velocity innovations. Yaw innovation magnitude is also included, which is likely what you're hitting.

I'll need to take a closer look to see why you're getting the "HIGH IMU ACCEL BIAS" error.

[dagar]

Odd, here's the EKF accel bias from https://review.px4.io/plot_app?log=a0fb96a2-de2f-4e2d-a4f6-0f930ea53248

[Kjkinney]

@dagar do you want me to post one of our good flights for comparison?

[dagar]

Yes, that might be helpful.

Look at the accel magnitude change with the FW_ARSP_MODE parameter change. The system wasn't moving. I'll do some bench testing/debugging.

[dagar]

Gyro as well. This doesn't correspond with any sensor calibration.

[dagar]

When did sensor recalibration happen relative to the above logs?

[Kjkinney]

The first two logs are based off calibrations done on the 10th of may which was our first flight test on the build. The last log was a flight on may 10th which was using calibrations from earlier that day in a lab environment.

After that last log, we recalibrated everything in the field and have yet to recalibrate since then.

[Kjkinney]

https://review.px4.io/plot_app?log=7f3d1365-11eb-485f-850b-b8824882e4ed

Here is another bad log. After digging through some more logs I'm noticing that a common theme is that if parameters are changed then the ekf freaks out for a bit.

[ryanjAA]

We are seeing the same PREFLIGHT FAIL: HIGH IMU ACCEL BIAS issue on 1.8 beta. Need to arm quickly and flies fine otherwise PREFLIGHT FAIL: HIGH IMU ACCEL BIAS shows up.

[Kjkinney]

We figured out how to repeat this error on multiple different aircraft running a slightly modified version of a recent master build.

Steps to repeat error

• Boot up vehicle

• Hold the vehicle directly nose down for greater than 10 seconds with vehicle running. From what I can tell you can do this at any point while the vehicle is disarmed and it will create this error.

• Place level and attempt to arm. Should trigger the "PREFLIGHT FAIL: HIGH IMU ACCEL BIAS" error.

Here are a bunch of logs from two different vehicles repeating the error.

https://review.px4.io/plot_app?log=4ca0d516-f9a7-4eed-9ea2-4f7a16d51ce1
https://review.px4.io/plot_app?log=3987ea04-d32a-4058-8739-a0b25a90f823
https://review.px4.io/plot_app?log=38587fee-ef9c-4a39-9a83-11186357980f

https://review.px4.io/plot_app?log=47c77c7d-8dcf-4c67-b52c-280b30e77b14
https://review.px4.io/plot_app?log=0c941809-fb0d-47b0-a2da-af65941d38c3
https://review.px4.io/plot_app?log=2436da7d-c47a-4da1-b7bf-09910b736723

[priseborough]

@Kjkinney You need to modify your autopilot installation so that the propeller slipstream does not produce 25 metre changes in baro height.

[dagar]

I'm guessing this was reopened because of my comment #9553.

I haven't been able to reproduce this on the bench, but there appears to be some edge case where calibration parameters aren't actually applied within drivers. PR #9553 adds some simple command line debugging helpers should we ever stumble across another suspected situation.

See comments #9474 (comment) and #9474 (comment) above.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing as stale.