IMU/Magnetometer does not always start

[anuppari]

Using the pixhawk mini, the imu and magnetometer seem to not start up consistently. Sometimes (using the wireless telemetry or USB with QGC) I can see the roll/pitch and heading indicators don't update. Also, IMU messages are not published by mavros. Stabilized mode also doesn't work.
It's usually fixed by just disconnecting and reconnecting the battery or USB cable a few times.

This is on Release 1.6 and using the latest (may 29) daily build of QGC, but I've been having this same problem for a few weeks with 1.6 RC3.

[anuppari]

I've also had this issue on multiple pixhawk mini's, so I don't think it's a hardware issue.

[anuppari]

I've tried uploading a log but I keep getting error 400 failed to parse errors

[dagar]

Are you using the new logger?
Set SYS_LOGGER 1 and SDLOG_MODE 1 to log immediately at boot.

[anuppari]

They were both already set to using the new logger and log from boot to shutdown. Is there anything I should be doing before pulling power to make sure the log file doesn't get corrupted if it is continuously logging until shutdown?

[LorenzMeier]

@anuppari If you arm and disarm the disarming will stop the log and prevent it getting corrupted.

Also make sure to use a QGC daily build - otherwise the file ending of the log file will be incorrect.

[LorenzMeier]

Also, I believe I fixed this yesterday - please re-test with 1.6.1

[anuppari]

I tested this with 1.6.1 and still have the same issue with the IMU and mag not starting up.

[anuppari]

Can we reopen this?

[dagar]

Yes, but we'll need more information. @d3z0rian have you experienced this?

[anuppari]

Here's a log showing the issue. Note the roll/pitch estimates do not show anything, and the raw mag output is zero.

What additional info do you need?

[dagar]

Can you try connecting the console via QGroundControl? Then you can try manually starting and stopping each driver. https://github.com/PX4/Firmware/blob/master/ROMFS/px4fmu_common/init.d/rc.sensors#L94

We need to figure out why these drivers are failing to start sometimes, but unfortunately I don't see a lot to go on from the existing logs.

[santiago3dr]

ran bench test on a pixhawk mini with 1.6.3-stable
http://logs.px4.io/plot_app?log=3d0a07f5-14e9-4ae3-be42-c243f5df3744

no gps/external mag connected
accel,gyro and internal mag (MPU9250) looks to be working just fine

[anuppari]

We get the issue intermittently, so usually after a power cycle, it works fine. We are also using FMUv3 with the LPE estimator, in case that's relevant, though it seems to be a sensor issue.

Were there any relevant changes from 1.6.1 to 1.6.3? It doesn't seem like a one-off hardware issue as we have the exact same problem on 5 separate pixmini's

[santiago3dr]

re-tested with fmu-v3 and lpe estimator; same results... will keep testing
http://logs.px4.io/plot_app?log=0d4787c1-57b3-4e4a-9074-2142fd02594a

[gbrunso]

anuppari and I tried resetting, starting, and stopping the sensor driver (mpu9250) from the mavlink console in QGC.
If the sensors start up correctly, the sensors and the console commands work appropriately. If the sensors do not start up, trying to reset or turn the drivers off and on does not seem to have any effect.
I updated to 1.6.3 and we are still having the same problem.

[dagar]

@anuppari which binary do you flash? I just noticed you're using LPE, which isn't in the default fmu-v2 binary.

First try changing SYS_MC_EST_GROUP to 2.

Can you try the latest flashing with the latest QGC daily build that only recently added support for flashing fmu-v3s?

[anuppari]

We are flashing v3, which based on #7324, can be used on the pixhawk mini to enable the LPE.

[anuppari]

And yes we've been using a QGC daily build from a few weeks ago

[dagar]

Have you been manually flashing it?

In your log I see accel and gryo, but no mag. The attitude indicator should still be updating, unless the estimator isn't starting.
Are you using an external mag? Is it always connected?

[anuppari]

Yes we are using the custom firmware option in QGC to flash the firmware.

The attitude and heading indicator both don't update in QGC when this issue happens. We are not using an external mag, just the pixhawk mini, without GPS/compass.

Is there a way to check if the attitude estimator is starting up correctly? From the log it appears the mag sensor is not starting up, so I think the issue is upstream of the estimator.

[anuppari]

A similar issue seems to have occurred previously. Not sure if that fix is related.

[dagar]

Ok, so it sounds like the mpu9250 is periodically failing to start for you, but the attitude indicator is a mystery.

Can you use the mavlink console in QGC when it happens to see what's running? It's in the log download tab.
Run ps, and ls /dev. Try it with and without the attitude indicator updating.

With access to the system console this would be easier because we could just see what's happening during boot.

[anuppari]

Working on that now.

Regarding the attitude indicator, isn't the IMU and mag all contained in one mavlink message? Does the attitude estimator wait until it receives at least one mag uOrb message before sending out an attitude estimate mavlink message?

Is there a way to access the system console during boot?

[gbrunso]

Here is the console output for ps and ls /dev when the issue is happening.

NuttShell (NSH)
nsh> ps
  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK  COMMAND
    0   0 FIFO     Kthread --- Ready              00000000 Idle Task
    1 192 FIFO     Kthread --- Waiting  Signal    00000000 hpwork
    2  50 FIFO     Kthread --- Waiting  Signal    00000000 lpwork
    3 100 FIFO     Task    --- Waiting  Semaphore 00000000 init
  292 250 FIFO     Task    --- Waiting  Semaphore 00000000 lp_estimator
  103 220 FIFO     Task    --- Waiting  Semaphore 00000000 gps 1
  296 250 FIFO     Task    --- Waiting  Semaphore 00000000 mc_att_control
  329 175 FIFO     pthread --- Waiting  Semaphore 00000000 mavlink_rcv_if2 0x200117a0
  298 250 FIFO     Task    --- Waiting  Semaphore 00000000 mc_pos_control
  107  90 FIFO     Task    --- Waiting  Semaphore 00000000 dataman
  172 100 FIFO     Task    --- Waiting  Signal    00000000 mavlink_if0 mavlink start -r 1200
  173 175 FIFO     pthread --- Waiting  Semaphore 00000000 mavlink_rcv_if0 0x100072e0
  334  50 FIFO     pthread --- Waiting  Semaphore 00000000 commander_low_prio 0x0
  335 100 FIFO     Task    --- Running            00000000 mavlink_shell
  240 250 FIFO     Task    --- Waiting  Semaphore 00000000 logger -b 16 -t -f logger start
  241  60 FIFO     pthread --- Waiting  Semaphore 00000000 log_writer_file 0x20010520
  306 105 FIFO     Task    --- Waiting  Semaphore 00000000 navigator
  149 250 FIFO     Task    --- Waiting  Semaphore 00000000 sensors
  151 140 FIFO     Task    --- Waiting  Signal    00000000 commander commander start
  184 100 FIFO     Task    --- Waiting  Signal    00000000 mavlink_if1 mavlink start -d /dev/ttyS2 -b 57600 -m osd -r 1000
  185 175 FIFO     pthread --- Waiting  Semaphore 00000000 mavlink_rcv_if1 0x2000dbb0
  284 250 FIFO             --- Waiting  Semaphore 00000000 attitude_estimator_q
  158 240 FIFO     Task    --- Waiting  Semaphore 00000000 px4io
  223 100 FIFO     Task    --- Waiting  Signal    00000000 mavlink_if2 mavlink start -r 800000 -d /dev/ttyACM0 -m config -x
nsh> ls /dev
/dev:
 accel0
 adc0
 baro0
 console
 gyro0
 led0
 mag0
 mmcsd0
 mpu9250_accel
 mpu9250_gyro
 mpu9250_mag
 ms5611_spi_int
 mtdblock0
 mtdblock1
 null
 pipe0
 pipe1
 pwm_output0
 pwm_output1
 px4fmu
 px4io
 ram0
 rgbled0
 tone_alarm0
 ttyACM0
 ttyS0
 ttyS1
 ttyS2
 ttyS3
 ttyS4
 ttyS5
 ttyS6
nsh> 

[gbrunso]

This should be a little easier to read.

NuttShell (NSH)
nsh> ps
PID PRI POLICY TYPE NPX STATE EVENT SIGMASK COMMAND
0 0 FIFO Kthread --- Ready 00000000 Idle Task
1 192 FIFO Kthread --- Waiting Signal 00000000 hpwork
2 50 FIFO Kthread --- Waiting Signal 00000000 lpwork
3 100 FIFO Task --- Waiting Semaphore 00000000 init
292 250 FIFO Task --- Waiting Semaphore 00000000 lp_estimator
103 220 FIFO Task --- Waiting Semaphore 00000000 gps 1
296 250 FIFO Task --- Waiting Semaphore 00000000 mc_att_control
329 175 FIFO pthread --- Waiting Semaphore 00000000 mavlink_rcv_if2 0x200117a0
298 250 FIFO Task --- Waiting Semaphore 00000000 mc_pos_control
107 90 FIFO Task --- Waiting Semaphore 00000000 dataman
172 100 FIFO Task --- Waiting Signal 00000000 mavlink_if0 mavlink start -r 1200
173 175 FIFO pthread --- Waiting Semaphore 00000000 mavlink_rcv_if0 0x100072e0
334 50 FIFO pthread --- Waiting Semaphore 00000000 commander_low_prio 0x0
335 100 FIFO Task --- Running 00000000 mavlink_shell
240 250 FIFO Task --- Waiting Semaphore 00000000 logger -b 16 -t -f logger start
241 60 FIFO pthread --- Waiting Semaphore 00000000 log_writer_file 0x20010520
306 105 FIFO Task --- Waiting Semaphore 00000000 navigator
149 250 FIFO Task --- Waiting Semaphore 00000000 sensors
151 140 FIFO Task --- Waiting Signal 00000000 commander commander start
184 100 FIFO Task --- Waiting Signal 00000000 mavlink_if1 mavlink start -d /dev/ttyS2 -b 57600 -m osd -r 1000
185 175 FIFO pthread --- Waiting Semaphore 00000000 mavlink_rcv_if1 0x2000dbb0
284 250 FIFO --- Waiting Semaphore 00000000 attitude_estimator_q
158 240 FIFO Task --- Waiting Semaphore 00000000 px4io
223 100 FIFO Task --- Waiting Signal 00000000 mavlink_if2 mavlink start -r 800000 -d /dev/ttyACM0 -m config -x
nsh> ls /dev
/dev:
accel0
adc0
baro0
console
gyro0
led0
mag0
mmcsd0
mpu9250_accel
mpu9250_gyro
mpu9250_mag
ms5611_spi_int
mtdblock0
mtdblock1
null
pipe0
pipe1
pwm_output0
pwm_output1
px4fmu
px4io
ram0
rgbled0
tone_alarm0
ttyACM0
ttyS0
ttyS1
ttyS2
ttyS3
ttyS4
ttyS5
ttyS6
nsh>

[davids5]

@gbrunso

With that PR, using a v3 build (the only one that should be used on a mini) you will need to use the external GPS that has the mag on it.

Is it connected?

[gbrunso]

No, we are not using the external GPS.

[davids5]

@gbrunso

We can not use the mpu9250 on the PixhawkMini because there is a HW errata on the PixhawkMini that prevents it from being reliable. This is not a problem because the ICM20608 is now supported along with the Mag on the included GPS.

Can you add the GPS to your configuration?

[gbrunso]

Yes, but we aren't using the compass or GPS. We use mocap for position and heading. Is possible to use the ICM20608 accelerometer and gyro without having to add an external compass?

[davids5]

https://github.com/PX4/Firmware/files/1194995/pr-bv3-c.txt - indicates the
icm20608_accel
icm20608_gyro
are started.

@r0gelion - are you able to reproduce #7327 (comment) ?

@dagar is there a parameter to override the need for the compass?

[r0gelion]

@davids5 Only if the GPS (which has the external mag) is disconnected. But I would expect that since the internal magnetometer is disabled.

[BRNKR]

Same issue here with latest master on v2 and v3 boards. It happens when you have ekf2 aid mask only with external vision and yaw fusion but without gps. It is not possible to fly the Copter in stabilize until you fuse the vision. The attitude estimator doesn't boot. Without gps module but only with gps fusion you can control pitch and roll with RC

[BRNKR]

Also on 1.6.5 stable. Took a lot of time to get rid of the issue :(

[davids5]

Same issue here with latest master on v2 and v3 boards.

Are seeing mpu9250 mag failures on an V2 board or just noting that without a GPS you do not have a MAG?

On a pixhawk mini you must uses the a px4fmu-v3_default build to not use the defective mpu9250 mag and have a GPS to get the external MAG.

[PX4BuildBot]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 30 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[giuanbalina]

Same issue here with 1.7.3, is there a fix?

[gbrunso]

I have not had any success with new firmware (we use QAV250 frames, pixhawk minis). We use 1.6.5 with mixed success. Currently we run a script that reboots the pixhawk repeatedly until we receive non-garbage sensor data. Unfortunately it is (to my knowledge) impossible to ignore the magnetometer. So if there are any magnetometer issues the entire sensor array will fail. This is particularly annoying when flying indoors, where magnetometer is useless.

[davids5]

Hi this is a hardware issue there's a pin that was not routed it was left open and is causing a problem with the mpu9250. I believe this was fixed on later revisions of the board

[giuanbalina]

Thanks, I bought the board two weeks ago, but probably it is an old revision. Do you know if there is a small supported external compass without GPS (No space left for it) I can use?

[pietrodn]

Also, is there a reliable way to distinguish between the flawed and the good hardware?

[hamishwillee]

@davids5 See above #7327 (comment) - "is there any way to detect a bad version" (or someone you recommend we ask?). This is for updating the docs: PX4/PX4-user_guide#171

[davids5]

@hamishwillee - I would imagine from being power off for a long time, one can try start the mpu9250 on the command line. If the board does not have the correct wiring, the driver will fail.

[davids5]

@ryanjAA - please open a new Issue, if there is a separate issue. This was very specific to the Pixhawk-Mini and a HW issue and we would not want to confuse this issue any further.

[ryanjAA]

Thanks. Already done. See 9151. That’s why I didn’t ask for this to be reopened but the issue is near identical in form if anyone else comes across this or another board.

[waynehardiman]

Hi,
I purchased a 3DR Pixhawk Mini (inc M8N GPS module) at the end of April and have finally got round to integrating it into a quadcopter build. I have spent a week messing around with this and reviewing the posts and still can not get either the internal mag or the external mag to work. The GPS functions correctly. I have re-flashed several times with the latest FMU-V3 firmware and have also tried FMU-V2 1.7.a Stable release. I have also tried a new Radiolink TS100 GPS module which includes the HMC5883 compass. Same result, the GPS works but the mag does not. Is there a solution to this that does not involve scrapping the Pixhawk Mini and replacing it with a Pixhawk 4.
Any assistance would be much appreciated.

[davids5]

@waynehardiman

The HMC5883 does work in a mini setup. The mag on the 9250 (onboard) is disabled due a HW problem.

I would check the wiring and keep in mind have a bad board.

You can use QGC to get the debug console and issue commands there to see what results you get are.

On the SD card you can place /fs/microsd/etc/rc.txt See https://dev.px4.io/en/advanced/system_startup.html.

In that file just start only uOrb and mavlink on the usb ttyACM to keep the debug console available and you can run the hmc5883 driver commands manually on the console and see if that give you a clue.