Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve feedback on sensor "failure" #2934

Closed
AndreasAntener opened this issue Oct 2, 2015 · 13 comments · Fixed by #3183
Closed

Improve feedback on sensor "failure" #2934

AndreasAntener opened this issue Oct 2, 2015 · 13 comments · Fixed by #3183

Comments

@AndreasAntener
Copy link
Contributor

@LorenzMeier
We see the message SENSOR FAILSAFE! RETURN TO LAND IMMEDIATELY with latest master now and then (in a setup that didn't have any problems so far).
As far as I can see this could relate to either gyro, accel or mag. It should output more info on what went wrong exactly. At the moment this message doesn't help.

Is maybe more info logged somewhere?
@mhkabir
Copy link
Member

mhkabir commented Oct 2, 2015

@AndreasAntener attitude_estimator_q status will give you what you need.

@LorenzMeier
Copy link
Member

Only to the console right now. The voter class can output a status though which we could add to the telemetry output.

@AndreasAntener
Copy link
Contributor Author

I see the print statements. How can I enable this?

@AndreasAntener
Copy link
Contributor Author

Never mind, it's in the status print

@AndreasAntener
Copy link
Contributor Author

It only pops up randomly but I just caught it with the status, seems to be mag:

INFO  gyro status:
INFO  validator: best: 0, prev best: 0, failsafe: NO (# 0)
INFO  sensor #0:

INFO    val:  -0.0086, lp:  -0.0085 mean dev:   0.0000 RMS:   0.0004

INFO    val:   0.0141, lp:   0.0141 mean dev:  -0.0000 RMS:   0.0004

INFO    val:  -0.0056, lp:  -0.0056 mean dev:   0.0000 RMS:   0.0003

INFO  sensor #1:

INFO    val:   0.0110, lp:   0.0118 mean dev:   0.0000 RMS:   0.0021

INFO    val:  -0.0110, lp:  -0.0108 mean dev:   0.0000 RMS:   0.0018

INFO    val:   0.0132, lp:   0.0134 mean dev:  -0.0000 RMS:   0.0015

INFO  sensor #2:

INFO    no data

INFO  accel status:
INFO  validator: best: 0, prev best: 0, failsafe: NO (# 0)
INFO  sensor #0:

INFO    val:   9.5105, lp:   9.5048 mean dev:   0.0000 RMS:   0.0066

INFO    val:  -0.2651, lp:  -0.2673 mean dev:   0.0000 RMS:   0.0072

INFO    val:   2.0955, lp:   2.0879 mean dev:   0.0000 RMS:   0.0107

INFO  sensor #1:

INFO    val:   9.5651, lp:   9.5471 mean dev:   0.0000 RMS:   0.0213

INFO    val:  -0.1404, lp:  -0.1540 mean dev:   0.0000 RMS:   0.0185

INFO    val:   2.5017, lp:   2.5063 mean dev:  -0.0000 RMS:   0.0268

INFO  sensor #2:

INFO    no data

INFO  mag status:
INFO  validator: best: 0, prev best: 1, failsafe: YES (# 2)
INFO  sensor #0:

INFO    val:  -0.3491, lp:  -0.3483 mean dev:   0.0000 RMS:   0.0018

INFO    val:   0.0533, lp:   0.0527 mean dev:  -0.0000 RMS:   0.0018

INFO    val:  -0.1703, lp:  -0.1700 mean dev:   0.0000 RMS:   0.0013

INFO  sensor #1:

INFO    val:  -0.4060, lp:  -0.4062 mean dev:  -0.0000 RMS:   0.0013

INFO    val:   0.0675, lp:   0.0670 mean dev:  -0.0000 RMS:   0.0023

INFO    val:  -0.1250, lp:  -0.1253 mean dev:  -0.0000 RMS:   0.0019

INFO  sensor #2:

INFO    no data

Is there any info in these values towards the problem? Is the detection too sensitive? Does the ID tell me if it's internal or external?

This btw happened now on boot via USB inside. We had the same (I assume) yesterday outside via battery.

@LorenzMeier
Copy link
Member

It currently uses the absolute error count to switch, rather than an interval error count. We should probably change this to an error density instead to allow for sporadic I2C bus errors, which happen in practice. The density would be errors divided by total reads over a sliding window. I'll have a go at this later today.

@LorenzMeier
Copy link
Member

I've changed this to error density on master. This should still disable a failing mag (please test!) but should accept a 1% error count under normal operating conditions.

@AndreasAntener
Copy link
Contributor Author

We'll probably fly the same setup again this week and see if it still switches or not. I can do a mag disconnect test later today.

@AndreasAntener
Copy link
Contributor Author

"best" switches from 0 to 1 if I pull the I2C plug. And it switched back to 0 after I reconnected (reported failsafe # 2). On the second try it didn't switch back to 0 anymore (reported failsafe # 3). This is the supposed behaviour?

@AndreasAntener
Copy link
Contributor Author

The failsafe counter increases on disconnect and reconnect.

@LorenzMeier
Copy link
Member

Most likely yes. Does it switch back after some time or not at all? There is a hard limit for the error count (10'000 I2C errors) at which point it won't attempt to switch back.

We're not trying to support hot-plug here!

@AndreasAntener
Copy link
Contributor Author

;) I was actually wondering why it switched back anyway. Once a sensor is in error state I probably don't want to use it anymore.

It switched back multiple times now before it stayed at 1. Not sure what was different before. I guess I somehow produced less i2c errors this time. Once it stays at 1 it doesn't seem to go back anymore.

@AndreasAntener
Copy link
Contributor Author

Wasn't there a command to check i2c errors? (maybe I'm imagining things..). I should probably check for those to see how many our setup produces..

@mhkabir mhkabir mentioned this issue Nov 14, 2015
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improved sensor failsafe reporting mhkabir/Firmware
3 participants
@LorenzMeier @mhkabir @AndreasAntener