Can't correctly mount SD card on a PX4 (V1) setup

[sjwilks]

The current master (via QGC but also locally built) on a V1 setup will show the microSD card as mounted but no files are visible. Master version will follow but it was flashed mid afternoon CET.

Stable works fine. The issue does not effect Pixhawks (V2).

[LorenzMeier]

Thanks for the report! Could you revert this diff and let us know if that changes anything?

@@ -417,7 +417,7 @@ CONFIG_PREALLOC_TIMERS=50
 # Stack and heap information
 #
 CONFIG_IDLETHREAD_STACKSIZE=3500
-CONFIG_USERMAIN_STACKSIZE=3000
+CONFIG_USERMAIN_STACKSIZE=2600
 CONFIG_PTHREAD_STACK_MIN=512
 CONFIG_PTHREAD_STACK_DEFAULT=2048

@@ -719,11 +719,11 @@ CONFIG_SCHED_WORKQUEUE=y
 CONFIG_SCHED_HPWORK=y
 CONFIG_SCHED_WORKPRIORITY=192
 CONFIG_SCHED_WORKPERIOD=5000
-CONFIG_SCHED_WORKSTACKSIZE=2048
+CONFIG_SCHED_WORKSTACKSIZE=1800
 CONFIG_SCHED_LPWORK=y
 CONFIG_SCHED_LPWORKPRIORITY=50
 CONFIG_SCHED_LPWORKPERIOD=50000
-CONFIG_SCHED_LPWORKSTACKSIZE=2048
+CONFIG_SCHED_LPWORKSTACKSIZE=1800
 # CONFIG_LIB_KBDCODEC is not set
 # CONFIG_LIB_SLCDCODEC is not set

[LorenzMeier]

Other than that a GIT bisect between master and stable would be great. I tried to linearize our history in the last weeks, so it should be a bit easier than previously.

[LorenzMeier]

@sjwilks Since you fixed the landing detector I presume you somehow were able to boot - what is the status of this issue?

[sjwilks]

This isn't fixed yet. I can't sit down for very long at one time at the moment but what I think is happening is there seems to be an issue with flashing updates to the PX4IO somehow. At one point I noticed the firmware update tone started and was overridden by something else. Now I can reproduce the failure by flashing a local vs continuous build one and microsd access isn't working until I force an IO flash (power on with safety button held down).

Of course this might just be introducing a delay and isn't directly related.

Still looking when I get free blocks of time. Up next, hook up the debug cable.

[LorenzMeier]

@davids5 Sounds like we have the timing related issue on the IO upgrade now also on master, not just nuttx_bringup.

@sjwilks I suspect a timing fail because the boot process is now faster, as we made the stack checking a debug option. Inspecting and fixing this is lined up.

[LorenzMeier]

@sjwilks When you are unable to run ls, what does free tell you? I expect some sort of secondary dependency due to not enough memory or similar.

[sjwilks]

Memory is fine (32k free). Let me check again with the land_detector fix branch vs current master via QGC to make sure the missing land_detector isn't having a strange effect...even though it's started much later than the mounting of the sd card.

[sjwilks]

Nope. So one of two things happen: either I can't cd to the microsd directory or I can and it appears to be empty.

Timing: can I add a delay somewhere to verify this?

[davids5]

I just built and installed master on px4fmu-v2 and the pio FW upgrade works from boot with pio running or boot with pio running the bootloader. Is this only on v1?

[sjwilks]

Yep. V2 is fine. It's only V1. I probably should try with a second V1 just
to be sure.

On Sat, 24 Jan 2015 17:24 David Sidrane [email protected] wrote:

I just built and installed master on px4fmu-v2 and the pio FW upgrade
works from boot with pio running or boot with pio running the bootloader.
Is this only on v1?

—
Reply to this email directly or view it on GitHub
#1691 (comment).

[davids5]

Is one of the symptoms that the fmu try to reprogram the pio on every reset or power on?

[sjwilks]

I don't think it is trying to everytime. The startup sequence below doesn't try to program it. Interesting is through that after mounting the microsd there are some ops (ie: dataman) that appear to succeed but once booting is complete the microsd isn't available.

[boot] Successfully initialized SPI port 1
[boot] Enabling IN12/13 instead of SPI2
[boot] Initializing SPI port 3
[boot] Successfully initialized SPI port 3
[boot] Successfully bound SPI port 3 to the MMCSD driver
[i] Looking for microSD...
[i] microSD mounted: /fs/microsd
[i] Init script not found: /fs/microsd/etc/rc.txt
[i] AUTOSTART mode
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
nshterm [6:70]
[uorb] ready
param: selected parameter default file /fs/mtd_params
[param] Loaded: /fs/mtd_params
rgbled: init failed
nsh: blinkm: command not found
nsh: pca8574: command not found
 1:   SYS_USE_IO: match
 4010: + SYS_AUTOSTART: match
[i] Config not found: /fs/microsd/etc/config.txt
[PX4IO_i2c] on I2C bus 2 at 0x1a
[PX4IO::checkcrc] CRCs match
dataman: Power on restart
dataman: Initialized, data manager file '/fs/microsd/dataman' size is 101504 bytes
commander: No RGB LED found at /dev/rgbled0
commander: loaded mission state: dataman_id=0, count=0, current=0
[PX4IO_i2c] on I2C bus 2 at 0x1a
[px4io] default PWM output device
[i] PX4IO started
ver hwcmp match: PX4FMU_V1
[i] Set PX4IO update rate limit: 200 Hz 
mavlink_if0: mode: 0, data rate: 1000 B/s on /dev/ttyS1 @ 57600B
param: Error: Parameter UAVCAN_ENABLE not found.
[MS5611_I2C] on I2C bus 2 at 0x76
<adc> init done
mpu6000: External SPI not available
[MPU6000] on SPI bus 1 at 3
l3gd20: External SPI not available
[L3GD20] on SPI bus 1 at 1
 1: + SENS_EXT_MAG: match
[HMC5883_I2C] on I2C bus 3 at 0x1e
meas_airspeed: no MS4525 airspeed sensor connected
nsh: ets_airspeed: command not found
nsh: ets_airspeed: command not found
nsh: px4flow: command not found
nsh: ll40ls: command not found
sensors_task: mag cal status changed
preflight_check [92:100]
ver hwcmp match: PX4FMU_V1
Start sdlog2 at 50Hz
[i] Start GPS
[i] MULTICOPTER
+ MAV_TYPE: curr: 2 -> new: 2
sdlog2: error writing file
sdlog2: log buffer size: 3072 bytes
[px4io] mixer sent
[i] Mixer: /etc/mixers/quad_x.main.mix
.
navigator: Initializing..
navigator: Could not clear geofence
[i] No addons script: /fs/microsd/etc/extras.txt
navigator: offboard mission updated: dataman_id=0, count=0, current_seq=0

NuttShell (NSH)
nsh> navigator: ERROR: invalid mission state
navigator: ERROR: can't save mission state
gps: ubx msg 0x0601 NAK
gps: not using NAV-PVT
gps: VER hash 0xe0cc2322
gps: VER hw  "  00040007"
gps: VER sw  "                  7.03 (45970)"
gps: VER ext "                   7.03 (45969)"
gps: module found: UBX
position_estimator_inav: baro offs: 290

nsh> ls /fs/microsd
/fs/microsd:
nsh: ls: no such directory: /fs/microsd

[sjwilks]

I forced a flash and this time it didn't help:

nsh> cd /fs
nsh> cd microsd
nsh: cd: chdir failed: No such file or directory
nsh> ls
/fs:
 microsd/
 mtd_params
 mtd_waypoints

I'll have more time tomorrow to look more systematically at this. I'll also do a git bisect.

[sjwilks]

OK, I found the problem but it took hours as there was some nasty behavior when flashing different versions as a good version would suddenly also fail if a bad version was flashed before returning to it. Seemed to be related to the flashed px4io firmware at the time. In the end carefully manually bisecting prioritising V1 changes narrowed it down to:

FMUv1: Disable stack checking (e62c8d7)

@LorenzMeier should this simply be re-enabled or were there followup changes that depend on this? Or an alternative approach?

[LorenzMeier]

No, it should not, as we have no resources to keep it enabled. There are two potential explanations: a) The stack checking wasn't fully disabled. But I did check in the completely compile output of all configs that it in fact is. And b) there was a race condition present for the whole time, which is now exposed. @davids5 Do you have FMUv1 hardware? If not, could you ask 3DR to ship you some?
While the support for those boards and SPI-SD is not critical, we need to be sure this is not a general issue.

@sjwilks Could you try nuttx_bringup and see if that works better for you? The new version might have fixed what you're currently running into.

[sjwilks]

With CONFIG_ARMV7M_STACKCHECK=n:
total used free largest
Mem: 173936 155280 18656 15072

With CONFIG_ARMV7M_STACKCHECK=y:
total used free largest
Mem: 173952 156048 17904 10448

After arming free space for the latter drops to 14k.

Using nuttx_bringup and then merging current master works with the stack check disabled.

My proposal would be re-enabling the stack check in master now until NuttX 7 is ready as master is currently broken for FMUV1 hardware.

[davids5]

@LorenzMeier I have requested V1 HW.

[davids5]

@sjwilks Did the memcpy fix that is currently on master and nuttx_bringup fix this issue on V1 HW (with stack checking off)?

[sjwilks]

I'll re - test this morning and let you know.

On Fri, 30 Jan 2015 00:42 David Sidrane [email protected] wrote:

@sjwilks https://github.com/sjwilks Did the memcpy fix that is
currently on master and nuttx_bringup fix this issue on V1 HW (with stack
checking off)?

—
Reply to this email directly or view it on GitHub
#1691 (comment).

[sjwilks]

This is still an issue on the current master. Flashing and then trying to access /fs/microsd fails.

I'm have issue building nuttx_bringup this time so I can't yet verify if this fixes it or not.

@davids5 did you receive a V1 setup?

[davids5]

Yes. I did. This may be related to a bug clobbering the heap. I am looking at now. If I can get to this today. I will, if not it will be beginning of next week

[LorenzMeier]

Very likely fixed here, please re-test with PX4/Firmware master.
https://github.com/PX4/NuttX/commits/master

[sjwilks]

A test with the latest master works :) I tried powering via USB as well as battery which according to one user caused different results.

Thanks!