Frequent large Log Dropouts - Cause?

[tops4u]

I'm currently running 1.7.3 and made a short flight today to test my Gimbal. I noticed, that there are rather long and frequent Log Dropouts.

How is this possible? I also noticed, that there are frequent Messages on my OSD about not Data available, I assume this has to do with each other.

I am also logging IMU High Rate and EKF Replay, so there is a bit a higher Stream Rate. My Card is an Industrial 8GB Kingston UHS-I (Class 10) --> https://www.kingston.com/en/flash/microsd_cards

My FMU is a mRo X2.1

Log is here: https://logs.px4.io/plot_app?log=c902429b-8467-42fb-996c-7bb2239ce229

[Antiheavy]

FWIW: we started using "U3" rated SDcards (4K video rated) and that reduced our logging drop outs significantly.

[bkueng]

I am also logging IMU High Rate and EKF Replay

This requires considerably higher bandwidth and can thus lead to more frequent dropouts. As @Antiheavy writes, I also suggest to use an U3, such as the SanDisk Extreme U3 32GB.

[tops4u]

I wonder why this is causing so much pain. The Thruput of a Class 10 should easily be enough for the amount of Data to be written.

Above Log is 16.3MB in size over a Period of 2:22 so this results in a rough Datarate of roughly 120kB/sec. Speed Class is per definition the minimal Throughput in a worst case szenario, so my card should be able to sustain a permanent Datarate of 10MB/s, so about 100x what is needed on average. Even if there is a larger chunk of Data coming thru (Buffer flush) there are Dropouts of about 4x10^5 us in which a total amount of 4MB could be written.

[LorenzMeier]

The SD card spec allows cards to not commit to memory for 500 ms and most cards have internal buffers. It’s not a FIFO you’re writing into.

[tops4u]

Thanks for commenting Lorenz. Fair enough, so this may explain the Dropouts of < 500ms, what about the larger ones > 1 sec? So after the initial 500ms max latency the card should be able to sustain a the specified Datarate (otherwise it would loose Frames in Videos for instance). Even so a faster Card would not help in this case. I'm still wondering if this has to do with data stream interruption towards the OSD.

[Antiheavy]

While debugging other system problems (e.g. spurious "Manual Control Lost" messages and other system warning messages) we've observed the Logger process spike. We've suspected the issue isn't so much that Class 10 can't keep up with the overall datarates, but more that Logger might be jamming up the processing pipeline in PX4 and other problems start occuring, e.g. logging drop outs. Unfortunately I don't have hard evidence that would satisfy the developers or help them know where to look, other than when we use U3 cards we see many fewer problems like that.

[Antiheavy]

@dagar and @priseborough recommended we use the U3 cards as well, maybe they've seen more specific issues?

[priseborough]

I see this phenomena regularly on logs I am sent for analysis. Buying a faster card does not necessarily make it better. Out of my collection of cards, it is the Sandisk Extreme 32GB U3 that works best in testing.

[priseborough]

Have a look at the section on SD cards here: https://dev.px4.io/en/log/logging.html

[tops4u]

Ok, done that. Max 702ms / Average 136KB/s... Guess I need another card.