Crazyfly 2.0 on the current master does not connect with QGC

[barzanisar]

@dennisss I have the same problem as in issue #7624 with the current master.
I followed the steps for flashing PX4 on crazyflie 2 from this link: https://docs.px4.io/en/flight_controller/crazyflie2.html

It finished flashing (erase, program, verify), but CF wasn’t able to connect with QGC, so I couldn't calibrate.

I tried the latest master, stable and beta (via QGroundControl as well), all of these finished flashing (erase, program, verify), but again CF wasn’t able to connect with QGC.

Then I did a binary search between the latest master and the commit ca82d36 described in the issue #7658, and found the last working commit: c82deaf Mon Sep 4 11:04:17 2017 that I can flash and connect with QGC, calibrate and so on.

[dagar]

Hi @barzanisar, I don't have one of these, but I might be able to help you debug and get it working again. Then we need to put something in place to keep it working.

[barzanisar]

yes please, thank you!

[dagar]

Would you like to discuss on the dev call or ping me on slack?

[dennisss]

Looks like there was a regression in the defconfig for this platform.
@barzanisar Can you try out this change and see if it resolves the issue: dennisss@fcb4ce4

[barzanisar]

Yes it works! Thank you!

[barzanisar]

@dennisss There's one more issue, when I arm the motors, they don't rotate. Even when the arming speed "PWM_MIN" parameter is above 1000us. I tried 1100,1200,1400 us.

Also, when I give some throttle and a bit of roll or pitch joystick command, the motors that are supposed to rotate with lower speed don't rotate at all.

[dennisss]

@barzanisar make sure that you are using the crazyflie airframe (SYS_AUTOSTART = 4900). The default settings should work and have PWM_MIN, PWM_DISARMED, PWM_MAX set to 0, 0, 255.

[LorenzMeier]

@Denisss we would like to have the props spinning when armed - what is the best way to achieve that? Without it the flight stability is not optimal in extreme maneuvers

[dennisss]

@LorenzMeier if you are referring to spinning props at 0% throttle, then pick your choice of value for PWM_MIN between 0-255 and adjust the value constraints configured here: https://github.com/PX4/Firmware/blob/master/src/drivers/boards/crazyflie/board_config.h#L169

[LorenzMeier]

Great, thanks! @barzanisar let me know when you need support on that.

[barzanisar]

Thanks @dennisss ! When I set PWM_MIN parameter in QGC or PWM_LOWEST_MIN value from
https://github.com/PX4/Firmware/blob/master/src/drivers/boards/crazyflie/board_config.h#L169
, the motors don't spin when armed. However, when I changed PWM_DEFAULT_MIN, the motors rotated when armed with the correct speed as set in PWM_DEFAULT_MIN. So I guess, PWM_DEFAULT_MIN does the job of PWM_MIN, but we have to change it from board_config.h every time.

Also, the cfbridge does not seem to properly communicate the joystick commands from QGC to CF2. When I use USB connection, joystick commands are sent properly and I can arm, disarm and change flight modes. However, when I use crazyradio PA with cfbridge, sometimes I am not able to arm/disarm.

Also, I noticed that when I first connect QGC with CF2 via USB (to calibrate etc) and then unplug usb and connect via crazyradio PA + cfbridge, the following happens:

In manual mode, when I arm using joystick, it denies the Arm command (eventhough throttle is zero from joystick). Then when I switch to altitude mode, it switches to LAND and ARMED mode, the motors start spinning fast. I disarm the motors using joystick button, which QGC reads and disarms the motors.

This doesn't happen when I reflash the firmware and only connect QGC and CF2 via crazyradio PA and cfbridge, i.e. without using the USB connection.

In short, if I flash the firmware, use USB connection and then unplug and use crazyradio PA, QGC is not able to communicate properly with CF2 and I observe the same pattern everytime so it is not a connection lost problem.
On the other hand, if I flash firmware, use crazyradio PA directly, QGC is able to communicate with CF2 but disarming command is not sent sometimes from some flight modes.

[barzanisar]

Also, when I first built cfbridge and tried "make run" command, it failed to find the radio. Then I followed the steps in https://github.com/bitcraze/crazyflie-lib-python#setting-udev-permissions to set udev permissions and restarted my computer. cfbridge could then find the radio and radio connected.

[barzanisar]

There's a delay of communication between crazyradio and CF2 and it seems like some messages are being dropped and sent late.

[dennisss]

@barzanisar I have noticed the disarm issue in manual flight modes as well, but I am unsure of the exact cause of it. I usually stick to offboard modes, where it is a non-issue for me. Radio loss is a known issue which I believe is due to the Crazyradio PA only being able to receive 31 bytes at a time, so large packets are split. Adding better packet sequencing to the protocol would probably help with this, although right now being conservative with message sending is the best solution I've found so far.

[LorenzMeier]

@bkueng @barzanisar should discuss in the afternoon. I think it needs some cleverness on both sides.

[barzanisar]

Update: @LorenzMeier @bkueng @dennisss

There were two issues:

1. Arm or disarm command sent from QGC was not received by CF2. We kind of dealt with this problem by reducing the joystick update rate from 25 Hz to 10 Hz in https://github.com/mavlink/qgroundcontrol/blob/9d5bed4842228dc7f247b1af90fd9020632b6d24/src/Joystick/Joystick.cc#L530 and keeping it consistent 10 Hz by changing the code in https://github.com/mavlink/qgroundcontrol/blob/9d5bed4842228dc7f247b1af90fd9020632b6d24/src/uas/UAS.cc#L1001-L1012 (setting sendCommand to true by default and only false for NAN values of joystick commands)

Sending less Joystick commands per second allows space for MAV_CMD_COMPONENT_ARM_DISARM command to be sent.

2. Once we send arm/disarm command, CF2 arms/disarms but COMMAND_ACK is not received by the cfbridge and hence QGC keeps waiting for COMMAND_ACK message and does not respond to other arm/disarm commands we send next. Only when we switch to another flight mode, QGC is programmed such that it stops looking for the ACK message and sends the arm/disarm commands queued up. This I found out from Vehicle.cc

We tried switching to "custom" mode Mavlink from "osd" in https://github.com/PX4/Firmware/blob/master/ROMFS/px4fmu_common/init.d/rcS#L603 and set the rate to 1000.

This was done to ensure that CF2 sends minimal data to QGC and hence the link has enough space for ACK message. "mavlink status" shows that the link has enough bandwidth for ACK message to be sent, yet the cfbridge doesn't receive it.

Now the next step I guess is to debug the bitcraze/crazyradio-firmware to see if it receives the ACK message? Any other suggestions?

[dagar]

Ideas

• new PX4 minimal mavlink mode (mavlink add minimal mode #8947) to send the bare minimum needed for functional QGC. Depends on your exact configuration, but this will likely reduce your mavlink telemetry rate (vehicle -> ground) by at least an order of magnitude.
• QGC throttle MANUAL_CONTROL rate based on stick movements or mav command queue depth
• QGC prevent excessive message queuing (might be hacky)
  • [WIP] Initial gimbal control mavlink/qgroundcontrol#6138
  • take a look at Vehicle::sendMavCommand() in this PR where additional commands of the same type are updated in the queue, rather than stacking deeper

[barzanisar]

@dagar

1. Custom mavlink mode sends even less data to the QGC as you can see here:
   . Yet, this didn't solve the problem of receiving Ack message from the vehicle.

2. I decreased the joystick update rate which sends less MANUAL_CONTROL id=69 messages to the vehicle so that there is space for MAV_CMD_COMPONENT_ARM_DISARM command to be sent. Is it the same thing you're talking about?

Also, I tested again with reduced joystick update rate, again sometimes MAV_CMD_COMPONENT_ARM_DISARM message is sent by QGC and cfbridge but is lost on the way from radio to the vehicle. I guess I'll have to decrease the joystick update rate even further but it will deteriorate the joytick command response.

3. I looked at your hack. Your hack works for my case only if I replace the first element of the Queue (which QGC has already sent to the vehicle and is waiting for its Ack message) with the latest command of the same type, i.e. by making these changes:
   if (_mavCommandQueue.size() >= 1) { for (int i = 0; i < _mavCommandQueue.size(); i++)

I was thinking about such a hack as a last resort but was worried about its implications since by doing this, we are simply ignoring the receipt of Ack message which removes the first command from the queue (see Vehicle::_handleCommandAck) and replacing that queued command with the latest command of the same type.

[barzanisar]

Update:
@dagar @LorenzMeier @bkueng

Issue1: COMMAND_ACK sent from CF2 is received into QGC but discarded

Observation: After digging into the lower level i.e. in syslink module, I found out that some bytes sent by the Syslink::send_message, are randomly not received by the cfbridge. I printed the bytes received in cfbridge by printing this buffer. For example, vehicle's heartbeat payload (of length 9 bytes) sent from CF2 to QGC is dropped sometimes and only header(10 bytes) and checksum(2 bytes) is received. And if the Ack message is unfortunately sent just after this bad hearbeat message, QGC keeps reading the Ack message thinking of it as the payload of the bad heartbeat in MAVLinkProtocol::receiveBytes and then discards forwarding the message to Vehicle::_mavlinkMessageReceived since the Ack message header is larger than the payload expected for the bad heartbeat.

Another possible issue: If some bytes are randomly dropped, it is possible that some bytes of the Ack message are also dropped sometimes. Though I never observed this and only observed payload dropping from heartbeat messages, since of course heartbeat message is more frequently sent.

Issue 2: COMMAND_LONG encoding MAV_CMD_COMPONENT_ARM_DISARM sent from QGC is either received partially or not received at all in CF2

Observation: Some messages sent from QGC and then cfbridge are sometimes not received in CF2 (they are dropped). Hence, the arm/disarm command message (that is longer then 32 bytes and hence is split into two messages in UdpClient::read is either not received at all in CF2 (i.e. both messages are dropped) or received partially (one of the messages is dropped) and this behavior appears to be random.

[barzanisar]

Solved, see document updated