posix_eagle broken on recent master (Snapdragon flight)

[RomanBapst]

The latest master does not work on Snapdragon anymore.
@nicovanduijn did a bisect and found the last working commit:
Last working firmware commit: 76bf9c6
First compilable commit that does not work: d8de624

The main changes between those commits are the following:

• moved geo and geo_lookup library into ecl submodule
• updated ecl and compile it as a library instead of a module (see px4_add_module() vs px4_add_library)

What happens on Snapdragon:

• it seems like the firmware on the dsp crashes because there is not response (qshell does not work) and there is also no IMU data visible

• in the shell of the dsp there are errors about symbols not being found
  

• there are also some krait errors appearing on the linaro side (will append a screenshot of that as well)

[dagar]

Would it be possible to get a snapdragon connected to Jenkins both for debugging this and catching these trivial breakages before they get merged to master?

[dagar]

Could someone post the verbose qurt build output of something in ecl (ekf or geo_lookup)? I could do it myself, but likely not until the weekend.

[RomanBapst]

@dagar Below you can find the raw ninja output for quart, bad.txt is from the first build which broke, good.txt is from the last commit which works.

good.txt

bad.txt

[LorenzMeier]

Can you please dump the symbols of the generated executable with @bkueng? That should provide pointers where the symbols got lost.

[RomanBapst]

@dagar We did some progress on this:

1. We moved geo and geo_lookup back to a C API (still in ecl) and that resolved the crashes on the DSP and the strange error messages about undefined symbols.

2. I then applied the same changes to master and we got some new errors:

[08500/03]  27:43.861  HAP:12351:undefined  symbol px4muorb_param_update_to_shmem (3) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.861  HAP:12351:undefined PLT symbol px4muorb_orb_initialize (1) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.861  HAP:12351:undefined PLT symbol px4muorb_set_absolute_time_offset (13) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.861  HAP:12351:undefined PLT symbol px4muorb_get_absolute_time (5) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.861  HAP:12351:undefined PLT symbol px4muorb_param_update_index_from_shmem (17) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.861  HAP:12351:undefined PLT symbol px4muorb_send_topic_data (21) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.861  HAP:12351:undefined PLT symbol px4muorb_is_subscriber_present (20) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.862  HAP:12351:undefined PLT symbol px4muorb_receive_msg (4) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.862  HAP:12351:undefined PLT symbol px4muorb_unblock_recieve_msg (9) /libpx4muorb_skel.so  0303  symbol.c
[08500/03]  27:43.862  HAP:12351:undefined PLT symbol px4muorb_receive_bulk_data (19) /libpx4muorb_skel.so  0303  symbol.c

Seems like there are undefined symbols in libpx4muorb_skel.so

Here are the ouputs of nm libpx4muorb_skel.so of master (broken) and a pre-version which still worked.

Master:

00003000 a _DYNAMIC
000030d0 d _GLOBAL_OFFSET_TABLE_
00003364 D __CTOR_END__
00003360 d __CTOR_LIST__
0000336c d __DTOR_END__
00003368 D __DTOR_LIST__
000025e0 r __EH_FRAME_BEGIN__
000025e0 r __EH_FRAME_END__
00000000 A __bss_start
00000000 A _edata
00000000 A _end
00002380 T _fini
00001c20 T _init
00003108 d methodArrays
000023e8 r methodStrings
000023c8 r methodStringsArrays
000031a8 d methods
00003160 d parameterArrays
000032e0 d parameters
         U px4muorb_add_subscriber
         U px4muorb_get_absolute_time
         U px4muorb_is_subscriber_present
         U px4muorb_orb_initialize
         U px4muorb_param_update_index_from_shmem
         U px4muorb_param_update_to_shmem
         U px4muorb_param_update_value_from_shmem
         U px4muorb_receive_bulk_data
         U px4muorb_receive_msg
         U px4muorb_remove_subscriber
         U px4muorb_send_topic_data
         U px4muorb_set_absolute_time_offset
00001d60 T px4muorb_skel_invoke
00003144 D px4muorb_slim
         U px4muorb_topic_advertised
         U px4muorb_topic_unadvertised
         U px4muorb_unblock_recieve_msg
00002438 r strings
000025b8 r types

Working version:

00003000 a _DYNAMIC
000030d0 d _GLOBAL_OFFSET_TABLE_
00003364 D __CTOR_END__
00003360 d __CTOR_LIST__
0000336c d __DTOR_END__
00003368 D __DTOR_LIST__
000025e0 r __EH_FRAME_BEGIN__
000025e0 r __EH_FRAME_END__
00000000 A __bss_start
00000000 A _edata
00000000 A _end
00002380 T _fini
00001c20 T _init
00003108 d methodArrays
000023e8 r methodStrings
000023c8 r methodStringsArrays
000031a8 d methods
00003160 d parameterArrays
000032e0 d parameters
00001d60 T px4muorb_skel_invoke
00003144 D px4muorb_slim
00002438 r strings
000025b8 r types

[dagar]

Fixed!