Builds failing on OSX and Linux with Github Actions CI

[smk762]

Recently our CI builds at https://github.com/KomodoPlatform/atomicDEX-Desktop/actions have been failing to build libwally.

I suspect it may have something to do with a recent libtools update https://savannah.gnu.org/forum/forum.php?forum_id=10139

related mac errors in log:

2022-03-21T15:15:44.8170920Z *** Warning: Linking the shared library libwallycore.la against the
2022-03-21T15:15:44.8262000Z *** static library secp256k1/.libs/libsecp256k1.a is not portable!
2022-03-21T15:15:44.8762380Z copying selected object files to avoid basename conflicts...
2022-03-21T15:15:44.9182980Z /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: libwallycore_la-elements.o has no symbols
2022-03-21T15:15:44.9184050Z /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: libwallycore_la-blech32.o has no symbols
2022-03-21T15:15:44.9576510Z /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libwallycore.a(libwallycore_la-elements.o) has no symbols
2022-03-21T15:15:44.9577510Z /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libwallycore.a(libwallycore_la-blech32.o) has no symbols

2022-03-22T10:49:20.9051540Z ./libtool: line 151: -o: command not found
2022-03-22T10:49:29.0185510Z   CCLD     libsecp256k1.la
2022-03-22T10:49:29.0256440Z ./libtool: line 151: -o: command not found
2022-03-22T10:49:30.3257350Z error: /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: no output file specified (specify with -o output)
2022-03-22T10:49:30.3258410Z Usage: /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-sacLT] [-no_warning_for_no_symbols]
2022-03-22T10:49:30.3259840Z Usage: /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-o output] [-install_name name] [-compatibility_version #] [-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table <filename>] [-seg_addr_table_filename <file_system_path>] [-all_load] [-noall_load]
2022-03-22T10:49:30.3281370Z make[2]: *** [libsecp256k1.la] Error 1
2022-03-22T10:49:30.3292420Z make[1]: *** [install-recursive] Error 1
2022-03-22T10:49:30.3301950Z make: *** [install-recursive] Error 1

Related Linux errors in log:

2022-03-22T15:40:02.5243502Z === configuring in src/secp256k1 (/home/runner/work/atomicDEX-Desktop/atomicDEX-Desktop/cmake-3.19.0-rc3-Linux-x86_64/libwally-core/src/secp256k1)
2022-03-22T15:40:02.5293435Z configure: WARNING: no configuration information is in src/secp256k1
2022-03-22T15:40:02.5804049Z Making install in src
2022-03-22T15:40:02.5865562Z make[1]: Entering directory '/home/runner/work/atomicDEX-Desktop/atomicDEX-Desktop/cmake-3.19.0-rc3-Linux-x86_64/libwally-core/src'
2022-03-22T15:40:02.5918363Z Making install in secp256k1
2022-03-22T15:40:02.5941948Z make[2]: *** No rule to make target 'install'.  Stop.
2022-03-22T15:40:02.5942747Z make[2]: Entering directory '/home/runner/work/atomicDEX-Desktop/atomicDEX-Desktop/cmake-3.19.0-rc3-Linux-x86_64/libwally-core/src/secp256k1'
2022-03-22T15:40:02.5945184Z make[2]: Leaving directory '/home/runner/work/atomicDEX-Desktop/atomicDEX-Desktop/cmake-3.19.0-rc3-Linux-x86_64/libwally-core/src/secp256k1'
2022-03-22T15:40:02.5950953Z Makefile:1666: recipe for target 'install-recursive' failed
2022-03-22T15:40:02.5952592Z make[1]: Leaving directory '/home/runner/work/atomicDEX-Desktop/atomicDEX-Desktop/cmake-3.19.0-rc3-Linux-x86_64/libwally-core/src'
2022-03-22T15:40:02.5953409Z make[1]: *** [install-recursive] Error 1
2022-03-22T15:40:02.5977357Z make: *** [install-recursive] Error 1
2022-03-22T15:40:02.5978115Z Makefile:434: recipe for target 'install-recursive' failed

I've attempted a few things to resolve this without success in our CI_libwally_test branch. For example:

• pinning older version of libtools
• using macos-10.15 instead of mac latest in https://github.com/KomodoPlatform/atomicDEX-Desktop/blob/CI_libwally_test/.github/workflows/atomicdex-desktop-ci.yml

Our windows builds are ok, because we used a cached prebuild.

Any hints you might have or insights regarding what might be missing / misconfigured in our workflow file or https://github.com/actions/virtual-environments would be appreciated

[smk762]

I figured it was worth trying to run your Github actions on a fork, seems similar issue - https://github.com/smk762/libwally-core/runs/5672778765?check_suite_focus=true#step:5:549

      Making all in secp256k1
      gcc -DHAVE_CONFIG_H -I/usr/local/opt/valgrind/include  -I. -I./src -O2  -std=c89 -pedantic -Wno-long-long -Wnested-externs -Wshadow -Wstrict-prototypes -Wundef -Wno-overlength-strings -Wall -Wno-unused-function -Wextra -Wcast-align -Wconditional-uninitialized -fvisibility=hidden   -c src/gen_context.c -o gen_context.o
      gcc -O2  -std=c89 -pedantic -Wno-long-long -Wnested-externs -Wshadow -Wstrict-prototypes -Wundef -Wno-overlength-strings -Wall -Wno-unused-function -Wextra -Wcast-align -Wconditional-uninitialized -fvisibility=hidden   -O2  gen_context.o -o gen_context
      ./gen_context
        CC       src/libsecp256k1_la-secp256k1.lo
      ./libtool: line 151: -o: command not found
        CCLD     libsecp256k1.la
      ./libtool: line 151: -o: command not found
      error: /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: no output file specified (specify with -o output)
      Usage: /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-sacLT] [-no_warning_for_no_symbols]
      Usage: /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-o output] [-install_name name] [-compatibility_version #] [-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table <filename>] [-seg_addr_table_filename <file_system_path>] [-all_load] [-noall_load]
      make[3]: *** [libsecp256k1.la] Error 1
      make[2]: *** [all-recursive] Error 1
      make[1]: *** [all] Error 2
      make: *** [all-recursive] Error 1
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-req-build-yr9k171x/setup.py", line 23, in <module>
          call('make -j{}'.format(multiprocessing.cpu_count()))
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-req-build-yr9k171x/setup.py", line 18, in call
          subprocess.check_call(cmd.split(' '), cwd=abs_path)
        File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 311, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['make', '-j3']' returned non-zero exit status 2.
      ----------------------------------------
  WARNING: Discarding file:///Users/runner/work/libwally-core/libwally-core. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
                                                             ✕ 62.29s
Error: Command ['python', '-m', 'pip', 'wheel', PosixPath('/Users/runner/work/libwally-core/libwally-core'), '--wheel-dir=/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheellcpz52lq/built_wheel', '--no-deps'] failed with code 1. None```

[jgriffiths]

Hi @smk762

Please check ./libtool: line 151: -o: command not found

Line 151 presumably references an environment variable that is expected to be set to a command (such as CC or LD). If it was empty I would expect this to be the error you get.

If that is the case, you can try setting that variable to the correct value before invoking ./configure.

[smk762]

Hi @smk762

Please check ./libtool: line 151: -o: command not found

Line 151 presumably references an environment variable that is expected to be set to a command (such as CC or LD). If it was empty I would expect this to be the error you get.

If that is the case, you can try setting that variable to the correct value before invoking ./configure.

The ./libtool: line 151: -o: command not found is also occurring when running the Github actions workflow via https://github.com/ElementsProject/libwally-core/blob/master/.github/workflows/wheels.yml on a fork of this repo.

[cdecker]

Found a possible solution to this issue. Recently libtool was updated from 2.4.6_4 to 2.4.7 (formula history) which broke libtool support for secp256k-zkp in Intel and M1 macs.

Checking with the upstream version secp256k1 I noticed that this error does not occur and asked whether there were any build changes that were not backported into secp256k1-zkp.
As a response @jonasnick opened PR BlockstreamResearch/secp256k1-zkp#174 which backports a couple of missing fixes, apparently including the required changes to get mac support working again.

Now secp256k1-zkp compiles fine in isolation, however when compiling libwally-core we get other errors relating to libtool:

./configure
[...]
checking the archiver (/usr/bin/libtool) interface... unknown
configure: error: could not determine /usr/bin/libtool interface
configure: error: ./configure failed for src/secp256k1

Initially I was under the impression it might be libtool as used by libwally-core but it seems it is somehow instrumenting the ./configure call in src/secp256k1 which then fails. Is there anything special about the way ./configure is invoked from libwally-cores ./configure?

[real-or-random]

Now secp256k1-zkp compiles fine in isolation, however when compiling libwally-core we get other errors relating to libtool:

./configure
[...]
checking the archiver (/usr/bin/libtool) interface... unknown
configure: error: could not determine /usr/bin/libtool interface
configure: error: ./configure failed for src/secp256k1

Initially I was under the impression it might be libtool as used by libwally-core but it seems it is somehow instrumenting the ./configure call in src/secp256k1 which then fails. Is there anything special about the way ./configure is invoked from libwally-cores ./configure?

Can you share a config.log for this error?

[jurvis]

+1 on this issue. currently trying to build libwally-core too on M1 Mac with [email protected]

here's my output when running ./build-libwally.sh -dsc:

/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-recursive
Making all in secp256k1
gcc -DHAVE_CONFIG_H -I. -I./src -O2  -std=c89 -pedantic -Wno-long-long -Wnested-externs -Wshadow -Wstrict-prototypes -Wundef -Wno-overlength-strings -Wall -Wno-unused-function -Wextra -Wcast-align -Wconditional-uninitialized -fvisibility=hidden -g -O2 -c src/gen_context.c -o gen_context.o
gcc -O2  -std=c89 -pedantic -Wno-long-long -Wnested-externs -Wshadow -Wstrict-prototypes -Wundef -Wno-overlength-strings -Wall -Wno-unused-function -Wextra -Wcast-align -Wconditional-uninitialized -fvisibility=hidden -g -O2  gen_context.o -o gen_context
./gen_context
  CC       src/libsecp256k1_la-secp256k1.lo
./libtool: line 151: -o: command not found
  CCLD     libsecp256k1.la
./libtool: line 151: -o: command not found
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: no output file specified (specify with -o output)

[jgriffiths]

@smk762 @cdecker @jurvis I've created a branch to fix these build issues, please see #321

If you could test this and ack for your use case(s) that would be great. I will test this with gdk and merge once I have confirmation everything is working as expected. Thanks!

[jurvis]

hey @jgriffiths, thanks for looking into this! I just noticed I mis-typed earlier where my logs were from; I was trying to build libwally-swift with their build script, not libwally-core.

Anyways, I was still able to produce that output with current HEAD of master with the following commands:

./configure --disable-shared --host=aarch64-apple-darwin --enable-static --disable-elements`
make clean
make

I checked out test_build_fix, and re-ran the commands above. Now, I am getting this error output instead:

*** Warning: Linking the shared library libwallycore.la against the
*** static library secp256k1/.libs/libsecp256k1.a is not portable!
copying selected object files to avoid basename conflicts...
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive member: .libs/libwallycore.a(libsecp256k1.a) fat file for cputype (16777223) cpusubtype (3) is not an object file (bad magic number)
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar: internal ranlib command failed
make[3]: *** [libwallycore.la] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1

hope this is helpful. let me know if there's more info I can help you with collecting

[jgriffiths]

@jurvis Thanks, I think this error is because its attempting to build a universal binary instead of a singular arch, and that doesn't seem to be supported for adding archive into archives.

Can you try limiting the arch to arm64 via CFLAGS="-march=arm64" or similar?

[jgriffiths]

@jurvis You can also try this commit 0dca9bf without -march which may fix fat linking for the shared lib.

[jurvis]

@jgriffiths yep, it looks like that was it. I was able to build libwally-core a-okay by just setting CFLAGS="-O3 -arch arm64 (trying to get as close as possible to command provided in libwally-swift's use-case).

However, when I try and build libwally-swift, I get this error:

*** static library secp256k1/.libs/libsecp256k1.a is not portable!
copying selected object files to avoid basename conflicts...
  CCLD     test_bech32
ld: warning: ignoring file ./.libs/libwallycore.a, building for iOS Simulator-arm64 but attempting to link with file built for iOS Simulator-arm64
Undefined symbols for architecture arm64:
  "_wally_addr_segwit_to_bytes", referenced from:
      _main in test_bech32-test_bech32.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [test_bech32] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1

here's what the script is doing (abbreviated):

export CFLAGS="-O3 -arch arm64 -fembed-bitcode-marker -mios-simulator-version-min=10.0 -isysroot `xcrun -sdk iphonesimulator --show-sdk-path`"
export CXXFLAGS="-O3 -arch arm64 -fembed-bitcode-marker -mios-simulator-version-min=10.0 -isysroot `xcrun -sdk iphonesimulator --show-sdk-path`"
./configure --disable-shared --host=$arch_target --enable-static --disable-elements
make clean
make

I apologize I don't really have knowledge about building universal binaries and how they're being managed to run on iOS. It looks like everything may be okay on libwally-core's end. If you agree, I think we can consider this issue fixed for me, and I can dig a little further on my end to see how I can fix the build script for libwally-swift

[jgriffiths]

@jurvis You may also need to set LDFLAGS=-arch arm64 so the linker knows what kind of ABI it is meant to be producing. In that case you would want to make 2 builds, one for x86_64 and one for arm64, rather than a single universal binary build.

Aside from allowing M1 builds for wally, my goal is to also have the CI produce wheels that we can release via PyPI so that building from source isn't needed on that platform. That is a question of CI hacking which I am experimenting on in #322. So while its good that you can build wally there, I'll probably consider this closed once we can produce working binaries on the CI, at least for arm64 specifically if not universal2.

[jurvis]

hi @jgriffiths I took another stab at building with this script:

export CFLAGS="-O3 -arch arm64 -arch arm64e -arch armv7 -arch armv7s -fembed-bitcode -mios-version-min=10.0 -isysroot `xcrun -sdk iphoneos --show-sdk-path`"
export CXXFLAGS="-O3 -arch arm64 -arch arm64e -arch armv7 -arch armv7s -isysroot -fembed-bitcode -mios-version-min=10.0 -isysroot `xcrun -sdk iphoneos --show-sdk-path`"
export LDFLAGS="-arch arm64 -arch arm64e -arch armv7 -arch armv7s"

./configure --disable-shared --host=$arch_target --enable-static --disable-elements
make clean
make

I now get the following error:

1 warning generated.
ccan/ccan/crypto/sha512/sha512.c:205:22: warning: cast from 'const unsigned char *' to 'const uint64_t *' (aka 'const unsigned long long *') increases required alignment from 1 to 4 [-Wcast-align]
                        Transform(ctx->s, (const uint64_t *)data);
                                          ^~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
  CC       ccan/ccan/str/hex/libwallycore_la-hex.lo
  CCLD     libwallycore.la
copying selected object files to avoid basename conflicts...
  CCLD     test_bech32
ld: file not found: secp256k1/src/libsecp256k1_precomputed_la-precomputed_ecmult_gen.o

lmk when that PR is ready to test, I will like to be able to make a contribution to fix downstream for libwally-swift too.

[jgriffiths]

@jurivs, at the point at which you get that error, what generated files are in src/secp256k1/src/?

Also I think you do not need --host, or if you do, you want to pass it as --build.

[jurvis]

@jgriffiths here's what I have when doing a tree on that directory: https://gist.github.com/jurvis/6a91f6f5a666af40d1936d3aa4c4603e

hm, I tried using --build, but got this error:

checking whether we are cross compiling... configure: error: in `/Users/jurvistan/Projects/libwally-swift/CLibWally/libwally-core':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details

[jgriffiths]

Thanks @jurvis looks like I will need to debug this on an M1 box, I'm arranging access to one.

[real-or-random]

1 warning generated.
ccan/ccan/crypto/sha512/sha512.c:205:22: warning: cast from 'const unsigned char *' to 'const uint64_t *' (aka 'const unsigned long long *') increases required alignment from 1 to 4 [-Wcast-align]
                        Transform(ctx->s, (const uint64_t *)data);
                                          ^~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
  CC       ccan/ccan/str/hex/libwallycore_la-hex.lo
  CCLD     libwallycore.la
copying selected object files to avoid basename conflicts...
  CCLD     test_bech32
ld: file not found: secp256k1/src/libsecp256k1_precomputed_la-precomputed_ecmult_gen.o

@jgriffiths here's what I have when doing a tree on that directory: gist.github.com/jurvis/6a91f6f5a666af40d1936d3aa4c4603e

This looks like you're mixing secp256k-zkp revisions. The error indicates that libwally is expecting a new version (BlockstreamResearch/secp256k1-zkp@21e2d65 or newer, merged 2022-02-05) because this introduces the "precomputed" files. The dir tree still shows an old revision before this commit.

[jgriffiths]

@real-or-random Thanks! that looks to be the root cause indeed.

@jurvis You need to update the secp submodule in your source tree, #321 is using current master 4c4463a7913289ce370603c5c2f53a7d8c9e89ce while you appear to have an older version checked out as @real-or-random says above.

I have one failure to debug/address currently then 321 should be mergable.

[jurvis]

@real-or-random @jgriffiths that was it! can't believe I didn't think of that 🤦‍♂️ thank you all!

[jurvis]

the script in libwally-swift needs a bit of tweaking for some Xcode-specific requirements and builds should work again with this release 😄

[jgriffiths]

@jurvis Would you able to smoke test the M1 python wheel for me so I can merge #321? Looks like it's going to take a while for me to get access to an M1 box.

$ # Set up a venv for testing
$ cd <path to wally src>
$ virtualenv venv
$ . venv/bin/activate

$ # Check self-built wheel
$ pip install .
$ python -c "import wallycore as wally; print(len(wally.witness_program_from_bytes(wally.hex_to_bytes('ff'*32),0)))"
34
$ pip uninstall -y wallycore

$ # Check CI built PyPI wheel
$ # Download artifact.zip from the bottom of https://github.com/ElementsProject/libwally-core/actions/runs/2107134659
$ # pip install the macos arm64 wheel matching your python version
$ python -c "import wallycore as wally; print(len(wally.witness_program_from_bytes(wally.hex_to_bytes('ff'*32),0)))"
34

$ # Cleanup
$ deactivate
$ rm -rf venv

[jurvis]

@jgriffiths sure, I can do it later tonight. will post results here 👍

[jurvis]

hey @jgriffiths it looks like it worked!

Self built wheel

(venv) libwally-core git:test_build_fix ❯ pip install .                                                                              ✭
Processing /Users/jurvistest/Projects/libwally-core
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: wallycore
  Building wheel for wallycore (setup.py) ... done
  Created wheel for wallycore: filename=wallycore-0.8.4-cp39-cp39-macosx_12_0_arm64.whl size=1550259 sha256=b0141c4b59c551d2bbcfdf8600867564140243eb59bdaa512f0949c760f58c4b
  Stored in directory: /Users/jurvistest/Library/Caches/pip/wheels/c8/fa/b3/8673d4b7947da7e7f52f558d1d7aacae6afecb096e95815d0d
Successfully built wallycore
Installing collected packages: wallycore
Successfully installed wallycore-0.8.4
(venv) libwally-core git:test_build_fix ❯ python -c "import wallycore as wally; print(len(wally.witness_program_from_bytes(wally.hex_to_bytes('ff'*32),0)))"
34

CI-built wheel:

(venv) artifact ❯ pip install wallycore-0.8.4-cp39-cp39-macosx_11_0_arm64.whl
Processing ./wallycore-0.8.4-cp39-cp39-macosx_11_0_arm64.whl
Installing collected packages: wallycore
Successfully installed wallycore-0.8.4
(venv) artifact ❯ python -c "import wallycore as wally; print(len(wally.witness_program_from_bytes(wally.hex_to_bytes('ff'*32),0)))"
34

[jgriffiths]

@jurvis, thanks very much! I will merge that PR shortly after a little more testing here.

[jurvis]

not sure if related but I built libwally-core with the following commands:

export CFLAGS="-O3 -arch arm64 -arch arm64e -arch armv7 -arch armv7s -fembed-bitcode -mios-version-min=10.0 -isysroot `xcrun -sdk iphoneos --show-sdk-path`"
export CXXFLAGS="-O3 -arch arm64 -arch arm64e -arch armv7 -arch armv7s -isysroot -fembed-bitcode -mios-version-min=10.0 -isysroot `xcrun -sdk iphoneos --show-sdk-path`"
./configure --disable-shared --host=aarch64-apple-darwin --enable-static --disable-elements
make clean
make

but ended up getting a bunch of errors for secp256k1, specifically "Undefined symbols for architecture arm64":

any ideas?

[jgriffiths]

@jurvis you will need to add the -arch items to LDFLAGS, and also add --disable-dependency-tracking to configure I think.

What are you targeting that requires all of the archs listed?

[jurvis]

@jgriffiths
So here's what worked/didn't work (in order which I tried them)
✅ Remove arm64 from arch list, add LDFLAGS and --disable-dependency-tracking
✅ Remove arm64 from arch list
🔴 Keep arm64 in arch list, add LDFLAGS and --disable-dependency-tracking

I believe arm64 is necessary because it is a possible architecture that Xcode will attempt to build for since it's listed as "Standard Architecture". I think while the arm64e ABI is what's being used by iOS/M1 devices, Xcode still tries to build with the arm64 arch

[jgriffiths]

@jurvis I am no expert on apple silicon/ABIs etc, but I haven't seen anyone explicitly targeting arm64e yet, e.g. python wheels for macos are either x86_64, arm64 or both together (so called universal2 fat binaries). I haven't seen anyone building for all of the arches listed, and tbh I don't quite understand why you'd want a single fat binary supporting all of them rather than separate files (with the exception of the original universal or universal2).

I'm guessing its possible that arm64 and arm64e are mutually exclusive, and that arm64 will run on the e variant. Its also possible that arm64 is interpreted as implying the e variant if you are building on that silicon, not sure.

[jurvis]

@jgriffiths no worries! it may just simply be how I'm managing the ABIs. I'm new to this too myself 😄 I'll continue to poke around and share what I learn when I figure it out. thanks again!

[jurvis]

@jgriffiths I was able to trace and identify the issue. It looks like I encountered my error because libsecp256k1's symbols were hidden in libwallycore.a, see nm libwallycore.a output:

libwallycore.a(libwallycore_la-sign.o):
---------------- s _MSG_PREFIX
                 U _pubkey_negate
                 U _pubkey_parse
                 U _pubkey_serialize
                 U _secp256k1_context_no_precomp
                 U _secp256k1_ec_pubkey_create
                 U _secp256k1_ec_seckey_verify
                 U _secp256k1_ecdsa_recover
                 U _secp256k1_ecdsa_recoverable_signature_parse_compact
                 U _secp256k1_ecdsa_recoverable_signature_serialize_compact
                 U _secp256k1_ecdsa_s2c_opening_parse
                 U _secp256k1_ecdsa_s2c_opening_serialize
                 U _secp256k1_ecdsa_s2c_sign
                 U _secp256k1_ecdsa_s2c_verify_commit
                 U _secp256k1_ecdsa_sign_recoverable
                 U _secp256k1_ecdsa_signature_normalize
                 U _secp256k1_ecdsa_signature_parse_compact
                 U _secp256k1_ecdsa_signature_parse_der
                 U _secp256k1_ecdsa_signature_serialize_compact
                 U _secp256k1_ecdsa_signature_serialize_der
                 U _secp256k1_ecdsa_verify
                 U _secp_ctx
                 U _wally_clear
                 U _wally_clear_2

I was able to fix it by just importing libwally-core/src/secp256k1/.libs/libsecp256k1.a to my Xcode project.

[jgriffiths]

@jurvis Makes sense, thanks for updating. The current changes add the secp object files to the wally shared library when built, but do not add them to the static library (since its not obvious how to do this in a safe way at this point).

gdk builds statically and adds the secp library also, so this solution should continue to work going forward.

[jgriffiths]

321 is merged, closing. Thanks all for your help resolving this.