Rust binding in Cryptography 3.4+ dynamically links with non-existent library on macOS #106
Comments
|
The executable bit is missing on Linux, too. |
postgres-mirror
pushed a commit
to pgadmin-org/pgadmin4
that referenced
this issue
Feb 10, 2021
… module as described at PyO3/setuptools-rust#106
|
Thanks for reporting. There's clearly one of two things going wrong. Either
I'm afraid I don't have access to a macOS computer right now to test on, so will need some help with identifying the fix. If you clone the cryptography source tree, invoke |
|
I can't get it to build unfortunately. It looks like the linker is failing to find what I assume are functions in the Python library, however even explicitly adding that to LDFLAGS isn't helping (in fact it looks like LDFLAGS is being ignored): Any ideas? I'm completely unfamiliar with rust/cargo. FYI, that was using the Python 3.9.1 distro from python.org (x86_64, not the Universal2 build), on an Intel mac running Big Sur. |
|
Ah, sorry, my bad. I'm not a regular macOS user, so forgot some linker configuration is needed. Python extensions require the Python symbols to be unresolved at link time, so we have to pass special flags to rustc. Full notes are at https://github.com/PyO3/pyo3#using-rust-from-python TLDR; use |
|
Thanks, that worked. The reference is there in the resulting build, albeit adjusted as you might expect for my build environment: The referenced library has also been built and is present at the path shown. Thanks. |
|
Great! Would you mind also checking the referenced library with I thought that the final library is just a copy of the referenced one, so it would also be interesting to know if there are any differences at all. For example, on my Linux (ubuntu) machine both libraries exist and are clearly the same file; a quick inspection shows they have the same size, mtime, and hash. |
|
Of course (and yes, they do have the same hash): So it really seems like that reference just shouldn't be there. |
|
Thanks for checking. Yes, it looks like it's probably an upstream Rust bug. I'll try to borrow my wife's gloriously underpowered macbook air at the weekend to see if I can narrow down a minimal repro for upstream. |
|
It seems that the non-existent library path comes from ❯ otool -D _rust.abi3.so
_rust.abi3.so:
/Users/messense/Projects/cryptography/src/rust/target/release/deps/libcryptography_rust.dylibInterestingly,
❯ otool -l _rust.abi3.so
_rust.abi3.so:
Load command 0
cmd LC_SEGMENT_64
cmdsize 632
segname __TEXT
vmaddr 0x0000000000000000
vmsize 0x0000000000040000
fileoff 0
filesize 262144
maxprot 0x00000005
initprot 0x00000005
nsects 7
flags 0x0
Section
sectname __text
segname __TEXT
addr 0x0000000000001268
size 0x0000000000033fec
offset 4712
align 2^2 (4)
reloff 0
nreloc 0
flags 0x80000400
reserved1 0
reserved2 0
Section
sectname __stubs
segname __TEXT
addr 0x0000000000035254
size 0x0000000000000414
offset 217684
align 2^2 (4)
reloff 0
nreloc 0
flags 0x80000408
reserved1 0 (index into indirect symbol table)
reserved2 12 (size of stubs)
Section
sectname __stub_helper
segname __TEXT
addr 0x0000000000035668
size 0x000000000000042c
offset 218728
align 2^2 (4)
reloff 0
nreloc 0
flags 0x80000400
reserved1 0
reserved2 0
Section
sectname __gcc_except_tab
segname __TEXT
addr 0x0000000000035a94
size 0x0000000000000d7c
offset 219796
align 2^2 (4)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __const
segname __TEXT
addr 0x0000000000036810
size 0x0000000000003940
offset 223248
align 2^4 (16)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __unwind_info
segname __TEXT
addr 0x000000000003a150
size 0x0000000000001fe8
offset 237904
align 2^2 (4)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __eh_frame
segname __TEXT
addr 0x000000000003c138
size 0x0000000000003ec8
offset 246072
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Load command 1
cmd LC_SEGMENT_64
cmdsize 232
segname __DATA_CONST
vmaddr 0x0000000000040000
vmsize 0x0000000000004000
fileoff 262144
filesize 16384
maxprot 0x00000003
initprot 0x00000003
nsects 2
flags 0x10
Section
sectname __got
segname __DATA_CONST
addr 0x0000000000040000
size 0x0000000000000030
offset 262144
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000006
reserved1 87 (index into indirect symbol table)
reserved2 0
Section
sectname __const
segname __DATA_CONST
addr 0x0000000000040030
size 0x00000000000020e8
offset 262192
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Load command 2
cmd LC_SEGMENT_64
cmdsize 632
segname __DATA
vmaddr 0x0000000000044000
vmsize 0x0000000000004000
fileoff 278528
filesize 16384
maxprot 0x00000003
initprot 0x00000003
nsects 7
flags 0x0
Section
sectname __la_symbol_ptr
segname __DATA
addr 0x0000000000044000
size 0x00000000000002b8
offset 278528
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000007
reserved1 93 (index into indirect symbol table)
reserved2 0
Section
sectname __data
segname __DATA
addr 0x00000000000442b8
size 0x0000000000000258
offset 279224
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __thread_vars
segname __DATA
addr 0x0000000000044510
size 0x00000000000000c0
offset 279824
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000013
reserved1 0
reserved2 0
Section
sectname __thread_data
segname __DATA
addr 0x00000000000445d0
size 0x0000000000000140
offset 280016
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000011
reserved1 0
reserved2 0
Section
sectname __thread_bss
segname __DATA
addr 0x0000000000044710
size 0x00000000000000a1
offset 0
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000012
reserved1 0
reserved2 0
Section
sectname __common
segname __DATA
addr 0x00000000000447b8
size 0x0000000000000030
offset 0
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000001
reserved1 0
reserved2 0
Section
sectname __bss
segname __DATA
addr 0x00000000000447e8
size 0x0000000000000090
offset 0
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000001
reserved1 0
reserved2 0
Load command 3
cmd LC_SEGMENT_64
cmdsize 72
segname __LINKEDIT
vmaddr 0x0000000000048000
vmsize 0x0000000000024000
fileoff 294912
filesize 133591
maxprot 0x00000001
initprot 0x00000001
nsects 0
flags 0x0
Load command 4
cmd LC_ID_DYLIB
cmdsize 120
name /Users/messense/Projects/cryptography/src/rust/target/release/deps/libcryptography_rust.dylib (offset 24)
time stamp 1 Thu Jan 1 08:00:01 1970
current version 0.0.0
compatibility version 0.0.0
Load command 5
cmd LC_DYLD_INFO_ONLY
cmdsize 48
rebase_off 294912
rebase_size 352
bind_off 295264
bind_size 144
weak_bind_off 0
weak_bind_size 0
lazy_bind_off 295408
lazy_bind_size 2064
export_off 297472
export_size 56
Load command 6
cmd LC_SYMTAB
cmdsize 24
symoff 298304
nsyms 2480
stroff 338704
strsize 86328
Load command 7
cmd LC_DYSYMTAB
cmdsize 80
ilocalsym 0
nlocalsym 2385
iextdefsym 2385
nextdefsym 2
iundefsym 2387
nundefsym 93
tocoff 0
ntoc 0
modtaboff 0
nmodtab 0
extrefsymoff 0
nextrefsyms 0
indirectsymoff 337984
nindirectsyms 180
extreloff 0
nextrel 0
locreloff 0
nlocrel 0
Load command 8
cmd LC_UUID
cmdsize 24
uuid 5308928A-547B-3A3C-BE3D-22D8652852AE
Load command 9
cmd LC_BUILD_VERSION
cmdsize 32
platform 1
minos 11.0
sdk 11.1
ntools 1
tool 3
version 609.8
Load command 10
cmd LC_SOURCE_VERSION
cmdsize 16
version 0.0
Load command 11
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 08:00:02 1970
current version 1292.60.1
compatibility version 1.0.0
Load command 12
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libresolv.9.dylib (offset 24)
time stamp 2 Thu Jan 1 08:00:02 1970
current version 1.0.0
compatibility version 1.0.0
Load command 13
cmd LC_FUNCTION_STARTS
cmdsize 16
dataoff 297528
datasize 776
Load command 14
cmd LC_DATA_IN_CODE
cmdsize 16
dataoff 298304
datasize 0
Load command 15
cmd LC_CODE_SIGNATURE
cmdsize 16
dataoff 425040
datasize 3463 |
|
Similar issue bytecodealliance/wasmtime#984 |
|
Note that you can use $ RUSTFLAGS='-C link-args=-Wl,-install_name,@rpath/libcryptography_rust.dylib' python setup.py build_ext -v
...
$ otool -D build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/bindings/_rust.abi3.so
build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/bindings/_rust.abi3.so:
@rpath/libcryptography_rust.dylib |
|
Thanks @messense for the investigation. Do you think we should be setting e.g. we presumably want this to be |
I am not sure. @dpage-edb Can you try change it using install_name_tool -id '@rpath/_rust.abi3.so' /path/to/_rust.abi3.so |
|
With a minor change to my build scripts to recognise the @rpath prefix as something to ignore along with @executable_path (which I'm fine with, as it's a generic change), yes, it seems to work fine. |
|
@davidhewitt would it be possible to get a release with this |
|
Thanks for the ping @alex. Apologies for the delay, I've been following up on a couple of other threads. I'm planning to release a |
|
Cool, sounds great. Thanks.
…On Thu, Mar 4, 2021 at 6:01 PM David Hewitt ***@***.***> wrote:
Thanks for the ping @alex. Apologies for the delay, I've been following up on a couple of other threads. I'm planning to release a setuptools-rust 0.12 this weekend once I merge a PR to improve the documentation a bit (probably add a readthedocs site).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
--
All that is necessary for evil to succeed is for good people to do nothing.
|
13 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue cloned from the Cryptography issue tracker per request, as setuptools-rust handles this for them: pyca/cryptography#5807
The Cryptography wheel for macOS includes _rust.abi3.so. This references a library that isn't shipped and has a path clearly from the build machine:
Whilst this may not break the package, it:
A) Is not good practice
B) Breaks build systems that automatically fixup library paths when creating relocatable appbundles.
B is the issue I'd like to see fixed, as the current situation requires me to modify code that's worked for many years (15+) to ignore errors when processing this particular file.
An additional issue, again likely benign for the general use case, is that _rust.abi3.so does not have the executable bit set as is generally the norm for shared libraries on *nix platforms. This also required me to change build code, as searches for binaries that need to be codesigned filtered possible targets based on the executable bit, before doing further tests to see if the file was actually an executable or library (which gets very expensive without that check, in a codebase with thousands of Python/JS/HTML files etc).
Probably not relevant, but here's my system info:
Thanks!
The text was updated successfully, but these errors were encountered: