crates with conditionally included pyo3 fail to link (/usr/bin/ld: cannot find -lpython3.10: No such file or directory)

[Trolldemorted]

Bug Description

I am building a parser in rust that needs to have bindings in multiple languages. Since I don't want unecessary copying overhead, I figured it would be nice if I could have one feature per (non-rust) language, and define the right structs and methods with conditional compilation.

Steps to Reproduce

1. Cargo.toml:

[package]
name = "pyo3bug"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
name = "pyo3bug"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version= "0.18.1", features = ["abi3-py37"], optional = true}

[features]
pythonbindings = ["pyo3"]

2. lib.rs:

use std::io;

#[cfg(not(feature="pythonbindings"))]
pub struct Foo {
    pub id: u32,
}

pub fn parse() -> io::Result<Foo> {
    Ok(Foo { id: 42 })
}

#[cfg(feature="pythonbindings")]
use pyo3::*;

#[cfg(feature="pythonbindings")]
#[pyclass]
pub struct Foo {
    #[pyo3(get)]
    id: u32,
}

#[cfg(feature="pythonbindings")]
#[pyfunction]
fn parse_py() -> PyResult<Foo> {
    let foo = parse()?;
    Ok(foo)
}

#[cfg(feature="pythonbindings")]
#[pymodule]
fn pyo3bug(_py: Python, m: &types::PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(parse_py, m)?)?;
    Ok(())
}

3. maturin develop --features pythonbindings

Backtrace

(.venv) benediktradtke@XXXXXX:~/repositories/pyo3test$ maturin develop --features pythonbindings
⚠️  Warning: You're building a library without activating pyo3's `extension-module` feature. See https://pyo3.rs/v0.18.2/building_and_distribution.html#linking
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
🐍 Not using a specific python interpreter
   Compiling pyo3bug v0.1.0 (/home/benediktradtke/repositories/pyo3test)
error: linking with `cc` failed: exit status: 15: pyo3bug                                                                                                                                                                                                                        
  |
  = note: "cc" "-Wl,--version-script=/tmp/rustcw25ZrV/list" "-m64" "/tmp/rustcw25ZrV/symbols.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.13bhaf9em9i0baql.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.1d44cnl99uq7h1bk.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.1l7exc6fc46x87e8.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.1x3tmzjtisau3wr3.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.1yzi7djcb3py0tmh.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.20ddh5ofk6z8l2gz.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.21of21gjmbigksq5.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.22y0ro5rc3g5dk79.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.22ytlhzvrpxkdgvj.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.23pu613xf9d8s8n1.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.243torcjhcug9yko.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.24p1kshw1qnlv9df.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.288nf7b9t20f4l4d.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.28o6s1ibobie6ovn.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.2bera4mhkt2m2gnt.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.2e7qvvltzfkf4jq5.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.2fw534w4jkjeq4gm.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.2h79p66vo5jfb5eg.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.2looqhyzeodpmtfy.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.2m214rh4sulqtxg0.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.2ug3mm9apdcea1fw.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.31hdlvqx208rps9p.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.37d5y0ukua44i4ih.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.3it2pvbco261paok.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.3iz8e1fl9zfyicrx.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.3nmb4oa9ob96w058.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.3s89se80p8bow1e5.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.4499yxgvh8fv0bhb.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.469sr79cvoukv9jy.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.484a4c7bf1kw5l4q.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.4a11my57k38tcs5c.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.4gjq94cnsxu0vxsp.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.4ja3638dqdngc52a.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.4r4nn9nb0n59j6bn.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.4yodhprgqt86h4jg.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.503dbec6y8h1lzfz.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.50zgiv1dhtfux6wm.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.52lkj4lvnfe3w2tg.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.54jckxmcobpteo5t.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.w7d96nuztnleb9z.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.x1llxothnwxu5g8.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.z688nrqpq4pvcp7.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.zgwirfwqfgrqjal.rcgu.o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/pyo3bug.1cts6ngs96iajhum.rcgu.o" "-Wl,--as-needed" "-L" "/home/benediktradtke/repositories/pyo3test/target/debug/deps" "-L" "/usr/lib/x86_64-linux-gnu" "-L" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libpyo3-cb9c86736f45cafd.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libmemoffset-5e291440bc97523c.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libparking_lot-da873ceac0e88c2b.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libparking_lot_core-5369079d00ce6272.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libcfg_if-f7efbe750c4643b0.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libsmallvec-dee5fb08fb47cfad.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/liblock_api-204a3923de61a77e.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libscopeguard-15ffe4de5d288c5a.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libpyo3_ffi-75e9522ed47322ce.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/liblibc-e68dd1e331a2af9d.rlib" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libunindent-408f2ed32440a1ca.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-ef39a505cf65c934.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-15e5018ac4168592.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-a48880ab9d595b7a.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-bc9d2daab0d39757.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-475fcb0e52485504.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-1a6b757a5eafba0e.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-c58dc586d7935f82.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-340edd7bc5378fb5.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-1140b6fd2dbbcaf2.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-0ec50699f9facc73.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-950575da53d6728c.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-94822031c9eedde2.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-8945cf70d2b634e2.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-3e02e71b2a2c7089.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-566a529bcdd2d0ae.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-aa7211f9d7756f6d.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-f09563bc45c7958a.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-a399e1d72279b781.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-0eb6c413e463c03b.rlib" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-91d9d5141f4e57a1.rlib" "-Wl,-Bdynamic" "-lpython3.10" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/benediktradtke/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/benediktradtke/repositories/pyo3test/target/debug/deps/libpyo3bug.so" "-Wl,--gc-sections" "-shared" "-Wl,-zrelro,-znow" "-nodefaultlibs"
  = note: /usr/bin/ld: cannot find -lpython3.10: No such file or directory
          collect2: error: ld returned 1 exit status
          


error: aborting due to previous error


error: could not compile `pyo3bug` due to 2 previous errors
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `PYTHON_SYS_EXECUTABLE="/home/benediktradtke/repositories/pyo3test/.venv/bin/python" "cargo" "rustc" "--features" "pythonbindings" "--manifest-path" "/home/benediktradtke/repositories/pyo3test/Cargo.toml" "--message-format" "json" "--lib"`

Your operating system and version

Ubuntu 22.04.1 LTS

Your Python version (python --version)

Python 3.10.6

Your Rust version (rustc --version)

rustc 1.66.1 (90743e729 2023-01-10)

Your PyO3 version

0.18.2

How did you install python? Did you use a virtualenv?

apt + python3 -m venv .venv

Additional Info

cargo build works fine, cargo build --features pythonbindings fails as expected.

maturin also fails to link if the pyo3 dependency is not optional:

[package]
name = "pyo3bug"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
name = "pyo3bug"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version= "0.18.1", features = ["abi3-py37"]}

[features]
pythonbindings = []

but it does link successfully if the default feature includes the pythonbindings feature:

[package]
name = "pyo3bug"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
name = "pyo3bug"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version= "0.18.1", features = ["abi3-py37"], optional = true}

[features]
pythonbindings = []
default = ["pythonbindings"]

I am at my wits' end, I don't even know whether this is a pyo3 or a tamurin bug.

[adamreichold]

Did you try Maturin's suggestion

⚠️ Warning: You're building a library without activating pyo3's extension-module feature. See https://pyo3.rs/v0.18.2/building_and_distribution.html#linking

to add features = ["extension-module", "abi3-py37"]?

[Trolldemorted]

Thanks for the hint! With

[package]
name = "pyo3bug"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
name = "pyo3bug"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version= "0.18.1", features = ["abi3-py37", "extension-module"], optional = true}

[features]
pythonbindings = ["pyo3"]

it works indeed, but why? Does maturin implicitly enable that feature if I don't use maturin --features foo? Or why would my own features interfere with maturin/pyo3-internal behaviour?

[davidhewitt]

The answer is that without the extension-module feature then PyO3 will try to link to your Python interpreter's shared library (as the assumption is you're building a binary which embeds Python). Presumably you have installed a Python 3.10 interpreter which hasn't shipped with a shared library, which is why the build fails to link.