pip install is broken with error from libsql-ffi

[avinassh]

I am trying to install the latest v0.0.34 on my machine (Intel Mac, Monterey and Python v3.8.19). It fails with following:

pip install libsql-experimental

Collecting libsql-experimental
  Using cached libsql_experimental-0.0.34.tar.gz (27 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: libsql-experimental
  Building wheel for libsql-experimental (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for libsql-experimental (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [236 lines of output]
      Running `maturin pep517 build-wheel -i /Users/avi/Downloads/venv/bin/python --compatibility off`
      📦 Including license file "/private/var/folders/8_/j9jcz5793r9ft9fy85ft3jn00000gn/T/pip-install-_b3xmnj0/libsql-experimental_6c73ee11e1884c928b0ce7bf6f27a0ff/LICENSE.md"
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.8 at /Users/avi/Downloads/venv/bin/python
      📡 Using build options features from pyproject.toml
      💻 Using `MACOSX_DEPLOYMENT_TARGET=10.7` for x86_64-apple-darwin by default
         Compiling proc-macro2 v1.0.70
         Compiling unicode-ident v1.0.12
         Compiling libc v0.2.151
         Compiling cfg-if v1.0.0
         Compiling version_check v0.9.4
         Compiling autocfg v1.1.0
         Compiling serde v1.0.193
         Compiling pin-project-lite v0.2.13
         Compiling smallvec v1.11.2
         Compiling once_cell v1.19.0
         Compiling log v0.4.20
         Compiling parking_lot_core v0.9.9
         Compiling futures-core v0.3.29
         Compiling scopeguard v1.2.0
         Compiling byteorder v1.5.0
         Compiling futures-task v0.3.29
         Compiling futures-sink v0.3.29
         Compiling futures-channel v0.3.29
         Compiling ahash v0.8.6
         Compiling tracing-core v0.1.32
         Compiling lock_api v0.4.11
         Compiling slab v0.4.9
         Compiling memchr v2.6.4
         Compiling futures-util v0.3.29
         Compiling futures-io v0.3.29
         Compiling getrandom v0.2.11
         Compiling mio v0.8.10
         Compiling num_cpus v1.16.0
         Compiling signal-hook-registry v1.4.1
         Compiling socket2 v0.5.5
         Compiling quote v1.0.33
         Compiling syn v2.0.41
         Compiling allocator-api2 v0.2.16
         Compiling pin-utils v0.1.0
         Compiling itoa v1.0.10
         Compiling either v1.9.0
         Compiling uncased v0.9.9
         Compiling spin v0.9.8
         Compiling parking_lot v0.12.1
         Compiling fnv v1.0.7
         Compiling untrusted v0.9.0
         Compiling rustversion v1.0.14
         Compiling glob v0.3.1
         Compiling prettyplease v0.2.15
         Compiling typenum v1.17.0
         Compiling tower-service v0.3.2
         Compiling rustix v0.38.28
         Compiling target-lexicon v0.12.12
         Compiling equivalent v1.0.1
         Compiling anyhow v1.0.75
         Compiling generic-array v0.14.7
         Compiling bitflags v2.4.1
         Compiling minimal-lexical v0.2.1
         Compiling regex-syntax v0.8.2
         Compiling core-foundation-sys v0.8.6
         Compiling clang-sys v1.6.1
         Compiling cc v1.0.83
         Compiling errno v0.3.8
         Compiling base64 v0.21.5
         Compiling httparse v1.8.0
         Compiling nom v7.1.3
         Compiling libloading v0.7.4
         Compiling rand_core v0.6.4
         Compiling indexmap v1.9.3
         Compiling home v0.5.9
         Compiling try-lock v0.2.5
         Compiling bindgen v0.66.1
         Compiling bitflags v1.3.2
         Compiling ppv-lite86 v0.2.17
         Compiling want v0.3.1
         Compiling pyo3-build-config v0.19.2
         Compiling security-framework-sys v2.9.1
         Compiling core-foundation v0.9.4
         Compiling ring v0.17.7
         Compiling rand_chacha v0.3.1
         Compiling which v4.4.2
         Compiling shlex v1.2.0
         Compiling lazycell v1.3.0
         Compiling async-trait v0.1.74
         Compiling peeking_take_while v0.1.2
         Compiling regex-automata v0.4.3
         Compiling lazy_static v1.4.0
         Compiling rustls-pki-types v1.4.0
         Compiling hashbrown v0.12.3
         Compiling tower-layer v0.3.2
         Compiling rustc-hash v1.1.0
         Compiling httpdate v1.0.3
         Compiling security-framework v2.9.2
         Compiling rand v0.8.5
         Compiling axum-core v0.3.4
         Compiling itertools v0.11.0
         Compiling cexpr v0.6.0
         Compiling siphasher v0.3.11
         Compiling rustls v0.21.10
         Compiling phf_shared v0.11.2
         Compiling block-padding v0.3.3
         Compiling regex v1.10.2
         Compiling axum v0.6.20
         Compiling mime v0.3.17
         Compiling inout v0.1.3
         Compiling crypto-common v0.1.6
         Compiling rustls-pemfile v1.0.4
         Compiling phf_generator v0.11.2
         Compiling rustls v0.22.2
         Compiling matchit v0.7.3
         Compiling sync_wrapper v0.1.2
         Compiling percent-encoding v2.3.1
         Compiling syn v1.0.109
         Compiling phf_codegen v0.11.2
         Compiling rustls-native-certs v0.6.3
         Compiling cipher v0.4.4
         Compiling pyo3-ffi v0.19.2
         Compiling fallible-streaming-iterator v0.1.9
         Compiling thiserror v1.0.51
         Compiling fallible-iterator v0.2.0
         Compiling zeroize v1.7.0
         Compiling subtle v2.5.0
         Compiling webpki-roots v0.25.3
         Compiling libsql-sqlite3-parser v0.11.1 (https://github.com/penberg/libsql/?rev=5a806fe53cc84a5a49853d4ae30bad64b7210c67#5a806fe5)
         Compiling rustls-pemfile v2.1.1
         Compiling memoffset v0.9.0
         Compiling cpufeatures v0.2.12
         Compiling serde_json v1.0.108
         Compiling http-range-header v0.3.1
         Compiling rustls-native-certs v0.7.0
         Compiling aes v0.8.4
         Compiling phf v0.11.2
         Compiling cbc v0.1.2
         Compiling pyo3 v0.19.2
         Compiling webpki-roots v0.26.1
         Compiling serde_derive v1.0.193
         Compiling tokio-macros v2.2.0
         Compiling zerocopy-derive v0.7.31
         Compiling futures-macro v0.3.29
         Compiling tracing-attributes v0.1.27
         Compiling pin-project-internal v1.1.3
         Compiling prost-derive v0.12.3
         Compiling zerocopy v0.7.31
         Compiling async-stream-impl v0.3.5
         Compiling thiserror-impl v1.0.51
         Compiling pin-project v1.1.3
         Compiling tracing v0.1.40
         Compiling async-stream v0.3.5
         Compiling hashbrown v0.14.3
         Compiling pyo3-macros-backend v0.19.2
         Compiling overload v0.1.1
         Compiling ryu v1.0.16
         Compiling fallible-iterator v0.3.0
         Compiling sharded-slab v0.1.7
         Compiling indexmap v2.1.0
         Compiling hashlink v0.8.4
         Compiling nu-ansi-term v0.46.0
         Compiling tracing-log v0.2.0
         Compiling sct v0.7.1
         Compiling rustls-webpki v0.101.7
         Compiling rustls-webpki v0.102.2
         Compiling thread_local v1.1.7
         Compiling indoc v1.0.9
         Compiling unindent v0.1.11
         Compiling tracing-subscriber v0.3.18
         Compiling futures-executor v0.3.29
         Compiling futures v0.3.29
         Compiling libsql-ffi v0.2.1 (https://github.com/penberg/libsql/?rev=5a806fe53cc84a5a49853d4ae30bad64b7210c67#5a806fe5)
         Compiling pyo3-macros v0.19.2
      error: failed to run custom build command for `libsql-ffi v0.2.1 (https://github.com/penberg/libsql/?rev=5a806fe53cc84a5a49853d4ae30bad64b7210c67#5a806fe5)`

      Caused by:
        process didn't exit successfully: `/private/var/folders/8_/j9jcz5793r9ft9fy85ft3jn00000gn/T/pip-install-_b3xmnj0/libsql-experimental_6c73ee11e1884c928b0ce7bf6f27a0ff/target/release/build/libsql-ffi-5955243ed0024a25/build-script-build` (exit status: 101)
        --- stdout
        cargo:rustc-link-lib=framework=Security
        cargo:rerun-if-changed=bundled/src/sqlite3.c
        cargo:rerun-if-changed=bundled/SQLite3MultipleCiphers/build/libsqlite3mc_static.a
        cargo:lib_dir=/private/var/folders/8_/j9jcz5793r9ft9fy85ft3jn00000gn/T/pip-install-_b3xmnj0/libsql-experimental_6c73ee11e1884c928b0ce7bf6f27a0ff/target/release/build/libsql-ffi-9350cb2489e39272/out
        Running `cmake` with options: -DCMAKE_BUILD_TYPE=Release -DSQLITE3MC_STATIC=ON -DCODEC_TYPE=AES256 -DSQLITE3MC_BUILD_SHELL=OFF -DSQLITE_SHELL_IS_UTF8=OFF -DSQLITE_USER_AUTHENTICATION=OFF -DSQLITE_SECURE_DELETE=OFF -DSQLITE_ENABLE_COLUMN_METADATA=ON -DSQLITE_USE_URI=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON
        -- The C compiler identification is AppleClang 14.0.0.14000029
        -- The CXX compiler identification is AppleClang 14.0.0.14000029
        -- Detecting C compiler ABI info
        -- Detecting C compiler ABI info - done
        -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
        -- Detecting C compile features
        -- Detecting C compile features - done
        -- Detecting CXX compiler ABI info
        -- Detecting CXX compiler ABI info - failed
        -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
        -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - broken
        -- Configuring incomplete, errors occurred!

        --- stderr
        CMake Error at /usr/local/Cellar/cmake/3.27.6/share/cmake/Modules/CMakeTestCXXCompiler.cmake:60 (message):
          The C++ compiler

            "/Library/Developer/CommandLineTools/usr/bin/c++"

          is not able to compile a simple test program.

          It fails with the following output:

            Change Dir: '/Users/avi/.cargo/git/checkouts/libsql-e519d6234c1a233a/5a806fe/libsql-ffi/bundled/SQLite3MultipleCiphers/build/CMakeFiles/CMakeScratch/TryCompile-UKacbq'

            Run Build Command(s): /usr/local/Cellar/cmake/3.27.6/bin/cmake -E env VERBOSE=1 /usr/bin/make -f Makefile cmTC_63e55/fast
            /Library/Developer/CommandLineTools/usr/bin/make  -f CMakeFiles/cmTC_63e55.dir/build.make CMakeFiles/cmTC_63e55.dir/build
            Building CXX object CMakeFiles/cmTC_63e55.dir/testCXXCompiler.cxx.o
            /Library/Developer/CommandLineTools/usr/bin/c++   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -mmacosx-version-min=10.7 -fPIE -MD -MT CMakeFiles/cmTC_63e55.dir/testCXXCompiler.cxx.o -MF CMakeFiles/cmTC_63e55.dir/testCXXCompiler.cxx.o.d -o CMakeFiles/cmTC_63e55.dir/testCXXCompiler.cxx.o -c /Users/avi/.cargo/git/checkouts/libsql-e519d6234c1a233a/5a806fe/libsql-ffi/bundled/SQLite3MultipleCiphers/build/CMakeFiles/CMakeScratch/TryCompile-UKacbq/testCXXCompiler.cxx
            clang: warning: include path for libstdc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
            Linking CXX executable cmTC_63e55
            /usr/local/Cellar/cmake/3.27.6/bin/cmake -E cmake_link_script CMakeFiles/cmTC_63e55.dir/link.txt --verbose=1
            /Library/Developer/CommandLineTools/usr/bin/c++  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -mmacosx-version-min=10.7 -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/cmTC_63e55.dir/testCXXCompiler.cxx.o -o cmTC_63e55
            clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
            ld: library not found for -lstdc++
            clang: error: linker command failed with exit code 1 (use -v to see invocation)
            make[1]: *** [cmTC_63e55] Error 1
            make: *** [cmTC_63e55/fast] Error 2





          CMake will not be able to correctly generate this project.
        Call Stack (most recent call first):
          CMakeLists.txt:2 (project)


        thread 'main' panicked at /Users/avi/.cargo/git/checkouts/libsql-e519d6234c1a233a/5a806fe/libsql-ffi/build.rs:360:9:
        Failed to run cmake with options: -DCMAKE_BUILD_TYPE=Release -DSQLITE3MC_STATIC=ON -DCODEC_TYPE=AES256 -DSQLITE3MC_BUILD_SHELL=OFF -DSQLITE_SHELL_IS_UTF8=OFF -DSQLITE_USER_AUTHENTICATION=OFF -DSQLITE_SECURE_DELETE=OFF -DSQLITE_ENABLE_COLUMN_METADATA=ON -DSQLITE_USE_URI=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON
        note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO MACOSX_DEPLOYMENT_TARGET="10.7" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.8-64bit" PYO3_PYTHON="/Users/avi/Downloads/venv/bin/python" PYTHON_SYS_EXECUTABLE="/Users/avi/Downloads/venv/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/private/var/folders/8_/j9jcz5793r9ft9fy85ft3jn00000gn/T/pip-install-_b3xmnj0/libsql-experimental_6c73ee11e1884c928b0ce7bf6f27a0ff/Cargo.toml" "--release" "--lib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/libsql_experimental.cpython-38-darwin.so"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Users/avi/Downloads/venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for libsql-experimental
Failed to build libsql-experimental
ERROR: Could not build wheels for libsql-experimental, which is required to install pyproject.toml-based projects

I tried installing an older version (0.0.28) which had worked earlier, but it is failing with the same error:

error: failed to run custom build command for `libsql-ffi v0.2.1`

      Caused by:
        process didn't exit successfully: `/private/var/folders/8_/j9jcz5793r9ft9fy85ft3jn00000gn/T/pip-install-f36ou_un/libsql-experimental_72f25a9da788466c9e173c4413dd6ac2/target/release/build/libsql-ffi-56ef16cdc9d4469c/build-script-build` (exit status: 101)

...
        thread 'main' panicked at /Users/avi/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libsql-ffi-0.2.1/build.rs:361:9:
        Failed to run cmake with options: -DCMAKE_BUILD_TYPE=Release -DSQLITE3MC_STATIC=ON -DCODEC_TYPE=AES256 -DSQLITE3MC_BUILD_SHELL=OFF -DSQLITE_SHELL_IS_UTF8=OFF -DSQLITE_USER_AUTHENTICATION=OFF -DSQLITE_SECURE_DELETE=OFF -DSQLITE_ENABLE_COLUMN_METADATA=ON -DSQLITE_USE_URI=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON
        note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO MACOSX_DEPLOYMENT_TARGET="10.7" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.8-64bit" PYO3_PYTHON="/Users/avi/Downloads/venv/bin/python" PYTHON_SYS_EXECUTABLE="/Users/avi/Downloads/venv/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/private/var/folders/8_/j9jcz5793r9ft9fy85ft3jn00000gn/T/pip-install-f36ou_un/libsql-experimental_72f25a9da788466c9e173c4413dd6ac2/Cargo.toml" "--release" "--lib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/libsql_experimental.cpython-38-darwin.so"`

Even the older versions point to the same libsql-ffi version (libsql-ffi v0.2.1). This highlights another issue, we are not pinning the dependencies.

---

first reported here - https://discord.com/channels/933071162680958986/1241858417757257809/1241858417757257809

[penberg]

The problem is somewhere in the sqlite3mc CMake-build in https://github.com/tursodatabase/libsql. Looks like some kind of a Apple toolchain change, discussed here for example: ponylang/ponyc#3684

[avinassh]

I found a temporary fix for macOS:

$ export MACOSX_DEPLOYMENT_TARGET=12.6
$ pip install libsql-experimental

Successfully installed libsql-experimental-0.0.34

[Marcus-Peterson]

Anyone being able to solve this on windows? I have the same problem (Windows 11)

[nikhil-swamix]

can confirm the same, this is bad . really bad. DX should be first thing a DBaaS company should focus on. *constructive criticism.
@penberg

[penberg]

@nikhil-swamix Yes. That is why this package is tagged as experimental.

[penberg]

I tweaked the README.md to make the experimental status more explicit: f4ff8ad

[nikhil-swamix]

agreed the package name is explicit enough. btw the bun guys were able to support windows eventually maybe some references can be taken from there. currently i want to use turso with python, if i use default sql library (inbuilt) or other flavors, lets say peewee or sqlalchemy,

1. will vsearch break? sync apis supported? or its squares and triangles...
2. any alternatives suggested?

bun was suggested as its sqlite driver is 2x faster than node, maybe it could be used as python cffi, rather than rust? feasible? @penberg

vector search+gold standard of local development ie sqlite, is one of the reason im testing before recommending (mostly startups), and also the generous free tier. please let me know very approx ETA as per your estimates on resolution of this.