import juliacall sometimes segfaults

[LilithHafner]

Affects: JuliaCall

Describe the bug
I sometimes get segfaults when importing julicall in combination with other packages.

x@x ~ % python -c 'import juliacall; import torch; from torch.autograd import Function'
x@x ~ % python -c 'import torch; from torch.autograd import Function; import juliacall'
x@x ~ % python -c 'import torch; from torch.autograd import Function; import juliacall'
zsh: segmentation fault  python3 -c 
x@x ~ % python -c 'import torch; from torch.autograd import Function; import juliacall'
zsh: segmentation fault  python3 -c 
x@x ~ % python -c 'import torch; from torch.autograd import Function; import juliacall'
zsh: segmentation fault  python3 -c 
x@x ~ % python -c 'import torch; from torch.autograd import Function; import juliacall'
zsh: segmentation fault  python3 -c 

Your system

x@x ~ % python --version
Python 3.11.2
x@x ~ % julia -e 'using InteractiveUtils; versioninfo()'
Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M2
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 4 virtual cores
Environment:
  JULIA_EDITOR = code
  JULIA_PYTHONCALL_EXE = /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
x@x ~ % python -m pip list                              
Package                   Version
------------------------- ---------
absl-py                   2.0.0
accelerate                0.23.0
aiohttp                   3.8.4
aiosignal                 1.3.1
astunparse                1.6.3
async-timeout             4.0.2
attrs                     22.2.0
cachetools                5.3.1
certifi                   2022.12.7
cffi                      1.15.1
charset-normalizer        3.1.0
click                     8.1.3
codespell                 2.2.6
comtypes                  1.1.14
contourpy                 1.0.7
cycler                    0.11.0
diffeqpy                  2.1.0
diffusers                 0.21.1
Faker                     19.10.0
filelock                  3.12.4
fire                      0.5.0
flatbuffers               23.5.26
fonttools                 4.39.3
frozenlist                1.3.3
fsspec                    2023.9.0
gast                      0.4.0
google-auth               2.23.0
google-auth-oauthlib      1.0.0
google-pasta              0.2.0
grpcio                    1.58.0
gTTS                      2.3.1
h5py                      3.9.0
huggingface-hub           0.17.1
idna                      3.4
importlib-metadata        6.8.0
jill                      0.11.5
Jinja2                    3.1.2
jsonschema                4.19.0
jsonschema-specifications 2023.7.1
juliacall                 0.9.14
juliapkg                  0.1.10
keras                     2.13.1
kiwisolver                1.4.4
libclang                  16.0.6
llvmlite                  0.41.0
Markdown                  3.4.4
MarkupSafe                2.1.3
matplotlib                3.7.1
mecab-python3             1.0.6
mpmath                    1.3.0
multidict                 6.0.4
nanoid                    2.0.0
networkx                  3.1
numba                     0.58.0
numpy                     1.24.2
oauthlib                  3.2.2
openai                    0.27.2
opencv-python             4.7.0.72
opt-einsum                3.3.0
packaging                 23.1
Pillow                    9.5.0
pip                       23.2.1
playsound                 1.3.0
protobuf                  4.24.3
psutil                    5.9.5
pyasn1                    0.5.0
pyasn1-modules            0.3.0
pycparser                 2.21
pyobjc-framework-Vision   9.0.1
pyparsing                 3.0.9
python-dateutil           2.8.2
python-gnupg              0.5.1
pyttsx3                   2.91
PyYAML                    6.0.1
referencing               0.30.2
regex                     2023.8.8
requests                  2.28.2
requests-futures          1.0.1
requests-oauthlib         1.3.1
rlvoice                   1.0.0
rpds-py                   0.10.2
rsa                       4.9
safetensors               0.3.3
semantic-version          2.10.0
setuptools                65.5.0
six                       1.16.0
sounddevice               0.4.6
srt                       3.5.3
sympy                     1.12
tensorboard               2.13.0
tensorboard-data-server   0.7.1
tensorflow                2.13.0
tensorflow-estimator      2.13.0
tensorflow-macos          2.13.0
termcolor                 2.3.0
tokenizers                0.13.3
torch                     2.0.1
tqdm                      4.65.0
transformers              4.33.1
typing_extensions         4.5.0
urllib3                   1.26.15
vosk                      0.3.44
websockets                10.4
Werkzeug                  2.3.7
wget                      3.2
wheel                     0.41.2
wrapt                     1.15.0
yarl                      1.8.2
zipp                      3.16.2
x@x ~ % julia                                           
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.3 (2023-08-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> juliapkg.status()
ERROR: UndefVarError: `juliapkg` not defined
Stacktrace:
 [1] top-level scope
   @ REPL[1]:1

julia> 
x@x ~ % python -c 'import juliacall; juliacall.juliapkg.status()'
┌ Warning: attempting to remove probably stale pidfile
│   path = "/Users/x/.julia/logs/manifest_usage.toml.pid"
└ @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.9.3+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/FileWatching/src/pidfile.jl:244
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'juliacall' has no attribute 'juliapkg'

[cjdoris]

Yeah it's a known issue. See also #215.

[cjdoris]

That issue suggested printing a warning if torch is loaded before juliacall, which was never implemented.

[willow-ahrens]

I am still seeing this error on macos, but I see the error when loading juliacall after numba. Here's the CI that is failing: pydata/sparse#767. I'm willing to put some development effort into this, but it's a little bit out of my wheelhouse. I suspect the issue is also with llvm symbols. Does anyone know what it would take to fix this? Is a PR to numba required?