SystemError: initialization of _heapq did not return an extension module #150

OliverEvans96 · 2018-03-25T14:36:48Z

I'm experiencing the same error mentioned by @sirgogo in #88 .

oliver@oliver-arch:~ % ipython
Python 3.5.5 |Anaconda, Inc.| (default, Mar 12 2018, 23:12:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import julia

In [2]: j = julia.Julia()
Error processing line 1 of /home/oliver/local/miniconda3/lib/python3.5/site-packages/PyUtilib-5.4.1-py2.7-nspkg.pth:

Failed to import the site module
Traceback (most recent call last):
  File "/home/oliver/local/miniconda3/lib/python3.5/site.py", line 167, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/home/oliver/local/miniconda3/lib/python3.5/types.py", line 166, in <module>
    import functools as _functools
  File "/home/oliver/local/miniconda3/lib/python3.5/functools.py", line 21, in <module>
    from collections import namedtuple
  File "/home/oliver/local/miniconda3/lib/python3.5/collections/__init__.py", line 29, in <module>
    import heapq as _heapq
  File "/home/oliver/local/miniconda3/lib/python3.5/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/oliver/local/miniconda3/lib/python3.5/site.py", line 559, in <module>
    main()
  File "/home/oliver/local/miniconda3/lib/python3.5/site.py", line 541, in main
    known_paths = addsitepackages(known_paths)
  File "/home/oliver/local/miniconda3/lib/python3.5/site.py", line 318, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/home/oliver/local/miniconda3/lib/python3.5/site.py", line 206, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/home/oliver/local/miniconda3/lib/python3.5/site.py", line 177, in addpackage
    import traceback
  File "/home/oliver/local/miniconda3/lib/python3.5/traceback.py", line 3, in <module>
    import collections
  File "/home/oliver/local/miniconda3/lib/python3.5/collections/__init__.py", line 29, in <module>
    import heapq as _heapq
  File "/home/oliver/local/miniconda3/lib/python3.5/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

I'm running Arch Linux and using the following versions:

Package	Version
python	3.5.5 :: Anaconda, Inc.
julia	0.6.2
PyJulia	0.1.5
PyCall	1.15.0

Any advice is appreciated!

Thanks,
Oliver

SHTILL · 2018-05-11T12:07:25Z

Have exactly the same error:
JULIA_HOME = /home/vasya/dist/julia/bin, libjulia_path = /home/vasya/dist/julia/bin/../lib/libjulia.so.0.6
calling jl_init_with_image(b'/home/vasya/anaconda3/lib/python3.6/site-packages/julia/fake-julia', /home/vasya/dist/julia/lib/julia/sys.so)
seems to work...
exception occured? None
exception occured? None
exception occured? None
exception occured? None
Error processing line 1 of /home/vasya/anaconda3/lib/python3.6/site-packages/matplotlib-2.1.2-py3.6-nspkg.pth

matteoacrossi · 2018-05-17T11:54:13Z

I have a similar error but it happens in matplotlib

SystemError: initialization of _heapq did not return an extension module

Python 3.5.5 |Anaconda custom (64-bit)| (default, Mar 12 2018, 16:25:05) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import julia
>>> j = julia.Julia()
Error processing line 1 of /Users/matteo/anaconda/lib/python3.5/site-packages/matplotlib-2.2.2-py3.5-nspkg.pth:

Failed to import the site module
Traceback (most recent call last):
  File "/Users/matteo/anaconda/lib/python3.5/site.py", line 167, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/Users/matteo/anaconda/lib/python3.5/types.py", line 166, in <module>
    import functools as _functools
  File "/Users/matteo/anaconda/lib/python3.5/functools.py", line 21, in <module>
    from collections import namedtuple
  File "/Users/matteo/anaconda/lib/python3.5/collections/__init__.py", line 29, in <module>
    import heapq as _heapq
  File "/Users/matteo/anaconda/lib/python3.5/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/matteo/anaconda/lib/python3.5/site.py", line 559, in <module>
    main()
  File "/Users/matteo/anaconda/lib/python3.5/site.py", line 541, in main
    known_paths = addsitepackages(known_paths)
  File "/Users/matteo/anaconda/lib/python3.5/site.py", line 318, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/Users/matteo/anaconda/lib/python3.5/site.py", line 206, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/Users/matteo/anaconda/lib/python3.5/site.py", line 177, in addpackage
    import traceback
  File "/Users/matteo/anaconda/lib/python3.5/traceback.py", line 3, in <module>
    import collections
  File "/Users/matteo/anaconda/lib/python3.5/collections/__init__.py", line 29, in <module>
    import heapq as _heapq
  File "/Users/matteo/anaconda/lib/python3.5/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

carmagnole · 2018-05-24T09:01:01Z

I have the same problem:

julia.Julia(debug = True)
JULIA_HOME = /Applications/Julia-0.6.app/Contents/Resources/julia/bin, libjulia_path = /Applications/Julia-0.6.app/Contents/Resources/julia/lib/libjulia.0.6.2.dylib
calling jl_init_with_image(b'/Applications/Julia-0.6.app/Contents/Resources/julia/bin', /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib)
seems to work...
exception occured? None
Failed to import the site module
Traceback (most recent call last):
File "/anaconda3/envs/Neuroscience/lib/python3.6/site.py", line 541, in
main()
File "/anaconda3/envs/Neuroscience/lib/python3.6/site.py", line 522, in main
known_paths = addusersitepackages(known_paths)
File "/anaconda3/envs/Neuroscience/lib/python3.6/site.py", line 282, in addusersitepackages
user_site = getusersitepackages()
File "/anaconda3/envs/Neuroscience/lib/python3.6/site.py", line 258, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/anaconda3/envs/Neuroscience/lib/python3.6/site.py", line 248, in getuserbase
USER_BASE = get_config_var('userbase')
File "/anaconda3/envs/Neuroscience/lib/python3.6/sysconfig.py", line 608, in get_config_var
return get_config_vars().get(name)
File "/anaconda3/envs/Neuroscience/lib/python3.6/sysconfig.py", line 587, in get_config_vars
import _osx_support
File "/anaconda3/envs/Neuroscience/lib/python3.6/_osx_support.py", line 4, in
import re
File "/anaconda3/envs/Neuroscience/lib/python3.6/re.py", line 122, in
import enum
File "/anaconda3/envs/Neuroscience/lib/python3.6/enum.py", line 2, in
from types import MappingProxyType, DynamicClassAttribute
File "/anaconda3/envs/Neuroscience/lib/python3.6/types.py", line 171, in
import functools as _functools
File "/anaconda3/envs/Neuroscience/lib/python3.6/functools.py", line 21, in
from collections import namedtuple
File "/anaconda3/envs/Neuroscience/lib/python3.6/collections/init.py", line 29, in
import heapq as _heapq
File "/anaconda3/envs/Neuroscience/lib/python3.6/heapq.py", line 587, in
from _heapq import *
SystemError: initialization of _heapq did not return an extension module

randy3k · 2018-05-26T19:22:14Z

I have a similar issue on my conda python, but not the homebrew python.

It may be because JuliaCall fails to detect if (anaconda/miniconda) python has been initialized.

I got the following from my miniconda python

In [1]: import ctypes

In [2]: libpython = ctypes.PyDLL("/Users/Randy/miniconda3/lib/libpython3.6m.dylib")

In [3]: libpython.Py_IsInitialized()
Out[3]: 0

I got the following from my homebrew python

In [1]: import ctypes

In [2]: libpython = ctypes.PyDLL("/usr/local/Cellar/python3/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/libpython3.6.dylib")

In [3]: libpython.Py_IsInitialized()
Out[4]: 1

UPDATE:

The following code works in miniconda python

In [1]: import ctypes

In [2]: libpython = ctypes.CDLL(None)

In [3]: libpython.Py_IsInitialized()
Out[3]: 1

And it seems that PyCall already loads in process symbols JuliaPy/PyCall.jl#293
Not sure the reason now.

bstellato · 2018-10-24T22:44:00Z

I am having the same issue with miniconda. I am also getting the same outputs as in #175 (comment) Any ideas why it cannot detect if python has been initialized?

tkf · 2018-10-25T01:53:31Z

Hmm... I thought I fixed #175 by this commit. Does it happen with PyJulia master as well?

I remember it happened with Julia 0.6 when the precompilation failed.

bstellato · 2018-10-25T03:34:40Z

Yes I am using the master and Julia 1.0. I deleted the ~/.julia/compiled/v1.0/PyCall/xxx.ji file before but it does not seem to help. The ctypes code here in #150 (comment) gives the same answer for me.

tkf · 2018-10-25T03:54:10Z

Julia 1.0 with miniconda is not supported ATM so it is supposed to print some informative error message. Are you using Linux?

If you use miniconda in Linux, python-jl is probably the best way to make it work: https://github.com/tkf/pyjulia/blob/readme/README.md#your-python-interpreter-is-statically-linked-to-libpython

bstellato · 2018-10-25T04:15:16Z

I am using Mac but it actually works with python-jl and Julia 1.0. Thanks a lot! However, it would be great to have the anaconda python working without that hack.

tkf · 2018-10-25T04:28:51Z

Good that it works!

BTW, in your comment you are mentioning you tried ldd, right? Meaning that

ldd PATH/TO/YOUR/python

doesn't show libpython? Do you actually have ldd in macOS? I thought it's different command.

Additionally, can you run the following in miniconda python (not python-jl) you were using?

from julia.find_libpython import find_libpython, linked_libpython
linked_libpython()  # what does it return?
find_libpython()    # what does it return?

I thought macOS didn't need python-jl. But if linked_libpython() above returns None it means our assumption was not correct...

it would be great to have the anaconda python working without that hack

Yeah I know... But I can't do anything now. 😿

bstellato · 2018-10-25T04:43:55Z

Sorry, I meant otool -L! Unfortunately it does not list anything related to libpython.

I have tried your commands.

linked_libpython() returns None.
find_libpython() returns the right library /Users/xxx/miniconda3/envs/python36/lib/libpython3.6m.dylib

Thanks for the quick responses by the way :)

tkf · 2018-10-25T04:54:42Z

You are welcome :) And thanks for trying the commands. Good to figure out that linked_libpython() can return None in macOS (I mean, it's bad, but better than not knowing it...).

It turned out macOS can have statically linked Python when it's installed via conda: JuliaPy#150 (comment) So it seems `linked_libpython` (which calls `libdl`) is the only way to reliably detect if the Python executable is statically linked or not. Since cd2e408 implements it for Windows as well, we can simplify core.py.

It turned out macOS can have statically linked Python when it's installed via conda: JuliaPy#150 (comment) So it seems `linked_libpython` (which calls `libdl`) is the only way to reliably detect if the Python executable is statically linked or not. Since cd2e408 implements it for Windows as well, we can simplify core.py a lot by relying on `linked_libpython`.

It turned out macOS can have statically linked Python when it's installed via conda: JuliaPy#150 (comment) So it seems `linked_libpython` (which calls `libdl`) is the only way to reliably detect if the Python executable is statically linked or not. Since cd2e408 implements it for Windows as well, we can now rely on `linked_libpython` everywhere which simplifies core.py.

It turned out macOS can have statically linked Python when it's installed via conda: #150 (comment) So it seems `linked_libpython` (which calls `libdl`) is the only way to reliably detect if the Python executable is statically linked or not. Since cd2e408 implements it for Windows as well, we can now rely on `linked_libpython` everywhere which simplifies core.py.

sblinz-zz · 2018-11-05T20:16:09Z

Removing ~/.julia/lib/v0.6/PyCall.jl worked for me. System info below:

OS: Ubuntu 16.04 (running inside a Docker container)
Julia: 0.6
Python: 3.5.2
Virtual Environment: Conda (rllab, https://github.com/rll/rllab/blob/master/environment.yml)

tkf mentioned this issue Aug 1, 2018

A script for reproducing the initialization error #175

Closed

tkf mentioned this issue Oct 25, 2018

Handle statically linked case in macOS as well #212

Merged

tkf closed this as completed in #212 Oct 28, 2018

ericphanson mentioned this issue Feb 6, 2023

Update model registry and "List of Models" to reflect addition of CatBoost models JuliaAI/MLJModels.jl#501

Closed

jakirkham mentioned this issue Aug 30, 2024

Test failures (cython-related?) on osx-64 with CPython 3.13 in conda-forge numpy/numpy#27314

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SystemError: initialization of _heapq did not return an extension module #150

SystemError: initialization of _heapq did not return an extension module #150

OliverEvans96 commented Mar 25, 2018 •

edited

Loading

SHTILL commented May 11, 2018 •

edited

Loading

matteoacrossi commented May 17, 2018

carmagnole commented May 24, 2018

randy3k commented May 26, 2018 •

edited

Loading

bstellato commented Oct 24, 2018

tkf commented Oct 25, 2018

bstellato commented Oct 25, 2018

tkf commented Oct 25, 2018

bstellato commented Oct 25, 2018

tkf commented Oct 25, 2018

bstellato commented Oct 25, 2018

tkf commented Oct 25, 2018

sblinz-zz commented Nov 5, 2018 •

edited

Loading

SystemError: initialization of _heapq did not return an extension module #150

SystemError: initialization of _heapq did not return an extension module #150

Comments

OliverEvans96 commented Mar 25, 2018 • edited Loading

SHTILL commented May 11, 2018 • edited Loading

matteoacrossi commented May 17, 2018

carmagnole commented May 24, 2018

randy3k commented May 26, 2018 • edited Loading

bstellato commented Oct 24, 2018

tkf commented Oct 25, 2018

bstellato commented Oct 25, 2018

tkf commented Oct 25, 2018

bstellato commented Oct 25, 2018

tkf commented Oct 25, 2018

bstellato commented Oct 25, 2018

tkf commented Oct 25, 2018

sblinz-zz commented Nov 5, 2018 • edited Loading

OliverEvans96 commented Mar 25, 2018 •

edited

Loading

SHTILL commented May 11, 2018 •

edited

Loading

randy3k commented May 26, 2018 •

edited

Loading

sblinz-zz commented Nov 5, 2018 •

edited

Loading