Python 3.12 support #6422

yarons · 2024-04-01T08:24:57Z

I'm not sure if this is an enhancement or a bug but I tried running the project inside a Python 3.12 container and I ran into several errors because of the NumPy version later on I encountered an error with the protobuf package.

paschembri · 2024-04-05T11:19:07Z

The requirements installation fails when using python 3.12 (homebrew / macos / apple silicon)

➜  llama.cpp git:(add-info-about-python-version) ✗ python3.12 -m venv venv12
➜  llama.cpp git:(add-info-about-python-version) ✗ source venv12/bin/activate
(venv12) ➜  llama.cpp git:(add-info-about-python-version) ✗ python3 -m pip install -r requirements.txt
Collecting numpy~=1.24.4 (from -r ./requirements/requirements-convert.txt (line 1))
  Using cached numpy-1.24.4.tar.gz (10.9 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
           ^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
                ^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 182, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
    super().__init__(
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 54, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 124, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 101, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 745, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/Users/pas/development/advanced-stack/experiments/llama.cpp/venv12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/private/var/folders/yq/85n50qvj5t5b40dhs8trfk500000gn/T/pip-build-env-mhm0v3jw/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'

MikeLP · 2024-04-06T00:12:32Z

Try to install setuptools first to fix distutils issue.

davidprokopec · 2024-04-17T11:35:39Z

Same error, tried running pip install setuptools first but ran into another error.

❯ pip install -r requirements.txt
Collecting numpy~=1.24.4 (from -r ./requirements/requirements-convert.txt (line 1))
  Using cached numpy-1.24.4.tar.gz (10.9 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "/Users/davidprokopec/work/repos/tools/ollama/llm/llama.cpp/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/davidprokopec/work/repos/tools/ollama/llm/llama.cpp/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/davidprokopec/work/repos/tools/ollama/llm/llama.cpp/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "/Users/davidprokopec/work/repos/tools/ollama/llm/llama.cpp/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 995, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/private/var/folders/3v/_lc4b1sj7656q1g5g8zlwj4m0000gn/T/pip-build-env-tjvz02n2/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
          import setuptools.version
        File "/private/var/folders/3v/_lc4b1sj7656q1g5g8zlwj4m0000gn/T/pip-build-env-tjvz02n2/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
          import pkg_resources
        File "/private/var/folders/3v/_lc4b1sj7656q1g5g8zlwj4m0000gn/T/pip-build-env-tjvz02n2/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

On M1 pro

jboero · 2024-04-27T14:27:54Z

This error is killing automation checks too. I'm not even doing anything with Python but an error kills the test Actions. I think this is a bigger probem if our own automation environment has been updated.

ModuleNotFoundError: No module named 'distutils'

bani · 2024-04-27T18:26:35Z

I was able to complete pip install and get the convert script to run after updating some requirements:

numpy~=1.26.4
torch~=2.3.0
sentencepiece~=0.2.0

I had also installed setuptools manually, but at this point I'm not sure if it was a required step with this setup.

But I wasn't able to run it successfully so far and I don't know if the problem is with the models or the script.

For the model in the tutorial (Vicuna), it ended in a BadZipFile Exception:

  File "/Users/bani/code/llm/llama.cpp/convert.py", line 899, in load
    return UnquantizedTensor(storage.load(storage_offset, elm_count).reshape(size))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bani/code/llm/llama.cpp/convert.py", line 883, in load
    with self.zip_file.open(info) as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/zipfile/__init__.py", line 1643, in open
    raise BadZipFile(f"Overlapped entries: {zinfo.orig_filename!r} (possible zip bomb)")
zipfile.BadZipFile: Overlapped entries: 'pytorch_model-00001-of-00003/data/2' (possible zip bomb)

And I also tried with OpenELM-3B-Instruct, which ended in
KeyError: 'transformer.layers.0.attn.k_norm.weight'

chrisalbertson · 2024-05-22T17:03:53Z

I was able to complete pip install and get the convert script to run after updating some requirements:

numpy~=1.26.4 torch~=2.3.0 sentencepiece~=0.2.0

This was very helpful and at least got the requirements to install. But now it seems I get the same "zip file" errors. It claims CRC-32 errors in the files. This will take some time to track down. I can't believe this stuff ever worked.

teleprint-me · 2024-05-23T01:33:16Z

Use safetensors. I looked into CRC, but haven't had enough time. convert-hf-to-gguf.py will eventually supersede convert.py.

oldgithubman · 2024-05-30T20:28:57Z

The python version requirement should be clearly stated in the main readme

beginor · 2024-06-01T10:20:45Z

I have created a venv with python 3.12 too, have the same issue too !

oldgithubman · 2024-06-01T19:22:05Z

I have created a venv with python 3.12 too, have the same issue too !

Use 3.11

Galunid · 2024-06-21T01:14:12Z

Should be fixed in #8041

yarons added the bug-unconfirmed label Apr 1, 2024

notatallshaw mentioned this issue Apr 17, 2024

AttributeError: module 'pkgutil' has no attribute 'ImpImporter' pypa/pip#12629

Closed

1 task

jboero mentioned this issue Apr 27, 2024

Added server example themes support with two sample themes and a favicon. #6848

Merged

pdevine mentioned this issue Apr 29, 2024

ModuleNotFoundError: No module named 'distutils' ollama/ollama#4024

Closed

Galunid closed this as completed Jun 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.12 support #6422

Python 3.12 support #6422

yarons commented Apr 1, 2024

paschembri commented Apr 5, 2024

MikeLP commented Apr 6, 2024

davidprokopec commented Apr 17, 2024 •

edited

Loading

jboero commented Apr 27, 2024 •

edited

Loading

bani commented Apr 27, 2024

chrisalbertson commented May 22, 2024

teleprint-me commented May 23, 2024

oldgithubman commented May 30, 2024

beginor commented Jun 1, 2024

oldgithubman commented Jun 1, 2024

Galunid commented Jun 21, 2024

Python 3.12 support #6422

Python 3.12 support #6422

Comments

yarons commented Apr 1, 2024

paschembri commented Apr 5, 2024

MikeLP commented Apr 6, 2024

davidprokopec commented Apr 17, 2024 • edited Loading

jboero commented Apr 27, 2024 • edited Loading

bani commented Apr 27, 2024

chrisalbertson commented May 22, 2024

teleprint-me commented May 23, 2024

oldgithubman commented May 30, 2024

beginor commented Jun 1, 2024

oldgithubman commented Jun 1, 2024

Galunid commented Jun 21, 2024

davidprokopec commented Apr 17, 2024 •

edited

Loading

jboero commented Apr 27, 2024 •

edited

Loading