RuntimeError: Ninja is required to load C++ extension #167

lilingge · 2019-01-17T13:43:41Z

Hi,author. I have followed the instructions on your page. I got your code by git clone, and run "python setup.py install" with no errors. However when I run "python3 demo.py", I got the error like this:

Traceback (most recent call last):
File "/home/llg/.local/lib/python3.5/site-packages/torch/utils/cpp_extension.py", line 873, in verify_ninja_availability
subprocess.check_call('ninja --version'.split(), stdout=devnull)
File "/usr/lib/python3.5/subprocess.py", line 576, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python3.5/subprocess.py", line 557, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 947, in init
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'ninja'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "1.py", line 2, in
import encoding
File "/home/llg/Documents/PyTorch-Encoding/encoding/init.py", line 13, in
from . import nn, functions, parallel, utils, models, datasets, transforms
File "/home/llg/Documents/PyTorch-Encoding/encoding/nn/init.py", line 12, in
from .encoding import *
File "/home/llg/Documents/PyTorch-Encoding/encoding/nn/encoding.py", line 18, in
from ..functions import scaled_l2, aggregate, pairwise_cosine
File "/home/llg/Documents/PyTorch-Encoding/encoding/functions/init.py", line 2, in
from .encoding import *
File "/home/llg/Documents/PyTorch-Encoding/encoding/functions/encoding.py", line 14, in
from .. import lib
File "/home/llg/Documents/PyTorch-Encoding/encoding/lib/init.py", line 15, in
], build_directory=cpu_path, verbose=False)
File "/home/llg/.local/lib/python3.5/site-packages/torch/utils/cpp_extension.py", line 645, in load
is_python_module)
File "/home/llg/.local/lib/python3.5/site-packages/torch/utils/cpp_extension.py", line 814, in _jit_compile
with_cuda=with_cuda)
File "/home/llg/.local/lib/python3.5/site-packages/torch/utils/cpp_extension.py", line 837, in _write_ninja_file_and_build
verify_ninja_availability()
File "/home/llg/.local/lib/python3.5/site-packages/torch/utils/cpp_extension.py", line 875, in verify_ninja_availability
raise RuntimeError("Ninja is required to load C++ extensions")
RuntimeError: Ninja is required to load C++ extension

How to overcome it? What's the "Ninja"?

zhanghang1989 · 2019-01-17T19:17:25Z

Please run following command:

wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
sudo unzip ninja-linux.zip -d /usr/local/bin/
sudo update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force

solution from https://www.claudiokuenzler.com/blog/756/install-newer-ninja-build-tools-ubuntu-14.04-trusty#.XEDUk89KjOB

EmmaSRH · 2019-03-07T01:45:04Z

Hi,author. Does the solution from https://www.claudiokuenzler.com/blog/756/install-newer-ninja-build-tools-ubuntu-14.04-trusty#.XEDUk89KjOB work? I tried it and it still doesn't work.

lilingge · 2019-03-08T08:40:55Z

Hi,author. Does the solution from https://www.claudiokuenzler.com/blog/756/install-newer-ninja-build-tools-ubuntu-14.04-trusty#.XEDUk89KjOB work? I tried it and it still doesn't work.

Hi, my problem was solved.However, the code still can't run. I tried many times, even ask an engineer of Dji for help. Finally, my personal opinion is that the stability of the code is poor. I hope you can run it successfully and let me know. Thanks

nemonameless · 2019-05-07T10:13:11Z

Please run following command:
wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
sudo unzip ninja-linux.zip -d /usr/local/bin/
sudo update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force 
solution from https://www.claudiokuenzler.com/blog/756/install-newer-ninja-build-tools-ubuntu-14.04-trusty#.XEDUk89KjOB

How to do this without sudo Permission?

SysOverdrive · 2020-05-08T10:17:17Z

I seem to have the same problem

Traceback (most recent call last):
Traceback (most recent call last):
File "setup.py", line 26, in
'build_ext': BuildExtension
File "\lib\site-packages\setuptools_init_.py", line 144, in setup
return distutils.core.setup(**attrs)
File "\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\install.py", line 67, in run
self.do_egg_install()
File "\lib\site-packages\setuptools\command\install.py", line 109, in do_egg_install
self.run_command('bdist_egg')
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\bdist_egg.py", line 173, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "\lib\site-packages\setuptools\command\bdist_egg.py", line 159, in call_command
self.run_command(cmdname)
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
self.build()
File "\lib\distutils\command\install_lib.py", line 107, in build
self.run_command('build_ext')
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\build_ext.py", line 87, in run
_build_ext.run(self)
File "\lib\distutils\command\build_ext.py", line 340, in run
self.build_extensions()
File "\lib\site-packages\torch\utils\cpp_extension.py", line 580, in build_extensions
build_ext.build_extensions(self)
File "\lib\distutils\command\build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "\lib\site-packages\setuptools\command\build_ext.py", line 208, in build_extension
_build_ext.build_extension(self, ext)
File "\lib\distutils\command\build_ext.py", line 534, in build_extension
depends=ext.depends)
File "\lib\site-packages\torch\utils\cpp_extension.py", line 562, in win_wrap_ninja_compile
with_cuda=with_cuda)
File "\lib\site-packages\torch\utils\cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "\lib\site-packages\torch\utils\cpp_extension.py", line 1413, in _run_ninja_build
raise RuntimeError(message)
RuntimeError: Error compiling objects for extension

How can I fix this on a windows machine ?

houhaol · 2020-06-01T02:32:36Z

Please run following command:
wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
sudo unzip ninja-linux.zip -d /usr/local/bin/
sudo update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force 
solution from https://www.claudiokuenzler.com/blog/756/install-newer-ninja-build-tools-ubuntu-14.04-trusty#.XEDUk89KjOB
How to do this without sudo Permission?

Just curious. Have you found a method to overcome this sudo permission issue? thanks

BooPatienze · 2020-08-31T07:16:39Z

Please run following command:
wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
sudo unzip ninja-linux.zip -d /usr/local/bin/
sudo update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force 
solution from https://www.claudiokuenzler.com/blog/756/install-newer-ninja-build-tools-ubuntu-14.04-trusty#.XEDUk89KjOB
How to do this without sudo Permission?

I just pip install ninja, and then no problem.

woctezuma · 2020-09-18T13:32:43Z

The simplest solution is sometimes overlooked. Thank you, BooPatienze!

csyhy1986 · 2021-03-23T12:54:33Z

I seem to have the same problem

Traceback (most recent call last):
Traceback (most recent call last):
File "setup.py", line 26, in
'build_ext': BuildExtension
File "\lib\site-packages\setuptools__init__.py", line 144, in setup
return distutils.core.setup(**attrs)
File "\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\install.py", line 67, in run
self.do_egg_install()
File "\lib\site-packages\setuptools\command\install.py", line 109, in do_egg_install
self.run_command('bdist_egg')
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\bdist_egg.py", line 173, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "\lib\site-packages\setuptools\command\bdist_egg.py", line 159, in call_command
self.run_command(cmdname)
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
self.build()
File "\lib\distutils\command\install_lib.py", line 107, in build
self.run_command('build_ext')
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\build_ext.py", line 87, in run
_build_ext.run(self)
File "\lib\distutils\command\build_ext.py", line 340, in run
self.build_extensions()
File "\lib\site-packages\torch\utils\cpp_extension.py", line 580, in build_extensions
build_ext.build_extensions(self)
File "\lib\distutils\command\build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "\lib\site-packages\setuptools\command\build_ext.py", line 208, in build_extension
_build_ext.build_extension(self, ext)
File "\lib\distutils\command\build_ext.py", line 534, in build_extension
depends=ext.depends)
File "\lib\site-packages\torch\utils\cpp_extension.py", line 562, in win_wrap_ninja_compile
with_cuda=with_cuda)
File "\lib\site-packages\torch\utils\cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "\lib\site-packages\torch\utils\cpp_extension.py", line 1413, in _run_ninja_build
raise RuntimeError(message)
RuntimeError: Error compiling objects for extension

How can I fix this on a windows machine ?

I also encounter with this problem on a windows platform, have you solved it bro?

zhanghang1989 · 2021-03-23T16:59:42Z

For Windows, I would recommend using docker https://hangzhang.org/PyTorch-Encoding/notes/compile.html#using-docker

cuge1995 · 2021-05-08T13:46:19Z

Please run following command:
wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
sudo unzip ninja-linux.zip -d /usr/local/bin/
sudo update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force 
solution from https://www.claudiokuenzler.com/blog/756/install-newer-ninja-build-tools-ubuntu-14.04-trusty#.XEDUk89KjOB

It works for me.

tommy-qichang · 2021-10-08T23:33:17Z

I found the subprocess does not include the lib path of conda environments. So I manually set the environments in the script, and it works now.
eg.

my_env = os.environ.copy()
my_env["PATH"] = "/path/to.your/conda/.conda/envs/py3/bin:" + my_env["PATH"]
os.environ.update(my_env)

nom57 · 2022-09-05T11:43:50Z

None of these solutions work for me on Windows

RenzeLou · 2022-09-19T16:05:11Z

I found the subprocess does not include the lib path of conda environments. So I manually set the environments in the script, and it works now. eg.
my_env = os.environ.copy()
my_env["PATH"] = "/path/to.your/conda/.conda/envs/py3/bin:" + my_env["PATH"]
os.environ.update(my_env)

Thanks. It works for me!

liyiersan · 2022-09-26T05:42:49Z

I seem to have the same problem
Traceback (most recent call last):
Traceback (most recent call last):
File "setup.py", line 26, in
'build_ext': BuildExtension
File "\lib\site-packages\setuptools__init__.py", line 144, in setup
return distutils.core.setup(**attrs)
File "\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\install.py", line 67, in run
self.do_egg_install()
File "\lib\site-packages\setuptools\command\install.py", line 109, in do_egg_install
self.run_command('bdist_egg')
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\bdist_egg.py", line 173, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "\lib\site-packages\setuptools\command\bdist_egg.py", line 159, in call_command
self.run_command(cmdname)
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
self.build()
File "\lib\distutils\command\install_lib.py", line 107, in build
self.run_command('build_ext')
File "\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "\lib\site-packages\setuptools\command\build_ext.py", line 87, in run
_build_ext.run(self)
File "\lib\distutils\command\build_ext.py", line 340, in run
self.build_extensions()
File "\lib\site-packages\torch\utils\cpp_extension.py", line 580, in build_extensions
build_ext.build_extensions(self)
File "\lib\distutils\command\build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "\lib\site-packages\setuptools\command\build_ext.py", line 208, in build_extension
_build_ext.build_extension(self, ext)
File "\lib\distutils\command\build_ext.py", line 534, in build_extension
depends=ext.depends)
File "\lib\site-packages\torch\utils\cpp_extension.py", line 562, in win_wrap_ninja_compile
with_cuda=with_cuda)
File "\lib\site-packages\torch\utils\cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "\lib\site-packages\torch\utils\cpp_extension.py", line 1413, in _run_ninja_build
raise RuntimeError(message)
RuntimeError: Error compiling objects for extension
How can I fix this on a windows machine ?

I also encounter with this problem on a windows platform, have you solved it bro?
Same problem on windows, did you solve it?

zhanghang1989 · 2022-09-26T23:16:13Z

Please use the docker solution if you ran into any troubles especially on windows https://hangzhang.org/PyTorch-Encoding/notes/compile.html#using-docker

adityamwagh · 2022-12-14T22:05:54Z

@nom57 A bit off topic, but you might want to use Windows Subsystem for Linux if you are on Windows. It greatly simplifies working with a machine learning workload on Windows.

I use it regularly on Windows 11. It's essentially a virtual machine, although with near native performance.

AndyJZhao · 2022-12-21T12:55:50Z

I found the subprocess does not include the lib path of conda environments. So I manually set the environments in the script, and it works now. eg.
my_env = os.environ.copy()
my_env["PATH"] = "/path/to.your/conda/.conda/envs/py3/bin:" + my_env["PATH"]
os.environ.update(my_env)

Just a few comments on the great answer of tommy-qichang which solves my problem immediately.

The code could be a little bit simpler, as below:

conda_path = "/path/to/your/conda/envs/py3/bin"
os.environ['PATH'] = f'{conda_path}:{os.environ["PATH"]}'

The code should be placed BEFORE GPU related packages are imported. In other words, there shouldn't be any torch/tensorflow/transformers packages before you've set the environment variables correctly.

Ren-Ma · 2023-05-10T02:58:12Z

This solution is the real answer to this question! Thank you @tommy-qichang @AndyJZhao ！

I found the subprocess does not include the lib path of conda environments. So I manually set the environments in the script, and it works now. eg.
my_env = os.environ.copy()
my_env["PATH"] = "/path/to.your/conda/.conda/envs/py3/bin:" + my_env["PATH"]
os.environ.update(my_env)
Just a few comments on the great answer of tommy-qichang which solves my problem immediately.

The code could be a little bit simpler, as below:
conda_path = "/path/to/your/conda/envs/py3/bin"
os.environ['PATH'] = f'{conda_path}:{os.environ["PATH"]}'
The code should be placed BEFORE GPU related packages are imported. In other words, there shouldn't be any torch/tensorflow/transformers packages before you've set the environment variables correctly.

fengchuanpeng · 2024-01-28T15:09:02Z

my_env = os.environ.copy()
my_env["PATH"] = "/path/to.your/conda/.conda/envs/py3/bin:" + my_env["PATH"]
os.environ.update(my_env)
@Ren-Ma Hello, which folder should these codes be placed in?

isrishtisingh · 2024-08-07T10:00:32Z

For Windows, I would recommend using docker https://hangzhang.org/PyTorch-Encoding/notes/compile.html#using-docker

@SysOverdrive and @csyhy1986 I also encountered this issue on my Windows system, doing the following steps has solved the issue for me.

Visit the URL: https://github.com/ninja-build/ninja/releases
Download the zip file named "ninja-win.zip"
Extract the file content from zip, and you will see a .exe file named ninja or ninja-win
Copy and paste the path of this file in your Windows environment path (both in user and system variables path)
Now restart your application/VS Code or whatever.

I am not sure if this is the right solution, but my error is gone now.

y455293511 · 2024-12-08T06:24:29Z

my_env = os.environ.copy()
my_env["PATH"] = "/path/to.your/conda/.conda/envs/py3/bin:" + my_env["PATH"]
os.environ.update(my_env)

This works！

zhanghang1989 mentioned this issue Aug 2, 2020

ADD Docker #305

Merged

zhanghang1989 closed this as completed in #305 Aug 2, 2020

hsfzxjy mentioned this issue Aug 31, 2020

RuntimeError: Ninja is required to load C++ extensions openseg-group/openseg.pytorch#32

Closed

sayakpaul mentioned this issue May 28, 2021

Cifar-10 example - RuntimeError: Error building extension 'fused_adam' microsoft/DeepSpeed#694

Closed

KMFODA mentioned this issue Dec 17, 2021

RuntimeError: Ninja is required to load C++ extensions microsoft/DeepSpeed#1644

Closed

mmuckley mentioned this issue Jan 7, 2022

Dependency and Build Fixes facebookresearch/NeuralCompression#171

Merged

sunset1995 mentioned this issue Feb 3, 2022

The custom CUDA extensions may be uncompatible with PyTorch 1.10.0, possible missing dependency ninja sunset1995/DirectVoxGO#9

Open

npurson mentioned this issue Apr 18, 2022

ModuleNotFoundError: No module named 'yolox.layers.fast_cocoeval' Megvii-BaseDetection/YOLOX#1253

Closed

DGideas mentioned this issue May 8, 2022

[scripts] Convert resharded MP checkpoints to unflattened. facebookresearch/metaseq#60

Merged

ananyahjha93 mentioned this issue Sep 5, 2022

Evaluation loop fails both with and without deepspeed mosaicml/composer#1472

Closed

MrForExample mentioned this issue Feb 26, 2024

ComfyUI Python Embeded Environment ERRORS MrForExample/ComfyUI-3D-Pack#61

Closed

GuanhuaWang mentioned this issue Oct 23, 2024

[BUG] 1: error: must run as root and 2: raise RuntimeError("Ninja is required to load C++ extensions") microsoft/DeepSpeed#5627

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RuntimeError: Ninja is required to load C++ extension #167

RuntimeError: Ninja is required to load C++ extension #167

lilingge commented Jan 17, 2019

zhanghang1989 commented Jan 17, 2019

EmmaSRH commented Mar 7, 2019

lilingge commented Mar 8, 2019

nemonameless commented May 7, 2019

SysOverdrive commented May 8, 2020

houhaol commented Jun 1, 2020

BooPatienze commented Aug 31, 2020

woctezuma commented Sep 18, 2020

csyhy1986 commented Mar 23, 2021

zhanghang1989 commented Mar 23, 2021

cuge1995 commented May 8, 2021

tommy-qichang commented Oct 8, 2021

nom57 commented Sep 5, 2022

RenzeLou commented Sep 19, 2022

liyiersan commented Sep 26, 2022

zhanghang1989 commented Sep 26, 2022

adityamwagh commented Dec 14, 2022

AndyJZhao commented Dec 21, 2022

Ren-Ma commented May 10, 2023

fengchuanpeng commented Jan 28, 2024

isrishtisingh commented Aug 7, 2024

y455293511 commented Dec 8, 2024 •

edited

Loading

RuntimeError: Ninja is required to load C++ extension #167

RuntimeError: Ninja is required to load C++ extension #167

Comments

lilingge commented Jan 17, 2019

zhanghang1989 commented Jan 17, 2019

EmmaSRH commented Mar 7, 2019

lilingge commented Mar 8, 2019

nemonameless commented May 7, 2019

SysOverdrive commented May 8, 2020

houhaol commented Jun 1, 2020

BooPatienze commented Aug 31, 2020

woctezuma commented Sep 18, 2020

csyhy1986 commented Mar 23, 2021

zhanghang1989 commented Mar 23, 2021

cuge1995 commented May 8, 2021

tommy-qichang commented Oct 8, 2021

nom57 commented Sep 5, 2022

RenzeLou commented Sep 19, 2022

liyiersan commented Sep 26, 2022

zhanghang1989 commented Sep 26, 2022

adityamwagh commented Dec 14, 2022

AndyJZhao commented Dec 21, 2022

Ren-Ma commented May 10, 2023

fengchuanpeng commented Jan 28, 2024

isrishtisingh commented Aug 7, 2024

y455293511 commented Dec 8, 2024 • edited Loading

y455293511 commented Dec 8, 2024 •

edited

Loading