ModuleNotFoundError: No module named "vllm._C"

[Kawai1Ace]

[WoosukKwon]

Hi @Kawai1Ace, it seems you are using the latest main branch of vLLM. Did you install vLLM from source? The vllm._C module is built when you install vLLM.

[allenhaozi]

in my case same install command, but install different python version

lead to the problem

https://github.com/vllm-project/vllm/blob/main/Dockerfile#L4
https://github.com/vllm-project/vllm/blob/main/Dockerfile#L48

Make sure they use the same version of python
if you use docker
change https://github.com/vllm-project/vllm/blob/main/Dockerfile#L4

RUN apt-get update -y \
    && apt-get install -y python3-pip

to

RUN apt-get update -y \
    && apt-get install -y python3.10 python3.10-dev

# Download and install pip for Python 3.10
RUN apt-get install -y curl && \
    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.10 get-pip.py

RUN ln -fs /usr/bin/python3.10 /usr/bin/python && ln -fs /usr/bin/python3.10 /usr/bin/python3

[Kawai1Ace]

@WoosukKwon
when I run the command pip install -e . ,ome errors occur

[hxdbf]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

[CheneyJeme]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

errors happens: git clone download the vllm project, and build failed; under the same folder, try pip install vllm, success; just remove the cloned vllm project.

[gunpal5]

I am trying to run vllm on WSL Ubuntu. I just installed vllm through pip and getting this error:

/usr/bin/python3: Error while finding module specification for 'vllm.entrypoints.openai.api_server' (ModuleNotFoundError: No module named 'vllm')

commands used:
pip install vllm
sudo python3 -m vllm.entrypoints.openai.api_server --model 01-ai/Yi-6B-Chat-8bits

[hxdbf]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

I think I should say it more clearly. There are two methods to install vllm, one is to use pip, and the other is to build and install from the source code. If using pip install vllm, you don't need and shouldn't clone this project because the "vllm" folder in the project will cause confusion during imports. If building and installing from the source, the '_C.cpython-xxx-linux-gnu.so' file will be built, and then you can import vllm._C.
In fact, there are some subtle differences between installing vllm with pip and installing it from the source code. For example, in the ninth line of utils.py, it's from vllm import cuda_utils under pip installation and from vllm._C import cuda_utils under source code installation. The latter is the reason for the error ModuleNotFoundError: No module named 'vllm._C'.
I assume you initially installed vllm using pip. Later, when you needed to run examples/llm_engine_example.py, you cloned the VLLM project, resulting in this situation—similar to my experience.

[njhill]

This should be fixed by #2845

[debate1]

@hxdbf If I encounter an issue when running the following command after cloning the repository:
venv-vllm/bin/python -m vllm.entrypoints.openai.api_server \ --model /hy-tmp/tigerbot-13b-chat-v5-4k \ --enable-lora \ --lora-modules sql-lora=/hy-tmp/tigerbot-13b-chat-v4-lora
what should I do？

[13140438775]

可以通过安装 python3-dev 包来解决这个问题：
sudo apt-get install python3-dev
对于其他操作系统，请确保安装了相应的 Python 开发包。

[zhu111-eng]

@WoosukKwon when I run the command pip install -e . ,ome errors occur

May I ask if you solved it, I had the same problem as you did

[yaokunkun]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

Good!

[Sanket-Kudtarkar-AI]

@gunpal5 Did you solve this issue?

[MojHnd]

@Sanket-Kudtarkar-AI @gunpal5
Did you solve this issue?

[MojHnd]

Hi @Kawai1Ace, it seems you are using the latest main branch of vLLM. Did you install vLLM from source? The vllm._C module is built when you install vLLM.

I successfully installed vllm-0.4.0.post1+neuron213.

In setup.py, there is this function:

if not _is_neuron():
    ext_modules.append(CMakeExtension(name="vllm._C"))

and

cmdclass={"build_ext": cmake_build_ext} if not _is_neuron() else {},

So, vllm._C won't be created. This results in ModuleNotFoundError: No module named 'vllm._C'.

How to fix it?

[papandadj]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

niubi

[MojHnd]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

niubi

I do not think so. Mine is like the following:

File ~/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/layers/activation.py:9
6 import torch.nn as nn
7 import torch.nn.functional as F
----> 9 from vllm._C import ops
10 from vllm.model_executor.layers.quantization import QuantizationConfig
11 from vllm.model_executor.parallel_utils.parallel_state import (
12 get_tensor_model_parallel_rank, get_tensor_model_parallel_world_size)

ModuleNotFoundError: No module named 'vllm._C'

[papandadj]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

niubi

I do not think so. Mine is like the following:

File ~/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/layers/activation.py:9
6 import torch.nn as nn
7 import torch.nn.functional as F
----> 9 from vllm._C import ops
10 from vllm.model_executor.layers.quantization import QuantizationConfig
11 from vllm.model_executor.parallel_utils.parallel_state import (
12 get_tensor_model_parallel_rank, get_tensor_model_parallel_world_size)

ModuleNotFoundError: No module named 'vllm._C'

I think you should run this command in the 'vllm' folder. You can create a new folder, such as 'run',and navigate into the 'run' folder.Then, you can run the command again

[MojHnd]

Thank you. I just solved it this way.

The problem is with from vllm._C import ops while there is no vllm._C.
We need ops that exists in your_environment_name/lib/python3.10/site-packages/vllm/model_executor/layers/ (see the figure below)

So, what we have to do is to change from vllm._C import ops to from vllm.model_executor.layers import ops in every single file of the package.
This solves the problem :)

[yaxu75]

Thank you. I just solved it this way.

The problem is with from vllm._C import ops while there is no vllm._C. We need ops that exists in your_environment_name/lib/python3.10/site-packages/vllm/model_executor/layers/ (see the figure below)

So, what we have to do is to change from vllm._C import ops to from vllm.model_executor.layers import ops in every single file of the package. This solves the problem :)

This will not work. from vllm._C import ops and from vllm.model_executor.layers import ops are totally different things. from vllm._C import ops are some CUDA operators written by C++, while from vllm.model_executor.layers import ops are just some utility codes written by Python.

You avoid the error by direct some empty import ops to another different one, so python won't notice if you don't actually use those operators. Once you actually use those CUDA operators, code will just not be able to run.

[eigen2017]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

Good!

solved my issue

[Msiavashi]

The issue still exists! Has anyone been able to find a solution so far?

[ghost]

NameError: name 'vllm_ops' is not defined 这个问题是只需要pip install vllm即可而无需git clone vllm项目 就能解决是吗

[clearsitedesigns]

Same issue today

[barnabasharris]

I had this issue today after running python -m pip install vllm in my activated environment

[pavelToman]

Hello!
I am trying to build and install v0.4.0 from sources.
Installation seems ok.
But I got the same error (No module named 'vllm._C) during pytest /tests.
As recommended before, I rename dir vllm to not interfere with vllm module. It works and the vllm._C problem disappears. But now I am hitting ModuleNotFoundError: No module named 'vllm.core.block' during the tests/core/block/test_block_space_manager.py test. Seems it is looking for dir vllm but can not find it since I rename it...
How to solve this, please?

[EXJUSTICE]

Having the same problem using the official AMD installation guide with Docker, with the error appearing during collect_env.py. Renaming vllm seems to have solved the problem for now.

[gokulcoder7]

Hi @Kawai1Ace, it seems you are using the latest main branch of vLLM. Did you install vLLM from source? The vllm._C module is built when you install vLLM.

(envv) F:\vllm\vllm>pip install -e .
Obtaining file:///F:/vllm/vllm
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: psutil in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (6.1.0)
Requirement already satisfied: sentencepiece in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.2.0)
Requirement already satisfied: numpy<2.0.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (1.26.4)
Requirement already satisfied: requests>=2.26.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (2.32.3)
Requirement already satisfied: tqdm in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (4.67.1)
Requirement already satisfied: py-cpuinfo in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (9.0.0)
Requirement already satisfied: transformers>=4.45.2 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (4.47.0)
Requirement already satisfied: tokenizers>=0.19.1 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.21.0)
Requirement already satisfied: protobuf in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (5.29.1)
Requirement already satisfied: fastapi!=0.113.,!=0.114.0,>=0.107.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.115.6)
Requirement already satisfied: aiohttp in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (3.11.10)
Requirement already satisfied: openai>=1.45.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (1.57.0)
Requirement already satisfied: uvicorn[standard] in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.32.1)
Requirement already satisfied: pydantic>=2.9 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (2.10.3)
Requirement already satisfied: pillow in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (10.4.0)
Requirement already satisfied: prometheus_client>=0.18.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.21.1)
Requirement already satisfied: prometheus-fastapi-instrumentator>=7.0.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (7.0.0)
Requirement already satisfied: tiktoken>=0.6.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.7.0)
Requirement already satisfied: lm-format-enforcer<0.11,>=0.10.9 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.10.9)
Requirement already satisfied: outlines<0.1,>=0.0.43 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.0.46)
Requirement already satisfied: typing_extensions>=4.10 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (4.12.2)
Requirement already satisfied: filelock>=3.16.1 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (3.16.1)
Requirement already satisfied: partial-json-parser in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.2.1.1.post4)
Requirement already satisfied: pyzmq in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (26.2.0)
Requirement already satisfied: msgspec in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.18.6)
Requirement already satisfied: gguf==0.10.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.10.0)
Requirement already satisfied: importlib_metadata in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (8.5.0)
Requirement already satisfied: mistral_common>=1.5.0 in f:\vllm\envv\lib\site-packages (from mistral_common[opencv]>=1.5.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.5.1)
Requirement already satisfied: pyyaml in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (6.0.2)
Requirement already satisfied: six>=1.16.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (1.17.0)
Requirement already satisfied: setuptools>=74.1.1 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (75.6.0)
Requirement already satisfied: einops in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.8.0)
Requirement already satisfied: compressed-tensors==0.8.0 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.8.0)
Requirement already satisfied: ray>=2.9 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (2.40.0)
Requirement already satisfied: nvidia-ml-py>=12.560.30 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (12.560.30)
Requirement already satisfied: torch==2.5.1 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (2.5.1)
Requirement already satisfied: torchvision==0.20.1 in f:\vllm\envv\lib\site-packages (from vllm==0.6.4.post2.dev271+gaf7c4a92) (0.20.1)
Requirement already satisfied: networkx in f:\vllm\envv\lib\site-packages (from torch==2.5.1->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.4.2)
Requirement already satisfied: jinja2 in f:\vllm\envv\lib\site-packages (from torch==2.5.1->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.1.4)
Requirement already satisfied: fsspec in f:\vllm\envv\lib\site-packages (from torch==2.5.1->vllm==0.6.4.post2.dev271+gaf7c4a92) (2024.9.0)
Requirement already satisfied: sympy==1.13.1 in f:\vllm\envv\lib\site-packages (from torch==2.5.1->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in f:\vllm\envv\lib\site-packages (from sympy==1.13.1->torch==2.5.1->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.3.0)
Requirement already satisfied: starlette<0.42.0,>=0.40.0 in f:\vllm\envv\lib\site-packages (from fastapi!=0.113.,!=0.114.0,>=0.107.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.41.3)
Requirement already satisfied: interegular>=0.3.2 in f:\vllm\envv\lib\site-packages (from lm-format-enforcer<0.11,>=0.10.9->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.3.3)
Requirement already satisfied: packaging in f:\vllm\envv\lib\site-packages (from lm-format-enforcer<0.11,>=0.10.9->vllm==0.6.4.post2.dev271+gaf7c4a92) (24.2)
Requirement already satisfied: jsonschema<5.0.0,>=4.21.1 in f:\vllm\envv\lib\site-packages (from mistral_common>=1.5.0->mistral_common[opencv]>=1.5.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (4.23.0)
Requirement already satisfied: opencv-python-headless<5.0.0,>=4.0.0 in f:\vllm\envv\lib\site-packages (from mistral_common[opencv]>=1.5.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (4.10.0.84)
Requirement already satisfied: anyio<5,>=3.5.0 in f:\vllm\envv\lib\site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (4.7.0)
Requirement already satisfied: distro<2,>=1.7.0 in f:\vllm\envv\lib\site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in f:\vllm\envv\lib\site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.28.1)
Requirement already satisfied: jiter<1,>=0.4.0 in f:\vllm\envv\lib\site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.8.0)
Requirement already satisfied: sniffio in f:\vllm\envv\lib\site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.3.1)
Requirement already satisfied: lark in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.2.2)
Requirement already satisfied: nest-asyncio in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.6.0)
Requirement already satisfied: cloudpickle in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.1.0)
Requirement already satisfied: diskcache in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (5.6.3)
Requirement already satisfied: numba in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.60.0)
Requirement already satisfied: referencing in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.35.1)
Requirement already satisfied: datasets in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.1.0)
Requirement already satisfied: pycountry in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (24.6.1)
Requirement already satisfied: pyairports in f:\vllm\envv\lib\site-packages (from outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (2.1.1)
Requirement already satisfied: annotated-types>=0.6.0 in f:\vllm\envv\lib\site-packages (from pydantic>=2.9->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.1 in f:\vllm\envv\lib\site-packages (from pydantic>=2.9->vllm==0.6.4.post2.dev271+gaf7c4a92) (2.27.1)
Requirement already satisfied: click>=7.0 in f:\vllm\envv\lib\site-packages (from ray>=2.9->vllm==0.6.4.post2.dev271+gaf7c4a92) (8.1.7)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in f:\vllm\envv\lib\site-packages (from ray>=2.9->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.1.0)
Requirement already satisfied: aiosignal in f:\vllm\envv\lib\site-packages (from ray>=2.9->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.3.1)
Requirement already satisfied: frozenlist in f:\vllm\envv\lib\site-packages (from ray>=2.9->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.5.0)
Requirement already satisfied: charset-normalizer<4,>=2 in f:\vllm\envv\lib\site-packages (from requests>=2.26.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in f:\vllm\envv\lib\site-packages (from requests>=2.26.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in f:\vllm\envv\lib\site-packages (from requests>=2.26.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in f:\vllm\envv\lib\site-packages (from requests>=2.26.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (2024.8.30)
Requirement already satisfied: regex>=2022.1.18 in f:\vllm\envv\lib\site-packages (from tiktoken>=0.6.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (2024.11.6)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in f:\vllm\envv\lib\site-packages (from tokenizers>=0.19.1->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.26.5)
Requirement already satisfied: colorama in f:\vllm\envv\lib\site-packages (from tqdm->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.4.6)
Requirement already satisfied: safetensors>=0.4.1 in f:\vllm\envv\lib\site-packages (from transformers>=4.45.2->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.4.5)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in f:\vllm\envv\lib\site-packages (from aiohttp->vllm==0.6.4.post2.dev271+gaf7c4a92) (2.4.4)
Requirement already satisfied: attrs>=17.3.0 in f:\vllm\envv\lib\site-packages (from aiohttp->vllm==0.6.4.post2.dev271+gaf7c4a92) (24.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in f:\vllm\envv\lib\site-packages (from aiohttp->vllm==0.6.4.post2.dev271+gaf7c4a92) (6.1.0)
Requirement already satisfied: propcache>=0.2.0 in f:\vllm\envv\lib\site-packages (from aiohttp->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.2.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in f:\vllm\envv\lib\site-packages (from aiohttp->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.18.3)
Requirement already satisfied: zipp>=3.20 in f:\vllm\envv\lib\site-packages (from importlib_metadata->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.21.0)
Requirement already satisfied: h11>=0.8 in f:\vllm\envv\lib\site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.14.0)
Requirement already satisfied: httptools>=0.6.3 in f:\vllm\envv\lib\site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.6.4)
Requirement already satisfied: python-dotenv>=0.13 in f:\vllm\envv\lib\site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.0.1)
Requirement already satisfied: watchfiles>=0.13 in f:\vllm\envv\lib\site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.0.0)
Requirement already satisfied: websockets>=10.4 in f:\vllm\envv\lib\site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev271+gaf7c4a92) (14.1)
Requirement already satisfied: httpcore==1.* in f:\vllm\envv\lib\site-packages (from httpx<1,>=0.23.0->openai>=1.45.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (1.0.7)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in f:\vllm\envv\lib\site-packages (from jsonschema<5.0.0,>=4.21.1->mistral_common>=1.5.0->mistral_common[opencv]>=1.5.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (2024.10.1)
Requirement already satisfied: rpds-py>=0.7.1 in f:\vllm\envv\lib\site-packages (from jsonschema<5.0.0,>=4.21.1->mistral_common>=1.5.0->mistral_common[opencv]>=1.5.0->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.22.3)
Requirement already satisfied: pyarrow>=15.0.0 in f:\vllm\envv\lib\site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (18.1.0)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in f:\vllm\envv\lib\site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.3.8)
Requirement already satisfied: pandas in f:\vllm\envv\lib\site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (2.2.3)
Requirement already satisfied: xxhash in f:\vllm\envv\lib\site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.5.0)
Requirement already satisfied: multiprocess<0.70.17 in f:\vllm\envv\lib\site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.70.16)
Requirement already satisfied: MarkupSafe>=2.0 in f:\vllm\envv\lib\site-packages (from jinja2->torch==2.5.1->vllm==0.6.4.post2.dev271+gaf7c4a92) (3.0.2)
Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in f:\vllm\envv\lib\site-packages (from numba->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (0.43.0)
Requirement already satisfied: python-dateutil>=2.8.2 in f:\vllm\envv\lib\site-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in f:\vllm\envv\lib\site-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in f:\vllm\envv\lib\site-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.4.post2.dev271+gaf7c4a92) (2024.2)
Building wheels for collected packages: vllm
Building editable for vllm (pyproject.toml) ... done
Created wheel for vllm: filename=vllm-0.6.4.post2.dev271+gaf7c4a92-0.editable-py3-none-any.whl size=11811 sha256=84a1c02fdec76713aca0378982ad058237d7e6f8290a6e636e0e9b974ccc7709
Stored in directory: C:\Users\Sushant\AppData\Local\Temp\pip-ephem-wheel-cache-d8nwwqpo\wheels\e6\8d\02\8116c93b68f513baab6a3f637831a88fabe297452aeb1dc62e
Successfully built vllm
Installing collected packages: vllm
Attempting uninstall: vllm
Found existing installation: vllm 0.6.4.post1
Uninstalling vllm-0.6.4.post1:
Successfully uninstalled vllm-0.6.4.post1
Successfully installed vllm-0.6.4.post2.dev271+gaf7c4a92

(envv) F:\vllm\vllm>vllm serve "LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct"
Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "F:\vllm\envv\Scripts\vllm.exe_main.py", line 4, in
File "F:\vllm\vllm\vllm_init.py", line 3, in
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
File "F:\vllm\vllm\vllm\engine\arg_utils.py", line 11, in
from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
File "F:\vllm\vllm\vllm\config.py", line 21, in
from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS,
File "F:\vllm\vllm\vllm\model_executor_init_.py", line 1, in
from vllm.model_executor.parameter import (BasevLLMParameter,
File "F:\vllm\vllm\vllm\model_executor\parameter.py", line 7, in
from vllm.distributed import get_tensor_model_parallel_rank
File "F:\vllm\vllm\vllm\distributed_init_.py", line 1, in
from .communication_op import *
File "F:\vllm\vllm\vllm\distributed\communication_op.py", line 6, in
from .parallel_state import get_tp_group
File "F:\vllm\vllm\vllm\distributed\parallel_state.py", line 38, in
import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer
File "F:\vllm\vllm\vllm\distributed\kv_transfer\kv_transfer_agent.py", line 15, in
from vllm.distributed.kv_transfer.kv_connector.factory import (
File "F:\vllm\vllm\vllm\distributed\kv_transfer\kv_connector\factory.py", line 3, in
from .base import KVConnectorBase
File "F:\vllm\vllm\vllm\distributed\kv_transfer\kv_connector\base.py", line 14, in
from vllm.sequence import IntermediateTensors
File "F:\vllm\vllm\vllm\sequence.py", line 16, in
from vllm.inputs import SingletonInputs, SingletonInputsAdapter
File "F:\vllm\vllm\vllm\inputs_init_.py", line 7, in
from .registry import (DummyData, InputContext, InputProcessingContext,
File "F:\vllm\vllm\vllm\inputs\registry.py", line 13, in
from vllm.transformers_utils.tokenizer import AnyTokenizer
File "F:\vllm\vllm\vllm\transformers_utils\tokenizer.py", line 16, in
from vllm.utils import make_async
File "F:\vllm\vllm\vllm\utils.py", line 44, in
from vllm.platforms import current_platform
File "F:\vllm\vllm\vllm\platforms_init_.py", line 100, in
from .cuda import CudaPlatform
File "F:\vllm\vllm\vllm\platforms\cuda.py", line 14, in
import vllm._C # noqa
^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'vllm._C'

(envv) F:\vllm\vllm>

[gokulcoder7]

@Kawai1Ace The error ModuleNotFoundError: No module named 'vllm._C' occurs because there is a folder named vllm, which makes the import command prioritize the functions in this folder vllm over the vllm library. You just need to rename this folder.

niubi

I do not think so. Mine is like the following:

File ~/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/layers/activation.py:9
6 import torch.nn as nn
7 import torch.nn.functional as F
----> 9 from vllm._C import ops
10 from vllm.model_executor.layers.quantization import QuantizationConfig
11 from vllm.model_executor.parallel_utils.parallel_state import (
12 get_tensor_model_parallel_rank, get_tensor_model_parallel_world_size)

ModuleNotFoundError: No module named 'vllm._C'

I think you should run this command in the 'vllm' folder. You can create a new folder, such as 'run',and navigate into the 'run' folder.Then, you can run the command again

F:\vllm\vllm\run>F:\vllm\envv\Scripts\activate

(envv) F:\vllm\vllm\run>vllm serve "LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct"
Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "F:\vllm\envv\Scripts\vllm.exe_main.py", line 4, in
File "F:\vllm\vllm\vllm_init.py", line 3, in
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
File "F:\vllm\vllm\vllm\engine\arg_utils.py", line 11, in
from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
File "F:\vllm\vllm\vllm\config.py", line 21, in
from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS,
File "F:\vllm\vllm\vllm\model_executor_init_.py", line 1, in
from vllm.model_executor.parameter import (BasevLLMParameter,
File "F:\vllm\vllm\vllm\model_executor\parameter.py", line 7, in
from vllm.distributed import get_tensor_model_parallel_rank
File "F:\vllm\vllm\vllm\distributed_init_.py", line 1, in
from .communication_op import *
File "F:\vllm\vllm\vllm\distributed\communication_op.py", line 6, in
from .parallel_state import get_tp_group
File "F:\vllm\vllm\vllm\distributed\parallel_state.py", line 38, in
import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer
File "F:\vllm\vllm\vllm\distributed\kv_transfer\kv_transfer_agent.py", line 15, in
from vllm.distributed.kv_transfer.kv_connector.factory import (
File "F:\vllm\vllm\vllm\distributed\kv_transfer\kv_connector\factory.py", line 3, in
from .base import KVConnectorBase
File "F:\vllm\vllm\vllm\distributed\kv_transfer\kv_connector\base.py", line 14, in
from vllm.sequence import IntermediateTensors
File "F:\vllm\vllm\vllm\sequence.py", line 16, in
from vllm.inputs import SingletonInputs, SingletonInputsAdapter
File "F:\vllm\vllm\vllm\inputs_init_.py", line 7, in
from .registry import (DummyData, InputContext, InputProcessingContext,
File "F:\vllm\vllm\vllm\inputs\registry.py", line 13, in
from vllm.transformers_utils.tokenizer import AnyTokenizer
File "F:\vllm\vllm\vllm\transformers_utils\tokenizer.py", line 16, in
from vllm.utils import make_async
File "F:\vllm\vllm\vllm\utils.py", line 44, in
from vllm.platforms import current_platform
File "F:\vllm\vllm\vllm\platforms_init_.py", line 100, in
from .cuda import CudaPlatform
File "F:\vllm\vllm\vllm\platforms\cuda.py", line 14, in
import vllm._C # noqa
^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'vllm._C'

(envv) F:\vllm\vllm\run>

[celsowm]

did a pip install -U vllm today
and:

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm_init_.py", line 3, in
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\engine\arg_utils.py", line 11, in
from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\config.py", line 22, in
from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS,
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\model_executor_init_.py", line 1, in
from vllm.model_executor.parameter import (BasevLLMParameter,
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\model_executor\parameter.py", line 7, in
from vllm.distributed import get_tensor_model_parallel_rank
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\distributed_init_.py", line 1, in
from .communication_op import *
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\distributed\communication_op.py", line 6, in
from .parallel_state import get_tp_group
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\distributed\parallel_state.py", line 38, in
import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\distributed\kv_transfer\kv_transfer_agent.py", line 15, in
from vllm.distributed.kv_transfer.kv_connector.factory import (
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\distributed\kv_transfer\kv_connector\factory.py", line 3, in
from .base import KVConnectorBase
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\distributed\kv_transfer\kv_connector\base.py", line 14, in
from vllm.sequence import IntermediateTensors
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\sequence.py", line 16, in
from vllm.inputs import SingletonInputs, SingletonInputsAdapter
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\inputs_init_.py", line 7, in
from .registry import (DummyData, InputContext, InputProcessingContext,
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\inputs\registry.py", line 13, in
from vllm.transformers_utils.tokenizer import AnyTokenizer
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\transformers_utils\tokenizer.py", line 16, in
from vllm.utils import make_async
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\utils.py", line 46, in
from vllm.platforms import current_platform
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\platforms_init_.py", line 100, in
from .cuda import CudaPlatform
File "C:\Users\celso\AppData\Roaming\Python\Python310\site-packages\vllm\platforms\cuda.py", line 15, in
import vllm._C # noqa
ModuleNotFoundError: No module named 'vllm._C'