[Feature]: support out tree multimodal models

[Jack47]

🚀 The feature, motivation and pitch

vllm >= 0.6 doesn't support out tree defined multimodal models:

rank0]:   File "/usr/local/lib/python3.8/dist-packages/vllm/entrypoints/llm.py", line 177, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 535, in from_engine_args
[rank0]:     engine_config = engine_args.create_engine_config()
[rank0]:   File "/usr/local/lib/python3.8/dist-packages/vllm/engine/arg_utils.py", line 792, in create_engine_config
[rank0]:     model_config = ModelConfig(
[rank0]:   File "/usr/local/lib/python3.8/dist-packages/vllm/config.py", line 230, in __init__
[rank0]:     self.multimodal_config = self._init_multimodal_config(
[rank0]:   File "/usr/local/lib/python3.8/dist-packages/vllm/config.py", line 251, in _init_multimodal_config
[rank0]:     raise ValueError(
[rank0]: ValueError: limit_mm_per_prompt is only supported for multimodal models.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[youkaichao]

how do you register the out of tree model?

[Jack47]

how do you register the out of tree model?

This is the feature I want to have, currently vllm 0.6 doesn't provide ways to register out of tree models?

One possible hack way:add my model type and class into _MULTIMODAL_MODELS in vllm

[youkaichao]

we do support out of tree registration, see https://docs.vllm.ai/en/latest/models/adding_model.html#out-of-tree-model-integration

[Jack47]

we do support out of tree registration, see https://docs.vllm.ai/en/latest/models/adding_model.html#out-of-tree-model-integration

sure and I've registered my multimodal models using that function. But the error I posted still occurs.

so vllm 0.6 doesn't provide ways to register out of tree multimodal registration?

[youkaichao]

cc @DarkLight1337 @ywang96
does multi-modal model support the out-of-tree registration?

[DarkLight1337]

Currently, you can do it by adding the model to _MULTIMODAL_MODELS. Just like regular OOT models, there's no public API for this.

[youkaichao]

@DarkLight1337 we do have a public api for this:

vllm/vllm/model_executor/models/__init__.py

Line 190 in 5e85f4f

def register_model(model_arch: str, model_cls: Type[nn.Module]):

[DarkLight1337]

@DarkLight1337 we do have a public api for this:

vllm/vllm/model_executor/models/__init__.py

Line 190 in 5e85f4f

def register_model(model_arch: str, model_cls: Type[nn.Module]):

The docs currently suggest to modify _MODELS directly.

[youkaichao]

@DarkLight1337 that is for registering your model when you fork vllm.

this issue is about out-of-tree registration, i.e. adding new model to vllm without a fork.

[ywang96]

@Jack47 Can you take a look and try #8717 to see if it fixes your issue?

[Jack47]

I'll try it later, thanks.
will post the result there.