vllm can't run peft model?

[Stealthwriter]

Hi, does vllm runs peft models? I'm trying to run them but im getting an error that the config.json is not available in the model main forlder

[viktor-ferenczi]

vLLM does not support PEFT/LoRA yet, but it is on the Development Roadmap

Related: #289, #815

[Stealthwriter]

vLLM does not support PEFT/LoRA yet, but it is on the Development Roadmap

Related: #289, #815

ok thanks!

[hllj]

You have to merge adapter from peft with their base model first.
This is my codebase for merging you can see here: https://github.com/Reasoning-Lab/Elementary-Math-Solving-Zalo-AI-2023/blob/79ed4742d91755b4a00fadd0079279394222928b/merge_peft_adapter.py
After that you will have your merged model and you can serve with vLLM from output_dir.

[rsong0606]

@hllj Hi there, does this mean the merged model either saving on local or pushing to the hugging face will have the same structure as the base model which means will have similar files and versions as the base model in hugging face?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!