[Feature] Initial support for multi-LoRA serving

[Ying1123]

This PR gives initial multi-LoRA serving support. Currently, it supports LoRA on attention (qkvo) and mlp (gate, up, down) linear layers. It supports dynamic loading and offloading, but it does not support unified memory. The memory pool for LoRA adapters is pre-allocated. Please use a smaller --mem-frac to launch server with larger --max-loras-per-batch.

Current example usage:

# launch server
python -m sglang.launch_server --model mistralai/Mistral-7B-Instruct-v0.3 --lora-paths /home/ying/test_lora /home/ying/test_lora_1 /home/ying/test_lora_2 /home/ying/test_lora_3 /home/ying/test_lora_4 --disable-radix --disable-cuda-graph --max-loras-per-batch 4

# send requests
# lora_path[i] specifies the LoRA used for text[i], so make sure they have the same length
# use None to specify base-only prompt, e.x. "lora_path": [None, "/home/ying/test_lora"]
import json
import requests

url = "http://127.0.0.1:30000"
json_data = {
        "text": ["prompt 1", "prompt 2", "prompt 3", "prompt 4", "prompt 5", "prompt 6", "prompt7"],
        "sampling_params": {"max_new_tokens": 32},
        "lora_path": ["/home/ying/test_lora", "/home/ying/test_lora_1", "/home/ying/test_lora_2", "/home/ying/test_lora_3", "/home/ying/test_lora_4", "/home/ying/test_lora", "/home/ying/test_lora_1"],
}
response = requests.post(
        url + "/generate",
        json=json_data,
)
print(json.dumps(response.json()))

You can expect the items below in the follow-up PRs.

• OpenAI compatible API
• compatibility with cuda graph
• compatibility with radix attention
• fully sharded LoRAs with tensor parallelism
• performance optimization
• memory optimization
• support LoRAs with different ranks
• add triton backend
• test cases enhancement
• Misc: path renaming [Feature] Support LoRA path renaming and add LoRA serving benchmarks #1433

References:
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
Punica: Multi-Tenant LoRA Serving

[binarycrayon]

Tested and looks great!

[upskyy]

@Ying1123 @merrymercy
I also get the same error when serving the gemma2 model as multi lora. Has the error been resolved?
I tested on image lmsysorg/sglang:v0.3.2-cu121.

python3 -m sglang.launch_server --model-path /base_model --tokenizer-path /base_model --lora-paths /lora_model0 /lora_model1  --disable-radix --disable-cuda-graph --max-loras-per-batch 2 --mem-fraction-static 0.5 --random-seed 0 --enable-torch-compile

AttributeError: 'Gemma2ForCausalLM' object has no attribute 'get_module_name'

Related issues: #1416

Resolved PR: #2330 (comment)

[xijiz]

@Ying1123 Does this feature support multimodal models such as Qwen2-VL? Currently vLLM does not support this.