Update bnb.md with example for OpenAI (vllm-project#11718)

abmfy · Jan 24, 2025 · b0ed9b4 · b0ed9b4
1 parent 71d9024
commit b0ed9b4
Showing 1 changed file with 7 additions and 0 deletions.
diff --git a/docs/source/quantization/bnb.md b/docs/source/quantization/bnb.md
@@ -37,3 +37,10 @@ model_id = "huggyllama/llama-7b"
 llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True, \
 quantization="bitsandbytes", load_format="bitsandbytes")
 ```
+## OpenAI Compatible Server
+
+Append the following to your 4bit model arguments:
+
+```
+--quantization bitsandbytes --load-format bitsandbytes
+```