docs: add instructions for Langchain

[mspronesti]

Hi! A month ago I made a few contributions to Langchain to support vLLM and its OpenAI-compatible server.

This PR aims at updating vLLM's documentation to showcase how to use vLLM with Langchain and to redirect to the tutorial I wrote there for further details. Hope you find this contribution meaningful.

[pvtoan]

Hi @mspronesti , does this LangChain-VLLM support quantized model?

Because the vllm-project already supported quantized model (AWQ format) as shown in #1032

However, when I use the same way and just pass "quantization='awq" to your LangChain-VLLM, it seems does not work and just show OOM.

model_path = "/home/quadrep/toan/projects/LLMs/weights/vicuna-33B-AWQ"
model = VLLM(model=model_path, tensor_parallel_size=2, trust_remote_code=True, max_new_tokens=512, quantization='awq')
--> Error: torch.cuda.OutOfMemoryError: CUDA out of memory.

[mspronesti]

Hi @pvtoan, quantization is not an explicit parameter of langchain's vLLM. You need to pass it as follows

model = VLLM(
  model=model_path, 
  tensor_parallel_size=2, 
  trust_remote_code=True, 
  max_new_tokens=512, 
  vllm_kwargs={"quantization": "awq"}
)

Also, notice that tensor_parallel_size=2 implies you want to serve the model in a distributed manner, with 2 GPUs. Hope this helps :)

[mspronesti]

Btw @WoosukKwon I have also resolved the conflicts in this PR, in case you are interested in merging It.

[pvtoan]

Hi @mspronesti , yes, it works now and I indeed uses 2 GPUs.

Thank you so much for your help!.

Besides, I'd like to ask if I have any problems when loading the model vicuna-33B-AWQ (model size is around, in total, 18GB with 2 shards) and one embedding model (size about 1.3GB) into 2 GPUs 4090 (each with 24GB) because all loaded models almost occupied 48GB from 2 GPUs.

In particular, two GPUs increased memory simultaneously from 0GB to around 9GB, then growed up to 14GB, and finally to 23.9GB.

My DRAM also increases from 6GB to around 28GB. My OS is Ubuntu 22.04.

Do you think it is normal for occupied memory including VRAM and DRAM?

[simon-mo]

Thank you for your contribution!