Question: CPU and quantization support?

[oleotiger]

Does vLLM support LLM inference on CPU? And with quantized model?

What's the difference between vLLM and GGML?

[hmellor]

Does vLLM support LLM inference on CPU?

No, but it's on the roadmap #2681

And with quantized model?

Yes: AWQ, GPTQ, SqueezeLLM

What's the difference between vLLM and GGML?

vLLM is a serving engine, GGLM is a tensor library for CPUs