[Usage]: I doubt about the meaning of --enable-prefix-caching

[chenchunhui97]

Your current environment

1. I doubt which version that starting support feat --enable-prefix-caching. (seems v0.3.0 still not support yet, and v0.4.0 has supported it already)
2. does --enable-prefix-caching mean to prefix system prompt only? or is it an implementation of RadixAttention (https://arxiv.org/abs/2312.07104)? what is the difference between prefix-caching and prefix-sharing (the following implementation)?

if prefix_len != None:
      # start inference
      if prompt_token_ids != None:
          outputs = llm.generate(prompt_token_ids=prompt_token_ids,
                                 sampling_params=sampling_params,
                                 prefix_pos=prefix_len * (len(prompts) // len(prefix_len)))
      else:
          outputs = llm.generate(prompts=prompts,
                                 sampling_params=sampling_params,
                                 prefix_pos=prefix_len * (len(prompts) // len(prefix_len)))
  else:
      outputs = llm.generate(prompts, sampling_params=sampling_params)

How would you like to use vllm

I want to know more details about --enable-prefix-caching and the releated paper.

[zhuohan123]

Please refer to #2614 for the details for now. We will publish a blogpost explaining our design soon. Stay tuned!

[timothylimyl]

@zhuohan123 look forward to the blog.

[samos123]

Was the blog post ever published? Please also update the flag documentation. I'm trying to understand what it actually does.

[Playerrrrr]

Where is the blog? @zhuohan123