[Bugfix / Core] Prefix Caching Guards #3903
Conversation
vllm/config.py
Outdated
| if self.disable_sliding_window: | ||
| return None |
There was a problem hiding this comment.
nit: Should there be a warning printed if sliding window is disabled even if it's enabled in config?
There was a problem hiding this comment.
+1, something along the line of "Sliding window is disabled per configuration. The model will ... "
I would assume the max content len is set to the sliding window size?
There was a problem hiding this comment.
Yea, it set to min(user_provided_max_model_len, sliding_window). The issue at current is that even if we set max_model_len=4096 for Mistral (which effectively disables sliding window), we still query the hf config to see if the model has sliding window
So this allows user to set it, therefore enabling usage of Mistral with APC
Yard1
changed the title
vllm/config.py
Outdated
| if self.disable_sliding_window: | ||
| return None |
There was a problem hiding this comment.
+1, something along the line of "Sliding window is disabled per configuration. The model will ... "
I would assume the max content len is set to the sliding window size?
| def _get_and_verify_dtype( | ||
| self, | ||
| config: PretrainedConfig, | ||
| dtype: Union[str, torch.dtype], | ||
| ) -> torch.dtype: | ||
| # NOTE: getattr(config, "torch_dtype", torch.float32) is not correct | ||
| # because config.torch_dtype can be None. | ||
| config_dtype = getattr(config, "torch_dtype", None) | ||
| if config_dtype is None: | ||
| config_dtype = torch.float32 | ||
|
|
||
| if isinstance(dtype, str): | ||
| dtype = dtype.lower() | ||
| if dtype == "auto": | ||
| if config_dtype == torch.float32: | ||
| # Following the common practice, we use float16 for float32 | ||
| # models. | ||
| torch_dtype = torch.float16 | ||
| else: | ||
| torch_dtype = config_dtype | ||
| else: | ||
| if dtype not in _STR_DTYPE_TO_TORCH_DTYPE: | ||
| raise ValueError(f"Unknown dtype: {dtype}") | ||
| torch_dtype = _STR_DTYPE_TO_TORCH_DTYPE[dtype] | ||
| elif isinstance(dtype, torch.dtype): | ||
| torch_dtype = dtype | ||
| else: | ||
| raise ValueError(f"Unknown dtype: {dtype}") | ||
|
|
||
| if is_hip() and torch_dtype == torch.float32: | ||
| rocm_supported_dtypes = [ | ||
| k for k, v in _STR_DTYPE_TO_TORCH_DTYPE.items() | ||
| if (k not in _ROCM_NOT_SUPPORTED_DTYPE) | ||
| ] | ||
| raise ValueError(f"dtype '{dtype}' is not supported in ROCm. " | ||
| f"Supported dtypes are {rocm_supported_dtypes}") | ||
|
|
||
| # Verify the dtype. | ||
| if torch_dtype != config_dtype: | ||
| if torch_dtype == torch.float32: | ||
| # Upcasting to float32 is allowed. | ||
| pass | ||
| elif config_dtype == torch.float32: | ||
| # Downcasting from float32 to float16 or bfloat16 is allowed. | ||
| pass | ||
| else: | ||
| # Casting between float16 and bfloat16 is allowed with warning. | ||
| logger.warning("Casting %s to %s.", config_dtype, torch_dtype) | ||
|
|
||
| return torch_dtype |
There was a problem hiding this comment.
@robertgshaw2-neuralmagic Do you still remember why you moved these methods from a global function to a method of ModelConfig?
There was a problem hiding this comment.
I noticed the function was only used by the ModelConfig and no where else, so I thought it made sense to encapulate
|
Close this PR since we merged #4846 |
Added checks for cases where prefix caching is not supported (due to limitations of
context_attention_fwd):Also added CLI for user to specify that they want to disable sliding window. This enables us to run mistral with prefix caching. Note - we could add onto this PR to disable sliding window if max_len < sliding_window_len
FIX #3880
cc @SageMoore @ElizaWszola @tlrmchlsmth
BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE
PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:
[Bugfix]for bug fixes.[CI/Build]for build or continuous integration improvements.[Doc]for documentation fixes and improvements.[Model]for adding a new model or improving an existing model. Model name should appear in the title.[Frontend]For changes on the vLLM frontend (e.g., OpenAI API server,LLMclass, etc.)[Kernel]for changes affecting CUDA kernels or other compute kernels.[Core]for changes in the core vLLM logic (e.g.,LLMEngine,AsyncLLMEngine,Scheduler, etc.)[Hardware][Vendor]for hardware-specific changes. Vendor name should appear in the prefix (e.g.,[Hardware][AMD]).[Misc]for PRs that do not fit the above categories. Please use this sparingly.Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:
format.shto format your code.docs/source/if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with
rfc-requiredand might not go through the PR.What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:
action-requiredlabel on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.Thank You
Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!