Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[torch.compile] store inductor compiled Python file #12182

Merged
merged 2 commits into from
Jan 19, 2025
Merged

[torch.compile] store inductor compiled Python file #12182

merged 2 commits into from
Jan 19, 2025

Conversation

youkaichao
Copy link
Member

@youkaichao youkaichao commented Jan 18, 2025 •
edited by github-actions bot
Loading

#11108 enabled the compilation cache, but it only stores a hash string, which is not human-readable.

this PR further stores the path of the python file compiled by inductor, so that we can manually check the result.

As a result, I can confirm that torch.compile does not do anything w.r.t. kv cache, thanks to #11677 . Therefore, we don't need to split the graph in v0. We can directly compile and load the full graph, which leads to faster compilation.

@youkaichao
fix splitting ops
9099e1f 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
update backends
ba3a7b8 
Signed-off-by: youkaichao <[email protected]>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

youkaichao
youkaichao commented Jan 18, 2025
View reviewed changes
vllm/compilation/backends.py
@dataclasses.dataclass
class InductorArtifact:
hash_str: str = ""
file_path: str = ""
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inside the file_path, we can find a call function. that is the ultimate function inductor compiled and called.

youkaichao
youkaichao commented Jan 18, 2025
View reviewed changes
vllm/config.py
"vllm.unified_attention_with_output",
]
# v0 uses full graph compilation
self.splitting_ops = []
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now that kv cache is completely hidden from torch.compile, changing gpu memory utilization (and thus changing kv cache shape) will not affect torch.compile . thus they can share the same compilation cache.

heheda12345
heheda12345 approved these changes Jan 19, 2025
View reviewed changes
Copy link
Collaborator

@heheda12345 heheda12345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Left one small comment.

vllm/compilation/backends.py
@@ -199,6 +219,7 @@ def wrap_inductor(graph: fx.GraphModule,
"Inductor cache lookup failed. Please remove"
f"the cache file {cache_data.cache_file_path} and try again." # noqa
)
inductor_artifact.file_path = inductor_compiled_graph.current_callable.__code__.co_filename # noqa
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we only update the file_path of graph_index 0? Do we only support fullgraph mode of torch.compile?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems to be misunderstanding. if graph_index == 0: is only for logging, this line is executed for every piece of graph.

@youkaichao youkaichao added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 19, 2025
@youkaichao
Copy link
Member Author

test failure is unrelated. merging.

@youkaichao youkaichao merged commit e66faf4 into vllm-project:main Jan 19, 2025
64 of 66 checks passed
@youkaichao youkaichao deleted the split_ops branch January 19, 2025 08:27
joennlae pushed a commit to 44ai-labs/vllm that referenced this pull request Jan 19, 2025
@youkaichao
[torch.compile] store inductor compiled Python file (vllm-project#12182)
4286078 
Signed-off-by: youkaichao <[email protected]>
joennlae pushed a commit to 44ai-labs/vllm that referenced this pull request Jan 19, 2025
@youkaichao
[torch.compile] store inductor compiled Python file (vllm-project#12182)
4d7d924 
Signed-off-by: youkaichao <[email protected]>
lckr pushed a commit to lckr/vllm that referenced this pull request Jan 19, 2025
@youkaichao
[torch.compile] store inductor compiled Python file (vllm-project#12182)
baea339 
Signed-off-by: youkaichao <[email protected]>
@youkaichao youkaichao mentioned this pull request Jan 21, 2025
Merged
@kzawora-intel kzawora-intel mentioned this pull request Jan 21, 2025
Merged
abmfy pushed a commit to abmfy/vllm-flashinfer that referenced this pull request Jan 24, 2025
@youkaichao @abmfy
[torch.compile] store inductor compiled Python file (vllm-project#12182)
d32d59a 
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Bowen Wang <[email protected]>
abmfy pushed a commit to abmfy/vllm-flashinfer that referenced this pull request Jan 24, 2025
@youkaichao @abmfy
[torch.compile] store inductor compiled Python file (vllm-project#12182)
4843f33 
Signed-off-by: youkaichao <[email protected]>
kzawora-intel added a commit to HabanaAI/vllm-fork that referenced this pull request Jan 28, 2025
@kzawora-intel @wallashss
@jikunshang @gshtras @youkaichao @hongxiayang @russellb @Isotr0py @ice-tong @simon-mo @madamczykhabana @DarkLight1337 @gujingit @ywang96 @imkero @MartinGleize @ShangmingCai @hmellor @terrytangyuan @heheda12345 @wangxiyuan @isikhi @Jason-CKY @jinzhen-lin @mgoin @tlrmchlsmth @NickLucche @jeejeelee
Rebase 2025.01.21 (#714)
c9db39b 
- **[Bugfix] Fix score api for missing max_model_len validation
(vllm-project#12119)**
- **[Bugfix] Mistral tokenizer encode accept list of str (vllm-project#12149)**
- **[AMD][FP8] Using MI300 FP8 format on ROCm for block_quant (vllm-project#12134)**
- **[torch.compile] disable logging when cache is disabled (vllm-project#12043)**
- **[misc] fix cross-node TP (vllm-project#12166)**
- **[AMD][CI/Build][Bugfix] use pytorch stale wheel (vllm-project#12172)**
- **[core] further polish memory profiling (vllm-project#12126)**
- **[Docs] Fix broken link in SECURITY.md (vllm-project#12175)**
- **[Model] Port deepseek-vl2 processor, remove dependency (vllm-project#12169)**
- **[core] clean up executor class hierarchy between v1 and v0
(vllm-project#12171)**
- **[Misc] Support register quantization method out-of-tree (vllm-project#11969)**
- **[V1] Collect env var for usage stats (vllm-project#12115)**
- **[BUGFIX] Move scores to float32 in case of running xgrammar on cpu
(vllm-project#12152)**
- **[Bugfix] Fix multi-modal processors for transformers 4.48 (vllm-project#12187)**
- **[torch.compile] store inductor compiled Python file (vllm-project#12182)**
- **benchmark_serving support --served-model-name param (vllm-project#12109)**
- **[Misc] Add BNB support to GLM4-V model (vllm-project#12184)**
- **[V1] Add V1 support of Qwen2-VL (vllm-project#12128)**
- **[Model] Support for fairseq2 Llama (vllm-project#11442)**
- **[Bugfix] Fix num_heads value for simple connector when tp enabled
(vllm-project#12074)**
- **[torch.compile] fix sym_tensor_indices (vllm-project#12191)**
- **Move linting to `pre-commit` (vllm-project#11975)**
- **[DOC] Fix typo in docstring and assert message (vllm-project#12194)**
- **[DOC] Add missing docstring in LLMEngine.add_request() (vllm-project#12195)**
- **[Bugfix] Fix incorrect types in LayerwiseProfileResults (vllm-project#12196)**
- **[Model] Add Qwen2 PRM model support (vllm-project#12202)**
- **[Core] Interface for accessing model from `VllmRunner` (vllm-project#10353)**
- **[misc] add placeholder format.sh (vllm-project#12206)**
- **[CI/Build] Remove dummy CI steps (vllm-project#12208)**
- **[CI/Build] Make pre-commit faster (vllm-project#12212)**
- **[Model] Upgrade Aria to transformers 4.48 (vllm-project#12203)**
- **[misc] print a message to suggest how to bypass commit hooks
(vllm-project#12217)**
- **[core][bugfix] configure env var during import vllm (vllm-project#12209)**
- **[V1] Remove `_get_cache_block_size` (vllm-project#12214)**
- **[Misc] Pass `attention` to impl backend (vllm-project#12218)**
- **[Bugfix] Fix `HfExampleModels.find_hf_info` (vllm-project#12223)**
- **[CI] Pass local python version explicitly to pre-commit mypy.sh
(vllm-project#12224)**
- **[Misc] Update CODEOWNERS (vllm-project#12229)**
- **fix: update platform detection for M-series arm based MacBook
processors (vllm-project#12227)**
- **[misc] add cuda runtime version to usage data (vllm-project#12190)**
- **[bugfix] catch xgrammar unsupported array constraints (vllm-project#12210)**
- **[Kernel] optimize moe_align_block_size for cuda graph and large
num_experts (e.g. DeepSeek-V3) (vllm-project#12222)**
- **Add quantization and guided decoding CODEOWNERS (vllm-project#12228)**
- **[AMD][Build] Porting dockerfiles from the ROCm/vllm fork (vllm-project#11777)**
- **[BugFix] Fix GGUF tp>1 when vocab_size is not divisible by 64
(vllm-project#12230)**
- **[ci/build] disable failed and flaky tests (vllm-project#12240)**
- **[Misc] Rename `MultiModalInputsV2 -> MultiModalInputs` (vllm-project#12244)**
- **[Misc]Add BNB quantization for PaliGemmaForConditionalGeneration
(vllm-project#12237)**
- **[Misc] Remove redundant TypeVar from base model (vllm-project#12248)**
- **[Bugfix] Fix mm_limits access for merged multi-modal processor
(vllm-project#12252)**

---------

Signed-off-by: Wallas Santos <[email protected]>
Signed-off-by: Kunshang Ji <[email protected]>
Signed-off-by: Gregory Shtrasberg <[email protected]>
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: hongxyan <[email protected]>
Signed-off-by: Russell Bryant <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Michal Adamczyk <[email protected]>
Signed-off-by: zibai <[email protected]>
Signed-off-by: Roger Wang <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Martin Gleize <[email protected]>
Signed-off-by: Shangming Cai <[email protected]>
Signed-off-by: Harry Mellor <[email protected]>
Signed-off-by: Yuan Tang <[email protected]>
Signed-off-by: Chen Zhang <[email protected]>
Signed-off-by: wangxiyuan <[email protected]>
Signed-off-by: isikhi <[email protected]>
Signed-off-by: Jason Cheng <[email protected]>
Signed-off-by: Jinzhen Lin <[email protected]>
Signed-off-by: mgoin <[email protected]>
Signed-off-by: NickLucche <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Konrad Zawora <[email protected]>
Co-authored-by: Wallas Henrique <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>
Co-authored-by: Gregory Shtrasberg <[email protected]>
Co-authored-by: youkaichao <[email protected]>
Co-authored-by: Hongxia Yang <[email protected]>
Co-authored-by: Russell Bryant <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Co-authored-by: yancong <[email protected]>
Co-authored-by: Simon Mo <[email protected]>
Co-authored-by: Michal Adamczyk <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>
Co-authored-by: gujing <[email protected]>
Co-authored-by: Roger Wang <[email protected]>
Co-authored-by: imkero <[email protected]>
Co-authored-by: Martin Gleize <[email protected]>
Co-authored-by: mgleize user <[email protected]>
Co-authored-by: shangmingc <[email protected]>
Co-authored-by: Harry Mellor <[email protected]>
Co-authored-by: Yuan Tang <[email protected]>
Co-authored-by: Chen Zhang <[email protected]>
Co-authored-by: wangxiyuan <[email protected]>
Co-authored-by: Işık <[email protected]>
Co-authored-by: Roger Wang <[email protected]>
Co-authored-by: Cheng Kuan Yong Jason <[email protected]>
Co-authored-by: Jinzhen Lin <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Tyler Michael Smith <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Nicolò Lucchesi <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>
rasmith pushed a commit to rasmith/vllm that referenced this pull request Jan 30, 2025
@youkaichao @rasmith
[torch.compile] store inductor compiled Python file (vllm-project#12182)
4f572e0 
Signed-off-by: youkaichao <[email protected]>
Isotr0py pushed a commit to Isotr0py/vllm that referenced this pull request Feb 2, 2025
@youkaichao @Isotr0py
[torch.compile] store inductor compiled Python file (vllm-project#12182)
c14abd1 
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@heheda12345 heheda12345 heheda12345 approved these changes

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@youkaichao @heheda12345