[Hardware][Gaudi][Bugfix] Fix HPU tensor parallelism, enable multiprocessing executor #12167
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Konrad Zawora <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
Signed-off-by: Konrad Zawora <[email protected]>
kzawora-intel
force-pushed
the
private/kzawora/tp_upstream_fix
branch
from
46d59ea to
68f1a43
Compare
youkaichao
reviewed
Jan 21, 2025
zhenwei-intel
added a commit
to zhenwei-intel/vllm
that referenced
this pull request
Jan 22, 2025
Signed-off-by: zhenwei <[email protected]>
Signed-off-by: Konrad Zawora <[email protected]>
Signed-off-by: Konrad Zawora <[email protected]>
Signed-off-by: Konrad Zawora <[email protected]>
WoosukKwon
approved these changes
Jan 22, 2025
github-actions
bot
added
the
ready
abmfy
pushed a commit
to abmfy/vllm-flashinfer
that referenced
this pull request
Jan 24, 2025
Signed-off-by: Konrad Zawora <[email protected]> Signed-off-by: Bowen Wang <[email protected]>
abmfy
pushed a commit
to abmfy/vllm-flashinfer
that referenced
this pull request
Jan 24, 2025
Signed-off-by: Konrad Zawora <[email protected]>
tjtanaa
pushed a commit
to EmbeddedLLM/vllm
that referenced
this pull request
Jan 27, 2025
Signed-off-by: Konrad Zawora <[email protected]>
tjtanaa
pushed a commit
to EmbeddedLLM/vllm
that referenced
this pull request
Jan 27, 2025
Signed-off-by: Konrad Zawora <[email protected]>
tjtanaa
pushed a commit
to EmbeddedLLM/vllm
that referenced
this pull request
Jan 28, 2025
Signed-off-by: Konrad Zawora <[email protected]>
rasmith
pushed a commit
to rasmith/vllm
that referenced
this pull request
Jan 30, 2025
Signed-off-by: Konrad Zawora <[email protected]>
Isotr0py
pushed a commit
to Isotr0py/vllm
that referenced
this pull request
Feb 2, 2025
Signed-off-by: Konrad Zawora <[email protected]> Signed-off-by: Isotr0py <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#11256 changed the input broadcasting flow a little bit, and in tensor parallel scenario we cannot assume that all workers will receive
execute_model_reqfrom the executor. Since broadcast from driver worker is now performed withinLocalOrDistributedWorkerBase.prepare_inputcalled byLocalOrDistributedWorkerBase.execute_model, for non-driver workers, we can expectexecute_model_reqto be None withinHPUWorker.execute_model(which is essentially a wrapper aroundLocalOrDistributedWorkerBase.execute_modelwith some additional HPU-specific profiling sugar). Asserting thatexecute_model_req is not Nonebreaks the tensor parallelism, and this PR fixes that. Additionally, it enables usage of multiprocessing executor on HPU, as it's fully functional on HPU (as introduced in #11030).