[Bugfix] Fix disagg hang caused by the prefill and decode communication issues #12723
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
houseroad
force-pushed
the
fb-sync-lufang
branch
from
8a54ef7 to
195c282
Compare
Signed-off-by: Lu Fang <[email protected]>
houseroad
force-pushed
the
fb-sync-lufang
branch
from
195c282 to
4779f98
Compare
|
@KuntaiDu, friendly ping for the review :-) |
houseroad
force-pushed
the
fb-sync-lufang
branch
from
4779f98 to
da06b0c
Compare
houseroad
requested review from
WoosukKwon,
robertgshaw2-redhat,
njhill,
ywang96,
comaniac,
alexm-redhat,
mgoin and
tlrmchlsmth
as code owners
houseroad
force-pushed
the
fb-sync-lufang
branch
from
da06b0c to
4779f98
Compare
Oh right, when the buffer is almost full, the prefill instance can finish inference of this request, but the KV cache of this request won't be added to the buffer, so the prefill instance will return |
| while not is_buffer_available(tokens_roi_recver): | ||
| self.buffer_cv.wait() |
There was a problem hiding this comment.
Would be nice if we can log here to make sure people know that the engine is waiting for the KV cache that is already generated but not entered into the lookup buffer?
There was a problem hiding this comment.
Added some logging.
| # repeatedly. | ||
| logger.debug("KV transfer buffer is full. Handling...") | ||
| while self.buffer_size > self.buffer_size_threshold: | ||
| self.full_handler() |
There was a problem hiding this comment.
Maybe also remove the code for full_handler if we don't use it.
|
Just tested, it works (and the performance gets much better). Thank you for contributing! |
Signed-off-by: Lu Fang <[email protected]>
github-actions
bot
added
the
ready
…on issues (vllm-project#12723) Signed-off-by: Lu Fang <[email protected]>
…on issues (vllm-project#12723) Signed-off-by: Lu Fang <[email protected]>
…on issues (vllm-project#12723) Signed-off-by: Lu Fang <[email protected]> Signed-off-by: SzymonOzog <[email protected]>
Now prefill acts as a server to wait for decode kv request. But when the kv is not ready, prefill return None and decode can proceed. This caused prefill kv buffer accumulate and block other prefill from moving forward. Fix by poll wait + relinquish to ensure kv ready and send the correct kv to decode.
The PR was created by @jiayisuse. He is in China for holiday, so submit the PR on his behalf to get review first.