Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VLM] Report multi_modal_placeholders in output #10407

Merged
merged 24 commits into from
Nov 18, 2024
Merged

[VLM] Report multi_modal_placeholders in output #10407

merged 24 commits into from
Nov 18, 2024

Conversation

lk-chen
Copy link
Contributor

@lk-chen lk-chen commented Nov 18, 2024 •
edited by github-actions bot
Loading

Report multi_modal_placeholders in output

This helps inspect VLM model performance, e.g. distinguish text tokens and multi-modal embeddings.

related: #8346

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

DarkLight1337
DarkLight1337 approved these changes Nov 18, 2024
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for improving the developer experience!

@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 18, 2024
@DarkLight1337
Copy link
Member

Do you get these test errors locally?

@lk-chen
Copy link
Contributor Author

lk-chen commented Nov 18, 2024

Do you get these test errors locally?

I cannot reproduce those errors due to lacking permission to llama models.
But by reading the errors, I made commit 8c5fdf8 and hope to pass the test

@DarkLight1337
Copy link
Member

Please fix the lint errors.

Linkun Chen and others added 20 commits November 18, 2024 05:52
Patch multi_modal_placeholders to RequestOutput
678c291 
* confirm that `offline_inference_vision_language.py` and `benchmark_throughput.py` runs
* FIXME: the placeholders in output, however, is empty
 - will fix in next commit

Signed-off-by: Linkun Chen <[email protected]>
pipe multi_modal_placeholders from intput to final output
a1cdcb3 
* add test for pixtral
* fix a typo

Signed-off-by: Linkun Chen <[email protected]>
@WoosukKwon
[V1] Add code owners for V1 (vllm-project#10397)
f60964a 
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@youkaichao
[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…
578e482 
…ect#10383)

Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@ywang96
[V1] Refactor model executable interface for all text-only language m…
7fa97cf 
…odels (vllm-project#10374)

Signed-off-by: Roger Wang <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@xuechendi
[CI/Build] Fix IDC hpu [Device not found] issue (vllm-project#10384)
629f512 
Signed-off-by: Chendi Xue <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@Isotr0py
[Bugfix][CPU] Fix CPU embedding runner with tensor parallel (vllm-pro…
7539ab8 
…ject#10394)

Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@youkaichao
[platforms] refactor cpu code (vllm-project#10402)
bce660d 
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@jikunshang
[Hardware] [HPU]add mark_step for hpu (vllm-project#10239)
305708b 
Signed-off-by: Kunshang Ji <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@imkero
[Bugfix] Fix mrope_position_delta in non-last prefill chunk (vllm-pro…
871a773 
…ject#10403)

Signed-off-by: imkero <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@wchen61
[Misc] Enhance offline_inference to support user-configurable paramet… (
242bb53 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@DarkLight1337
Fix initialization
f5312d3 
Signed-off-by: Cyrus Leung <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@DarkLight1337
Run isort
439e324 
Signed-off-by: Linkun Chen <[email protected]>
@DarkLight1337
isort
60815f2 
Signed-off-by: Linkun Chen <[email protected]>
@DarkLight1337
isort
ec46755 
Signed-off-by: Linkun Chen <[email protected]>
@Isotr0py
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
ce3ae6f 
…m-project#10327)

Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@HollowMan6
[Bugfix] Ignore ray reinit error when current platform is ROCm or XPU (…
466b2cf 
…vllm-project#10375)

Signed-off-by: Hollow Man <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
update RequestOutput.__init__() to take multi_modal_placeholders as…
3f092ce 
… optional argument

also require it to be passed as kwargs, to avoid breaking existing code.

Signed-off-by: Linkun Chen <[email protected]>
update RequestOutput.__init__() to take multi_modal_placeholders as…
dd8427e 
… optional argument

also require it to be passed as kwargs, to avoid breaking existing code.

Signed-off-by: Linkun Chen <[email protected]>
update RequestOutput.__init__() to take multi_modal_placeholders as…
76ac8b0 
… optional argument

also require it to be passed as kwargs, to avoid breaking existing code.

Signed-off-by: Linkun Chen <[email protected]>
@lk-chen lk-chen requested a review from WoosukKwon as a code owner November 18, 2024 05:52
@mergify mergify bot added the ci/build label Nov 18, 2024
@lk-chen lk-chen marked this pull request as draft November 18, 2024 05:55
disable mypy type check
c963a25 
mypy is not smart enough to validate kwargs

Signed-off-by: Linkun Chen <[email protected]>
@lk-chen lk-chen marked this pull request as ready for review November 18, 2024 06:00
disable mypy type check
470fbd3 
mypy is not smart enough to validate kwargs

Signed-off-by: Linkun Chen <[email protected]>
@lk-chen
Copy link
Contributor Author

lk-chen commented Nov 18, 2024 •
edited
Loading

Please fix the lint errors.

Done. Could you help me remove the accidentally-added reviewers? Those were added when I tried to fix DCO

remove unnecessary debug code
550be23 
Signed-off-by: Linkun Chen <[email protected]>
comaniac
comaniac approved these changes Nov 18, 2024
View reviewed changes
Copy link
Collaborator

@comaniac comaniac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The reviewers were added automatically due to code owners so it's fine.

Also cc @ywang96 @mgoin

@DarkLight1337 DarkLight1337 merged commit c7dec92 into vllm-project:main Nov 18, 2024
50 checks passed
coolkp pushed a commit to coolkp/vllm that referenced this pull request Nov 20, 2024
@lk-chen @coolkp
[VLM] Report multi_modal_placeholders in output (vllm-project#10407)
dd84f40 
Signed-off-by: Linkun Chen <[email protected]>
KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024
@lk-chen
[VLM] Report multi_modal_placeholders in output (vllm-project#10407)
add442e 
Signed-off-by: Linkun Chen <[email protected]>
mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 20, 2024
@lk-chen @mfournioux
[VLM] Report multi_modal_placeholders in output (vllm-project#10407)
dad15c3 
Signed-off-by: Linkun Chen <[email protected]>
Signed-off-by: Maxime Fournioux <[email protected]>
rickyyx pushed a commit to rickyyx/vllm that referenced this pull request Nov 20, 2024
@lk-chen @rickyyx
[VLM] Report multi_modal_placeholders in output (vllm-project#10407)
40753ce 
Signed-off-by: Linkun Chen <[email protected]>
Signed-off-by: rickyx <[email protected]>
tlrmchlsmth pushed a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024
@lk-chen @tlrmchlsmth
[VLM] Report multi_modal_placeholders in output (vllm-project#10407)
4a4c938 
Signed-off-by: Linkun Chen <[email protected]>
Signed-off-by: Tyler Michael Smith <[email protected]>
prashantgupta24 pushed a commit to opendatahub-io/vllm that referenced this pull request Dec 3, 2024
@lk-chen @prashantgupta24
[VLM] Report multi_modal_placeholders in output (vllm-project#10407)
663fb57 
Signed-off-by: Linkun Chen <[email protected]>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@lk-chen
[VLM] Report multi_modal_placeholders in output (vllm-project#10407)
2813a5f 
Signed-off-by: Linkun Chen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@comaniac comaniac comaniac approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

Assignees
No one assigned
Labels
ci/build ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.