[Core] Make encoder-decoder inputs a nested structure to be more composable #9604

DarkLight1337 · 2024-10-23T05:01:59Z

This PR updates encoder-decoder inputs: from a flat schema that inherits from decoder-only inputs with additional encoder-* fields, to a nested structure with separate encoder and decoder components.

With separated encoder and decoder components, it becomes easier to support input embeddings for decoder-only models first in #6869.

github-actions · 2024-10-23T05:02:10Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

DarkLight1337 · 2024-10-23T07:41:29Z

@heheda12345 PTAL at the changes to mllama processor to ensure that the semantics remain the same. I can successfully run the example script, so it's probably fine.

vllm/model_executor/models/mllama.py

heheda12345 · 2024-10-23T20:57:40Z

@DarkLight1337 I've verified that test_mllama.py is successful, and the input of MllamaForConditionalGeneration.forward does not changed, so I think this pr does not change the behavior of mllama.

vllm/inputs/preprocess.py

DarkLight1337 · 2024-10-26T02:28:54Z

To unblock the next steps, @robertgshaw2-neuralmagic @njhill can you review this?

njhill

Thanks @DarkLight1337, looks good

njhill · 2024-10-30T22:34:47Z

vllm/sequence.py

+        if inputs["type"] == "token":
+            return inputs.get("prompt")

-        return cast(Optional[str], self.inputs.get(prompt_key))
+        assert_never(inputs)


Is there a reason to not write these like this? (which is clearer imo)

assert inputs["type"] == "token" return inputs.get("prompt")

It's to reduce diffs when introducing #6869 which adds a new inputs["type"] == "embed" branch.

vllm/model_executor/models/mllama.py

vllm/inputs/preprocess.py

njhill · 2024-10-30T22:54:28Z

vllm/inputs/preprocess.py

+        if encoder_inputs["type"] == "token":
+            pass
+        else:
+            assert_never(encoder_inputs)


Presumably this is needed for static type checking and this wouldn't suffice?

assert encoder_inputs["type"] == "token"

See my first comment.

mergify · 2024-10-30T22:57:48Z

This pull request has merge conflicts that must be resolved before it can be
merged. @DarkLight1337 please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

DarkLight1337 · 2024-11-01T08:16:46Z

@njhill I have finished addressing your comments.

Signed-off-by: DarkLight1337 <[email protected]>

njhill

Thanks @DarkLight1337

…osable (#9604) Signed-off-by: DarkLight1337 <[email protected]>

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Loc Huynh <[email protected]>

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Sumit Dubey <[email protected]>

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]>

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Maxime Fournioux <[email protected]>

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]>

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]>

DarkLight1337 added 2 commits October 23, 2024 04:59

Make encoder-decoder inputs a composed structure

0c872b3

Merge branch 'main' into refactor-preprocessing

9287a1b

DarkLight1337 requested review from WoosukKwon, zhuohan123, youkaichao, alexm-redhat, comaniac and njhill as code owners October 23, 2024 05:01

DarkLight1337 added 2 commits October 23, 2024 05:11

Rename

fa5ad17

Fix type error

44fd058

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 23, 2024

DarkLight1337 added 2 commits October 23, 2024 07:05

Fix test

b73a345

Fix llama-3.2

fa968b5

DarkLight1337 mentioned this pull request Oct 23, 2024

[Core] generate from input embeds #6869

Open

heheda12345 reviewed Oct 23, 2024

View reviewed changes

vllm/model_executor/models/mllama.py Show resolved Hide resolved

heheda12345 reviewed Oct 23, 2024

View reviewed changes

vllm/inputs/preprocess.py Outdated Show resolved Hide resolved

DarkLight1337 added 4 commits October 24, 2024 02:29

Remove force_bos

906ee1e

Add example output

005ad95

format

a5f0c16

Fix

6ab44e4

DarkLight1337 requested a review from robertgshaw2-redhat October 26, 2024 02:29

Merge branch 'main' into refactor-preprocessing

21be11f

njhill reviewed Oct 30, 2024

View reviewed changes

mergify bot added the needs-rebase label Oct 30, 2024

Merge branch 'main' into refactor-preprocessing

1f927d2

DarkLight1337 added 3 commits October 31, 2024 01:31

Fix merge

760db05

Update mllama processing

acb8e6f

Fix line

3bed519

mergify bot removed the needs-rebase label Oct 31, 2024

DarkLight1337 added 7 commits October 31, 2024 01:33

format

ea861e0

Avoid repeated lookups

f654421

Remove unused import

594794e

Fix mypy

08ea824

Merge branch 'main' into refactor-preprocessing

b622f41

Merge branch 'main' into refactor-preprocessing

800960d

Fix merge

283bc2c

DarkLight1337 and others added 7 commits November 2, 2024 17:06

Merge branch 'main' into refactor-preprocessing

e8169ea

Signed-off-by: DarkLight1337 <[email protected]>

Merge branch 'main' into refactor-preprocessing

61bf1d1

Fix missing import

b45cdc9

Signed-off-by: DarkLight1337 <[email protected]>

Improve error message

4d33b1e

Signed-off-by: DarkLight1337 <[email protected]>

Add missing export

0a549e5

Signed-off-by: DarkLight1337 <[email protected]>

Improve error message.

f741a75

Format

cd231fa

njhill approved these changes Nov 4, 2024

View reviewed changes

DarkLight1337 merged commit bbc3619 into main Nov 5, 2024
60 checks passed

DarkLight1337 deleted the refactor-preprocessing branch November 5, 2024 02:07

DarkLight1337 added a commit that referenced this pull request Nov 5, 2024

[Core] Make encoder-decoder inputs a nested structure to be more comp…

71e5996

…osable (#9604) Signed-off-by: DarkLight1337 <[email protected]>

JC1DA pushed a commit to JC1DA/vllm that referenced this pull request Nov 11, 2024

[Core] Make encoder-decoder inputs a nested structure to be more comp…

a2e9819

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Loc Huynh <[email protected]>

sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024

[Core] Make encoder-decoder inputs a nested structure to be more comp…

0970400

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Sumit Dubey <[email protected]>

KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024

[Core] Make encoder-decoder inputs a nested structure to be more comp…

92dbfb4

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]>

sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024

[Core] Make encoder-decoder inputs a nested structure to be more comp…

7f72481

…osable (vllm-project#9604) Signed-off-by: DarkLight1337 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Make encoder-decoder inputs a nested structure to be more composable #9604

[Core] Make encoder-decoder inputs a nested structure to be more composable #9604

DarkLight1337 commented Oct 23, 2024 •

edited

Loading

github-actions bot commented Oct 23, 2024

DarkLight1337 commented Oct 23, 2024 •

edited

Loading

heheda12345 commented Oct 23, 2024

DarkLight1337 commented Oct 26, 2024

njhill left a comment

njhill Oct 30, 2024

DarkLight1337 Oct 31, 2024

njhill Oct 30, 2024

DarkLight1337 Oct 31, 2024

mergify bot commented Oct 30, 2024

DarkLight1337 commented Nov 1, 2024

njhill left a comment

[Core] Make encoder-decoder inputs a nested structure to be more composable #9604

[Core] Make encoder-decoder inputs a nested structure to be more composable #9604

Conversation

DarkLight1337 commented Oct 23, 2024 • edited Loading

github-actions bot commented Oct 23, 2024

DarkLight1337 commented Oct 23, 2024 • edited Loading

heheda12345 commented Oct 23, 2024

DarkLight1337 commented Oct 26, 2024

njhill left a comment

Choose a reason for hiding this comment

njhill Oct 30, 2024

Choose a reason for hiding this comment

DarkLight1337 Oct 31, 2024

Choose a reason for hiding this comment

njhill Oct 30, 2024

Choose a reason for hiding this comment

DarkLight1337 Oct 31, 2024

Choose a reason for hiding this comment

mergify bot commented Oct 30, 2024

DarkLight1337 commented Nov 1, 2024

njhill left a comment

Choose a reason for hiding this comment

DarkLight1337 commented Oct 23, 2024 •

edited

Loading

DarkLight1337 commented Oct 23, 2024 •

edited

Loading