[AMD][Build] Porting dockerfiles from the ROCm/vllm fork #11777

gshtras · 2025-01-06T19:43:57Z

An attempt to unify the build process with how it is done in ROCm/vllm
Split the build process into:

Building the required dependency libraries using Dockerfile.rocm_base. Not needed for the end user, done by AMD.
Building the vLLM itself on top of it.

In addition to not building the libraries each time, the new base image is now much smaller (7GB on docker hub vs 18GB previously), which allows to make the build process much faster (~10 minutes)

Signed-off-by: Gregory Shtrasberg <[email protected]>

github-actions · 2025-01-06T19:44:08Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

Signed-off-by: Gregory Shtrasberg <[email protected]>

Dockerfile.rocm

Signed-off-by: Gregory Shtrasberg <[email protected]>

DaBossCoda · 2025-01-12T19:50:31Z

nice

mergify · 2025-01-12T22:40:51Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gshtras.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Gregory Shtrasberg <[email protected]>

robertgshaw2-redhat · 2025-01-13T15:56:42Z

@SageMoore - could you take a quick look through this?

SageMoore

Looks reasonable to me. I assume the configs are copied over from the rocm/vllm-dev container and are expected to be faster than what we have on main? Do you have any performance results that you can share?

Dockerfile.rocm_base

SageMoore · 2025-01-13T16:36:04Z

Dockerfile.rocm_base

+ARG FA_BRANCH
+ARG FA_REPO
+RUN git clone ${PYTORCH_REPO} pytorch
+RUN cd pytorch && git checkout ${PYTORCH_BRANCH} && \


Does this version of pytorch support rocm 6.3? If not, would it make more sense to pull in the 6.2 wheel from pypy instead of building from source?

This is the same base image and therefore a combination of versions used in the AMD weekly dockers - rocm/vllm-dev:main (out of ROCm/vllm.git)
6.2 wheel is a all-in-one whl that brings with it an entire ROCm worth of .so's, including the old less performant hipblaslt

gshtras · 2025-01-13T21:43:33Z

Looks reasonable to me. I assume the configs are copied over from the rocm/vllm-dev container and are expected to be faster than what we have on main? Do you have any performance results that you can share?

Not sure which configs do you mean. The change to the triton tuning configs in this PR is due to the triton version change. Triton 3.2+ deprecated the use of num_stages:0 in favor of num_stages:2

mgoin · 2025-01-14T21:11:24Z

It looks like the CI isn't able to find the container?

gshtras · 2025-01-14T21:17:49Z

It looks like the CI isn't able to find the container?

All GPU nodes of AMD CI are currently down due to a network issue in the compute cluster used there
It looks like the build of the container is the only one that succeeded as it doesn't use GPU nodes

Signed-off-by: Gregory Shtrasberg <[email protected]>

gshtras · 2025-01-17T00:03:47Z

To unbreak the CI we need vllm-project/buildkite-ci#57

Signed-off-by: Gregory Shtrasberg <[email protected]>

mergify · 2025-01-18T03:19:14Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gshtras.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

DarkLight1337 · 2025-01-20T12:21:53Z

Can you fix the merge conflicts?

Signed-off-by: Gregory Shtrasberg <[email protected]>

…t#11777) Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: Bowen Wang <[email protected]>

…t#11777) Signed-off-by: Gregory Shtrasberg <[email protected]>

- **[Bugfix] Fix score api for missing max_model_len validation (vllm-project#12119)** - **[Bugfix] Mistral tokenizer encode accept list of str (vllm-project#12149)** - **[AMD][FP8] Using MI300 FP8 format on ROCm for block_quant (vllm-project#12134)** - **[torch.compile] disable logging when cache is disabled (vllm-project#12043)** - **[misc] fix cross-node TP (vllm-project#12166)** - **[AMD][CI/Build][Bugfix] use pytorch stale wheel (vllm-project#12172)** - **[core] further polish memory profiling (vllm-project#12126)** - **[Docs] Fix broken link in SECURITY.md (vllm-project#12175)** - **[Model] Port deepseek-vl2 processor, remove dependency (vllm-project#12169)** - **[core] clean up executor class hierarchy between v1 and v0 (vllm-project#12171)** - **[Misc] Support register quantization method out-of-tree (vllm-project#11969)** - **[V1] Collect env var for usage stats (vllm-project#12115)** - **[BUGFIX] Move scores to float32 in case of running xgrammar on cpu (vllm-project#12152)** - **[Bugfix] Fix multi-modal processors for transformers 4.48 (vllm-project#12187)** - **[torch.compile] store inductor compiled Python file (vllm-project#12182)** - **benchmark_serving support --served-model-name param (vllm-project#12109)** - **[Misc] Add BNB support to GLM4-V model (vllm-project#12184)** - **[V1] Add V1 support of Qwen2-VL (vllm-project#12128)** - **[Model] Support for fairseq2 Llama (vllm-project#11442)** - **[Bugfix] Fix num_heads value for simple connector when tp enabled (vllm-project#12074)** - **[torch.compile] fix sym_tensor_indices (vllm-project#12191)** - **Move linting to `pre-commit` (vllm-project#11975)** - **[DOC] Fix typo in docstring and assert message (vllm-project#12194)** - **[DOC] Add missing docstring in LLMEngine.add_request() (vllm-project#12195)** - **[Bugfix] Fix incorrect types in LayerwiseProfileResults (vllm-project#12196)** - **[Model] Add Qwen2 PRM model support (vllm-project#12202)** - **[Core] Interface for accessing model from `VllmRunner` (vllm-project#10353)** - **[misc] add placeholder format.sh (vllm-project#12206)** - **[CI/Build] Remove dummy CI steps (vllm-project#12208)** - **[CI/Build] Make pre-commit faster (vllm-project#12212)** - **[Model] Upgrade Aria to transformers 4.48 (vllm-project#12203)** - **[misc] print a message to suggest how to bypass commit hooks (vllm-project#12217)** - **[core][bugfix] configure env var during import vllm (vllm-project#12209)** - **[V1] Remove `_get_cache_block_size` (vllm-project#12214)** - **[Misc] Pass `attention` to impl backend (vllm-project#12218)** - **[Bugfix] Fix `HfExampleModels.find_hf_info` (vllm-project#12223)** - **[CI] Pass local python version explicitly to pre-commit mypy.sh (vllm-project#12224)** - **[Misc] Update CODEOWNERS (vllm-project#12229)** - **fix: update platform detection for M-series arm based MacBook processors (vllm-project#12227)** - **[misc] add cuda runtime version to usage data (vllm-project#12190)** - **[bugfix] catch xgrammar unsupported array constraints (vllm-project#12210)** - **[Kernel] optimize moe_align_block_size for cuda graph and large num_experts (e.g. DeepSeek-V3) (vllm-project#12222)** - **Add quantization and guided decoding CODEOWNERS (vllm-project#12228)** - **[AMD][Build] Porting dockerfiles from the ROCm/vllm fork (vllm-project#11777)** - **[BugFix] Fix GGUF tp>1 when vocab_size is not divisible by 64 (vllm-project#12230)** - **[ci/build] disable failed and flaky tests (vllm-project#12240)** - **[Misc] Rename `MultiModalInputsV2 -> MultiModalInputs` (vllm-project#12244)** - **[Misc]Add BNB quantization for PaliGemmaForConditionalGeneration (vllm-project#12237)** - **[Misc] Remove redundant TypeVar from base model (vllm-project#12248)** - **[Bugfix] Fix mm_limits access for merged multi-modal processor (vllm-project#12252)** --------- Signed-off-by: Wallas Santos <[email protected]> Signed-off-by: Kunshang Ji <[email protected]> Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: youkaichao <[email protected]> Signed-off-by: hongxyan <[email protected]> Signed-off-by: Russell Bryant <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Michal Adamczyk <[email protected]> Signed-off-by: zibai <[email protected]> Signed-off-by: Roger Wang <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Martin Gleize <[email protected]> Signed-off-by: Shangming Cai <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Signed-off-by: Yuan Tang <[email protected]> Signed-off-by: Chen Zhang <[email protected]> Signed-off-by: wangxiyuan <[email protected]> Signed-off-by: isikhi <[email protected]> Signed-off-by: Jason Cheng <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: NickLucche <[email protected]> Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: Konrad Zawora <[email protected]> Co-authored-by: Wallas Henrique <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> Co-authored-by: Gregory Shtrasberg <[email protected]> Co-authored-by: youkaichao <[email protected]> Co-authored-by: Hongxia Yang <[email protected]> Co-authored-by: Russell Bryant <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: yancong <[email protected]> Co-authored-by: Simon Mo <[email protected]> Co-authored-by: Michal Adamczyk <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: gujing <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: imkero <[email protected]> Co-authored-by: Martin Gleize <[email protected]> Co-authored-by: mgleize user <[email protected]> Co-authored-by: shangmingc <[email protected]> Co-authored-by: Harry Mellor <[email protected]> Co-authored-by: Yuan Tang <[email protected]> Co-authored-by: Chen Zhang <[email protected]> Co-authored-by: wangxiyuan <[email protected]> Co-authored-by: Işık <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Cheng Kuan Yong Jason <[email protected]> Co-authored-by: Jinzhen Lin <[email protected]> Co-authored-by: Michael Goin <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Michael Goin <[email protected]> Co-authored-by: Nicolò Lucchesi <[email protected]> Co-authored-by: Jee Jee Li <[email protected]>

…t#11777) Signed-off-by: Gregory Shtrasberg <[email protected]>

…t#11777) Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: Isotr0py <[email protected]>

Porting dockerfiles from the ROCm/vllm fork

fb0011a

Signed-off-by: Gregory Shtrasberg <[email protected]>

Removed references to the obsolete patch for old rocm versions

a309129

Signed-off-by: Gregory Shtrasberg <[email protected]>

mergify bot added documentation Improvements or additions to documentation ci/build labels Jan 6, 2025

gshtras force-pushed the dockerfile_unification branch from 8016502 to a309129 Compare January 6, 2025 19:44

mgoin self-requested a review January 6, 2025 21:19

Adjust configs for new triton version

0b9f7f8

Signed-off-by: Gregory Shtrasberg <[email protected]>

DarkLight1337 mentioned this pull request Jan 7, 2025

Update amd-installation.md #11470

Closed

hongxiayang added the rocm label Jan 7, 2025

mgoin requested a review from robertgshaw2-redhat January 8, 2025 22:08

mgoin reviewed Jan 8, 2025

View reviewed changes

Dockerfile.rocm Outdated Show resolved Hide resolved

Dockerfile.rocm Outdated Show resolved Hide resolved

Remove punica and use the right repo

c500477

Signed-off-by: Gregory Shtrasberg <[email protected]>

gshtras requested a review from mgoin January 9, 2025 18:22

DarkLight1337 mentioned this pull request Jan 10, 2025

[Doc] links Tensorizer example #11918

Open

gshtras mentioned this pull request Jan 10, 2025

[platforms] enable platform plugins #11602

Merged

mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 12, 2025

mergify bot added the needs-rebase label Jan 12, 2025

Merge remote-tracking branch 'origin/main' into dockerfile_unification

25e1db2

mergify bot removed the needs-rebase label Jan 13, 2025

Documentation lint

907392f

Signed-off-by: Gregory Shtrasberg <[email protected]>

SageMoore reviewed Jan 13, 2025

View reviewed changes

robertgshaw2-redhat approved these changes Jan 13, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into dockerfile_unification

5c36cb8

gshtras force-pushed the dockerfile_unification branch from 1092866 to 5c36cb8 Compare January 16, 2025 20:52

Test stage in the dockerfile to be used in the CI

7b90ed5

Signed-off-by: Gregory Shtrasberg <[email protected]>

gshtras added 4 commits January 17, 2025 15:54

Merge remote-tracking branch 'origin/main' into dockerfile_unification

9df47f7

Merge remote-tracking branch 'origin/main' into dockerfile_unification

35dea97

Fix pytest module lookup by removing vllm subfolder

a69c453

Signed-off-by: Gregory Shtrasberg <[email protected]>

Installing lm_eval in the test image

0773f53

Signed-off-by: Gregory Shtrasberg <[email protected]>

mergify bot added the needs-rebase label Jan 18, 2025

Merge remote-tracking branch 'origin/main' into dockerfile_unification

afe691c

mergify bot removed the needs-rebase label Jan 20, 2025

gshtras requested review from SageMoore and robertgshaw2-redhat January 20, 2025 18:11

Base advanced to 6.3.1

aa6c73d

Signed-off-by: Gregory Shtrasberg <[email protected]>

mgoin approved these changes Jan 21, 2025

View reviewed changes

mgoin enabled auto-merge (squash) January 21, 2025 01:42

youkaichao disabled auto-merge January 21, 2025 04:22

youkaichao merged commit d4b62d4 into vllm-project:main Jan 21, 2025
68 of 70 checks passed

kzawora-intel mentioned this pull request Jan 21, 2025

Rebase 2025.01.21 HabanaAI/vllm-fork#714

Merged

gshtras deleted the dockerfile_unification branch January 22, 2025 18:49

abmfy pushed a commit to abmfy/vllm-flashinfer that referenced this pull request Jan 24, 2025

[AMD][Build] Porting dockerfiles from the ROCm/vllm fork (vllm-projec…

0adfbae

…t#11777) Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: Bowen Wang <[email protected]>

abmfy pushed a commit to abmfy/vllm-flashinfer that referenced this pull request Jan 24, 2025

[AMD][Build] Porting dockerfiles from the ROCm/vllm fork (vllm-projec…

c179187

…t#11777) Signed-off-by: Gregory Shtrasberg <[email protected]>

rasmith pushed a commit to rasmith/vllm that referenced this pull request Jan 30, 2025

[AMD][Build] Porting dockerfiles from the ROCm/vllm fork (vllm-projec…

fbfe8c3

…t#11777) Signed-off-by: Gregory Shtrasberg <[email protected]>

Isotr0py pushed a commit to Isotr0py/vllm that referenced this pull request Feb 2, 2025

[AMD][Build] Porting dockerfiles from the ROCm/vllm fork (vllm-projec…

329fb52

…t#11777) Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: Isotr0py <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMD][Build] Porting dockerfiles from the ROCm/vllm fork #11777

[AMD][Build] Porting dockerfiles from the ROCm/vllm fork #11777

gshtras commented Jan 6, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Jan 6, 2025

DaBossCoda commented Jan 12, 2025

mergify bot commented Jan 12, 2025

robertgshaw2-redhat commented Jan 13, 2025

SageMoore left a comment

SageMoore Jan 13, 2025

gshtras Jan 13, 2025

gshtras commented Jan 13, 2025

mgoin commented Jan 14, 2025

gshtras commented Jan 14, 2025 •

edited by mgoin

Loading

gshtras commented Jan 17, 2025

mergify bot commented Jan 18, 2025

DarkLight1337 commented Jan 20, 2025

[AMD][Build] Porting dockerfiles from the ROCm/vllm fork #11777

[AMD][Build] Porting dockerfiles from the ROCm/vllm fork #11777

Conversation

gshtras commented Jan 6, 2025 • edited by github-actions bot Loading

github-actions bot commented Jan 6, 2025

DaBossCoda commented Jan 12, 2025

mergify bot commented Jan 12, 2025

robertgshaw2-redhat commented Jan 13, 2025

SageMoore left a comment

Choose a reason for hiding this comment

SageMoore Jan 13, 2025

Choose a reason for hiding this comment

gshtras Jan 13, 2025

Choose a reason for hiding this comment

gshtras commented Jan 13, 2025

mgoin commented Jan 14, 2025

gshtras commented Jan 14, 2025 • edited by mgoin Loading

gshtras commented Jan 17, 2025

mergify bot commented Jan 18, 2025

DarkLight1337 commented Jan 20, 2025

gshtras commented Jan 6, 2025 •

edited by github-actions bot

Loading

gshtras commented Jan 14, 2025 •

edited by mgoin

Loading