Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create PlacementGroup for steps using vLLM #842

Merged
merged 7 commits into from
Jul 30, 2024
Merged

Conversation

gabrielmbmb
Copy link
Member

@gabrielmbmb gabrielmbmb commented Jul 30, 2024

Description

This PR updates RayPipeline to create a PlacementGroup when an step is using vLLM. The created placement group, contains as many GPU bundles as tensor_parallel_size specified in the vLLM initialisation and it uses the STRICT_PACK to have the GPU bundles in the same node. It also creates the placement group specifying the _soft_target_node_id, assuring that the _RayStepWrapper actor for the step using vLLM will be created in a specific node. This avoid having vLLM raising the exception Ray does not allocate any GPUs on the driver node. Consider adjusting the Ray placement group or running the driver on a GPU node, as it assures that the driver _StepWrapperRay created resides in the same node as the ray actors created by vLLM for the distributed inference.

@gabrielmbmb gabrielmbmb added the enhancement New feature or request label Jul 30, 2024
@gabrielmbmb gabrielmbmb added this to the 1.3.0 milestone Jul 30, 2024
@gabrielmbmb gabrielmbmb self-assigned this Jul 30, 2024
Copy link

Documentation for this PR has been built. You can view it at: https://distilabel.argilla.io/pr-842/

Copy link

codspeed-hq bot commented Jul 30, 2024

CodSpeed Performance Report

Merging #842 will not alter performance

Comparing ray-placement-group (f56a9bb) with develop (2aa977f)

Summary

✅ 1 untouched benchmarks

@gabrielmbmb gabrielmbmb marked this pull request as ready for review July 30, 2024 12:42
@gabrielmbmb gabrielmbmb merged commit be61d20 into develop Jul 30, 2024
7 checks passed
@gabrielmbmb gabrielmbmb deleted the ray-placement-group branch July 30, 2024 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

1 participant