Create PlacementGroup
for steps using vLLM
#842
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR updates
RayPipeline
to create a PlacementGroup when an step is usingvLLM
. The created placement group, contains as many GPU bundles astensor_parallel_size
specified in thevLLM
initialisation and it uses theSTRICT_PACK
to have the GPU bundles in the same node. It also creates the placement group specifying the_soft_target_node_id
, assuring that the_RayStepWrapper
actor for the step usingvLLM
will be created in a specific node. This avoid havingvLLM
raising the exceptionRay does not allocate any GPUs on the driver node. Consider adjusting the Ray placement group or running the driver on a GPU node
, as it assures that the driver_StepWrapperRay
created resides in the same node as the ray actors created byvLLM
for the distributed inference.