Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify input/output types #295

Merged
merged 11 commits into from
Nov 21, 2023
Merged

Unify input/output types #295

merged 11 commits into from
Nov 21, 2023

Conversation

mrwyattii
Copy link
Contributor

@mrwyattii mrwyattii commented Nov 13, 2023

Several changes in this PR, with the main goal to make the inputs/outputs of pipeline and client.generate match. That is, each can take str|List[str] as input and will return List[Response] where Response is mii.batching.data_classes.Response.

In addition, I've modified the entire chain of communication between the MII user-facing APIs to the inference server to better represent the inputs and outputs. This includes removing many protobuf message types that are using in MII-Legacy, but not the latest MII.

Full summary of changes:

  • Move Response, Request, etc. classes to new data_classes.py file
  • Made the output of client.generate match that of pipeline
  • Removed old GRPC/proto code that is unused
  • Changed how we represent response messages to better capture generation details

Note: This looks like a lot of change, but it's mostly moving things around the recompiling the proto stuff

TODO:

Copy link
Contributor

@tohtana tohtana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This achievement is great, thank you!
Since we don't have response.response any more in the client code, we need to make sure we update the benchmark scripts in DeepSpeedExamples after this PR is merged or the package on PyPI is updated

@mrwyattii mrwyattii marked this pull request as ready for review November 17, 2023 22:43
@mrwyattii mrwyattii merged commit f34b772 into main Nov 21, 2023
@mrwyattii mrwyattii deleted the mrwyattii/unify-input-output branch November 21, 2023 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants