Update TFS EI example. #660
Conversation
There was a problem hiding this comment.
There are references to the sagemaker-tensorflow-container in this notebook, although the corresponding repo should be sagemaker-tensorflow-serving-container.
There is a line in regards to export/Servo, that might not be true anymore. It looks like the name it is expecting is based on the model object now: https://github.com/aws/sagemaker-tensorflow-serving-container/blob/master/container/sagemaker/serve.py#L55
Can we reference the PR in Python SDK and the TFS container in this PR? We should mention that this notebook is pending release on the Python SDK PR.
|
I removed the sentence since it was originally referencing TF container, and now we are using TFS container which doesnt expect Export/Servo as model anymore. Although the predefined TFS container starts with a default model, but it doesnt suit this example's case, so I'm not mentioning it here. |
There was a problem hiding this comment.
I think it might be worth introducing that this container's is doing inferences with TensorFlow serving using the rest api.
Lets add reference to the tensorflow-serving-container in here, that way the user knows that a predefined SageMaker TensorFlow serving container is being used here.
| @@ -167,7 +163,7 @@ | |||
| "import numpy as np\n", | |||
| "random_input = np.random.rand(1, 1, 3, 3)\n", | |||
| "\n", | |||
| "prediction = predictor.predict({'input': random_input.tolist()})\n", | |||
| "prediction = predictor.predict({'inputs': random_input.tolist()})\n", | |||
There was a problem hiding this comment.
Can you explain why the 'inputs' keyword is provided here?
There was a problem hiding this comment.
TFS container checks if input data is labeled as 'instances/inputs/examples' (https://github.com/aws/sagemaker-tensorflow-serving-container/blob/master/container/sagemaker/tensorflow-serving.js#L131). If yes, it will pass the data as it is. Otherwise, for example, if the label is 'input', it prepends an 'instances' to the data (https://github.com/aws/sagemaker-tensorflow-serving-container/blob/master/container/sagemaker/tensorflow-serving.js#L139).
There was a problem hiding this comment.
We should mention that in the notebook, as it seems this functionality is TFS container specific.
There was a problem hiding this comment.
Actually it is a functionality of TFS: https://www.tensorflow.org/tfx/serving/api_rest#specifying_input_tensors_in_row_format
| @@ -38,7 +38,7 @@ | |||
| "\n", | |||
| "The pre-trained model we will be using for this example is a NCHW ResNet-50 model from the [official Tensorflow model Github repository](https://github.com/tensorflow/models/tree/master/official/resnet#pre-trained-model). For more information in regards to deep residual networks, please check [here](https://github.com/tensorflow/models/tree/master/official/resnet). It isn't a requirement to train our model on SageMaker to use SageMaker for serving our model.\n", | |||
| "\n", | |||
| "SageMaker expects our models to be compressed in a tar.gz format in S3. Thankfully, our model already comes in that format.\n", | |||
| "SageMaker expects our models to be compressed in a tar.gz format in S3. Thankfully, our model already comes in that format. The predefined TensorFlow Serving containers use REST API for handling inferences, for more informationm, please see [predefined SageMaker TensorFlow Serving](https://github.com/aws/sagemaker-tensorflow-serving-container/blob/master/container/sagemaker/serve.py).\n", | |||
There was a problem hiding this comment.
Issue #, if available:
Description of changes:
Update the tensorflow_using_elastic_inference_with_your_own_model.ipynb
This example ran successfully with the hardcoded image uri: '562200002688.dkr.ecr.us-west-2.amazonaws.com/sagemaker-tensorflow-serving-eia:{}-cpu''. But it won't work until the TFS EIA images are distributed and the Python SDK change merged (listed below) .
TFS Container:
aws/sagemaker-tensorflow-serving-container#10
aws/sagemaker-tensorflow-serving-container#12
Python SDK:
aws/sagemaker-python-sdk#682
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.