feat: executor to sagemaker custom container

[deepankarm]

Goals

The PR enables using an Executor as a sagemaker custom container for inference by implementing 2 endpoints- POST /invocations & GET /ping and accepting a serve command on port 8080.

To separate sagemaker inference, I've added a --provider [NONE|SAGEMAKER] argument to the CLI which can be expanded later to azureml, vertexai etc. When provider is sagemaker, we start a custom fastapi app that implements the GET /ping endpoint.

• If /invocations route is already added to the Executor, we add it to the fastapi app.
• If /invocations is not added and only one route is added to the Executor, we add an /invocations route that points to the route the Executor implements.
• Else raise an error.

Sagemaker needs the image to be pushed to ECR. To use an already pushed Executor with sagemaker, one can do the following and push the image.

# Dockerfile
FROM registry.hubble.jina.ai/myimage

ENTRYPOINT [ "jina", "executor", "--uses", "config.yml", "--provider", "sagemaker" ]

During run, sagemaker adds serve as a CMD to the container entrypoint. I haven't added any arguments to the jina cli for this as serve is ignored (if we follow the above syntax in the entrypoint). While implementing other providers, I'll evaluate the need of the serve command.

End-to-end test

I've manually tested the Executor after pushing a model to S3, the Executor image to ECR & running real-time & serverless inference via sagemaker endpoints.

Pending

Support for batch-transform jobs is not added in this PR, will be followed in another PR.

---

• check and update documentation. See guide and ask the team.

[deepankarm]

Without this update, I keep getting an error during commit. SO link

RuntimeError: The Poetry configuration is invalid:
            - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

[codecov]

Codecov Report

Patch coverage: 29.69% and project coverage change: -3.44% ⚠️

Comparison is base (d4343d0) 77.14% compared to head (812667b) 73.70%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6046      +/-   ##
==========================================
- Coverage   77.14%   73.70%   -3.44%     
==========================================
  Files         144      143       -1     
  Lines       13777    13855      +78     
==========================================
- Hits        10628    10212     -416     
- Misses       3149     3643     +494     

Flag	Coverage Δ
jina	73.70% <29.69%> (-3.44%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
jina/orchestrate/flow/base.py	88.16% <ø> (+1.25%)	⬆️
jina/serve/runtimes/worker/http_sagemaker_app.py	0.00% <0.00%> (ø)
jina/serve/runtimes/worker/request_handling.py	71.50% <11.76%> (-10.44%)	⬇️
jina/serve/executors/__init__.py	62.26% <35.29%> (-5.58%)	⬇️
jina/orchestrate/deployments/__init__.py	81.27% <42.85%> (+0.43%)	⬆️
jina/serve/runtimes/asyncio.py	86.95% <71.42%> (-1.66%)	⬇️
jina/serve/runtimes/servers/http.py	90.43% <71.42%> (-1.95%)	⬇️
jina/serve/runtimes/servers/__init__.py	93.04% <83.33%> (-1.65%)	⬇️
jina/enums.py	88.39% <100.00%> (+0.31%)	⬆️
jina/parsers/orchestrate/pod.py	100.00% <100.00%> (ø)

... and 38 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[JoanFM]

Is port 8080 the only way this can happen?