diff --git a/docs/modelserving/v1beta1/serving_runtime.md b/docs/modelserving/v1beta1/serving_runtime.md index 42b707b63..4d647d972 100644 --- a/docs/modelserving/v1beta1/serving_runtime.md +++ b/docs/modelserving/v1beta1/serving_runtime.md @@ -33,20 +33,20 @@ These can also be found in the respective [runtime YAML](https://github.com/kser For model frameworks using the KServe serving runtime, the specific default version can be found in [kserve/python](https://github.com/kserve/kserve/tree/master/python). In a given serving runtime directory the pyproject.toml file contains the exact model framework version used. For example, in [kserve/python/lgbserver](https://github.com/kserve/kserve/tree/master/python/lgbserver) the [pyproject.toml](https://github.com/kserve/kserve/blob/master/python/lgbserver/pyproject.toml) file sets the model framework version to 3.3.2, `lightgbm ~= 3.3.2`. -| Model Serving Runtime | Exported model | HTTP | gRPC | Default Serving Runtime Version | Supported Framework (Major) Version(s) | Examples | -|-------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|---------------------|-------------|------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------| -| [Custom ModelServer](https://github.com/kserve/kserve/tree/master/python/kserve/kserve) | -- | v1, v2 | v2 | -- | -- | [Custom Model](custom/custom_model) | -| [LightGBM MLServer](https://mlserver.readthedocs.io/en/latest/runtimes/lightgbm.html) | [Saved LightGBM Model](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Booster.html#lightgbm.Booster.save_model) | v2 | v2 | v1.3.2 (MLServer) | 3 | [LightGBM Iris V2](./lightgbm) | -| [LightGBM ModelServer](https://github.com/kserve/kserve/tree/master/python/lgbserver) | [Saved LightGBM Model](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Booster.html#lightgbm.Booster.save_model) | v1, v2 | v2 | v0.11 (KServe) | 3 | [LightGBM Iris](./lightgbm) | -| [MLFlow ModelServer](https://docs.seldon.io/projects/seldon-core/en/latest/servers/mlflow.html) | [Saved MLFlow Model](https://www.mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.save_model) | v2 | v2 | v1.3.2 (MLServer) | 1 | [MLFLow wine-classifier](./mlflow) | -| [PMML ModelServer](https://github.com/kserve/kserve/tree/master/python/pmmlserver) | [PMML](http://dmg.org/pmml/v4-4-1/GeneralStructure.html) | v1, v2 | v2 | v0.11 (KServe) | 3, 4 ([PMML4.4.1](https://github.com/autodeployai/pypmml)) | [SKLearn PMML](./pmml) | -| [SKLearn MLServer](https://github.com/SeldonIO/MLServer) | [Pickled Model](https://scikit-learn.org/stable/modules/model_persistence.html) | v2 | v2 | v1.3.2 (MLServer) | 1 | [SKLearn Iris V2](./sklearn/v2) | -| [SKLearn ModelServer](https://github.com/kserve/kserve/tree/master/python/sklearnserver) | [Pickled Model](https://scikit-learn.org/stable/modules/model_persistence.html) | v1, v2 | v2 | v0.11 (KServe) | 1.3 | [SKLearn Iris](./sklearn/v2) | -| [TFServing](https://www.tensorflow.org/tfx/guide/serving) | [TensorFlow SavedModel](https://www.tensorflow.org/guide/saved_model) | v1 | *tensorflow | 2.6.2 ([TFServing Versions](https://github.com/tensorflow/serving/releases)) | 2 | [TensorFlow flower](./tensorflow) | -| [TorchServe](https://pytorch.org/serve/server.html) | [Eager Model/TorchScript](https://pytorch.org/docs/master/generated/torch.save.html) | v1, v2, *torchserve | *torchserve | 0.8.0 (TorchServe) | 2 | [TorchServe mnist](./torchserve) | -| [Triton Inference Server](https://github.com/triton-inference-server/server) | [TensorFlow,TorchScript,ONNX](https://github.com/triton-inference-server/server/blob/r21.09/docs/model_repository.md) | v2 | v2 | 23.05-py3 (Triton) | 8 (TensoRT), 1, 2 (TensorFlow), 2 (PyTorch), 2 (Triton) [Compatibility Matrix](https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html) | [Torchscript cifar](triton/torchscript) | -| [XGBoost MLServer](https://github.com/SeldonIO/MLServer) | [Saved Model](https://xgboost.readthedocs.io/en/latest/tutorials/saving_model.html) | v2 | v2 | v1.3.2 (MLServer) | 1 | [XGBoost Iris V2](./xgboost) | -| [XGBoost ModelServer](https://github.com/kserve/kserve/tree/master/python/xgbserver) | [Saved Model](https://xgboost.readthedocs.io/en/latest/tutorials/saving_model.html) | v1, v2 | v2 | v0.11 (KServe) | 1 | [XGBoost Iris](./xgboost) | +| Model Serving Runtime | Exported model | HTTP | gRPC | Default Serving Runtime Version | Supported Framework (Major) Version(s) | Examples | +|-------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|---------------------|-------------|------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------| +| [Custom ModelServer](https://github.com/kserve/kserve/tree/master/python/kserve/kserve) | -- | v1, v2 | v2 | -- | -- | [Custom Model](custom/custom_model/README.md) | +| [LightGBM MLServer](https://mlserver.readthedocs.io/en/latest/runtimes/lightgbm.html) | [Saved LightGBM Model](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Booster.html#lightgbm.Booster.save_model) | v2 | v2 | v1.3.2 (MLServer) | 3 | [LightGBM Iris V2](./lightgbm/README.md) | +| [LightGBM ModelServer](https://github.com/kserve/kserve/tree/master/python/lgbserver) | [Saved LightGBM Model](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Booster.html#lightgbm.Booster.save_model) | v1, v2 | v2 | v0.11 (KServe) | 3 | [LightGBM Iris](./lightgbm/README.md) | +| [MLFlow ModelServer](https://docs.seldon.io/projects/seldon-core/en/latest/servers/mlflow.html) | [Saved MLFlow Model](https://www.mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.save_model) | v2 | v2 | v1.3.2 (MLServer) | 1 | [MLFLow wine-classifier](./mlflow/v2/README.md) | +| [PMML ModelServer](https://github.com/kserve/kserve/tree/master/python/pmmlserver) | [PMML](http://dmg.org/pmml/v4-4-1/GeneralStructure.html) | v1, v2 | v2 | v0.11 (KServe) | 3, 4 ([PMML4.4.1](https://github.com/autodeployai/pypmml)) | [SKLearn PMML](./pmml/README.md) | +| [SKLearn MLServer](https://github.com/SeldonIO/MLServer) | [Pickled Model](https://scikit-learn.org/stable/modules/model_persistence.html) | v2 | v2 | v1.3.2 (MLServer) | 1 | [SKLearn Iris V2](./sklearn/v2/README.md) | +| [SKLearn ModelServer](https://github.com/kserve/kserve/tree/master/python/sklearnserver) | [Pickled Model](https://scikit-learn.org/stable/modules/model_persistence.html) | v1, v2 | v2 | v0.11 (KServe) | 1.3 | [SKLearn Iris](./sklearn/v2/README.md) | +| [TFServing](https://www.tensorflow.org/tfx/guide/serving) | [TensorFlow SavedModel](https://www.tensorflow.org/guide/saved_model) | v1 | *tensorflow | 2.6.2 ([TFServing Versions](https://github.com/tensorflow/serving/releases)) | 2 | [TensorFlow flower](./tensorflow/README.md) | +| [TorchServe](https://pytorch.org/serve/server.html) | [Eager Model/TorchScript](https://pytorch.org/docs/master/generated/torch.save.html) | v1, v2, *torchserve | *torchserve | 0.8.0 (TorchServe) | 2 | [TorchServe mnist](./torchserve/README.md) | +| [Triton Inference Server](https://github.com/triton-inference-server/server) | [TensorFlow,TorchScript,ONNX](https://github.com/triton-inference-server/server/blob/r21.09/docs/model_repository.md) | v2 | v2 | 23.05-py3 (Triton) | 8 (TensoRT), 1, 2 (TensorFlow), 2 (PyTorch), 2 (Triton) [Compatibility Matrix](https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html) | [Torchscript cifar](triton/torchscript/README.md) | +| [XGBoost MLServer](https://github.com/SeldonIO/MLServer) | [Saved Model](https://xgboost.readthedocs.io/en/latest/tutorials/saving_model.html) | v2 | v2 | v1.3.2 (MLServer) | 1 | [XGBoost Iris V2](./xgboost/README.md) | +| [XGBoost ModelServer](https://github.com/kserve/kserve/tree/master/python/xgbserver) | [Saved Model](https://xgboost.readthedocs.io/en/latest/tutorials/saving_model.html) | v1, v2 | v2 | v0.11 (KServe) | 1 | [XGBoost Iris](./xgboost/README.md) | diff --git a/docs/python_runtime_api/docs/index.md b/docs/python_runtime_api/docs/index.md new file mode 100644 index 000000000..a025a7817 --- /dev/null +++ b/docs/python_runtime_api/docs/index.md @@ -0,0 +1,58 @@ +# KServe Python Runtime API +KServe's python runtime API implements a standardized python model server API following [open inference protocol](https://github.com/kserve/open-inference-protocol). +It encapsulates data plane API definitions and storage retrieval for models. + +It provides many functionalities, including among others: + +* Implements the data plane API following open inference protocol. +* Provide extensible model server and model API. +* Allow customizing pre-processing, prediction and post-processing handlers. +* Readiness and liveness Handlers. + +## Installation + +KServe Python SDK can be installed by `pip` or `poetry`. + +### pip install + +```sh +pip install kserve +``` + +### Poetry + +Checkout KServe GitHub repository and Install via [poetry](https://python-poetry.org/). + +```sh +cd kserve/python/kserve +peotry install +``` + +## API Reference +::: kserve.model_server +::: kserve.model + +## Storage API +The storage API is used by KServe `Storage Initializer` which supports the following cloud storage providers. + +The storage package is optional and can be installed via +```sh +pip install kserve[storage] +``` + +* Google Cloud Storage with a prefix: "gs://" + * By default, it uses `GOOGLE_APPLICATION_CREDENTIALS` environment variable for user authentication. + * If `GOOGLE_APPLICATION_CREDENTIALS` is not provided, anonymous client will be used to download the artifacts. +* S3 Compatible Object Storage with a prefix "s3://" + * For static credentials it uses `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY` environment variables for authentication. +* Azure Blob Storage with the format: `https://{$STORAGE_ACCOUNT_NAME}.blob.core.windows.net/{$CONTAINER}/{$PATH}` + * By default, it uses anonymous client to download the artifacts. + * For e.g. https://kfserving.blob.core.windows.net/triton/simple_string/ +* Persistent Volume Claim (PVC) with the format `pvc://{$pvcname}/[path]`. + * The `pvcname` is the name of the PVC that contains the model. + * The `[path]` is the relative path to the model on the PVC. + * For e.g. `pvc://mypvcname/model/path/on/pvc` +* Generic URI over either `HTTP` prefixed with `http://` or `HTTPS` prefixed with `https://`. + For example: + * `https://.com/model.joblib` + * `http://.com/model.zip` \ No newline at end of file diff --git a/docs/sdk_docs/sdk_doc.md b/docs/sdk_docs/sdk_doc.md index ef43023e8..5cfec71d8 100644 --- a/docs/sdk_docs/sdk_doc.md +++ b/docs/sdk_docs/sdk_doc.md @@ -1,9 +1,9 @@ # KServe Python SDK -Python SDK for KServe Server and Client. +Python SDK for KServe controller plane client and data plane serving runtime API. ## Installation -KServe Python SDK can be installed by `pip` or `Setuptools`. +KServe Python SDK can be installed by `pip` or `poetry`. ### pip install @@ -11,69 +11,41 @@ KServe Python SDK can be installed by `pip` or `Setuptools`. pip install kserve ``` -### Setuptools +### Poetry -Install via [Setuptools](http://pypi.python.org/pypi/setuptools). +Checkout KServe GitHub repository and Install via [poetry](https://python-poetry.org/). ```sh -python setup.py install --user +cd kserve/python/kserve +peotry install ``` -(or `sudo python setup.py install` to install the package for all users) +## KServe Serving Runtime API +KServe's python serving runtime API implements the [open inference protocol](https://github.com/kserve/open-inference-protocol) +using `FastAPI`, see [Serving Runtime API docs](../python_runtime_api/docs/index.md) for more details. -## KServe Python Server -KServe's python server libraries implement a standardized library that is extended by model serving frameworks such as Scikit Learn, XGBoost and PyTorch. It encapsulates data plane API definitions and storage retrieval for models. - -It provides many functionalities, including among others: - -* Registering a model and starting the server -* Prediction Handler -* Pre/Post Processing Handler -* Liveness Handler -* Readiness Handlers - -It supports the following storage providers: - -* Google Cloud Storage with a prefix: "gs://" - * By default, it uses `GOOGLE_APPLICATION_CREDENTIALS` environment variable for user authentication. - * If `GOOGLE_APPLICATION_CREDENTIALS` is not provided, anonymous client will be used to download the artifacts. -* S3 Compatible Object Storage with a prefix "s3://" - * By default, it uses `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY` environment variables for user authentication. -* Azure Blob Storage with the format: "https://{$STORAGE_ACCOUNT_NAME}.blob.core.windows.net/{$CONTAINER}/{$PATH}" - * By default, it uses anonymous client to download the artifacts. - * For e.g. https://kfserving.blob.core.windows.net/triton/simple_string/ -* Local filesystem either without any prefix or with a prefix "file://". For example: - * Absolute path: `/absolute/path` or `file:///absolute/path` - * Relative path: `relative/path` or `file://relative/path` - * For local filesystem, we recommended to use relative path without any prefix. -* Persistent Volume Claim (PVC) with the format "pvc://{$pvcname}/[path]". - * The `pvcname` is the name of the PVC that contains the model. - * The `[path]` is the relative path to the model on the PVC. - * For e.g. `pvc://mypvcname/model/path/on/pvc` -* Generic URI, over either `HTTP`, prefixed with `http://` or `HTTPS`, prefixed with `https://`. For example: - * `https://.com/model.joblib` - * `http://.com/model.joblib` - -## KServe Client +## KServe Client API +KServe's python client interacts with KServe control plane APIs for executing operations on a remote KServe cluster, +such as creating, patching and deleting of a InferenceService instance. ### Getting Started -KServe's python client interacts with KServe control plane APIs for executing operations on a remote KServe cluster, such as creating, patching and deleting of a InferenceService instance. See the [Sample for Python SDK Client](./samples/kserve_sdk_v1beta1_sample.ipynb) to get started. +Please see the [Sample for Python SDK Client](./samples/kserve_sdk_v1beta1_sample.ipynb) to get started. -### Documentation for Client API +### KServe Client API Reference -Class | Method | Description ------------- | ------------- | ------------- -[KServeClient](docs/KServeClient.md) | [set_credentials](docs/KServeClient.md#set_credentials) | Set Credentials| -[KServeClient](docs/KServeClient.md) | [create](docs/KServeClient.md#create) | Create InferenceService| -[KServeClient](docs/KServeClient.md) | [get](docs/KServeClient.md#get) | Get or watch the specified InferenceService or all InferenceServices in the namespace | -[KServeClient](docs/KServeClient.md) | [patch](docs/KServeClient.md#patch) | Patch the specified InferenceService| -[KServeClient](docs/KServeClient.md) | [replace](docs/KServeClient.md#replace) | Replace the specified InferenceService| -[KServeClient](docs/KServeClient.md) | [delete](docs/KServeClient.md#delete) | Delete the specified InferenceService | -[KServeClient](docs/KServeClient.md) | [wait_isvc_ready](docs/KServeClient.md#wait_isvc_ready) | Wait for the InferenceService to be ready | -[KServeClient](docs/KServeClient.md) | [is_isvc_ready](docs/KServeClient.md#is_isvc_ready) | Check if the InferenceService is ready | +| Class | Method | Description | +|--------------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------------| +| [KServeClient](docs/KServeClient.md) | [set_credentials](docs/KServeClient.md#set_credentials) | Set Credentials | +| [KServeClient](docs/KServeClient.md) | [create](docs/KServeClient.md#create) | Create InferenceService | +| [KServeClient](docs/KServeClient.md) | [get](docs/KServeClient.md#get) | Get or watch the specified InferenceService or all InferenceServices in the namespace | +| [KServeClient](docs/KServeClient.md) | [patch](docs/KServeClient.md#patch) | Patch the specified InferenceService | +| [KServeClient](docs/KServeClient.md) | [replace](docs/KServeClient.md#replace) | Replace the specified InferenceService | +| [KServeClient](docs/KServeClient.md) | [delete](docs/KServeClient.md#delete) | Delete the specified InferenceService | +| [KServeClient](docs/KServeClient.md) | [wait_isvc_ready](docs/KServeClient.md#wait_isvc_ready) | Wait for the InferenceService to be ready | +| [KServeClient](docs/KServeClient.md) | [is_isvc_ready](docs/KServeClient.md#is_isvc_ready) | Check if the InferenceService is ready | -## Documentation For Models +### Reference for Generated Data Models - [KnativeAddressable](docs/KnativeAddressable.md) - [KnativeCondition](docs/KnativeCondition.md) diff --git a/mkdocs.yml b/mkdocs.yml index fbef25170..fd5d9d982 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -92,6 +92,7 @@ nav: - Control Plane API: reference/api.md - Open Inference Protocol API Spec: reference/swagger-ui.md - Python Client SDK: sdk_docs/sdk_doc.md + - Python Runtime Server SDK: python_runtime_api/docs/index.md - Developer Guide: - How to contribute: developer/developer.md - Debugging guide: developer/debug.md @@ -160,6 +161,11 @@ copyright: "Copyright © 2021 The KServe Authors" plugins: - search + - mkdocstrings: + default_handler: python + handlers: + python: + load_external_modules: true - mike: version_selector: true css_dir: css diff --git a/requirements.txt b/requirements.txt index 45d715ab7..7c9e2f1d3 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,3 +6,4 @@ mkdocs-redirects>=1.0.3 mike>=1.1.2 mkdocs-swagger-ui-tag markdown-include +mkdocstrings-python