From dc783947c2206bb550f80edb2c2ff9c7a815bd6f Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 30 Jul 2019 10:45:15 -0700 Subject: [PATCH 01/23] Updates --- .../TensorRT-ExecutionProvider.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/execution_providers/TensorRT-ExecutionProvider.md b/docs/execution_providers/TensorRT-ExecutionProvider.md index 37c4c75ff58fa..68ff6946419f7 100644 --- a/docs/execution_providers/TensorRT-ExecutionProvider.md +++ b/docs/execution_providers/TensorRT-ExecutionProvider.md @@ -1,11 +1,11 @@ -## TensortRT Execution Provider (preview) +## TensortRT Execution Provider -The TensorRT execution provider in the ONNX Runtime will make use of NVIDIA's [TensortRT](https://developer.nvidia.com/tensorrt) Deep Learning inferencing engine to accelerate ONNX model in their family of GPUs. Microsoft and NVIDIA worked closely to integrate the TensorRT execution provider with ONNX Runtime. +The TensorRT execution provider in the ONNX Runtime makes use of NVIDIA's [TensortRT](https://developer.nvidia.com/tensorrt) Deep Learning inferencing engine to accelerate ONNX model in their family of GPUs. Microsoft and NVIDIA worked closely to integrate the TensorRT execution provider with ONNX Runtime. -This execution provider release is currently in preview but, we have validated support for all the ONNX Models in the model zoo. With the TensorRT execution provider, the ONNX Runtime delivers better inferencing performance on the same hardware compared to generic GPU acceleration. +With the TensorRT execution provider, the ONNX Runtime delivers better inferencing performance on the same hardware compared to generic GPU acceleration. ### Build TensorRT execution provider -Developers can now tap into the power of TensorRT through ONNX Runtime to accelerate inferencing of ONNX models. Instructions to build the TensorRT execution provider from source is available [here](https://github.com/Microsoft/onnxruntime/blob/master/BUILD.md#build). +Developers can now tap into the power of TensorRT through ONNX Runtime to accelerate inferencing of ONNX models. Instructions to build the TensorRT execution provider from source are available [here](https://github.com/Microsoft/onnxruntime/blob/master/BUILD.md#build). [Dockerfiles](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles#tensorrt-version-preview) are available for convenience. ### Using the TensorRT execution provider #### C/C++ @@ -18,7 +18,10 @@ status = session_object.Load(model_file_name); The C API details are [here](https://github.com/Microsoft/onnxruntime/blob/master/docs/C_API.md#c-api). ### Python -When using the python wheel from the ONNX Runtime build with TensorRT execution provider, it will be automatically prioritized over the default GPU or CPU execution providers. There is no need to separately register the execution provider. Python APIs details are [here](https://github.com/Microsoft/onnxruntime/blob/master/docs/python/api_summary.rst#api-summary). +When using the Python wheel from the ONNX Runtime build with TensorRT execution provider, it will be automatically prioritized over the default GPU or CPU execution providers. There is no need to separately register the execution provider. Python APIs details are [here](https://microsoft.github.io/onnxruntime/api_summary.html). -### Using onnxruntime_perf_test -You can test the performance for your ONNX Model with the TensorRT execution provider. Use the flag `-e tensorrt` in [onnxruntime_perf_test](https://github.com/Microsoft/onnxruntime/tree/master/onnxruntime/test/perftest#onnxruntime-performance-test). +### Performance Tuning +To test the performance of your ONNX Model with the TensorRT execution provider, use the flag `-e tensorrt` in [onnxruntime_perf_test](https://github.com/Microsoft/onnxruntime/tree/master/onnxruntime/test/perftest#onnxruntime-performance-test). + +### Sample +Please see [this Notebook](https://github.com/microsoft/onnxruntime/blob/master/docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb) for an example of running a model on GPU using ONNX Runtime through Azure Machine Learning Services. From 15e0fe9c471209999185d015d23f4a4c06399d21 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 30 Jul 2019 10:49:38 -0700 Subject: [PATCH 02/23] Remove preview texts --- dockerfiles/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/dockerfiles/README.md b/dockerfiles/README.md index 7a7a2642b6fde..e3c4cbcbe68fc 100644 --- a/dockerfiles/README.md +++ b/dockerfiles/README.md @@ -8,7 +8,7 @@ - [OpenVINO](Dockerfile.openvino) - [ONNX Runtime Server](Dockerfile.server) -## Build from Source Version (Preview) +## Build from Source #### Linux 16.04, CPU, Python Bindings 1. Build the docker image from the Dockerfile in this repository. @@ -26,7 +26,7 @@ docker run -it onnxruntime-source ``` -## CUDA Version (Preview) +## CUDA #### Linux 16.04, CUDA 10.0, CuDNN 7 1. Build the docker image from the Dockerfile in this repository. @@ -44,7 +44,7 @@ docker run -it onnxruntime-cuda ``` -## nGraph Version (Preview) +## nGraph (Preview) #### Linux 16.04, Python Bindings 1. Build the docker image from the Dockerfile in this repository. @@ -62,7 +62,7 @@ docker run -it onnxruntime-ngraph ``` -## TensorRT Version (Preview) +## TensorRT #### Linux 16.04, TensorRT 5.0.2 1. Build the docker image from the Dockerfile in this repository. @@ -80,7 +80,7 @@ docker run -it onnxruntime-trt ``` -## OpenVINO Version (Preview) +## OpenVINO (Preview) #### Linux 16.04, Python Bindings 1. Build the onnxruntime image for all the accelerators supported as below @@ -104,7 +104,7 @@ | MYRIAD_FP16 | Intel MovidiusTM USB sticks | | VAD-R_FP16 | Intel Vision Accelerator Design based on MovidiusTM MyriadX VPUs | -## CPU Version +## CPU 1. Retrieve your docker image in one of the following ways. @@ -122,7 +122,7 @@ docker run -it onnxruntime-cpu ``` -## GPU Version +## GPU 1. Retrieve your docker image in one of the following ways. - Build the docker image from the DockerFile in this repository. @@ -138,7 +138,7 @@ ``` docker run -it --device /dev/dri:/dev/dri onnxruntime-gpu:latest ``` -## Myriad VPU Accelerator Version +## Myriad VPU Accelerator 1. Retrieve your docker image in one of the following ways. - Build the docker image from the DockerFile in this repository. @@ -155,7 +155,7 @@ docker run -it --network host --privileged -v /dev:/dev onnxruntime-myriad:latest ``` -## VAD-R Accelerator Version +## VAD-R Accelerator 1. Retrieve your docker image in one of the following ways. - Build the docker image from the DockerFile in this repository. From d51d967b480ee2d0f279aa3b92e0ddb80b1ab98c Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 30 Jul 2019 11:05:58 -0700 Subject: [PATCH 03/23] Update README.md --- .../c_cxx/fns_candy_style_transfer/README.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/samples/c_cxx/fns_candy_style_transfer/README.md b/samples/c_cxx/fns_candy_style_transfer/README.md index 4211aa8d1ec59..3e8529e7659d7 100644 --- a/samples/c_cxx/fns_candy_style_transfer/README.md +++ b/samples/c_cxx/fns_candy_style_transfer/README.md @@ -1,19 +1,18 @@ -# Build +# FNS Candy +FNS Candy is a style transfer model. In this sample application, we use the ONNX Runtime C API to process an image using the FNS Candy model in ONNX format. + +# Build Instructions See [../README.md](../README.md) # Prepare data -Please download the model from (candy.onnx)[https://raw.githubusercontent.com/microsoft/Windows-Machine-Learning/master/Samples/FNSCandyStyleTransfer/UWP/cs/Assets/candy.onnx] +First, download the FNS Candy ONNX model from [here](https://raw.githubusercontent.com/microsoft/Windows-Machine-Learning/master/Samples/FNSCandyStyleTransfer/UWP/cs/Assets/candy.onnx). -Then prepare an image: -1. In png format -2. With dimension of 720x720 +Then, prepare an image: +1. PNG format +2. Dimension of 720x720 # Run +Command to run the application: ``` fns_candy_style_transfer.exe ``` - - - - - From 5f4ff2acc5508f33598c1951f029d8aeda242fb0 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 30 Jul 2019 11:20:07 -0700 Subject: [PATCH 04/23] Updates --- README.md | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index b1a715dbfe95d..cacd0056ea268 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,9 @@ **Getting Started** * [Getting ONNX Models](#getting-onnx-models) * [Deploying ONNX Runtime](#deploying-onnx-runtime) -* [Examples and Tutorials](#examples-and-tutorials) +* [Performance Tuning](#performance-tuning) + +**[Examples and Tutorials](#examples-and-tutorials)** **More Info** * [Technical Design Details](#technical-design-details) @@ -43,19 +45,19 @@ In addition to DNN models, ONNX Runtime fully supports the [ONNX-ML profile](htt ONNX Runtime supports both CPU and GPU. Using various graph optimizations and accelerators, ONNX Runtime can provide lower latency compared to other runtimes for faster end-to-end customer experiences and minimized machine utilization costs. Currently ONNX Runtime supports the following accelerators: -* CPU - * MLAS (Microsoft Linear Algebra Subprograms) - * MKL-DNN - * MKL-ML - * [Intel nGraph](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/nGraph-ExecutionProvider.md) -* GPU - * CUDA - * [TensorRT](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/TensorRT-ExecutionProvider.md) +* MLAS (Microsoft Linear Algebra Subprograms) +* [MKL-DNN](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/MKL-DNN-ExecutionProvider.md) - [subgraph optimization](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/MKL-DNN-Subgraphs.md) +* MKL-ML +* [Intel nGraph](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/nGraph-ExecutionProvider.md) +* CUDA +* [TensorRT](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/TensorRT-ExecutionProvider.md) +* [OpenVINO](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/OpenVINO-ExecutionProvider.md) -Not all variations are supported in the [official release builds](#apis-and-official-builds), but can be built from source following [these instructions](https://github.com/Microsoft/onnxruntime/blob/master/BUILD.md). +Not all variations are supported in the [official release builds](#apis-and-official-builds), but can be built from source following [these instructions](https://github.com/Microsoft/onnxruntime/blob/master/BUILD.md). Find Dockerfiles [here](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles). We are continuously working to integrate new execution providers for further improvements in latency and efficiency. If you are interested in contributing a new execution provider, please see [this page](docs/AddingExecutionProvider.md). + ### Cross Platform [API documentation and package installation](https://github.com/microsoft/onnxruntime#installation) @@ -115,8 +117,12 @@ ONNX Runtime can be deployed to the cloud for model inferencing using [Azure Mac **ONNX Runtime Server (beta)** is a hosted application for serving ONNX models using ONNX Runtime, providing a REST API for prediction. Usage details can be found [here](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md), and image installation instructions are [here](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles#onnx-runtime-server-preview). -## Examples and Tutorials -### Python +## Performance Tuning +ONNX Runtime is open and extensible, supporting a broad set of configurations and execution providers for model acceleration. For performance tuning guidance, please see [this page](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Perf_Tuning.md). + + +# Examples and Tutorials +## Python * [Basic Inferencing Sample](https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/simple_onnxruntime_inference.ipynb) * [Inferencing (Resnet50)](https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/resnet50_modelzoo_onnxruntime_inference.ipynb) * [Inferencing samples](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem/inference_demos) using [ONNX-Ecosystem Docker image](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem) @@ -132,11 +138,11 @@ ONNX Runtime can be deployed to the cloud for model inferencing using [Azure Mac * [FER+ on Azure Kubernetes Service with TensorRT](https://github.com/microsoft/onnxruntime/blob/master/docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb) -### C# +## C# * [Inferencing Tutorial](https://github.com/microsoft/onnxruntime/blob/master/docs/CSharp_API.md#getting-started) -### C/C++ +## C/C++ * [Basic Inferencing (SqueezeNet) - C](https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp) * [Basic Inferencing (SqueezeNet) - C++](https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/CXX_Api_Sample.cpp) * [Inferencing (MNIST) - C++](https://github.com/microsoft/onnxruntime/tree/master/samples/c_cxx/MNIST) @@ -153,6 +159,7 @@ ONNX Runtime can be deployed to the cloud for model inferencing using [Azure Mac transform](include/onnxruntime/core/optimizer/graph_transformer.h) * [Add a new rewrite rule](include/onnxruntime/core/optimizer/rewrite_rule.h) + # Contribute We welcome contributions! Please see the [contribution guidelines](CONTRIBUTING.md). From 6bc64bc1b713fc0b8d7943d6d6cfefa3cc33b2ba Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 30 Jul 2019 11:28:29 -0700 Subject: [PATCH 05/23] Update README.md --- README.md | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index cacd0056ea268..693e99ac7bf65 100644 --- a/README.md +++ b/README.md @@ -11,12 +11,14 @@ [ONNX](https://onnx.ai) is an interoperable format for machine learning models supported by various ML and DNN frameworks and tools. The universal format makes it easier to interoperate between frameworks and maximize the reach of hardware optimization investments. *** +**[Key Features](#key-features)** + **Setup** * [Installation](#installation) * [APIs and Official Binaries](#apis-and-official-builds) * [Building from Source](#building-from-source) -**Getting Started** +**Usage** * [Getting ONNX Models](#getting-onnx-models) * [Deploying ONNX Runtime](#deploying-onnx-runtime) * [Performance Tuning](#performance-tuning) @@ -31,8 +33,8 @@ **[License](#license)** *** -## Key Features -### Run any ONNX model +# Key Features +## Run any ONNX model ONNX Runtime provides comprehensive support of the ONNX spec and can be used to run all models based on ONNX v1.2.1 and higher. See version compatibility details [here](https://github.com/microsoft/onnxruntime/blob/master/docs/Versioning.md). *Note: Some operators not supported in the current ONNX version may be available as a [Contrib Operator](https://github.com/microsoft/onnxruntime/blob/master/docs/ContribOperators.md)* @@ -41,7 +43,7 @@ ONNX Runtime provides comprehensive support of the ONNX spec and can be used to In addition to DNN models, ONNX Runtime fully supports the [ONNX-ML profile](https://github.com/onnx/onnx/blob/master/docs/Operators-ml.md) of the ONNX spec for traditional ML scenarios. -### High Performance +## High Performance ONNX Runtime supports both CPU and GPU. Using various graph optimizations and accelerators, ONNX Runtime can provide lower latency compared to other runtimes for faster end-to-end customer experiences and minimized machine utilization costs. Currently ONNX Runtime supports the following accelerators: @@ -58,12 +60,12 @@ Not all variations are supported in the [official release builds](#apis-and-offi We are continuously working to integrate new execution providers for further improvements in latency and efficiency. If you are interested in contributing a new execution provider, please see [this page](docs/AddingExecutionProvider.md). -### Cross Platform +## Cross Platform [API documentation and package installation](https://github.com/microsoft/onnxruntime#installation) ONNX Runtime is available for Linux, Windows, Mac with Python, C#, and C APIs, with more to come! If you have specific scenarios that are not currently supported, please share your suggestions and scenario details via [Github Issues](https://github.com/microsoft/onnxruntime/issues). - +*** # Installation **Quick Start:** The [ONNX-Ecosystem Docker container image](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem) is available on Dockerhub and includes ONNX Runtime (CPU, Python), dependencies, tools to convert from various frameworks, and Jupyter notebooks to help get started. @@ -98,12 +100,15 @@ system. `locale-gen en_US.UTF-8` `update-locale LANG=en_US.UTF-8` * Follow similar procedure to configure other locales on other platforms. - + ## Building from Source If additional build flavors are needed, please find instructions on building from source at [Build ONNX Runtime](BUILD.md). For production scenarios, it's strongly recommended to build from an [official release branch](https://github.com/microsoft/onnxruntime/releases). Dockerfiles are available [here](https://github.com/microsoft/onnxruntime/tree/faxu-doc-updates/tools/ci_build/github/linux/docker) to help you get started. +*** +# Usage + ## Getting ONNX Models * The [ONNX Model Zoo](https://github.com/onnx/models) has popular ready-to-use pre-trained models. * To export or convert a trained ONNX model trained from various frameworks, see [ONNX Tutorials](https://github.com/onnx/tutorials). Versioning comptability information can be found under [Versioning](docs/Versioning.md#tool-compatibility) @@ -120,7 +125,7 @@ ONNX Runtime can be deployed to the cloud for model inferencing using [Azure Mac ## Performance Tuning ONNX Runtime is open and extensible, supporting a broad set of configurations and execution providers for model acceleration. For performance tuning guidance, please see [this page](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Perf_Tuning.md). - +*** # Examples and Tutorials ## Python * [Basic Inferencing Sample](https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/simple_onnxruntime_inference.ipynb) @@ -147,7 +152,7 @@ ONNX Runtime is open and extensible, supporting a broad set of configurations an * [Basic Inferencing (SqueezeNet) - C++](https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/CXX_Api_Sample.cpp) * [Inferencing (MNIST) - C++](https://github.com/microsoft/onnxruntime/tree/master/samples/c_cxx/MNIST) - +*** # Technical Design Details * [High level architectural design](docs/HighLevelDesign.md) * [Versioning](docs/Versioning.md) @@ -159,7 +164,7 @@ ONNX Runtime is open and extensible, supporting a broad set of configurations an transform](include/onnxruntime/core/optimizer/graph_transformer.h) * [Add a new rewrite rule](include/onnxruntime/core/optimizer/rewrite_rule.h) - +*** # Contribute We welcome contributions! Please see the [contribution guidelines](CONTRIBUTING.md). @@ -170,6 +175,6 @@ For any feedback or to report a bug, please file a [GitHub Issue](https://github This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. - +*** # License [MIT License](LICENSE) From 4f3c5eeab1c114189a458b1402b6df0b39b5f38b Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 30 Jul 2019 11:33:11 -0700 Subject: [PATCH 06/23] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 693e99ac7bf65..26489ade9c6de 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ Additional dockerfiles for some features can be found [here](https://github.com/ |---|:---|:---|:---| | **Python** | **[pypi: onnxruntime](https://pypi.org/project/onnxruntime)**

Windows (x64)
Linux (x64)
Mac OS X (x64) | -- | **[pypi: onnxruntime-gpu](https://pypi.org/project/onnxruntime-gpu)**

Windows (x64)
Linux (x64) | | **C#** | **[Nuget: Microsoft.ML.OnnxRuntime](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime/)**

Windows (x64, x86)
Linux (x64, x86)
Mac OS X (x64) | **[Nuget: Microsoft.ML.OnnxRuntime.MKLML](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.MKLML/)**

Windows (x64)
Linux (x64)
Mac OS X (x64) | **[Nuget: Microsoft.ML.OnnxRuntime.Gpu](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.Gpu/)**

Windows (x64)
Linux (x64) | -| **C** | **[Nuget: Microsoft.ML.OnnxRuntime](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime)**

**[.zip, .tgz](https://aka.ms/onnxruntime-release)**

Windows (x64, x86)
Linux (x64, x86)
Mac OS X (x64 | **[Nuget: Microsoft.ML.OnnxRuntime.MKLML](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.MKLML/)**

Windows (x64)
Linux (x64)
Mac OS X (x64) | **[Nuget: Microsoft.ML.OnnxRuntime.Gpu](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.Gpu/)**

**[.zip, .tgz](https://aka.ms/onnxruntime-release)**

Windows (x64)
Linux (x64) | +| **C/C++ wrapper** | **[Nuget: Microsoft.ML.OnnxRuntime](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime)**

**[.zip, .tgz](https://aka.ms/onnxruntime-release)**

Windows (x64, x86)
Linux (x64, x86)
Mac OS X (x64) | **[Nuget: Microsoft.ML.OnnxRuntime.MKLML](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.MKLML/)**

Windows (x64)
Linux (x64)
Mac OS X (x64) | **[Nuget: Microsoft.ML.OnnxRuntime.Gpu](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.Gpu/)**

**[.zip, .tgz](https://aka.ms/onnxruntime-release)**

Windows (x64)
Linux (x64) | #### System Requirements (pre-requisite dependencies) * ONNX Runtime binaries in the CPU packages use OpenMP and depend on the library being available at runtime in the From ab44efa935565d0dfec98e9644cd9f336c01dc89 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Wed, 31 Jul 2019 21:08:24 -0700 Subject: [PATCH 07/23] Minor wording update --- dockerfiles/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dockerfiles/README.md b/dockerfiles/README.md index e3c4cbcbe68fc..8fdc9b68e9fad 100644 --- a/dockerfiles/README.md +++ b/dockerfiles/README.md @@ -44,7 +44,7 @@ docker run -it onnxruntime-cuda ``` -## nGraph (Preview) +## nGraph (Public Preview) #### Linux 16.04, Python Bindings 1. Build the docker image from the Dockerfile in this repository. @@ -80,7 +80,7 @@ docker run -it onnxruntime-trt ``` -## OpenVINO (Preview) +## OpenVINO (PUblic Preview) #### Linux 16.04, Python Bindings 1. Build the onnxruntime image for all the accelerators supported as below @@ -172,7 +172,7 @@ docker run -it --device --mount type=bind,source=/var/tmp,destination=/var/tmp --device /dev/ion:/dev/ion onnxruntime-hddl:latest ``` -## ONNX Runtime Server (Preview) +## ONNX Runtime Server (Public Preview) #### Linux 16.04 1. Build the docker image from the Dockerfile in this repository From 014d8dd72275e4c712d3f5059253334171e43890 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Wed, 31 Jul 2019 21:08:48 -0700 Subject: [PATCH 08/23] Update README.md --- dockerfiles/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dockerfiles/README.md b/dockerfiles/README.md index 8fdc9b68e9fad..afcf16c029ae4 100644 --- a/dockerfiles/README.md +++ b/dockerfiles/README.md @@ -80,7 +80,7 @@ docker run -it onnxruntime-trt ``` -## OpenVINO (PUblic Preview) +## OpenVINO (Public Preview) #### Linux 16.04, Python Bindings 1. Build the onnxruntime image for all the accelerators supported as below From 31def18b359da082b948aa2bb698e4d35032954b Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Mon, 5 Aug 2019 18:07:32 -0700 Subject: [PATCH 09/23] Update doc on CUDA version --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 26489ade9c6de..afb5bfec9450a 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,10 @@ system. * For Windows, **OpenMP** support comes as part of VC runtime. It is also available as redist packages: [vc_redist.x64.exe](https://aka.ms/vs/15/release/vc_redist.x64.exe) and [vc_redist.x86.exe](https://aka.ms/vs/15/release/vc_redist.x86.exe) * For Linux, the system must have **libgomp.so.1** which can be installed using `apt-get install libgomp1`. -* GPU builds require the **CUDA 10.0 and cuDNN 7.3** runtime libraries being installed on the system. Older releases used 9.1/7.1 - please refer to [release notes](https://github.com/microsoft/onnxruntime/releases) for more details. +* GPU builds require CUDA runtime libraries being installed on the system: + * Version: **CUDA 10.0** and **cuDNN 7.3** + * Note: Linux Python packages require **CUDA 10.1** and **cuDNN 7.3** +Older ONNX Runtime releases: used **CUDA 9.1** and **cuDNN 7.1** - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details. * Python binaries are compatible with **Python 3.5-3.7**. See [Python Dev Notes](https://github.com/microsoft/onnxruntime/blob/master/docs/Python_Dev_Notes.md) * Certain operators makes use of system locales. Installation of the **English language package** and configuring `en_US.UTF-8 locale` is required. * For Ubuntu install [language-pack-en package](https://packages.ubuntu.com/search?keywords=language-pack-en) From f4d7d7b2933de3021d31416f3bbefe285d897f30 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Mon, 5 Aug 2019 18:22:29 -0700 Subject: [PATCH 10/23] revert update --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index afb5bfec9450a..c214724d306f1 100644 --- a/README.md +++ b/README.md @@ -92,9 +92,7 @@ system. * For Windows, **OpenMP** support comes as part of VC runtime. It is also available as redist packages: [vc_redist.x64.exe](https://aka.ms/vs/15/release/vc_redist.x64.exe) and [vc_redist.x86.exe](https://aka.ms/vs/15/release/vc_redist.x86.exe) * For Linux, the system must have **libgomp.so.1** which can be installed using `apt-get install libgomp1`. -* GPU builds require CUDA runtime libraries being installed on the system: - * Version: **CUDA 10.0** and **cuDNN 7.3** - * Note: Linux Python packages require **CUDA 10.1** and **cuDNN 7.3** +* GPU builds require **CUDA 10.0 and cuDNN 7.3** runtime libraries being installed on the system. Older releases used 9.1/7.1 - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details. Older ONNX Runtime releases: used **CUDA 9.1** and **cuDNN 7.1** - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details. * Python binaries are compatible with **Python 3.5-3.7**. See [Python Dev Notes](https://github.com/microsoft/onnxruntime/blob/master/docs/Python_Dev_Notes.md) * Certain operators makes use of system locales. Installation of the **English language package** and configuring `en_US.UTF-8 locale` is required. From 68432c5400dc8e63e6300ad27fc0bd24708707a7 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 6 Aug 2019 18:06:14 -0700 Subject: [PATCH 11/23] Update readme for issue #1558 --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c214724d306f1..f2ee924e823e5 100644 --- a/README.md +++ b/README.md @@ -92,8 +92,10 @@ system. * For Windows, **OpenMP** support comes as part of VC runtime. It is also available as redist packages: [vc_redist.x64.exe](https://aka.ms/vs/15/release/vc_redist.x64.exe) and [vc_redist.x86.exe](https://aka.ms/vs/15/release/vc_redist.x86.exe) * For Linux, the system must have **libgomp.so.1** which can be installed using `apt-get install libgomp1`. -* GPU builds require **CUDA 10.0 and cuDNN 7.3** runtime libraries being installed on the system. Older releases used 9.1/7.1 - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details. -Older ONNX Runtime releases: used **CUDA 9.1** and **cuDNN 7.1** - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details. +* GPU builds require CUDA runtime libraries being installed on the system: + * Version: **CUDA 10.0** and **cuDNN 7.3** + * Linux Python packages require **CUDA 10.1** and **cuDNN 7.3** + * Older ONNX Runtime releases: used **CUDA 9.1** and **cuDNN 7.1** - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details. * Python binaries are compatible with **Python 3.5-3.7**. See [Python Dev Notes](https://github.com/microsoft/onnxruntime/blob/master/docs/Python_Dev_Notes.md) * Certain operators makes use of system locales. Installation of the **English language package** and configuring `en_US.UTF-8 locale` is required. * For Ubuntu install [language-pack-en package](https://packages.ubuntu.com/search?keywords=language-pack-en) From 76e94e045ecba092d936732ca366decbe1b3c539 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 6 Aug 2019 18:19:04 -0700 Subject: [PATCH 12/23] Clean up example section --- README.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index f2ee924e823e5..dcc3a52c6f678 100644 --- a/README.md +++ b/README.md @@ -141,9 +141,17 @@ ONNX Runtime is open and extensible, supporting a broad set of configurations an **Deployment with AzureML** -* Inferencing: [Inferencing Facial Expression Recognition](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-facial-expression-recognition-deploy.ipynb), [Inferencing MNIST Handwritten Digits](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.ipynb), [ Resnet50 Image Classification](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-modelzoo-aml-deploy-resnet50.ipynb), [TinyYolo](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb) -* [Train and Inference MNIST from Pytorch](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-train-pytorch-aml-deploy-mnist.ipynb) -* [FER+ on Azure Kubernetes Service with TensorRT](https://github.com/microsoft/onnxruntime/blob/master/docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb) +* Inferencing using [ONNX Model Zoo](https://github.com/onnx/models) models: + * [Facial Expression Recognition](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-facial-expression-recognition-deploy.ipynb) + * [MNIST Handwritten Digits](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.ipynb) + * [Resnet50 Image Classification](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-modelzoo-aml-deploy-resnet50.ipynb) +* Convert existing model for Inferencing: + * [TinyYolo](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb) +* Train a model with PyTorch and Inferencing: + * [MNIST](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-train-pytorch-aml-deploy-mnist.ipynb) + +* Inferencing with TensorRT Execution Provider on GPU (AKS) + * [FER+](https://github.com/microsoft/onnxruntime/blob/master/docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb) ## C# From 9b4955a7b4a1a4d437633a0521f0fbc15f209a24 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Wed, 7 Aug 2019 10:30:17 -0700 Subject: [PATCH 13/23] Cosmetic updates - Add a index of build instructions for browsability - Update build CUDA version from 9.1 to 10 --- BUILD.md | 54 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/BUILD.md b/BUILD.md index f4d1650ee03ad..037d0ac56d36a 100644 --- a/BUILD.md +++ b/BUILD.md @@ -30,7 +30,8 @@ OS/Compiler Matrix: ONNX Runtime python binding only supports Python 3.5, 3.6 and 3.7. -## Getting Started + +# Getting Started You may either get a prebuilt onnxruntime from nuget.org, or do it yourself using the following steps: 1. Checkout the source tree: ``` @@ -39,7 +40,8 @@ You may either get a prebuilt onnxruntime from nuget.org, or do it yourself usin ``` 2. Install cmake-3.13 or better from https://cmake.org/download/. -On Windows: +**On Windows:** + 3. (optional) Install protobuf 3.6.1 from source code (cmake/external/protobuf). CMake flag protobuf\_BUILD\_SHARED\_LIBS must be turned OFF. After the installation, you should have the 'protoc' executable in your PATH. 4. (optional) Install onnx from source code (cmake/external/onnx) ``` @@ -49,7 +51,8 @@ On Windows: ``` 5. Run `build.bat --config RelWithDebInfo --build_shared_lib --parallel`. -On Linux: +**On Linux:** + 3. (optional) Install protobuf 3.6.1 from source code (cmake/external/protobuf). CMake flag protobuf\_BUILD\_SHARED\_LIBS must be turned ON. After the installation, you should have the 'protoc' executable in your PATH. It is recommended to run `ldconfig` to make sure protobuf libraries are found. 4. If you installed your protobuf in a non standard location it would be helpful to set the following env var:`export CMAKE_ARGS="-DONNX_CUSTOM_PROTOC_EXECUTABLE=full path to protoc"` so ONNX build can find it. Also run `ldconfig ` so the linker can find protobuf libraries. 5. (optional) Install onnx from source code (cmake/external/onnx) @@ -64,18 +67,20 @@ The build script runs all unit tests by default (for native builds and skips tes The complete list of build options can be found by running `./build.sh (or ./build.bat) --help` +--- +# Build Instructions +* [x86](#Build-x86) +* [CI & Test Builds](#Build/Test-Flavors-for-CI) +* [Additional Common Build Flavors](#Additional-Build-Flavors) +* [ARM](#ARM-Builds) +* [Android](#Android-Builds) +* [ONNX Runtime Server (Linux)](#Build-ONNX-Runtime-Server-on-Linux) +--- + ## Build x86 - For Windows, just add --x86 argument when launching build.bat - For Linux, it must be built out of a x86 os, --x86 argument also needs be specified to build.sh -## Build ONNX Runtime Server on Linux - -1. ONNX Runtime server (and only the server) requires you to have Go installed to build, due to building BoringSSL. - See https://golang.org/doc/install for installation instructions. -2. In the ONNX Runtime root folder, run `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel` -3. ONNX Runtime Server supports sending log to [rsyslog](https://www.rsyslog.com/) daemon. To enable it, please build with an additional parameter: `--cmake_extra_defines onnxruntime_USE_SYSLOG=1`. The build command will look like this: `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel --cmake_extra_defines onnxruntime_USE_SYSLOG=1` - - ## Build/Test Flavors for CI ### CI Build Environments @@ -94,14 +99,14 @@ The complete list of build flavors can be seen by running `./build.sh --help` or The default generator on Windows is Visual Studio 2017, but you can also use the newer Visual Studio 2019 by passing `--cmake_generator "Visual Studio 16 2019"` to build.bat. ### Windows CUDA Build -ONNX Runtime supports CUDA builds. You will need to download and install [CUDA](https://developer.nvidia.com/cuda-toolkit) and [CUDNN](https://developer.nvidia.com/cudnn). +ONNX Runtime supports CUDA builds. You will need to download and install [CUDA](https://developer.nvidia.com/cuda-toolkit) and [cuDNN](https://developer.nvidia.com/cudnn). -ONNX Runtime is built and tested with CUDA 9.1 and CUDNN 7.1 using the Visual Studio 2017 14.11 toolset (i.e. Visual Studio 2017 v15.3). -CUDA versions from 9.1 up to 10.0, and CUDNN versions from 7.1 up to 7.4 should also work with Visual Studio 2017. +ONNX Runtime is built and tested with CUDA 10.0 and cuDNN 7.3 using the Visual Studio 2017 14.11 toolset (i.e. Visual Studio 2017 v15.3). +CUDA versions from 9.1 up to 10.1, and cuDNN versions from 7.1 up to 7.4 should also work with Visual Studio 2017. - The path to the CUDA installation must be provided via the CUDA_PATH environment variable, or the `--cuda_home parameter`. - - The path to the CUDNN installation (include the `cuda` folder in the path) must be provided via the CUDNN_PATH environment variable, or `--cudnn_home parameter`. The CUDNN path should contain `bin`, `include` and `lib` directories. - - The path to the CUDNN bin directory must be added to the PATH environment variable so that cudnn64_7.dll is found. + - The path to the cuDNN installation (include the `cuda` folder in the path) must be provided via the cuDNN_PATH environment variable, or `--cudnn_home parameter`. The cuDNN path should contain `bin`, `include` and `lib` directories. + - The path to the cuDNN bin directory must be added to the PATH environment variable so that cudnn64_7.dll is found. You can build with: @@ -110,7 +115,7 @@ You can build with: ./build.bat --use_cuda --cudnn_home --cuda_home (Windows) ``` -Depending on compatibility between the CUDA, CUDNN, and Visual Studio 2017 versions you are using, you may need to explicitly install an earlier version of the MSVC toolset. +Depending on compatibility between the CUDA, cuDNN, and Visual Studio 2017 versions you are using, you may need to explicitly install an earlier version of the MSVC toolset. - CUDA 10.0 is known to work with toolsets from 14.11 up to 14.16 (Visual Studio 2017 15.9), and should continue to work with future Visual Studio versions - https://devblogs.microsoft.com/cppblog/cuda-10-is-now-available-with-support-for-the-latest-visual-studio-2017-versions/ - CUDA 9.2 is known to work with the 14.11 MSVC toolset (Visual Studio 15.3 and 15.4) @@ -140,19 +145,19 @@ To build ONNX Runtime using MKL-DNN built with dependency on MKL small libraries ONNX runtime with nGraph as an execution provider (released as preview) can be built on Linux as follows : `./build.sh --use_ngraph`. Similarly, on Windows use `.\build.bat --use_ngraph`. ### TensorRT -ONNX Runtime supports the TensorRT execution provider (released as preview). You will need to download and install [CUDA](https://developer.nvidia.com/cuda-toolkit), [CUDNN](https://developer.nvidia.com/cudnn) and [TensorRT](https://developer.nvidia.com/nvidia-tensorrt-download). +ONNX Runtime supports the TensorRT execution provider (released as preview). You will need to download and install [CUDA](https://developer.nvidia.com/cuda-toolkit), [cuDNN](https://developer.nvidia.com/cudnn) and [TensorRT](https://developer.nvidia.com/nvidia-tensorrt-download). -The TensorRT execution provider for ONNX Runtime is built and tested with CUDA 9.0/CUDA 10.0, CUDNN 7.1 and TensorRT 5.0.2.6. +The TensorRT execution provider for ONNX Runtime is built and tested with CUDA 9.0/CUDA 10.0, cuDNN 7.1 and TensorRT 5.0.2.6. - The path to the CUDA installation must be provided via the CUDA_PATH environment variable, or the `--cuda_home parameter`. The CUDA path should contain `bin`, `include` and `lib` directories. - The path to the CUDA `bin` directory must be added to the PATH environment variable so that `nvcc` is found. - - The path to the CUDNN installation (path to folder that contains libcudnn.so) must be provided via the CUDNN_PATH environment variable, or `--cudnn_home parameter`. + - The path to the cuDNN installation (path to folder that contains libcudnn.so) must be provided via the cuDNN_PATH environment variable, or `--cudnn_home parameter`. - The path to TensorRT installation must be provided via the `--tensorrt_home parameter`. You can build from source on Linux by using the following `cmd` from the onnxruntime directory: ``` -./build.sh --cudnn_home --cuda_home --use_tensorrt --tensorrt_home (Linux) +./build.sh --cudnn_home --cuda_home --use_tensorrt --tensorrt_home (Linux) ``` ### OpenVINO Build @@ -397,3 +402,10 @@ ls -l /code/onnxruntime/build/Linux/MinSizeRel/dist/*.whl 3. Denote the unzip destination in step 1 as $ANDROID_NDK, append `-DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DONNX_CUSTOM_PROTOC_EXECUTABLE=path/to/protoc` to your cmake args, run cmake and make to build it. Note: For 32-bit devices, replace `-DANDROID_ABI=arm64-v8a` to `-DANDROID_ABI=armeabi-v7a`. + +## Build ONNX Runtime Server on Linux +Read more about ONNX Runtime Server [here](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md) +1. ONNX Runtime server (and only the server) requires you to have Go installed to build, due to building BoringSSL. + See https://golang.org/doc/install for installation instructions. +2. In the ONNX Runtime root folder, run `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel` +3. ONNX Runtime Server supports sending log to [rsyslog](https://www.rsyslog.com/) daemon. To enable it, please build with an additional parameter: `--cmake_extra_defines onnxruntime_USE_SYSLOG=1`. The build command will look like this: `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel --cmake_extra_defines onnxruntime_USE_SYSLOG=1` From 98bdb36026747bdd044181f51efd91bf3f7f25cb Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Wed, 7 Aug 2019 10:32:10 -0700 Subject: [PATCH 14/23] Fix broken link --- BUILD.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/BUILD.md b/BUILD.md index 037d0ac56d36a..10fb8bcdc4600 100644 --- a/BUILD.md +++ b/BUILD.md @@ -70,7 +70,7 @@ The complete list of build options can be found by running `./build.sh (or ./bui --- # Build Instructions * [x86](#Build-x86) -* [CI & Test Builds](#Build/Test-Flavors-for-CI) +* [CI & Test Builds](#Build-and-Test-Flavors-for-CI) * [Additional Common Build Flavors](#Additional-Build-Flavors) * [ARM](#ARM-Builds) * [Android](#Android-Builds) @@ -81,7 +81,7 @@ The complete list of build options can be found by running `./build.sh (or ./bui - For Windows, just add --x86 argument when launching build.bat - For Linux, it must be built out of a x86 os, --x86 argument also needs be specified to build.sh -## Build/Test Flavors for CI +## Build and Test Flavors for CI ### CI Build Environments From 430577f2abca406fae0656e3c974f7e75b074877 Mon Sep 17 00:00:00 2001 From: Hariharan Seshadri Date: Wed, 7 Aug 2019 17:04:33 -0700 Subject: [PATCH 15/23] Update README to reflect upgrade to pip requirement --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index dcc3a52c6f678..5d60a1b1b511d 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ system. * Version: **CUDA 10.0** and **cuDNN 7.3** * Linux Python packages require **CUDA 10.1** and **cuDNN 7.3** * Older ONNX Runtime releases: used **CUDA 9.1** and **cuDNN 7.1** - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details. -* Python binaries are compatible with **Python 3.5-3.7**. See [Python Dev Notes](https://github.com/microsoft/onnxruntime/blob/master/docs/Python_Dev_Notes.md) +* Python binaries are compatible with **Python 3.5-3.7**. See [Python Dev Notes](https://github.com/microsoft/onnxruntime/blob/master/docs/Python_Dev_Notes.md). If using `pip` to be download the Python binaries, run `pip install --upgrade pip` prior to downloading. * Certain operators makes use of system locales. Installation of the **English language package** and configuring `en_US.UTF-8 locale` is required. * For Ubuntu install [language-pack-en package](https://packages.ubuntu.com/search?keywords=language-pack-en) * Run the following commands: From 1463d4e738218af288818b7fd880c0b4a6db2110 Mon Sep 17 00:00:00 2001 From: Hariharan Seshadri Date: Thu, 8 Aug 2019 16:47:42 -0700 Subject: [PATCH 16/23] Update CuDNN version for Linux Python packages --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5d60a1b1b511d..eb4498da02da1 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ system. * For Linux, the system must have **libgomp.so.1** which can be installed using `apt-get install libgomp1`. * GPU builds require CUDA runtime libraries being installed on the system: * Version: **CUDA 10.0** and **cuDNN 7.3** - * Linux Python packages require **CUDA 10.1** and **cuDNN 7.3** + * Linux Python packages require **CUDA 10.1** and **cuDNN 7.6** * Older ONNX Runtime releases: used **CUDA 9.1** and **cuDNN 7.1** - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details. * Python binaries are compatible with **Python 3.5-3.7**. See [Python Dev Notes](https://github.com/microsoft/onnxruntime/blob/master/docs/Python_Dev_Notes.md). If using `pip` to be download the Python binaries, run `pip install --upgrade pip` prior to downloading. * Certain operators makes use of system locales. Installation of the **English language package** and configuring `en_US.UTF-8 locale` is required. From 645d192396144b9129b05e3c95b329c9afd89d8f Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Thu, 15 Aug 2019 11:54:37 -0700 Subject: [PATCH 17/23] Clean up content Updated ordering and add table of contents --- BUILD.md | 234 ++++++++++++++++++++++++++++--------------------------- 1 file changed, 119 insertions(+), 115 deletions(-) diff --git a/BUILD.md b/BUILD.md index 10fb8bcdc4600..8ec96098789eb 100644 --- a/BUILD.md +++ b/BUILD.md @@ -1,38 +1,9 @@ -# Build ONNX Runtime -Dockerfiles are available [here](https://github.com/microsoft/onnxruntime/tree/master/tools/ci_build/github/linux/docker) to help you get started. +# Building ONNX Runtime - Getting Started +*Dockerfiles are available [here](https://github.com/microsoft/onnxruntime/tree/master/tools/ci_build/github/linux/docker) to help you get started.* -## Supported architectures +*Pre-built packages are available the locations indicated [here](https://github.com/microsoft/onnxruntime#official-builds).* -| | x86_32 | x86_64 | ARM32v7 | ARM64 | -|-----------|:------------:|:------------:|:------------:|:------------:| -|Windows | YES | YES | YES | YES | -|Linux | YES | YES | YES | YES | -|Mac OS X | NO | YES | NO | NO | - -## Supported dev environments - -| OS | Supports CPU | Supports GPU| Notes | -|-------------|:------------:|:------------:|------------------------------------| -|Windows 10 | YES | YES | VS2019 through the latest VS2015 are supported | -|Windows 10
Subsystem for Linux | YES | NO | | -|Ubuntu 16.x | YES | YES | Also supported on ARM32v7 (experimental) | - -* Red Hat Enterprise Linux and CentOS are not supported. -* Other version of Ubuntu might work but we don't support them officially. -* GCC 4.x and below are not supported. - -OS/Compiler Matrix: - -| OS/Compiler | Supports VC | Supports GCC | -|-------------|:------------:|:----------------:| -|Windows 10 | YES | Not tested | -|Linux | NO | YES(gcc>=5.0) | - -ONNX Runtime python binding only supports Python 3.5, 3.6 and 3.7. - - -# Getting Started -You may either get a prebuilt onnxruntime from nuget.org, or do it yourself using the following steps: +**To build the baseline CPU version of ONNX Runtime from source:** 1. Checkout the source tree: ``` git clone --recursive https://github.com/Microsoft/onnxruntime @@ -51,6 +22,8 @@ You may either get a prebuilt onnxruntime from nuget.org, or do it yourself usin ``` 5. Run `build.bat --config RelWithDebInfo --build_shared_lib --parallel`. +*Note: The default Windows CMake Generator is Visual Studio 2017, but you can also use the newer Visual Studio 2019 by passing `--cmake_generator "Visual Studio 16 2019"` to build.bat.* + **On Linux:** 3. (optional) Install protobuf 3.6.1 from source code (cmake/external/protobuf). CMake flag protobuf\_BUILD\_SHARED\_LIBS must be turned ON. After the installation, you should have the 'protoc' executable in your PATH. It is recommended to run `ldconfig` to make sure protobuf libraries are found. @@ -65,40 +38,103 @@ You may either get a prebuilt onnxruntime from nuget.org, or do it yourself usin The build script runs all unit tests by default (for native builds and skips tests by default for cross-compiled builds). -The complete list of build options can be found by running `./build.sh (or ./build.bat) --help` +# Supported architectures and build environments + +## Architectures + +| | x86_32 | x86_64 | ARM32v7 | ARM64 | +|-----------|:------------:|:------------:|:------------:|:------------:| +|Windows | YES | YES | YES | YES | +|Linux | YES | YES | YES | YES | +|Mac OS X | NO | YES | NO | NO | + +## Environments + +| OS | Supports CPU | Supports GPU| Notes | +|-------------|:------------:|:------------:|------------------------------------| +|Windows 10 | YES | YES | VS2019 through the latest VS2015 are supported | +|Windows 10
Subsystem for Linux | YES | NO | | +|Ubuntu 16.x | YES | YES | Also supported on ARM32v7 (experimental) | + +* Red Hat Enterprise Linux and CentOS are not supported. +* Other version of Ubuntu might work but we don't support them officially. +* GCC 4.x and below are not supported. + +### OS/Compiler Matrix: + +| OS/Compiler | Supports VC | Supports GCC | +|-------------|:------------:|:----------------:| +|Windows 10 | YES | Not tested | +|Linux | NO | YES(gcc>=5.0) | + +ONNX Runtime Python bindings support Python 3.5, 3.6 and 3.7. --- -# Build Instructions -* [x86](#Build-x86) -* [CI & Test Builds](#Build-and-Test-Flavors-for-CI) -* [Additional Common Build Flavors](#Additional-Build-Flavors) -* [ARM](#ARM-Builds) -* [Android](#Android-Builds) +# Additional Build Instructions +The complete list of build options can be found by running `./build.sh (or ./build.bat) --help` + +* [Docker on Linux](#Docker-on-Linux) * [ONNX Runtime Server (Linux)](#Build-ONNX-Runtime-Server-on-Linux) + +**Execution Providers** +* [NVIDIA CUDA](#CUDA) +* [NVIDIA TensorRT](#TensorRT) +* [Intel MKL-DNN/MKL-ML](#MKL-DNN/MKL-ML) +* [Intel nGraph](#nGraph) +* [Intel OpenVINO](#openvino) + +**Options** +* [OpenMP](#OpenMP) +* [OpenBLAS](#OpenBLAS) + +**Architectures** +* [x86](#x86) +* [ARM](#ARM) +* [Android](#Android) --- +## Docker on Linux +Install Docker: `https://docs.docker.com/install/` -## Build x86 - - For Windows, just add --x86 argument when launching build.bat - - For Linux, it must be built out of a x86 os, --x86 argument also needs be specified to build.sh +## Build ONNX Runtime Server on Linux +Read more about ONNX Runtime Server [here](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md) +1. ONNX Runtime server (and only the server) requires you to have Go installed to build, due to building BoringSSL. + See https://golang.org/doc/install for installation instructions. +2. In the ONNX Runtime root folder, run `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel` +3. ONNX Runtime Server supports sending log to [rsyslog](https://www.rsyslog.com/) daemon. To enable it, please build with an additional parameter: `--cmake_extra_defines onnxruntime_USE_SYSLOG=1`. The build command will look like this: `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel --cmake_extra_defines onnxruntime_USE_SYSLOG=1` -## Build and Test Flavors for CI +**CPU** +``` +cd tools/ci_build/github/linux/docker +docker build -t onnxruntime_dev --build-arg OS_VERSION=16.04 -f Dockerfile.ubuntu . +docker run --rm -it onnxruntime_dev /bin/bash +``` -### CI Build Environments +**GPU** +If you need GPU support, please also install: +1. nvidia driver. Before doing this please add `nomodeset rd.driver.blacklist=nouveau` to your linux [kernel boot parameters](https://www.kernel.org/doc/html/v4.17/admin-guide/kernel-parameters.html). +2. nvidia-docker2: [Install doc](`https://github.com/NVIDIA/nvidia-docker/wiki/Installation-(version-2.0)`) -| Build Job Name | Environment | Dependency | Test Coverage | Scripts | -|--------------------|---------------------|---------------------------------|--------------------------|------------------------------------------| -| Linux_CI_Dev | Ubuntu 16.04 | python=3.5 | Unit tests; ONNXModelZoo | [script](tools/ci_build/github/linux/run_build.sh) | -| Linux_CI_GPU_Dev | Ubuntu 16.04 | python=3.5; nvidia-docker | Unit tests; ONNXModelZoo | [script](tools/ci_build/github/linux/run_build.sh) | -| Windows_CI_Dev | Windows Server 2016 | python=3.5 | Unit tests; ONNXModelZoo | [script](build.bat) | -| Windows_CI_GPU_Dev | Windows Server 2016 | cuda=9.1; cudnn=7.1; python=3.5 | Unit tests; ONNXModelZoo | [script](build.bat) | +To test if your nvidia-docker works: +``` +docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi +``` + +Then build a docker image. We provided a sample for use: +``` +cd tools/ci_build/github/linux/docker +docker build -t cuda_dev -f Dockerfile.ubuntu_gpu . +``` + +Then run it +``` +./tools/ci_build/github/linux/run_dockerbuild.sh +``` -## Additional Build Flavors -The complete list of build flavors can be seen by running `./build.sh --help` or `./build.bat --help`. Here are some common flavors. +## Execution Providers -### Windows CMake Generator -The default generator on Windows is Visual Studio 2017, but you can also use the newer Visual Studio 2019 by passing `--cmake_generator "Visual Studio 16 2019"` to build.bat. +### CUDA +For Linux, please use [this Dockerfile](https://github.com/microsoft/onnxruntime/blob/master/tools/ci_build/github/linux/docker/Dockerfile.ubuntu_gpu) and refer to instructions above for [building with Docker on Linux](#Docker-on-Linux) -### Windows CUDA Build ONNX Runtime supports CUDA builds. You will need to download and install [CUDA](https://developer.nvidia.com/cuda-toolkit) and [cuDNN](https://developer.nvidia.com/cudnn). ONNX Runtime is built and tested with CUDA 10.0 and cuDNN 7.3 using the Visual Studio 2017 14.11 toolset (i.e. Visual Studio 2017 v15.3). @@ -137,13 +173,6 @@ _Side note: If you have multiple versions of CUDA installed on a Windows machine e.g. C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\BuildCustomizations\. If you want to build with an earlier version, you must temporarily remove the 'CUDA x.y.*' files for later versions from this directory._ -### MKL-DNN/MKLML -To build ONNX Runtime with MKL-DNN support, build it with `./build.sh --use_mkldnn` -To build ONNX Runtime using MKL-DNN built with dependency on MKL small libraries, build it with `./build.sh --use_mkldnn --use_mklml` - -### nGraph -ONNX runtime with nGraph as an execution provider (released as preview) can be built on Linux as follows : `./build.sh --use_ngraph`. Similarly, on Windows use `.\build.bat --use_ngraph`. - ### TensorRT ONNX Runtime supports the TensorRT execution provider (released as preview). You will need to download and install [CUDA](https://developer.nvidia.com/cuda-toolkit), [cuDNN](https://developer.nvidia.com/cudnn) and [TensorRT](https://developer.nvidia.com/nvidia-tensorrt-download). @@ -158,9 +187,16 @@ You can build from source on Linux by using the following `cmd` from the onnxrun ``` ./build.sh --cudnn_home --cuda_home --use_tensorrt --tensorrt_home (Linux) - ``` -### OpenVINO Build + +### MKL-DNN/MKL-ML +To build ONNX Runtime with MKL-DNN support, build it with `./build.sh --use_mkldnn` +To build ONNX Runtime using MKL-DNN built with dependency on MKL small libraries, build it with `./build.sh --use_mkldnn --use_mklml` + +### nGraph +ONNX runtime with nGraph as an execution provider (released as preview) can be built on Linux as follows : `./build.sh --use_ngraph`. Similarly, on Windows use `.\build.bat --use_ngraph` + +### OpenVINO ONNX Runtime supports OpenVINO Execution Provider to enable deep learning inference using Intel® OpenVINOTM Toolkit. This execution provider supports several Intel hardware device types - CPU, integrated GPU, Intel® MovidiusTM VPUs and Intel® Vision accelerator Design with 8 Intel MovidiusTM MyriadX VPUs. @@ -199,58 +235,33 @@ The OpenVINO Execution Provider can be built using the following commands: | VAD-M_FP16 | Intel® Vision Accelerator Design based on 8 MovidiusTM MyriadX VPUs | For more information on OpenVINO Execution Provider's ONNX Layer support, Topology support, and Intel hardware enabled, please refer to the document OpenVINO-ExecutionProvider.md in $onnxruntime_root/docs/execution_providers - + +## Options ### OpenBLAS -#### Windows +**Windows** Instructions how to build OpenBLAS for windows can be found here https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio#build-openblas-for-universal-windows-platform. Once you have the OpenBLAS binaries, build ONNX Runtime with `./build.bat --use_openblas` -#### Linux +**Linux** For Linux (e.g. Ubuntu 16.04), install libopenblas-dev package `sudo apt-get install libopenblas-dev` and build with `./build.sh --use_openblas` - + ### OpenMP ``` ./build.sh --use_openmp (for Linux) ./build.bat --use_openmp (for Windows) ``` -### Build with Docker on Linux -Install Docker: `https://docs.docker.com/install/` - -#### CPU -``` -cd tools/ci_build/github/linux/docker -docker build -t onnxruntime_dev --build-arg OS_VERSION=16.04 -f Dockerfile.ubuntu . -docker run --rm -it onnxruntime_dev /bin/bash -``` - -#### GPU -If you need GPU support, please also install: -1. nvidia driver. Before doing this please add `nomodeset rd.driver.blacklist=nouveau` to your linux [kernel boot parameters](https://www.kernel.org/doc/html/v4.17/admin-guide/kernel-parameters.html). -2. nvidia-docker2: [Install doc](`https://github.com/NVIDIA/nvidia-docker/wiki/Installation-(version-2.0)`) - -To test if your nvidia-docker works: -``` -docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi -``` - -Then build a docker image. We provided a sample for use: -``` -cd tools/ci_build/github/linux/docker -docker build -t cuda_dev -f Dockerfile.ubuntu_gpu . -``` - -Then run it -``` -./tools/ci_build/github/linux/run_dockerbuild.sh -``` +## Architectures +### x86 + - For Windows, just add --x86 argument when launching build.bat + - For Linux, it must be built out of a x86 os, --x86 argument also needs be specified to build.sh -## ARM Builds +### ARM We have experimental support for Linux ARM builds. Windows on ARM is well tested. -### Cross compiling for ARM with Docker (Linux/Windows - FASTER, RECOMMENDED) +#### Cross compiling for ARM with Docker (Linux/Windows - FASTER, RECOMMENDED) This method allows you to compile using a desktop or cloud VM. This is much faster than compiling natively and avoids out-of-memory issues that may be encountered when on lower-powered ARM devices. The resulting ONNX Runtime Python wheel (.whl) file is then deployed to an ARM device where it can be invoked in Python 3 scripts. The Dockerfile used in these instructions specifically targets Raspberry Pi 3/3+ running Raspbian Stretch. The same approach should work for other ARM devices, but may require some changes to the Dockerfile such as choosing a different base image (Line 0: `FROM ...`). @@ -301,7 +312,7 @@ The Dockerfile used in these instructions specifically targets Raspberry Pi 3/3+ ``` 10. Test installation by following the instructions [here](https://microsoft.github.io/onnxruntime/) -### Cross compiling on Linux (without Docker) +#### Cross compiling on Linux (without Docker) 1. Get the corresponding toolchain. For example, if your device is Raspberry Pi and the device os is Ubuntu 16.04, you may use gcc-linaro-6.3.1 from [https://releases.linaro.org/components/toolchain/binaries](https://releases.linaro.org/components/toolchain/binaries) 2. Setup env vars ```bash @@ -327,7 +338,7 @@ The Dockerfile used in these instructions specifically targets Raspberry Pi 3/3+ 6. Append `-DONNX_CUSTOM_PROTOC_EXECUTABLE=/path/to/protoc -DCMAKE_TOOLCHAIN_FILE=path/to/tool.cmake` to your cmake args, run cmake and make to build it. -### Native compiling on Linux ARM device (SLOWER) +#### Native compiling on Linux ARM device (SLOWER) Docker build runs on a Raspberry Pi 3B with Raspbian Stretch Lite OS (Desktop version will run out memory when linking the .so file) will take 8-9 hours in total. ```bash sudo apt-get update @@ -379,19 +390,19 @@ ls -l /code/onnxruntime/build/Linux/MinSizeRel/*.so ls -l /code/onnxruntime/build/Linux/MinSizeRel/dist/*.whl ``` -### Cross compiling on Windows -#### Using Visual C++ compilers +#### Cross compiling on Windows +**Using Visual C++ compilers** 1. Download and install Visual C++ compilers and libraries for ARM(64). If you have Visual Studio installed, please use the Visual Studio Installer (look under the section `Individual components` after choosing to `modify` Visual Studio) to download and install the corresponding ARM(64) compilers and libraries. 2. Use `build.bat` and specify `--arm` or `--arm64` as the build option to start building. Preferably use `Developer Command Prompt for VS` or make sure all the installed cross-compilers are findable from the command prompt being used to build using the PATH environmant variable. -### Using other compilers +**Using other compilers** (TODO) -## Android Builds +### Android -### Cross compiling on Linux +#### Cross compiling on Linux 1. Get Android NDK from https://developer.android.com/ndk/downloads. Please unzip it after downloading. @@ -402,10 +413,3 @@ ls -l /code/onnxruntime/build/Linux/MinSizeRel/dist/*.whl 3. Denote the unzip destination in step 1 as $ANDROID_NDK, append `-DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DONNX_CUSTOM_PROTOC_EXECUTABLE=path/to/protoc` to your cmake args, run cmake and make to build it. Note: For 32-bit devices, replace `-DANDROID_ABI=arm64-v8a` to `-DANDROID_ABI=armeabi-v7a`. - -## Build ONNX Runtime Server on Linux -Read more about ONNX Runtime Server [here](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md) -1. ONNX Runtime server (and only the server) requires you to have Go installed to build, due to building BoringSSL. - See https://golang.org/doc/install for installation instructions. -2. In the ONNX Runtime root folder, run `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel` -3. ONNX Runtime Server supports sending log to [rsyslog](https://www.rsyslog.com/) daemon. To enable it, please build with an additional parameter: `--cmake_extra_defines onnxruntime_USE_SYSLOG=1`. The build command will look like this: `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel --cmake_extra_defines onnxruntime_USE_SYSLOG=1` From be7ad6885df5322e0eeffcda8e3860bd65576a97 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Thu, 15 Aug 2019 12:06:35 -0700 Subject: [PATCH 18/23] Minor format fixes --- BUILD.md | 58 +++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 17 deletions(-) diff --git a/BUILD.md b/BUILD.md index 8ec96098789eb..4a4137b0f883d 100644 --- a/BUILD.md +++ b/BUILD.md @@ -3,7 +3,7 @@ *Pre-built packages are available the locations indicated [here](https://github.com/microsoft/onnxruntime#official-builds).* -**To build the baseline CPU version of ONNX Runtime from source:** +## To build the baseline CPU version of ONNX Runtime from source: 1. Checkout the source tree: ``` git clone --recursive https://github.com/Microsoft/onnxruntime @@ -38,6 +38,8 @@ The build script runs all unit tests by default (for native builds and skips tests by default for cross-compiled builds). +--- + # Supported architectures and build environments ## Architectures @@ -70,6 +72,7 @@ The build script runs all unit tests by default (for native builds and skips tes ONNX Runtime Python bindings support Python 3.5, 3.6 and 3.7. --- + # Additional Build Instructions The complete list of build options can be found by running `./build.sh (or ./build.bat) --help` @@ -79,7 +82,7 @@ The complete list of build options can be found by running `./build.sh (or ./bui **Execution Providers** * [NVIDIA CUDA](#CUDA) * [NVIDIA TensorRT](#TensorRT) -* [Intel MKL-DNN/MKL-ML](#MKL-DNN/MKL-ML) +* [Intel MKL-DNN/MKL-ML](#MKLDNN-and-MKLML) * [Intel nGraph](#nGraph) * [Intel OpenVINO](#openvino) @@ -95,13 +98,6 @@ The complete list of build options can be found by running `./build.sh (or ./bui ## Docker on Linux Install Docker: `https://docs.docker.com/install/` -## Build ONNX Runtime Server on Linux -Read more about ONNX Runtime Server [here](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md) -1. ONNX Runtime server (and only the server) requires you to have Go installed to build, due to building BoringSSL. - See https://golang.org/doc/install for installation instructions. -2. In the ONNX Runtime root folder, run `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel` -3. ONNX Runtime Server supports sending log to [rsyslog](https://www.rsyslog.com/) daemon. To enable it, please build with an additional parameter: `--cmake_extra_defines onnxruntime_USE_SYSLOG=1`. The build command will look like this: `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel --cmake_extra_defines onnxruntime_USE_SYSLOG=1` - **CPU** ``` cd tools/ci_build/github/linux/docker @@ -130,6 +126,17 @@ Then run it ./tools/ci_build/github/linux/run_dockerbuild.sh ``` +--- + +## Build ONNX Runtime Server on Linux +Read more about ONNX Runtime Server [here](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md) +1. ONNX Runtime server (and only the server) requires you to have Go installed to build, due to building BoringSSL. + See https://golang.org/doc/install for installation instructions. +2. In the ONNX Runtime root folder, run `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel` +3. ONNX Runtime Server supports sending log to [rsyslog](https://www.rsyslog.com/) daemon. To enable it, please build with an additional parameter: `--cmake_extra_defines onnxruntime_USE_SYSLOG=1`. The build command will look like this: `./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel --cmake_extra_defines onnxruntime_USE_SYSLOG=1` + +--- + ## Execution Providers ### CUDA @@ -173,6 +180,8 @@ _Side note: If you have multiple versions of CUDA installed on a Windows machine e.g. C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\BuildCustomizations\. If you want to build with an earlier version, you must temporarily remove the 'CUDA x.y.*' files for later versions from this directory._ +--- + ### TensorRT ONNX Runtime supports the TensorRT execution provider (released as preview). You will need to download and install [CUDA](https://developer.nvidia.com/cuda-toolkit), [cuDNN](https://developer.nvidia.com/cudnn) and [TensorRT](https://developer.nvidia.com/nvidia-tensorrt-download). @@ -189,13 +198,19 @@ You can build from source on Linux by using the following `cmd` from the onnxrun ./build.sh --cudnn_home --cuda_home --use_tensorrt --tensorrt_home (Linux) ``` -### MKL-DNN/MKL-ML +--- + +### MKLDNN and MKLML To build ONNX Runtime with MKL-DNN support, build it with `./build.sh --use_mkldnn` To build ONNX Runtime using MKL-DNN built with dependency on MKL small libraries, build it with `./build.sh --use_mkldnn --use_mklml` +--- + ### nGraph ONNX runtime with nGraph as an execution provider (released as preview) can be built on Linux as follows : `./build.sh --use_ngraph`. Similarly, on Windows use `.\build.bat --use_ngraph` +--- + ### OpenVINO ONNX Runtime supports OpenVINO Execution Provider to enable deep learning inference using Intel® OpenVINOTM Toolkit. This execution provider supports several Intel hardware device types - CPU, integrated GPU, Intel® MovidiusTM VPUs and Intel® Vision accelerator Design with 8 Intel MovidiusTM MyriadX VPUs. @@ -236,7 +251,17 @@ The OpenVINO Execution Provider can be built using the following commands: For more information on OpenVINO Execution Provider's ONNX Layer support, Topology support, and Intel hardware enabled, please refer to the document OpenVINO-ExecutionProvider.md in $onnxruntime_root/docs/execution_providers +--- + ## Options +### OpenMP +``` +./build.sh --use_openmp (for Linux) +./build.bat --use_openmp (for Windows) +``` + +--- + ### OpenBLAS **Windows** Instructions how to build OpenBLAS for windows can be found here https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio#build-openblas-for-universal-windows-platform. @@ -246,18 +271,16 @@ Once you have the OpenBLAS binaries, build ONNX Runtime with `./build.bat --use_ **Linux** For Linux (e.g. Ubuntu 16.04), install libopenblas-dev package `sudo apt-get install libopenblas-dev` and build with `./build.sh --use_openblas` - -### OpenMP -``` -./build.sh --use_openmp (for Linux) -./build.bat --use_openmp (for Windows) -``` + +--- ## Architectures ### x86 - For Windows, just add --x86 argument when launching build.bat - For Linux, it must be built out of a x86 os, --x86 argument also needs be specified to build.sh +--- + ### ARM We have experimental support for Linux ARM builds. Windows on ARM is well tested. @@ -337,7 +360,6 @@ The Dockerfile used in these instructions specifically targets Raspberry Pi 3/3+ ``` 6. Append `-DONNX_CUSTOM_PROTOC_EXECUTABLE=/path/to/protoc -DCMAKE_TOOLCHAIN_FILE=path/to/tool.cmake` to your cmake args, run cmake and make to build it. - #### Native compiling on Linux ARM device (SLOWER) Docker build runs on a Raspberry Pi 3B with Raspbian Stretch Lite OS (Desktop version will run out memory when linking the .so file) will take 8-9 hours in total. ```bash @@ -400,6 +422,8 @@ ls -l /code/onnxruntime/build/Linux/MinSizeRel/dist/*.whl **Using other compilers** (TODO) +--- + ### Android #### Cross compiling on Linux From 84d547658a54cb08e8110cd8a68184ff83f28ff8 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Thu, 15 Aug 2019 12:08:37 -0700 Subject: [PATCH 19/23] Move Android NNAPI under EP section --- BUILD.md | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/BUILD.md b/BUILD.md index 4a4137b0f883d..91af979863c6c 100644 --- a/BUILD.md +++ b/BUILD.md @@ -85,6 +85,7 @@ The complete list of build options can be found by running `./build.sh (or ./bui * [Intel MKL-DNN/MKL-ML](#MKLDNN-and-MKLML) * [Intel nGraph](#nGraph) * [Intel OpenVINO](#openvino) +* [Android NNAPI](#Android) **Options** * [OpenMP](#OpenMP) @@ -93,7 +94,7 @@ The complete list of build options can be found by running `./build.sh (or ./bui **Architectures** * [x86](#x86) * [ARM](#ARM) -* [Android](#Android) + --- ## Docker on Linux Install Docker: `https://docs.docker.com/install/` @@ -253,6 +254,22 @@ For more information on OpenVINO Execution Provider's ONNX Layer support, To --- +### Android + +#### Cross compiling on Linux + +1. Get Android NDK from https://developer.android.com/ndk/downloads. Please unzip it after downloading. + +2. Get a pre-compiled protoc: + + You may get it from https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protoc-3.6.1-linux-x86_64.zip. Please unzip it after downloading. + +3. Denote the unzip destination in step 1 as $ANDROID_NDK, append `-DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DONNX_CUSTOM_PROTOC_EXECUTABLE=path/to/protoc` to your cmake args, run cmake and make to build it. + +Note: For 32-bit devices, replace `-DANDROID_ABI=arm64-v8a` to `-DANDROID_ABI=armeabi-v7a`. + +--- + ## Options ### OpenMP ``` @@ -421,19 +438,3 @@ ls -l /code/onnxruntime/build/Linux/MinSizeRel/dist/*.whl **Using other compilers** (TODO) - ---- - -### Android - -#### Cross compiling on Linux - -1. Get Android NDK from https://developer.android.com/ndk/downloads. Please unzip it after downloading. - -2. Get a pre-compiled protoc: - - You may get it from https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protoc-3.6.1-linux-x86_64.zip. Please unzip it after downloading. - -3. Denote the unzip destination in step 1 as $ANDROID_NDK, append `-DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DONNX_CUSTOM_PROTOC_EXECUTABLE=path/to/protoc` to your cmake args, run cmake and make to build it. - -Note: For 32-bit devices, replace `-DANDROID_ABI=arm64-v8a` to `-DANDROID_ABI=armeabi-v7a`. From 2d914f0cb44a046252bf017c158343f548f1dcf9 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Thu, 15 Aug 2019 12:13:46 -0700 Subject: [PATCH 20/23] Add link to operator support documentation --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index eb4498da02da1..0657adbf0ffec 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,8 @@ ## Run any ONNX model ONNX Runtime provides comprehensive support of the ONNX spec and can be used to run all models based on ONNX v1.2.1 and higher. See version compatibility details [here](https://github.com/microsoft/onnxruntime/blob/master/docs/Versioning.md). +For the full set of operators and types supported, please see [ + *Note: Some operators not supported in the current ONNX version may be available as a [Contrib Operator](https://github.com/microsoft/onnxruntime/blob/master/docs/ContribOperators.md)* **Traditional ML support** From 6089c463412c427590771f1ecce2c7b8234a6f89 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Thu, 15 Aug 2019 12:15:12 -0700 Subject: [PATCH 21/23] Fix typo --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 0657adbf0ffec..3c5695ec19223 100644 --- a/README.md +++ b/README.md @@ -37,14 +37,15 @@ ## Run any ONNX model ONNX Runtime provides comprehensive support of the ONNX spec and can be used to run all models based on ONNX v1.2.1 and higher. See version compatibility details [here](https://github.com/microsoft/onnxruntime/blob/master/docs/Versioning.md). -For the full set of operators and types supported, please see [ - -*Note: Some operators not supported in the current ONNX version may be available as a [Contrib Operator](https://github.com/microsoft/onnxruntime/blob/master/docs/ContribOperators.md)* - **Traditional ML support** In addition to DNN models, ONNX Runtime fully supports the [ONNX-ML profile](https://github.com/onnx/onnx/blob/master/docs/Operators-ml.md) of the ONNX spec for traditional ML scenarios. +For the full set of operators and types supported, please see [operator documentation](https://github.com/microsoft/onnxruntime/blob/master/docs/OperatorKernels.md) + +*Note: Some operators not supported in the current ONNX version may be available as a [Contrib Operator](https://github.com/microsoft/onnxruntime/blob/master/docs/ContribOperators.md)* + + ## High Performance ONNX Runtime supports both CPU and GPU. Using various graph optimizations and accelerators, ONNX Runtime can provide lower latency compared to other runtimes for faster end-to-end customer experiences and minimized machine utilization costs. From 04e762c40983895f71264f901d9ea7c0e41ac8e6 Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 27 Aug 2019 10:27:08 -0700 Subject: [PATCH 22/23] typo fix --- BUILD.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/BUILD.md b/BUILD.md index 91af979863c6c..3691e18d1b2e4 100644 --- a/BUILD.md +++ b/BUILD.md @@ -1,7 +1,7 @@ # Building ONNX Runtime - Getting Started *Dockerfiles are available [here](https://github.com/microsoft/onnxruntime/tree/master/tools/ci_build/github/linux/docker) to help you get started.* -*Pre-built packages are available the locations indicated [here](https://github.com/microsoft/onnxruntime#official-builds).* +*Pre-built packages are available at the locations indicated [here](https://github.com/microsoft/onnxruntime#official-builds).* ## To build the baseline CPU version of ONNX Runtime from source: 1. Checkout the source tree: From 3b0efca9912d40d4ffec40fd611cb6e61321ac0c Mon Sep 17 00:00:00 2001 From: Faith Xu Date: Tue, 27 Aug 2019 10:28:12 -0700 Subject: [PATCH 23/23] remove todo section --- BUILD.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/BUILD.md b/BUILD.md index 3691e18d1b2e4..220cd8b4653ab 100644 --- a/BUILD.md +++ b/BUILD.md @@ -436,5 +436,3 @@ ls -l /code/onnxruntime/build/Linux/MinSizeRel/dist/*.whl 2. Use `build.bat` and specify `--arm` or `--arm64` as the build option to start building. Preferably use `Developer Command Prompt for VS` or make sure all the installed cross-compilers are findable from the command prompt being used to build using the PATH environmant variable. -**Using other compilers** -(TODO)