From ac54b30c07bd0fc82251e166199d8e483a1e0a40 Mon Sep 17 00:00:00 2001 From: colorjam Date: Fri, 26 Mar 2021 07:16:32 +0000 Subject: [PATCH 01/16] Update readme nnidoc link --- README.md | 136 +++++++++++++++++++++++++++--------------------------- 1 file changed, 68 insertions(+), 68 deletions(-) diff --git a/README.md b/README.md index fb7148af00..9bad71830d 100644 --- a/README.md +++ b/README.md @@ -14,9 +14,9 @@ [NNI Doc](https://nni.readthedocs.io/) | [简体中文](README_zh_CN.md) -**NNI (Neural Network Intelligence)** is a lightweight but powerful toolkit to help users **automate** Feature Engineering, Neural Architecture Search, Hyperparameter Tuning and Model Compression. +**NNI (Neural Network Intelligence)** is a lightweight but powerful toolkit to help users **automate** Feature Engineering, Neural Architecture Search, Hyperparameter Tuning and Model Compression. -The tool manages automated machine learning (AutoML) experiments, **dispatches and runs** experiments' trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in **different training environments** like Local Machine, Remote Servers, OpenPAI, Kubeflow, FrameworkController on K8S (AKS etc.), DLWorkspace (aka. DLTS), AML (Azure Machine Learning), AdaptDL (aka. ADL) , other cloud options and even Hybrid mode. +The tool manages automated machine learning (AutoML) experiments, **dispatches and runs** experiments' trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in **different training environments** like Local Machine, Remote Servers, OpenPAI, Kubeflow, FrameworkController on K8S (AKS etc.), DLWorkspace (aka. DLTS), AML (Azure Machine Learning), AdaptDL (aka. ADL) , other cloud options and even Hybrid mode. ## **Who should consider using NNI** @@ -72,7 +72,7 @@ Within the following table, we summarized the current NNI capabilities, we are g
  • TensorFlow
  • MXNet
  • Caffe2
    • - More...
    + More...
    - Hyperparameter Tuning + Hyperparameter Tuning - Neural Architecture Search + Neural Architecture Search - Model Compression + Model Compression - Feature Engineering (Beta) + Feature Engineering (Beta) - Early Stop Algorithms + Early Stop Algorithms @@ -198,21 +198,21 @@ Within the following table, we summarized the current NNI capabilities, we are g @@ -244,8 +244,8 @@ For detail system requirements of NNI, please refer to [here](https://nni.readth Note: * If there is any privilege issue, add `--user` to install NNI in the user directory. -* Currently NNI on Windows supports local, remote and pai mode. Anaconda or Miniconda is highly recommended to install [NNI on Windows](docs/en_US/Tutorial/InstallationWin.rst). -* If there is any error like `Segmentation fault`, please refer to [FAQ](docs/en_US/Tutorial/FAQ.rst). For FAQ on Windows, please refer to [NNI on Windows](docs/en_US/Tutorial/InstallationWin.rst#faq). +* Currently NNI on Windows supports local, remote and pai mode. Anaconda or Miniconda is highly recommended to install [NNI on Windows](https://nni.readthedocs.io/en/latest/Tutorial/InstallationWin.html). +* If there is any error like `Segmentation fault`, please refer to [FAQ](https://nni.readthedocs.io/en/latest/Tutorial/FAQ.html). For FAQ on Windows, please refer to [NNI on Windows](https://nni.readthedocs.io/en/latest/Tutorial/InstallationWin.html#faq). ### **Verify installation** @@ -297,7 +297,7 @@ You can use these commands to get more information about the experiment ----------------------------------------------------------------------- ``` -* Open the `Web UI url` in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below. [Here](docs/en_US/Tutorial/WebUI.rst) are more Web UI pages. +* Open the `Web UI url` in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below. [Here](https://nni.readthedocs.io/en/latest/Tutorial/WebUI.html) are more Web UI pages. From 3ef10bf426e2ec49c264b32c220ab8cec5138f61 Mon Sep 17 00:00:00 2001 From: colorjam Date: Fri, 26 Mar 2021 07:45:52 +0000 Subject: [PATCH 02/16] Clarify compression doc --- docs/en_US/Compression/Overview.rst | 5 +++-- docs/en_US/Compression/Tutorial.rst | 12 ++++++++++-- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/en_US/Compression/Overview.rst b/docs/en_US/Compression/Overview.rst index e08c57e97d..d2738662c2 100644 --- a/docs/en_US/Compression/Overview.rst +++ b/docs/en_US/Compression/Overview.rst @@ -14,7 +14,8 @@ NNI provides a model compression toolkit to help user compress and speed up thei * Provide friendly and easy-to-use compression utilities for users to dive into the compression process and results. * Concise interface for users to customize their own compression algorithms. -*Note that the interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future.* +*Note that NNI compresses the model through a simulation process. To obtain a truly compact model, users should conduct model speedup. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future.* + Supported Algorithms -------------------- @@ -24,7 +25,7 @@ The algorithms include pruning algorithms and quantization algorithms. Pruning Algorithms ^^^^^^^^^^^^^^^^^^ -Pruning algorithms compress the original network by removing redundant weights or channels of layers, which can reduce model complexity and address the over-fitting issue. +Pruning algorithms compress the original network by removing redundant weights or channels of layers, which can reduce model complexity and address the over-fitting issue. .. list-table:: :header-rows: 1 diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 2cb874620f..0d75d65386 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -23,8 +23,17 @@ There are different keys in a ``dict``. Some of them are common keys supported b Some other keys are often specific to a certain algorithm, users can refer to `pruning algorithms <./Pruner.rst>`__ and `quantization algorithms <./Quantizer.rst>`__ for the keys allowed by each algorithm. -A simple example of configuration is shown below: +To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can be write as: +.. code-block:: python + + [ + { + 'sparsity': 0.6, + 'op_types': ['Conv2d'] + }, + ] +To control the sparsity of specific layers, the configuration can be writed as: .. code-block:: python [ @@ -41,7 +50,6 @@ A simple example of configuration is shown below: 'op_names': ['op_name3'] } ] - It means following the algorithm's default setting for compressed operations with sparsity 0.8, but for ``op_name1`` and ``op_name2`` use sparsity 0.6, and do not compress ``op_name3``. Quantization specific keys From 7c3186c492f4629defbea49f376eea157beaf6cd Mon Sep 17 00:00:00 2001 From: colorjam Date: Fri, 26 Mar 2021 08:49:15 +0000 Subject: [PATCH 03/16] Fix warning --- docs/en_US/Compression/Tutorial.rst | 37 +++++++++++++---------------- 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 0d75d65386..9ffda93117 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -26,30 +26,27 @@ Some other keys are often specific to a certain algorithm, users can refer to `p To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can be write as: .. code-block:: python - [ - { - 'sparsity': 0.6, - 'op_types': ['Conv2d'] - }, - ] + [{ + 'sparsity': 0.6, + 'op_types': ['Conv2d'] + }] To control the sparsity of specific layers, the configuration can be writed as: .. code-block:: python - [ - { - 'sparsity': 0.8, - 'op_types': ['default'] - }, - { - 'sparsity': 0.6, - 'op_names': ['op_name1', 'op_name2'] - }, - { - 'exclude': True, - 'op_names': ['op_name3'] - } - ] + [{ + 'sparsity': 0.8, + 'op_types': ['default'] + }, + { + 'sparsity': 0.6, + 'op_names': ['op_name1', 'op_name2'] + }, + { + 'exclude': True, + 'op_names': ['op_name3'] + }] + It means following the algorithm's default setting for compressed operations with sparsity 0.8, but for ``op_name1`` and ``op_name2`` use sparsity 0.6, and do not compress ``op_name3``. Quantization specific keys From 46b99c0f1da1f76043e2cb1b1fa0941088ce235e Mon Sep 17 00:00:00 2001 From: colorjam Date: Mon, 29 Mar 2021 02:09:04 +0000 Subject: [PATCH 04/16] Update according to comments & fix warning --- README.md | 148 ++++++++++++++-------------- docs/en_US/Compression/Tutorial.rst | 18 ++-- 2 files changed, 83 insertions(+), 83 deletions(-) diff --git a/README.md b/README.md index 9bad71830d..488d21d0e7 100644 --- a/README.md +++ b/README.md @@ -5,18 +5,18 @@ ----------- [![MIT licensed](https://img.shields.io/badge/license-MIT-brightgreen.svg)](LICENSE) -[![Build Status](https://msrasrg.visualstudio.com/NNIOpenSource/_apis/build/status/full%20test%20-%20linux?branchName=master)](https://msrasrg.visualstudio.com/NNIOpenSource/_build/latest?definitionId=62&branchName=master) +[![Build Status](https://msrasrg.visualstudio.com/NNIOpenSource/_apis/build/status/full%20test%20-%20linux?branchName=master)](https://msrasrg.visualstudio.com/NNIOpenSource/_build/stable?definitionId=62&branchName=master) [![Issues](https://img.shields.io/github/issues-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen) [![Bugs](https://img.shields.io/github/issues/Microsoft/nni/bug.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen+label%3Abug) [![Pull Requests](https://img.shields.io/github/issues-pr-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/pulls?q=is%3Apr+is%3Aopen) [![Version](https://img.shields.io/github/release/Microsoft/nni.svg)](https://github.com/Microsoft/nni/releases) [![Join the chat at https://gitter.im/Microsoft/nni](https://badges.gitter.im/Microsoft/nni.svg)](https://gitter.im/Microsoft/nni?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -[![Documentation Status](https://readthedocs.org/projects/nni/badge/?version=latest)](https://nni.readthedocs.io/en/latest/?badge=latest) +[![Documentation Status](https://readthedocs.org/projects/nni/badge/?version=stable)](https://nni.readthedocs.io/en/stable/?badge=stable) [NNI Doc](https://nni.readthedocs.io/) | [简体中文](README_zh_CN.md) -**NNI (Neural Network Intelligence)** is a lightweight but powerful toolkit to help users **automate** Feature Engineering, Neural Architecture Search, Hyperparameter Tuning and Model Compression. +**NNI (Neural Network Intelligence)** is a lightweight but powerful toolkit to help users **automate** Feature Engineering, Neural Architecture Search, Hyperparameter Tuning and Model Compression. -The tool manages automated machine learning (AutoML) experiments, **dispatches and runs** experiments' trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in **different training environments** like Local Machine, Remote Servers, OpenPAI, Kubeflow, FrameworkController on K8S (AKS etc.), DLWorkspace (aka. DLTS), AML (Azure Machine Learning), AdaptDL (aka. ADL) , other cloud options and even Hybrid mode. +The tool manages automated machine learning (AutoML) experiments, **dispatches and runs** experiments' trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in **different training environments** like Local Machine, Remote Servers, OpenPAI, Kubeflow, FrameworkController on K8S (AKS etc.), DLWorkspace (aka. DLTS), AML (Azure Machine Learning), AdaptDL (aka. ADL) , other cloud options and even Hybrid mode. ## **Who should consider using NNI** @@ -72,7 +72,7 @@ Within the following table, we summarized the current NNI capabilities, we are g
  • TensorFlow
  • MXNet
  • Caffe2
    • - More...
    + More...
      @@ -81,7 +81,7 @@ Within the following table, we summarized the current NNI capabilities, we are g
    • Scikit-learn
    • XGBoost
    • LightGBM
      • - More...
      + More...
     @@ -197,22 +197,22 @@ Within the following table, we summarized the current NNI capabilities, we are g @@ -237,15 +237,15 @@ Windows python -m pip install --upgrade nni ``` -If you want to try latest code, please [install NNI](https://nni.readthedocs.io/en/latest/installation.html) from source code. +If you want to try stable code, please [install NNI](https://nni.readthedocs.io/en/stable/installation.html) from source code. -For detail system requirements of NNI, please refer to [here](https://nni.readthedocs.io/en/latest/Tutorial/InstallationLinux.html#system-requirements) for Linux & macOS, and [here](https://nni.readthedocs.io/en/latest/Tutorial/InstallationWin.html#system-requirements) for Windows. +For detail system requirements of NNI, please refer to [here](https://nni.readthedocs.io/en/stable/Tutorial/InstallationLinux.html#system-requirements) for Linux & macOS, and [here](https://nni.readthedocs.io/en/stable/Tutorial/InstallationWin.html#system-requirements) for Windows. Note: * If there is any privilege issue, add `--user` to install NNI in the user directory. -* Currently NNI on Windows supports local, remote and pai mode. Anaconda or Miniconda is highly recommended to install [NNI on Windows](https://nni.readthedocs.io/en/latest/Tutorial/InstallationWin.html). -* If there is any error like `Segmentation fault`, please refer to [FAQ](https://nni.readthedocs.io/en/latest/Tutorial/FAQ.html). For FAQ on Windows, please refer to [NNI on Windows](https://nni.readthedocs.io/en/latest/Tutorial/InstallationWin.html#faq). +* Currently NNI on Windows supports local, remote and pai mode. Anaconda or Miniconda is highly recommended to install [NNI on Windows](https://nni.readthedocs.io/en/stable/Tutorial/InstallationWin.html). +* If there is any error like `Segmentation fault`, please refer to [FAQ](https://nni.readthedocs.io/en/stable/Tutorial/FAQ.html). For FAQ on Windows, please refer to [NNI on Windows](https://nni.readthedocs.io/en/stable/Tutorial/InstallationWin.html#faq). ### **Verify installation** @@ -297,7 +297,7 @@ You can use these commands to get more information about the experiment ----------------------------------------------------------------------- ``` -* Open the `Web UI url` in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below. [Here](https://nni.readthedocs.io/en/latest/Tutorial/WebUI.html) are more Web UI pages. +* Open the `Web UI url` in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below. [Here](https://nni.readthedocs.io/en/stable/Tutorial/WebUI.html) are more Web UI pages.
    drawing - Hyperparameter Tuning + Hyperparameter Tuning - Neural Architecture Search + Neural Architecture Search - Model Compression + Model Compression - Feature Engineering (Beta) + Feature Engineering (Beta) - Early Stop Algorithms + Early Stop Algorithms
    diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 9ffda93117..604c448e88 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -35,17 +35,17 @@ To control the sparsity of specific layers, the configuration can be writed as: .. code-block:: python [{ - 'sparsity': 0.8, - 'op_types': ['default'] - }, + 'sparsity': 0.8, + 'op_types': ['default'] + }, { - 'sparsity': 0.6, - 'op_names': ['op_name1', 'op_name2'] - }, + 'sparsity': 0.6, + 'op_names': ['op_name1', 'op_name2'] + }, { - 'exclude': True, - 'op_names': ['op_name3'] - }] + 'exclude': True, + 'op_names': ['op_name3'] + }] It means following the algorithm's default setting for compressed operations with sparsity 0.8, but for ``op_name1`` and ``op_name2`` use sparsity 0.6, and do not compress ``op_name3``. From f0154bedf18a13cb8f0603e35a9da62ffa1629f5 Mon Sep 17 00:00:00 2001 From: colorjam Date: Mon, 29 Mar 2021 02:31:41 +0000 Subject: [PATCH 05/16] Add a blank line --- docs/en_US/Compression/Tutorial.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 604c448e88..8064c862db 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -47,6 +47,7 @@ To control the sparsity of specific layers, the configuration can be writed as: 'op_names': ['op_name3'] }] + It means following the algorithm's default setting for compressed operations with sparsity 0.8, but for ``op_name1`` and ``op_name2`` use sparsity 0.6, and do not compress ``op_name3``. Quantization specific keys From e8f8e57d26db5db80dbc2f38ef00639570884e06 Mon Sep 17 00:00:00 2001 From: colorjam Date: Mon, 29 Mar 2021 02:52:42 +0000 Subject: [PATCH 06/16] Fix warning --- docs/en_US/Compression/Tutorial.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 8064c862db..604c448e88 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -47,7 +47,6 @@ To control the sparsity of specific layers, the configuration can be writed as: 'op_names': ['op_name3'] }] - It means following the algorithm's default setting for compressed operations with sparsity 0.8, but for ``op_name1`` and ``op_name2`` use sparsity 0.6, and do not compress ``op_name3``. Quantization specific keys From d6dcaa57875c534d70decd1b6ff1c4f41320d30f Mon Sep 17 00:00:00 2001 From: colorjam Date: Mon, 29 Mar 2021 03:32:27 +0000 Subject: [PATCH 07/16] Update according to comments --- README.md | 2 +- docs/en_US/Compression/Tutorial.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 488d21d0e7..a918b174b7 100644 --- a/README.md +++ b/README.md @@ -237,7 +237,7 @@ Windows python -m pip install --upgrade nni ``` -If you want to try stable code, please [install NNI](https://nni.readthedocs.io/en/stable/installation.html) from source code. +If you want to try latest code, please [install NNI](https://nni.readthedocs.io/en/stable/installation.html) from source code. For detail system requirements of NNI, please refer to [here](https://nni.readthedocs.io/en/stable/Tutorial/InstallationLinux.html#system-requirements) for Linux & macOS, and [here](https://nni.readthedocs.io/en/stable/Tutorial/InstallationWin.html#system-requirements) for Windows. diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 604c448e88..39fd35ef38 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -29,7 +29,7 @@ To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can b [{ 'sparsity': 0.6, 'op_types': ['Conv2d'] - }] + }] To control the sparsity of specific layers, the configuration can be writed as: .. code-block:: python From a5ee770cf0093010127f9c7aa216e95b82881613 Mon Sep 17 00:00:00 2001 From: colorjam Date: Mon, 29 Mar 2021 05:56:30 +0000 Subject: [PATCH 08/16] Fix minor bugs --- docs/en_US/Compression/Overview.rst | 2 +- docs/en_US/Compression/Tutorial.rst | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/en_US/Compression/Overview.rst b/docs/en_US/Compression/Overview.rst index d2738662c2..a833db7b05 100644 --- a/docs/en_US/Compression/Overview.rst +++ b/docs/en_US/Compression/Overview.rst @@ -14,7 +14,7 @@ NNI provides a model compression toolkit to help user compress and speed up thei * Provide friendly and easy-to-use compression utilities for users to dive into the compression process and results. * Concise interface for users to customize their own compression algorithms. -*Note that NNI compresses the model through a simulation process. To obtain a truly compact model, users should conduct model speedup. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future.* +.. Note:: NNI compresses the model through a simulation process. To obtain a truly compact model, users should conduct model speedup. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future. Supported Algorithms diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 39fd35ef38..30b514296c 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -24,6 +24,7 @@ There are different keys in a ``dict``. Some of them are common keys supported b Some other keys are often specific to a certain algorithm, users can refer to `pruning algorithms <./Pruner.rst>`__ and `quantization algorithms <./Quantizer.rst>`__ for the keys allowed by each algorithm. To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can be write as: + .. code-block:: python [{ @@ -32,6 +33,7 @@ To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can b }] To control the sparsity of specific layers, the configuration can be writed as: + .. code-block:: python [{ From ff02c77c8f897fe3095bd7a764059882f51065d7 Mon Sep 17 00:00:00 2001 From: colorjam Date: Mon, 29 Mar 2021 06:01:38 +0000 Subject: [PATCH 09/16] Unify config list --- docs/en_US/Compression/Tutorial.rst | 57 +++++++++++++++-------------- 1 file changed, 29 insertions(+), 28 deletions(-) diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 30b514296c..8eea26555b 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -39,12 +39,12 @@ To control the sparsity of specific layers, the configuration can be writed as: [{ 'sparsity': 0.8, 'op_types': ['default'] - }, - { + }, + { 'sparsity': 0.6, 'op_names': ['op_name1', 'op_name2'] - }, - { + }, + { 'exclude': True, 'op_names': ['op_name3'] }] @@ -69,10 +69,10 @@ bits length of quantization, key is the quantization type, value is the quantiza .. code-block:: bash { - quant_bits: { - 'weight': 8, - 'output': 4, - }, + quant_bits: { + 'weight': 8, + 'output': 4, + }, } when the value is int type, all quantization types share same bits length. eg. @@ -80,7 +80,7 @@ when the value is int type, all quantization types share same bits length. eg. .. code-block:: bash { - quant_bits: 8, # weight or output quantization are all 8 bits + quant_bits: 8, # weight or output quantization are all 8 bits } The following example shows a more complete ``config_list``\ , it uses ``op_names`` (or ``op_types``\ ) to specify the target layers along with the quantization bits for those layers. @@ -88,25 +88,26 @@ The following example shows a more complete ``config_list``\ , it uses ``op_name .. code-block:: bash config_list = [{ - 'quant_types': ['weight'], - 'quant_bits': 8, - 'op_names': ['conv1'] - }, { - 'quant_types': ['weight'], - 'quant_bits': 4, - 'quant_start_step': 0, - 'op_names': ['conv2'] - }, { - 'quant_types': ['weight'], - 'quant_bits': 3, - 'op_names': ['fc1'] - }, - { - 'quant_types': ['weight'], - 'quant_bits': 2, - 'op_names': ['fc2'] - } - ] + 'quant_types': ['weight'], + 'quant_bits': 8, + 'op_names': ['conv1'] + }, + { + 'quant_types': ['weight'], + 'quant_bits': 4, + 'quant_start_step': 0, + 'op_names': ['conv2'] + }, + { + 'quant_types': ['weight'], + 'quant_bits': 3, + 'op_names': ['fc1'] + }, + { + 'quant_types': ['weight'], + 'quant_bits': 2, + 'op_names': ['fc2'] + }] In this example, 'op_names' is the name of layer and four layers will be quantized to different quant_bits. From 740b8e2424f0613f4f98f072ea5e669c73533bc9 Mon Sep 17 00:00:00 2001 From: colorjam Date: Mon, 29 Mar 2021 06:06:03 +0000 Subject: [PATCH 10/16] Add conduct model speedup link --- docs/en_US/Compression/Overview.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/en_US/Compression/Overview.rst b/docs/en_US/Compression/Overview.rst index a833db7b05..54f1d20e99 100644 --- a/docs/en_US/Compression/Overview.rst +++ b/docs/en_US/Compression/Overview.rst @@ -14,7 +14,8 @@ NNI provides a model compression toolkit to help user compress and speed up thei * Provide friendly and easy-to-use compression utilities for users to dive into the compression process and results. * Concise interface for users to customize their own compression algorithms. -.. Note:: NNI compresses the model through a simulation process. To obtain a truly compact model, users should conduct model speedup. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future. +.. note:: +NNI compresses the model through a simulation process. To obtain a truly compact model, users should conduct `model speedup <./ModelSpeedup.rst>`__. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future. Supported Algorithms From c4a70dd9fd305413c34910ab6fbf9e3409a3625e Mon Sep 17 00:00:00 2001 From: colorjam Date: Mon, 29 Mar 2021 06:20:08 +0000 Subject: [PATCH 11/16] Fix minor bugs --- docs/en_US/Compression/Overview.rst | 2 +- docs/en_US/Compression/Tutorial.rst | 2 +- examples/model_compress/pruning/.gitignore | 2 ++ 3 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 examples/model_compress/pruning/.gitignore diff --git a/docs/en_US/Compression/Overview.rst b/docs/en_US/Compression/Overview.rst index 54f1d20e99..1a1509e2a7 100644 --- a/docs/en_US/Compression/Overview.rst +++ b/docs/en_US/Compression/Overview.rst @@ -15,7 +15,7 @@ NNI provides a model compression toolkit to help user compress and speed up thei * Concise interface for users to customize their own compression algorithms. .. note:: -NNI compresses the model through a simulation process. To obtain a truly compact model, users should conduct `model speedup <./ModelSpeedup.rst>`__. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future. + NNI compresses the model through a simulation process. To obtain a truly compact model, users should conduct `model speedup <./ModelSpeedup.rst>`__. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future. Supported Algorithms diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 8eea26555b..4b0aeae580 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -17,7 +17,7 @@ The ``dict``\ s in the ``list`` are applied one by one, that is, the configurati There are different keys in a ``dict``. Some of them are common keys supported by all the compression algorithms: -* **op_types**\ : This is to specify what types of operations to be compressed. 'default' means following the algorithm's default setting. +* **op_types**\ : This is to specify what types of operations to be compressed. 'default' means following the algorithm's default setting. All suported module types are defined in :githublink:`default_layers.py ` for pytorch. * **op_names**\ : This is to specify by name what operations to be compressed. If this field is omitted, operations will not be filtered by it. * **exclude**\ : Default is False. If this field is True, it means the operations with specified types and names will be excluded from the compression. diff --git a/examples/model_compress/pruning/.gitignore b/examples/model_compress/pruning/.gitignore new file mode 100644 index 0000000000..b4bd197eaa --- /dev/null +++ b/examples/model_compress/pruning/.gitignore @@ -0,0 +1,2 @@ +cifar-10-python.tar.gz +cifar-10-batches-py/ \ No newline at end of file From 4ec0f5f4953c4d5e5502e7cfc0a6fbd79be12ac2 Mon Sep 17 00:00:00 2001 From: colorjam Date: Tue, 30 Mar 2021 01:43:47 +0000 Subject: [PATCH 12/16] Fix dead link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a918b174b7..882088b569 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ ----------- [![MIT licensed](https://img.shields.io/badge/license-MIT-brightgreen.svg)](LICENSE) -[![Build Status](https://msrasrg.visualstudio.com/NNIOpenSource/_apis/build/status/full%20test%20-%20linux?branchName=master)](https://msrasrg.visualstudio.com/NNIOpenSource/_build/stable?definitionId=62&branchName=master) +[![Build Status](https://msrasrg.visualstudio.com/NNIOpenSource/_apis/build/status/full%20test%20-%20linux?branchName=master)](https://msrasrg.visualstudio.com/NNIOpenSource/_build/latest?definitionId=62&branchName=master) [![Issues](https://img.shields.io/github/issues-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen) [![Bugs](https://img.shields.io/github/issues/Microsoft/nni/bug.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen+label%3Abug) [![Pull Requests](https://img.shields.io/github/issues-pr-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/pulls?q=is%3Apr+is%3Aopen) From c23c63e317605fbd31b4c4a5262520f889e32318 Mon Sep 17 00:00:00 2001 From: colorjam Date: Tue, 30 Mar 2021 01:51:00 +0000 Subject: [PATCH 13/16] Fix typos. --- docs/en_US/Compression/Overview.rst | 4 ++-- docs/en_US/Compression/Tutorial.rst | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en_US/Compression/Overview.rst b/docs/en_US/Compression/Overview.rst index 1a1509e2a7..bd2b1d97da 100644 --- a/docs/en_US/Compression/Overview.rst +++ b/docs/en_US/Compression/Overview.rst @@ -5,7 +5,7 @@ Model Compression with NNI As larger neural networks with more layers and nodes are considered, reducing their storage and computational cost becomes critical, especially for some real-time applications. Model compression can be used to address this problem. -NNI provides a model compression toolkit to help user compress and speed up their model with state-of-the-art compression algorithms and strategies. There are several core features supported by NNI model compression: +NNI provides a model compression toolkit to help user compress and speed up their model with state-of-the-art compression algorithms and strategies. The algorithms simulate the process of pruning and quantization without truly compressing model and reducing latency. There are several core features supported by NNI model compression: * Support many popular pruning and quantization algorithms. @@ -15,7 +15,7 @@ NNI provides a model compression toolkit to help user compress and speed up thei * Concise interface for users to customize their own compression algorithms. .. note:: - NNI compresses the model through a simulation process. To obtain a truly compact model, users should conduct `model speedup <./ModelSpeedup.rst>`__. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future. + Since NNI compression algorithms are not meant to compress model while NNI speedup tool can truly compress model and reduce latency. To obtain a truly compact model, users should conduct `model speedup <./ModelSpeedup.rst>`__. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future. Supported Algorithms diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 4b0aeae580..8c60d337fc 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -23,7 +23,7 @@ There are different keys in a ``dict``. Some of them are common keys supported b Some other keys are often specific to a certain algorithm, users can refer to `pruning algorithms <./Pruner.rst>`__ and `quantization algorithms <./Quantizer.rst>`__ for the keys allowed by each algorithm. -To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can be write as: +To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can be written as: .. code-block:: python @@ -32,7 +32,7 @@ To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can b 'op_types': ['Conv2d'] }] -To control the sparsity of specific layers, the configuration can be writed as: +To control the sparsity of specific layers, the configuration can be written as: .. code-block:: python From c194bd47c7aba7e25a3b6b4e59b10a1ebe3e5a49 Mon Sep 17 00:00:00 2001 From: colorjam Date: Tue, 30 Mar 2021 01:54:36 +0000 Subject: [PATCH 14/16] Fix typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 882088b569..e08512cefe 100644 --- a/README.md +++ b/README.md @@ -297,7 +297,7 @@ You can use these commands to get more information about the experiment ----------------------------------------------------------------------- ``` -* Open the `Web UI url` in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below. [Here](https://nni.readthedocs.io/en/stable/Tutorial/WebUI.html) are more Web UI pages. +* Open the `Web UI url` in your browser, you can view detailed information of the experiment and all the submitted trial jobs as shown below. [Here](https://nni.readthedocs.io/en/stable/Tutorial/WebUI.html) are more Web UI pages.
    drawing
    From 6c1542c96f258c9564622dd698093bd91ea91de0 Mon Sep 17 00:00:00 2001 From: colorjam Date: Tue, 30 Mar 2021 02:42:40 +0000 Subject: [PATCH 15/16] Fix dead links --- .../CommunitySharings/ModelCompressionComparison.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en_US/CommunitySharings/ModelCompressionComparison.rst b/docs/en_US/CommunitySharings/ModelCompressionComparison.rst index 758acabeb3..de22b86390 100644 --- a/docs/en_US/CommunitySharings/ModelCompressionComparison.rst +++ b/docs/en_US/CommunitySharings/ModelCompressionComparison.rst @@ -13,7 +13,7 @@ The experiments are performed with the following pruners/datasets/models: * - Models: :githublink:`VGG16, ResNet18, ResNet50 ` + Models: :githublink:`VGG16, ResNet18, ResNet50 ` * Datasets: CIFAR-10 @@ -96,14 +96,14 @@ Implementation Details This avoids potential issues of counting them of masked models. * - The experiment code can be found :githublink:`here `. + The experiment code can be found :githublink:`here `. Experiment Result Rendering ^^^^^^^^^^^^^^^^^^^^^^^^^^^ * - If you follow the practice in the :githublink:`example `\ , for every single pruning experiment, the experiment result will be saved in JSON format as follows: + If you follow the practice in the :githublink:`example `\ , for every single pruning experiment, the experiment result will be saved in JSON format as follows: .. code-block:: json @@ -114,8 +114,8 @@ Experiment Result Rendering } * - The experiment results are saved :githublink:`here `. - You can refer to :githublink:`analyze ` to plot new performance comparison figures. + The experiment results are saved :githublink:`here `. + You can refer to :githublink:`analyze ` to plot new performance comparison figures. Contribution ------------ From 1c76fe090f68403ec088fc2896f18bd6e5e1d154 Mon Sep 17 00:00:00 2001 From: colorjam Date: Tue, 30 Mar 2021 06:57:00 +0000 Subject: [PATCH 16/16] Update --- docs/en_US/Compression/Overview.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en_US/Compression/Overview.rst b/docs/en_US/Compression/Overview.rst index bd2b1d97da..4cb29533fe 100644 --- a/docs/en_US/Compression/Overview.rst +++ b/docs/en_US/Compression/Overview.rst @@ -5,7 +5,7 @@ Model Compression with NNI As larger neural networks with more layers and nodes are considered, reducing their storage and computational cost becomes critical, especially for some real-time applications. Model compression can be used to address this problem. -NNI provides a model compression toolkit to help user compress and speed up their model with state-of-the-art compression algorithms and strategies. The algorithms simulate the process of pruning and quantization without truly compressing model and reducing latency. There are several core features supported by NNI model compression: +NNI provides a model compression toolkit to help user compress and speed up their model with state-of-the-art compression algorithms and strategies. There are several core features supported by NNI model compression: * Support many popular pruning and quantization algorithms.
    drawing