Update docs for aml and nasbenchmark, re-submit from #2647

docs/en_US/NAS/Benchmarks.md

@@ -7,40 +7,25 @@
     Example Usages <BenchmarksExample>
 ```
 
+## Introduction
+To imporve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as [NAS-Bench-101](https://arxiv.org/abs/1902.09635), [NAS-Bench-201](https://arxiv.org/abs/2001.00326), [NDS](https://arxiv.org/abs/1905.13214), etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
+
 ## Prerequisites
 
 * Please prepare a folder to household all the benchmark databases. By default, it can be found at `${HOME}/.nni/nasbenchmark`. You can place it anywhere you like, and specify it in `NASBENCHMARK_DIR` before importing NNI.
 * Please install `peewee` via `pip install peewee`, which NNI uses to connect to database.
 
 ## Data Preparation
 
-To avoid storage and legal issues, we do not provide any prepared databases. We strongly recommend users to use docker to run the generation scripts, to ease the burden of installing multiple dependencies. Please follow the following steps.
-
-**Step 1.** Clone NNI repo. Replace `${NNI_VERSION}` with a released version name or branch name, e.g., `v1.6`.
-
-```bash
-git clone -b ${NNI_VERSION} https://github.com/microsoft/nni
-```
-
-**Step 2.** Run docker.
-
-For NAS-Bench-101,
+To avoid storage and legality issues, we do not provide any prepared databases. Please follow the following steps.
 
-```bash
-docker run -v ${HOME}/.nni/nasbenchmark:/outputs -v /path/to/your/nni:/nni tensorflow/tensorflow:1.15.2-py3 /bin/bash /nni/examples/nas/benchmarks/nasbench101.sh
+1. Clone NNI to your machine and enter `examples/nas/benchmarks` directory.
 ```
-
-For NAS-Bench-201,
-
-```bash
-docker run -v ${HOME}/.nni/nasbenchmark:/outputs -v /path/to/your/nni:/nni ufoym/deepo:pytorch-cpu /bin/bash /nni/examples/nas/benchmarks/nasbench201.sh
-```
-
-For NDS,
-
-```bash
-docker run -v ${HOME}/.nni/nasbenchmark:/outputs -v /path/to/your/nni:/nni python:3.7 /bin/bash /nni/examples/nas/benchmarks/nds.sh
+git clone https://github.com/microsoft/nni.git
+cd nni/examples/nas/benchmarks
 ```
+2. Install dependencies via `pip3 install -r xxx.requirements.txt`. `xxx` can be `nasbench101`, `nasbench201` or `nds`.
+3. Generate the database via `./xxx.sh`. The directory that stores the benchmark file can be configured with `NASBENCHMARK_DIR` environment variable, which defaults to `~/.nni/nasbenchmark`. Note that the NAS-Bench-201 dataset will be downloaded from a google drive.
 
 Please make sure there is at least 10GB free disk space and note that the conversion process can take up to hours to complete.
 

docs/en_US/NAS/BenchmarksExample.ipynb

@@ -1,27 +1,4 @@
 {
- "nbformat": 4,
- "nbformat_minor": 2,
- "metadata": {
-  "language_info": {
-   "name": "python",
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "version": "3.6.10-final"
-  },
-  "orig_nbformat": 2,
-  "file_extension": ".py",
-  "mimetype": "text/x-python",
-  "name": "python",
-  "npconvert_exporter": "python",
-  "pygments_lexer": "ipython3",
-  "version": 3,
-  "kernelspec": {
-   "name": "python361064bitnnilatestcondabff8d66a619a4d26af34fe0fe687c7b0",
-   "display_name": "Python 3.6.10 64-bit ('nnilatest': conda)"
-  }
- },
  "cells": [
   {
    "cell_type": "markdown",
@@ -53,6 +30,14 @@
     "## NAS-Bench-101"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use the following architecture as an example:<br>\n",
+    "![nas-101](../../img/nas-bench-101-example.png)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 2,
@@ -82,13 +67,28 @@
     "    pprint.pprint(t)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "An architecture of NAS-Bench-101 could be trained more than once. Each element of the returned generator is a dict which contains one of the training results of this trial config (architecture + hyper-parameters) including train/valid/test accuracy, training time, number of epochs, etc. The results of NAS-Bench-201 and NDS follow similar formats."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## NAS-Bench-201"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use the following architecture as an example:<br>\n",
+    "![nas-201](../../img/nas-bench-201-example.png)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 3,
@@ -120,6 +120,16 @@
     "## NDS"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use the following architecture as an example:<br>\n",
+    "![nds](../../img/nas-bench-nds-example.png)\n",
+    "\n",
+    "Here, `bot_muls`, `ds`, `num_gs`, `ss` and `ws` stand for \"bottleneck multipliers\", \"depths\", \"number of groups\", \"strides\" and \"widths\" respectively."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 4,
@@ -273,5 +283,28 @@
     "print('Elapsed time: ', time.time() - ti, 'seconds')"
    ]
   }
- ]
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python",
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "version": "3.6.10-final"
+  },
+  "orig_nbformat": 2,
+  "file_extension": ".py",
+  "mimetype": "text/x-python",
+  "name": "python",
+  "npconvert_exporter": "python",
+  "pygments_lexer": "ipython3",
+  "version": 3,
+  "kernelspec": {
+   "name": "python361064bitnnilatestcondabff8d66a619a4d26af34fe0fe687c7b0",
+   "display_name": "Python 3.6.10 64-bit ('nnilatest': conda)"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
 }

docs/en_US/TrainingService/AMLMode.md

@@ -5,12 +5,22 @@ NNI supports running an experiment on [AML](https://azure.microsoft.com/en-us/se
 ## Setup environment
 Step 1. Install NNI, follow the install guide [here](../Tutorial/QuickStart.md).   
 
-Step 2. Create AML account, follow the document [here](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-manage-workspace-cli).
+Step 2. Create an Azure account/subscription using this [link](https://azure.microsoft.com/en-us/free/services/machine-learning/). If you already have an Azure account/subscription, skip this step.
 
-Step 3. Get your account information.
-![](../../img/aml_account.png)
+Step 3. Install the Azure CLI on your machine, follow the install guide [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest).
 
-Step4. Install AML package environment.
+Step 4. Authenticate to your Azure subscription from the CLI. To authenticate interactively, open a command line or terminal and use the following command:
+```
+az login
+```
+
+Step 5. Log into your Azure account with a web browser and create a Machine Learning resource. You will need to choose a resource group and specific a workspace name. Then download `config.json` which will be used later.
+![](../../img/aml_workspace.png)
+
+Step 6. Create an AML cluster as the computeTarget.
+![](../../img/aml_cluster.png)
+
+Step 7. Open a command line and install AML package environment.
 ```
 python3 -m pip install azureml --user
 python3 -m pip install azureml-sdk --user
@@ -52,9 +62,9 @@ Note: You should set `trainingServicePlatform: aml` in NNI config YAML file if y
 
 Compared with [LocalMode](LocalMode.md) trial configuration in aml mode have these additional keys:
 * computeTarget
-    * required key. The computer cluster name you want to use in your AML workspace.
+    * required key. The compute cluster name you want to use in your AML workspace. See Step 6.
 * image
-    * required key. The docker image name used in job.
+    * required key. The docker image name used in job. The image `msranni/nni` of this example only support GPU computeTargets.
 
 amlConfig:
 * subscriptionId
@@ -63,4 +73,15 @@ amlConfig:
     * the resourceGroup of your account
 * workspaceName
     * the workspaceName of your account
-
+
+The required information of amlConfig could be found in the downloaded `config.json` in Step 5.
+
+Run the following commands to start the example experiment:
+```
+git clone https://github.com/microsoft/nni.git
+cd nni/examples/trials/mnist-tfv1
+
+# modify config_aml.yml ...
+
+nnictl create --config config_aml.yml
+```