Merge pull request #116 from Microsoft/master

merge master

README.md

@@ -14,9 +14,71 @@
 NNI (Neural Network Intelligence) is a toolkit to help users run automated machine learning (AutoML) experiments. 
 The tool dispatches and runs trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in different environments like local machine, remote servers and cloud.
 
+### **NNI [v0.5](https://github.com/Microsoft/nni/releases) has been released!**
 <p align="center">
-<img src="./docs/img/nni_arch_overview.png" alt="drawing"/>
+  <a href=#><img src="https://rawgit.com/QuanluZhang/nni/update-doc11/overview.svg" /></a>
 </p>
+<table>
+  <tbody>
+    <tr align="center">
+      <td>
+        <b>User Code + SDK( import nni )</b>
+        <img src="https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png"/>
+      </td>
+      <td>
+        <b>Tunning Algorithm Extensions</b>
+        <img src="https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png"/>
+      </td>
+      <td>
+        <b>Training Service Extensions</b>
+        <img src="https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png"/>
+      </td>
+    </tr>
+    <tr/>
+    <tr valign="top">
+      <td>
+      <ul>
+        <li>CNTK</li>
+        <li>Tensorflow</li>
+        <li>PyTorch</li>
+        <li>Keras</li>
+        <li>...</li>
+        </ul>
+        (Python based frameworks)
+      </td>
+      <td>
+        <a href="docs/HowToChooseTuner.md">Tuner</a>
+        <ul>
+          <li><a href="docs/HowToChooseTuner.md#TPE">TPE</a></li>
+          <li><a href="docs/HowToChooseTuner.md#Random">Random Search</a></li>
+          <li><a href="docs/HowToChooseTuner.md#Anneal">Anneal</a></li>
+          <li><a href="docs/HowToChooseTuner.md#Evolution">Naive Evolution</a></li>
+          <li><a href="docs/HowToChooseTuner.md#SMAC">SMAC</a></li>
+          <li><a href="docs/HowToChooseTuner.md#Batch">Batch</a></li>
+          <li><a href="docs/HowToChooseTuner.md#Grid">Grid Search</a></li>
+          <li><a href="docs/HowToChooseTuner.md#Hyperband">Hyperband</a></li>
+          <li><a href="docs/HowToChooseTuner.md#NetworkMorphism">Network Morphism</a></li>
+          <li><a href="examples/tuners/enas_nni/README.md">ENAS</a></li>
+          <li><a href="docs/HowToChooseTuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
+        </ul> 
+          <a href="docs/HowToChooseTuner.md#assessor">Assessor</a> 
+        <ul>
+          <li><a href="docs/HowToChooseTuner.md#Medianstop">Median Stop</a></li>
+          <li><a href="docs/HowToChooseTuner.md#Curvefitting">Curve Fitting</a></li>
+        </ul>
+      </td>
+      <td>
+      <ul>
+        <li><a href="docs/tutorial_1_CR_exp_local_api.md">Local Machine</a></li>
+        <li><a href="docs/tutorial_2_RemoteMachineMode.md">Remote Servers</a></li>
+        <li><a href="docs/PAIMode.md">OpenPAI</a></li>
+        <li><a href="docs/KubeflowMode.md">Kubeflow</a></li>
+        <li><a href="docs/KubeflowMode.md">FrameworkController on K8S (AKS etc.)</a></li>
+      </ul>
+      </td>
+    </tr>
+  </tbody>
+</table>
 
 ## **Who should consider using NNI**
 * Those who want to try different AutoML algorithms in their training code (model) at their local machine.
@@ -35,12 +97,14 @@ We encourage researchers and students leverage these projects to accelerate the
 
 **Install through pip** 	
 * We support Linux and MacOS in current stage, Ubuntu 16.04 or higher, along with MacOS 10.14.1 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.	
-```bash	
+
+```bash
     python3 -m pip install --upgrade nni
 ```
-* Note:
-  * If you are in docker container (as root), please remove `--user` from the installation command.
-  * If there is any error like `Segmentation fault`, please refer to [FAQ](docs/FAQ.md)
+Note:
+
+* `--user` can be added if you want to install NNI in your home directory, which does not require any special privileges.
+* If there is any error like `Segmentation fault`, please refer to [FAQ](docs/FAQ.md)
 
 **Install through source code**
 * We support Linux (Ubuntu 16.04 or higher), MacOS (10.14.1) in our current stage. 

azure-pipelines.yml

@@ -1,35 +1,45 @@
 trigger:
+- dev-it
 - master
 - dev-remote-ci
 
 jobs:
-
 - job: 'Ubuntu_16_04'
-  pool:
-    vmImage: 'Ubuntu 16.04'
-  strategy:
-    matrix:
-      Python36:
-        PYTHON_VERSION: '3.6'
+  pool: 'NNI CI GPU'
 
   steps:
-  - script: python3 -m pip install --upgrade pip setuptools
+  - script: python3 -m pip install --upgrade pip setuptools --user
     displayName: 'Install python tools'
   - script: |
       source install.sh
     displayName: 'Install nni toolkit via source code'
+  - script: |
+      python3 -m pip install scikit-learn==0.20.0 --user
+      python3 -m pip install torch==0.4.1 --user
+      python3 -m pip install torchvision==0.2.1 --user
+      python3 -m pip install keras==2.1.6 --user
+      python3 -m pip install tensorflow-gpu==1.10.0 --user
+    displayName: 'Install dependencies for integration tests'
   - script: |
       cd test
       source unittest.sh
     displayName: 'Unit test'
   - script: |
       cd test
       PATH=$HOME/.local/bin:$PATH python3 naive_test.py
-    displayName: 'Integration tests'
+    displayName: 'Naive test'
+  - script: |
+      cd test
+      PATH=$HOME/.local/bin:$PATH python3 tuner_test.py
+    displayName: 'Built-in tuners / assessors tests'
+  - script: |
+      cd test
+      PATH=$HOME/.local/bin:$PATH python3 config_test.py --ts local
+    displayName: 'Examples and advanced features tests on local machine'
   - script: |
       cd test
-      PATH=$HOME/.local/bin:$PATH python3 sdk_test.py
-    displayName: 'Built-in dispatcher tests'
+      PATH=$HOME/.local/bin:$PATH python3 metrics_test.py
+    displayName: 'Trial job metrics test'
 
 - job: 'macOS_10_13'
   pool:
@@ -52,8 +62,8 @@ jobs:
   - script: |
       cd test
       PATH=$HOME/Library/Python/3.7/bin:$PATH python3 naive_test.py
-    displayName: 'Integration tests'
+    displayName: 'Naive test'
   - script: |
       cd test
-      PATH=$HOME/Library/Python/3.7/bin:$PATH python3 sdk_test.py
-    displayName: 'Built-in dispatcher tests'
+      PATH=$HOME/Library/Python/3.7/bin:$PATH python3 tuner_test.py
+    displayName: 'Built-in tuners / assessors tests'

docs/GetStarted.md

@@ -110,3 +110,4 @@ The experiment has been running now, NNI provides WebUI for you to view experime
 * [How to run an experiment on local (with multiple GPUs)?](tutorial_1_CR_exp_local_api.md)
 * [How to run an experiment on multiple machines?](tutorial_2_RemoteMachineMode.md)
 * [How to run an experiment on OpenPAI?](PAIMode.md)
+* [How to create a multi-phase experiment](multiPhase.md)

docs/HowToChooseTuner.md

@@ -244,7 +244,7 @@ _Usage_:
       optimize_mode: maximize
 ```
 
-
+<a name="assessor"></a>
 # How to use Assessor that NNI supports?
 
 For now, NNI has supported the following assessor algorithms.

docs/NNICTLDOC.md

@@ -1,6 +1,5 @@
-nnictl
+# nnictl
 
-===
 
 ## Introduction
 

docs/multiPhase.md

@@ -0,0 +1,43 @@
+## Create multi-phase experiment
+
+Typically each trial job gets single set of configuration (e.g. hyper parameters) from tuner and do some kind of experiment, let's say train a model with that hyper parameter and reports its result to tuner. Sometimes you may want to train multiple models within one trial job to share information between models or saving system resource by creating less trial jobs, for example:
+1. Train multiple models sequentially in one trial job, so that later models can leverage the weights or other information of prior models and may use different hyper parameters.
+2. Train large amount of models on limited system resource, combine multiple models together to save system resource to create large amount of trial jobs.
+3. Any other scenario that you would like to train multiple models with different hyper parameters in one trial job, be aware that if you allocate multiple GPUs to a trial job and you train multiple models concurrently within on trial job, you need to allocate GPU resource properly by your trial code.
+
+In above cases, you can leverage NNI multi-phase experiment to train multiple models with different hyper parameters within each trial job.
+
+Multi-phase experiments refer to experiments whose trial jobs request multiple hyper parameters from tuner and report multiple final results to NNI.
+
+To use multi-phase experiment, please follow below steps:
+
+1.  Implement nni.multi_phase.MultiPhaseTuner. For example, this [ENAS tuner](https://github.com/countif/enas_nni/blob/master/nni/examples/tuners/enas/nni_controller_ptb.py) is a multi-phase Tuner which implements nni.multi_phase.MultiPhaseTuner. While implementing your MultiPhaseTuner, you may want to use the trial_job_id parameter of generate_parameters method to generate hyper parameters for each trial job.
+
+2. Set ```multiPhase``` field to ```true```, and configure your tuner implemented in step 1 as customized tuner in configuration file, for example:
+
+```yml
+...
+multiPhase: true
+tuner:
+  codeDir: tuners/enas
+  classFileName: nni_controller_ptb.py
+  className: ENASTuner
+  classArgs:
+    say_hello: "hello"
+...
+```
+
+
+3. Invoke nni.get_next_parameter() API for multiple times as needed in a trial, for example:
+
+```python
+for i in range(5):
+    # get parameter from tuner
+    tuner_param = nni.get_next_parameter()
+
+    # consume the params
+    # ...
+    # report final result somewhere for the parameter retrieved above
+    nni.report_final_result()
+    # ...
+```

examples/trials/cifar10_pytorch/main.py

@@ -1,6 +1,6 @@
 '''Train CIFAR10 with PyTorch.'''
 from __future__ import print_function
-
+import argparse
 import torch
 import torch.nn as nn
 import torch.optim as optim
@@ -174,6 +174,10 @@ def test(epoch):
 
 
 if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--epochs", type=int, default=200)
+    args, _ = parser.parse_known_args()
+
     try:
         RCV_CONFIG = nni.get_next_parameter()
         #RCV_CONFIG = {'lr': 0.1, 'optimizer': 'Adam', 'model':'senet18'}
@@ -182,7 +186,7 @@ def test(epoch):
         prepare(RCV_CONFIG)
         acc = 0.0
         best_acc = 0.0
-        for epoch in range(start_epoch, start_epoch+200):
+        for epoch in range(start_epoch, start_epoch+args.epochs):
             train(epoch)
             acc, best_acc = test(epoch)
             nni.report_intermediate_result(acc)

examples/trials/mnist-annotation/mnist.py

@@ -1,5 +1,6 @@
 """A deep MNIST classifier using convolutional layers."""
 
+import argparse
 import logging
 import math
 import tempfile
@@ -180,7 +181,7 @@ def main(params):
     test_acc = 0.0
     with tf.Session() as sess:
         sess.run(tf.global_variables_initializer())
-        """@nni.variable(nni.choice(1, 4, 8, 16, 32), name=batch_size)"""
+        """@nni.variable(nni.choice(16, 32), name=batch_size)"""
         batch_size = params['batch_size']
         for i in range(params['batch_num']):
             batch = mnist.train.next_batch(batch_size)
@@ -210,29 +211,27 @@ def main(params):
         logger.debug('Final result is %g', test_acc)
         logger.debug('Send final result done.')
 
-
-def generate_default_params():
-    '''
-    Generate default parameters for mnist network.
-    '''
-    params = {
-        'data_dir': '/tmp/tensorflow/mnist/input_data',
-        'dropout_rate': 0.5,
-        'channel_1_num': 32,
-        'channel_2_num': 64,
-        'conv_size': 5,
-        'pool_size': 2,
-        'hidden_size': 1024,
-        'learning_rate': 1e-4,
-        'batch_num': 2000,
-        'batch_size': 32}
-    return params
-
+def get_params():
+    ''' Get parameters from command line '''
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--data_dir", type=str, default='/tmp/tensorflow/mnist/input_data', help="data directory")
+    parser.add_argument("--dropout_rate", type=float, default=0.5, help="dropout rate")
+    parser.add_argument("--channel_1_num", type=int, default=32)
+    parser.add_argument("--channel_2_num", type=int, default=64)
+    parser.add_argument("--conv_size", type=int, default=5)
+    parser.add_argument("--pool_size", type=int, default=2)
+    parser.add_argument("--hidden_size", type=int, default=1024)
+    parser.add_argument("--learning_rate", type=float, default=1e-4)
+    parser.add_argument("--batch_num", type=int, default=2000)
+    parser.add_argument("--batch_size", type=int, default=32)
+
+    args, _ = parser.parse_known_args()
+    return args
 
 if __name__ == '__main__':
     '''@nni.get_next_parameter()'''
     try:
-        main(generate_default_params())
+        main(vars(get_params()))
     except Exception as exception:
         logger.exception(exception)
         raise

examples/trials/mnist/mnist.py

@@ -1,5 +1,6 @@
 """A deep MNIST classifier using convolutional layers."""
 
+import argparse
 import logging
 import math
 import tempfile
@@ -198,33 +199,30 @@ def main(params):
         logger.debug('Final result is %g', test_acc)
         logger.debug('Send final result done.')
 
-
-def generate_default_params():
-    '''
-    Generate default parameters for mnist network.
-    '''
-    params = {
-        'data_dir': '/tmp/tensorflow/mnist/input_data',
-        'dropout_rate': 0.5,
-        'channel_1_num': 32,
-        'channel_2_num': 64,
-        'conv_size': 5,
-        'pool_size': 2,
-        'hidden_size': 1024,
-        'learning_rate': 1e-4,
-        'batch_num': 2000,
-        'batch_size': 32}
-    return params
-
+def get_params():
+    ''' Get parameters from command line '''
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--data_dir", type=str, default='/tmp/tensorflow/mnist/input_data', help="data directory")
+    parser.add_argument("--dropout_rate", type=float, default=0.5, help="dropout rate")
+    parser.add_argument("--channel_1_num", type=int, default=32)
+    parser.add_argument("--channel_2_num", type=int, default=64)
+    parser.add_argument("--conv_size", type=int, default=5)
+    parser.add_argument("--pool_size", type=int, default=2)
+    parser.add_argument("--hidden_size", type=int, default=1024)
+    parser.add_argument("--learning_rate", type=float, default=1e-4)
+    parser.add_argument("--batch_num", type=int, default=2000)
+    parser.add_argument("--batch_size", type=int, default=32)
+
+    args, _ = parser.parse_known_args()
+    return args
 
 if __name__ == '__main__':
     try:
         # get parameters form tuner
-        RCV_PARAMS = nni.get_next_parameter()
-        logger.debug(RCV_PARAMS)
-        # run
-        params = generate_default_params()
-        params.update(RCV_PARAMS)
+        tuner_params = nni.get_next_parameter()
+        logger.debug(tuner_params)
+        params = vars(get_params())
+        params.update(tuner_params)
         main(params)
     except Exception as exception:
         logger.exception(exception)