Exp stop refactor (#161)

* Update RemoteMachineMode.md (#63) * Remove unused classes for SQuAD QA example. * Remove more unused functions for SQuAD QA example. * Fix default dataset config. * Add Makefile README (#64) * update document (#92) * Edit readme.md * updated a word * Update GetStarted.md * Update GetStarted.md * refact readme, getstarted and write your trial md. * Update README.md * Update WriteYourTrial.md * Update WriteYourTrial.md * Update WriteYourTrial.md * Update WriteYourTrial.md * Fix nnictl bugs and add new feature (#75) * fix nnictl bug * fix nnictl create bug * add experiment status logic * add more information for nnictl * fix Evolution Tuner bug * refactor code * fix code in updater.py * fix nnictl --help * fix classArgs bug * update check response.status_code logic * remove Buffer warning (#100) * update readme in ga_squad * update readme * fix typo * Update README.md * Update README.md * Update README.md * Add support for debugging mode * fix setup.py (#115) * Add DAG model configuration format for SQuAD example. * Explain config format for SQuAD QA model. * Add more detailed introduction about the evolution algorithm. * Fix install.sh add add trial log path (#109) * fix nnictl bug * fix nnictl create bug * add experiment status logic * add more information for nnictl * fix Evolution Tuner bug * refactor code * fix code in updater.py * fix nnictl --help * fix classArgs bug * update check response.status_code logic * show trial log path * update document * fix install.sh * set default vallue for maxTrialNum and maxExecDuration * fix nnictl * Dev smac (#116) * support package install (#91) * fix nnictl bug * support package install * update * update package install logic * Fix package install issue (#95) * fix nnictl bug * fix pakcage install * support SMAC as a tuner on nni (#81) * update doc * update doc * update doc * update hyperopt installation * update doc * update doc * update description in setup.py * update setup.py * modify encoding * encoding * add encoding * remove pymc3 * update doc * update builtin tuner spec * support smac in sdk, fix logging issue * support smac tuner * add optimize_mode * update config in nnictl * add __init__.py * update smac * update import path * update setup.py: remove entry_point * update rest server validation * fix bug in nnictl launcher * support classArgs: optimize_mode * quick fix bug * test travis * add dependency * add dependency * add dependency * add dependency * create smac python package * fix trivial points * optimize import of tuners, modify nnictl accordingly * fix bug: incorrect algorithm_name * trivial refactor * for debug * support virtual * update doc of SMAC * update smac requirements * update requirements * change debug mode * update doc * update doc * refactor based on comments * fix comments * modify example config path to relative path and increase maxTrialNum (#94) * modify example config path to relative path and increase maxTrialNum * add document * support conda (#90) (#110) * support install from venv and travis CI * support install from venv and travis CI * support install from venv and travis CI * support conda * support conda * modify example config path to relative path and increase maxTrialNum * undo messy commit * undo messy commit * Support pip install as root (#77) * Typo on #58 (#122) * PAI Training Service implementation (#128) * PAI Training service implementation **1. Implement PAITrainingService **2. Add trial-keeper python module, and modify setup.py to install the module **3. Add PAItrainingService rest server to collect metrics from PAI container. * fix datastore for multiple final result (#129) * Update NNI v0.2 release notes (#132) Update NNI v0.2 release notes * Update setup.py Makefile and documents (#130) * update makefile and setup.py * update makefile and setup.py * update document * update document * Update Makefile no travis * update doc * update doc * fix convert from ss to pcs (#133) * Fix bugs about webui (#131) * Fix webui bugs * Fix tslint * webui logpath and document (#135) * Add webui document and logpath as a href * fix tslint * fix comments by Chengmin * Pai training service bug fix and enhancement (#136) * Add NNI installation scripts * Update pai script, update NNI_out_dir * Update NNI dir in nni sdk local.py * Create .nni folder in nni sdk local.py * Add check before creating .nni folder * Fix typo for PAI_INSTALL_NNI_SHELL_FORMAT * Improve annotation (#138) * Improve annotation * Minor bugfix * Selectively install through pip (#139) Selectively install through pip * update setup.py * fix paiTrainingService bugs (#137) * fix nnictl bug * add hdfs host validation * fix bugs * fix dockerfile * fix install.sh * update install.sh * fix dockerfile * Set timeout for HDFSUtility exists function * remove unused TODO * fix sdk * add optional for outputDir and dataDir * refactor dockerfile.base * Remove unused import in hdfsclientUtility * Add documentation for NNI PAI mode experiment (#141) * Add documentation for NNI PAI mode * Fix typo based on PR comments * Exit with subprocess return code of trial keeper * Remove additional exit code * Fix typo based on PR comments * update doc for smac tuner (#140) * Revert "Selectively install through pip (#139)" due to potential pip install issue (#142) * Revert "Selectively install through pip (#139)" This reverts commit 1d17483. * Add exit code of subprocess for trial_keeper * Update README, add link to PAImode doc * Merge branch V0.2 to Master (#143) * webui logpath and document (#135) * Add webui document and logpath as a href * fix tslint * fix comments by Chengmin * Pai training service bug fix and enhancement (#136) * Add NNI installation scripts * Update pai script, update NNI_out_dir * Update NNI dir in nni sdk local.py * Create .nni folder in nni sdk local.py * Add check before creating .nni folder * Fix typo for PAI_INSTALL_NNI_SHELL_FORMAT * Improve annotation (#138) * Improve annotation * Minor bugfix * Selectively install through pip (#139) Selectively install through pip * update setup.py * fix paiTrainingService bugs (#137) * fix nnictl bug * add hdfs host validation * fix bugs * fix dockerfile * fix install.sh * update install.sh * fix dockerfile * Set timeout for HDFSUtility exists function * remove unused TODO * fix sdk * add optional for outputDir and dataDir * refactor dockerfile.base * Remove unused import in hdfsclientUtility * Add documentation for NNI PAI mode experiment (#141) * Add documentation for NNI PAI mode * Fix typo based on PR comments * Exit with subprocess return code of trial keeper * Remove additional exit code * Fix typo based on PR comments * update doc for smac tuner (#140) * Revert "Selectively install through pip (#139)" due to potential pip install issue (#142) * Revert "Selectively install through pip (#139)" This reverts commit 1d17483. * Add exit code of subprocess for trial_keeper * Update README, add link to PAImode doc * fix bug (#147) * Refactor nnictl and add config_pai.yml (#144) * fix nnictl bug * add hdfs host validation * fix bugs * fix dockerfile * fix install.sh * update install.sh * fix dockerfile * Set timeout for HDFSUtility exists function * remove unused TODO * fix sdk * add optional for outputDir and dataDir * refactor dockerfile.base * Remove unused import in hdfsclientUtility * add config_pai.yml * refactor nnictl create logic and add colorful print * fix nnictl stop logic * add annotation for config_pai.yml * add document for start experiment * fix config.yml * fix document * Fix trial keeper wrongly exit issue (#152) * Fix trial keeper bug, use actual exitcode to exit rather than 1 * Fix bug of table sort (#145) * Update doc for PAIMode and v0.2 release notes (#153) * Update v0.2 documentation regards to release note and PAI training service * Update document to describe NNI docker image * fix antd (#159) * refactor experiment stopping logic * support change concurrency * remove trialJobs.ts * trivial changes * fix bugs * fix bug * support updating maxTrialNum
microsoft · Oct 7, 2018 · fc5dce8 · fc5dce8
1 parent 45650c4
commit fc5dce8
Showing 133 changed files with 6,022 additions and 1,154 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -4,9 +4,9 @@ language: python
 python:
   - "3.6"
 before_install:
-  - wget https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.xz
-  - tar xf node-v10.9.0-linux-x64.tar.xz
-  - sudo mv node-v10.9.0-linux-x64 /usr/local/node
+  - wget https://nodejs.org/dist/v10.10.0/node-v10.10.0-linux-x64.tar.xz
+  - tar xf node-v10.10.0-linux-x64.tar.xz
+  - sudo mv node-v10.10.0-linux-x64 /usr/local/node
   - export PATH=/usr/local/node/bin:$PATH
   - sudo sh -c 'PATH=/usr/local/node/bin:$PATH yarn global add serve'
 install:

diff --git a/Makefile b/Makefile
@@ -1,6 +1,14 @@
 # Setting variables
 
 SHELL := /bin/bash
+PIP_INSTALL := python3 -m pip install
+PIP_UNINSTALL := python3 -m pip uninstall
+
+## Colorful output
+_INFO := $(shell echo -e '\e[1;36m')
+_WARNING := $(shell echo -e '\e[1;33m')
+_END := $(shell echo -e '\e[0m')
+
 
 ## Install directories
 ifeq ($(shell id -u), 0)  # is root
@@ -20,7 +28,7 @@ else  # is normal user
 endif
 
 ## Dependency information
-NODE_VERSION ?= v10.9.0
+NODE_VERSION ?= v10.10.0
 NODE_TARBALL ?= node-$(NODE_VERSION)-linux-x64.tar.xz
 NODE_PATH ?= $(INSTALL_PREFIX)/nni/node
 
@@ -35,41 +43,37 @@ SERVE_PATH ?= $(INSTALL_PREFIX)/nni/serve
 
 ## Check if dependencies have been installed globally
 ifeq (, $(shell command -v node 2>/dev/null))
-    $(info Node.js not found)
+    $(info $(_INFO) Node.js not found $(_END))
     _MISS_DEPS := 1  # node not found
 else
     _VER := $(shell node --version)
     _NEWER := $(shell echo -e "$(NODE_VERSION)\n$(_VER)" | sort -Vr | head -n 1)
     ifneq ($(_VER), $(_NEWER))
-        $(info Node.js version not match)
+        $(info $(_INFO) Node.js version not match $(_END))
         _MISS_DEPS := 1  # node outdated
     endif
 endif
 ifeq (, $(shell command -v yarnpkg 2>/dev/null))
-    $(info Yarn not found)
+    $(info $(_INFO) Yarn not found $(_END))
     _MISS_DEPS := 1  # yarn not found
 endif
 ifeq (, $(shell command -v serve 2>/dev/null))
-    $(info Serve not found)
+    $(info $(_INFO) Serve not found $(_END))
     _MISS_DEPS := 1  # serve not found
 endif
 
 ifdef _MISS_DEPS
-    $(info Missing dependencies, use local toolchain)
+    $(info $(_INFO) Missing dependencies, use local toolchain $(_END))
     NODE := $(NODE_PATH)/bin/node
     YARN := PATH=$${PATH}:$(NODE_PATH)/bin $(YARN_PATH)/bin/yarn
     SERVE := $(SERVE_PATH)/serve
 else
-    $(info All dependencies found, use global toolchain)
+    $(info $(_INFO) All dependencies found, use global toolchain $(_END))
     NODE := node
     YARN := yarnpkg
     SERVE := serve
 endif
 
-## Colorful output
-_INFO := $(shell echo -e '\e[1;36m')
-_WARNING := $(shell echo -e '\e[1;33m')
-_END := $(shell echo -e '\e[0m')
 
 # Setting variables end
 
@@ -90,7 +94,6 @@ build:
 	#$(_INFO) Building nnictl $(_END)
 	cd tools && python3 setup.py build
 
-
 # Standard installation target
 # Must be invoked after building
 .PHONY: install
@@ -109,17 +112,15 @@ remote-machine-install:
 	cd src/sdk/pynni && python3 setup.py install $(PIP_MODE)
 
 
-# All-in-one target
+# All-in-one target for non-expert users
 # Installs NNI as well as its dependencies, and update bashrc to set PATH
 .PHONY: easy-install
 easy-install: check-perm
 easy-install: install-dependencies
 easy-install: build
-easy-install: install-python-modules
-easy-install: install-node-modules
-easy-install: install-scripts
-easy-install: install-examples
-easy-install: update-bashrc
+easy-install: install
+easy-install: update-bash-config
+
 easy-install:
 	#$(_INFO) Complete! #(_END)
 
@@ -132,6 +133,7 @@ pip-install: build
 pip-install: install-node-modules
 pip-install: install-scripts
 pip-install: install-examples
+pip-install: update-bash-config
 
 
 # Target for NNI developers
@@ -146,8 +148,8 @@ dev-install:
 
 .PHONY: uninstall
 uninstall:
-	-pip3 uninstall -y nni
-	-pip3 uninstall -y nnictl
+	-$(PIP_UNINSTALL) -y nni
+	-$(PIP_UNINSTALL) -y nnictl
 	-rm -rf $(INSTALL_PREFIX)/nni
 	-rm -f $(BIN_PATH)/nnimanager
 	-rm -f $(BIN_PATH)/nnictl
@@ -206,7 +208,6 @@ install-python-modules:
 	#$(_INFO) Installing nnictl $(_END)
 	cd tools && python3 setup.py install $(PIP_MODE)
 
-
 .PHONY: install-node-modules
 install-node-modules:
 	mkdir -p $(INSTALL_PREFIX)/nni
@@ -222,11 +223,11 @@ install-node-modules:
 .PHONY: install-dev-modules
 install-dev-modules:
 	#$(_INFO) Installing Python SDK $(_END)
-	cd src/sdk/pynni && pip3 install $(PIP_MODE) -e .
+	cd src/sdk/pynni && $(PIP_INSTALL) $(PIP_MODE) -e .
 
 	#$(_INFO) Installing nnictl $(_END)
-	cd tools && pip3 install $(PIP_MODE) -e .
-	
+	cd tools && $(PIP_INSTALL) $(PIP_MODE) -e .
+
 	mkdir -p $(INSTALL_PREFIX)/nni
 
 	#$(_INFO) Installing NNI Manager $(_END)
@@ -254,9 +255,6 @@ install-scripts:
 	chmod +x $(BIN_PATH)/nnictl
 
 	install -Dm644 tools/bash-completion $(BASH_COMP_SCRIPT)
-ifndef _ROOT
-	echo '[[ -f $(BASH_COMP_SCRIPT) ]] && source $(BASH_COMP_SCRIPT)' >> ~/.bash_completion
-endif
 
 
 .PHONY: install-examples
@@ -265,16 +263,20 @@ install-examples:
 	[ $(EXAMPLES_PATH) = ${PWD}/examples ] || cp -rT examples $(EXAMPLES_PATH)
 
 
-.PHONY: update-bashrc
-ifeq (, $(shell echo $$PATH | tr ':' '\n' | grep -x '$(BIN_PATH)'))  # $(BIN_PATH) not in PATH
-    ifdef _ROOT
-        $(error $(BIN_PATH) not in PATH as root, which should never happen)
-    endif
-update-bashrc:
+.PHONY: update-bash-config
+ifndef _ROOT
+update-bash-config:
+	#$(_INFO) Updating bash configurations $(_END)
+    ifeq (, $(shell echo $$PATH | tr ':' '\n' | grep -x '$(BIN_PATH)'))  # $(BIN_PATH) not in PATH
 	#$(_WARNING) NOTE: adding $(BIN_PATH) to PATH in bashrc $(_END)
 	echo 'export PATH="$$PATH:$(BIN_PATH)"' >> ~/.bashrc
-else  # $(BIN_PATH) already in PATH
-update-bashrc: ;
+    endif
+    ifeq (, $(shell (source ~/.bash_completion ; command -v _nnictl) 2>/dev/null))  # completion not installed
+	#$(_WARNING) NOTE: adding $(BASH_COMP_SCRIPT) to ~/.bash_completion $(_END)
+	echo '[[ -f $(BASH_COMP_SCRIPT) ]] && source $(BASH_COMP_SCRIPT)' >> ~/.bash_completion
+    endif
+else
+update-bash-config: ;
 endif
 
 
@@ -297,7 +299,7 @@ ifdef _ROOT
 	$(error You should not develop NNI as root)
 endif
 ifdef _MISS_DEPS
-	$(error Please install Node.js, Yarn, and Serve to develop NNI)
+#	$(error Please install Node.js, Yarn, and Serve to develop NNI)
 endif
 	#$(_INFO) Pass! $(_END)
 

diff --git a/README.Makefile.md b/README.Makefile.md
@@ -0,0 +1,91 @@
+# Makefile and Installation Setup
+
+NNI uses GNU make for building and installing.
+
+The `Makefile` offers standard targets `build`, `install`, and `uninstall`, as well as alternative installation targets for different setup:
+
+* `easy-install`: target for non-expert users, which handles everything automatically;
+* `pip-install`: target in favor of `setup.py`;
+* `dev-install`: target for NNI contributors, which installs NNI as symlinks instead of copying files;
+* `remote-machine-install`: target that only installs core Python library for remote machine workers.
+
+The targets will be detailed later.
+
+## Dependencies
+
+NNI requires at least Node.js, Yarn, and setuptools to build, while PIP and TypeScript are also recommended.
+
+NNI requires Node.js, serve, and all dependency libraries to run.
+Required Node.js libraries (including TypeScript) can be installed by Yarn, and required Python libraries can be installed by setuptools or PIP.
+
+For NNI *users*, `make install-dependencies` can be used to install Node.js, Yarn, and serve.
+This will install Node.js and serve to NNI's installation directory, and install Yarn to `/tmp/nni-yarn`.
+This target requires wget to work.
+
+For NNI *developers*, it is recommended to install Node.js, Yarn, and serve manually.
+See their official sites for installation guide.
+
+## Building NNI
+
+Simply run `make` when dependencies are ready.
+
+## Installation
+
+### Directory Hierarchy
+
+The main parts of NNI project consist of two Node.js modules (`nni_manager`, `webui`) and two Python packages (`nni`, `nnictl`).
+
+By default the Node.js modules are installed to `/usr/share/nni` for all users or installed to `~/.local/nni` for current user.
+
+The Python packages are installed with setuptools and therefore the location depends on Python configuration.
+When install as non-priviledged user and virtualenv is not detected, `--user` flag will be used.
+
+In addition, `nnictl` offers a bash completion scripts, which will be installed to `/usr/share/bash-completion/completions` or `~/.bash_completion.d`.
+
+In some configuration, NNI will also install Node.js and the serve module to `/usr/share/nni`.
+
+All directories mentioned above are configurable. See next section for details.
+
+### Configuration
+
+The `Makefile` uses environment variables to override default settings.
+
+Available variables are listed below:
+
+| Name               | Description                                             | Default for normal user           | Default for root                                |
+|--------------------|---------------------------------------------------------|-----------------------------------|-------------------------------------------------|
+| `BIN_PATH`         | Path for executables                                    | `~/.local/bin`                    | `/usr/bin`                                      |
+| `INSTALL_PREFIX`   | Path for Node.js modules (a suffix `nni` will be added) | `~/.local`                        | `/usr/share`                                    |
+| `EXAMPLES_PATH`    | Path for NNI examples                                   | `~/nni/examples`                  | `$INSTALL_PREFIX/nni/examples`                  |
+| `BASH_COMP_SCRIPT` | Path of bash completion script                          | `~/.bash_completion.d/nnictl`     | `/usr/share/bash-completion/completions/nnictl` |
+| `PIP_MODE`         | Arguments for `python3 setup.py install`                | `--user` if `VIRTUAL_ENV` not set | (empty)                                         |
+| `NODE_PATH`        | Path to install Node.js runtime                         | `$INSTALL_PREFIX/nni/node`        | `$INSTALL_PREFIX/nni/node`                      |
+| `SERVE_PATH`       | Path to install serve package                           | `$INSTALL_PREFIX/nni/serve`       | `$INSTALL_PREFIX/nni/serve`                     |
+| `YARN_PATH`        | Path to install Yarn                                    | `/tmp/nni-yarn`                   | `/tmp/nni-yarn`                                 |
+| `NODE`             | Node.js command                                         | see source file                   | see source file                                 |
+| `SERVE`            | serve command                                           | see source file                   | see source file                                 |
+| `YARN`             | Yarn command                                            | see source file                   | see source file                                 |
+
+Note that these variables will influence installation destination as well as generated `nnictl` and `nnimanager` scripts.
+If the path to copy files is different from where they will run (e.g. when creating a distro package), please generate `nnictl` and `nnimanager` manually.
+
+### Targets
+
+The workflow of each installation targets is listed below:
+
+| Target                   | Workflow                                                             |
+|--------------------------|----------------------------------------------------------------------|
+| `install`                | Install Python packages, Node.js modules, NNI scripts, and examples  |
+| `easy-install`           | Install dependencies, build, install NNI, and edit `~/.bashrc`       |
+| `pip-install`            | Install dependencies, build, install NNI excluding Python packages   |
+| `dev-install`            | Install Python and Node.js modules as symlinks, then install scripts |
+| `remote-machine-install` | Install `nni` Python package                                         |
+
+## TODO
+
+* `clean` target
+* `test` target
+* `lint` target
+* Exclude tuners and their dependencies from `remote-machine-install`
+* Test cases for each target
+* Review variables
diff --git a/README.md b/README.md
@@ -26,35 +26,45 @@ The tool dispatches and runs trial jobs that generated by tuning algorithms to s
 * As a researcher and data scientist, you want to implement your own AutoML algorithms and compare with other algorithms
 * As a ML platform owner, you want to support AutoML in your platform
 
-# Getting Started with NNI
+# Get Started with NNI
 
 ## **Installation**
-Install through python pip. (the current version only supports linux, nni on ubuntu 16.04 or newer has been well tested)
-* requirements: python >= 3.5, git, wget
+pip Installation Prerequisites
+* linux (ubuntu 16.04 or newer version has been well tested)
+* python >= 3.5
+* git, wget
+
 ```
-pip3 install -v --user git+https://github.com/Microsoft/nni.git@v0.1
+python3 -m pip install -v --user git+https://github.com/Microsoft/nni.git@v0.2
 source ~/.bashrc
 ```
 
+## **Quick start: run your first experiment at local**
+It only requires 3 steps to start an experiment on NNI:
+![](./docs/3_steps.jpg)
+
+
+NNI provides a set of examples in the package to get you familiar with the above process. In the following example [/examples/trials/mnist], we had already set up the configuration and updated the training codes for you. You can directly run the following command to start an experiment. 
 
-## **Quick start: run an experiment at local**
-Requirements:
-* NNI installed on your local machine
-* tensorflow installed
+**NOTE**: The following example is an experiment built on TensorFlow, make sure you have **TensorFlow installed** before running the following command.
 
-Run the following command to create an experiment for [mnist]
+Try it out:
 ```bash
-    nnictl create --config ~/nni/examples/trials/mnist-annotation/config.yml
+    nnictl create --config ~/nni/examples/trials/mnist/config.yml
 ```
-This command will start an experiment and a WebUI. The WebUI endpoint will be shown in the output of this command (for example, `http://localhost:8080`). Open this URL in your browser. You can analyze your experiment through WebUI, or browse trials' tensorboard. 
+
+In the command output, find out the **Web UI url** and open it in your browser. You can analyze your experiment through WebUI, or browse trials' tensorboard. 
+
+To learn more about how this example was constructed and how to analyze the experiment results in NNI Web UI, please refer to [How to write a trial run on NNI (MNIST as an example)?](docs/WriteYourTrial.md)
 
 ## **Please refer to [Get Started Tutorial](docs/GetStarted.md) for more detailed information.**
 ## More tutorials
-* [How to write a trial running on NNI (Mnist as an example)?](docs/WriteYourTrial.md)
+
 * [Tutorial of NNI python annotation.](tools/nni_annotation/README.md)
 * [Tuners supported by NNI.](src/sdk/pynni/nni/README.md)
 * [How to enable early stop (i.e. assessor) in an experiment?](docs/EnableAssessor.md)
 * [How to run an experiment on multiple machines?](docs/RemoteMachineMode.md)
+* [How to run an experiment on OpenPAI?](docs/PAIMode.md)
 * [How to write a customized tuner?](docs/CustomizedTuner.md)
 * [How to write a customized assessor?](examples/assessors/README.md)
 * [How to resume an experiment?](docs/NNICTLDOC.md)

diff --git a/deployment/Dockerfile b/deployment/Dockerfile
@@ -10,4 +10,6 @@ RUN pip3 --no-cache-dir install tensorflow-gpu==1.10.0
 #
 #Keras 2.1.6
 #
-RUN pip3 --no-cache-dir install Keras==2.1.6
+RUN pip3 --no-cache-dir install Keras==2.1.6
+
+WORKDIR /root