Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

Add doc of TextNAS #2260

Merged
merged 14 commits into from
Apr 3, 2020
56 changes: 56 additions & 0 deletions docs/en_US/NAS/TextNAS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# TextNAS

## Introduction

This is the implementation of the TextNAS algorithm proposed in the paper [TextNAS: A Neural Architecture Search Space tailored for Text Representation](https://arxiv.org/pdf/1912.10729.pdf). TextNAS is a neural architecture search algorithm tailored for text representation, more specifically, TextNAS is based on a novel search space consists of operators widely adopted to solve various NLP tasks, and TextNAS also supports multi-path ensemble within a single network to balance the width and depth of the architecture.

The search space of TextNAS contains:

* 1-D convolutional operator with filter size 1, 3, 5, 7
* recurrent operator (bi-directional GRU)
* self-attention operator
* pooling operator (max/average)

Following the ENAS algorithm, TextNAS also utilizes parameter sharing to accelerate the search speed and adopts a reinforcement-learning controller for the architecture sampling and generation. Please refer to the paper for more details of TextNAS.

## Examples

### Search Space

[Example code](https://github.com/microsoft/nni/tree/master/examples/nas/textnas)

```bash
# In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
git clone https://github.com/Microsoft/nni.git

# search the best architecture
cd examples/nas/textnas

# view more options for search
python3 search.py -h
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could @pkuyym or @ultmaster add the guideline for preparing data?

```

### retrain

```bash
# In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
git clone https://github.com/Microsoft/nni.git

# search the best architecture
cd examples/nas/textnas

# default to retrain on sst-2
sh run_retrain.sh
```

## Reference

### PyTorch

```eval_rst
.. autoclass:: nni.nas.pytorch.enas.EnasTrainer
:members:

.. autoclass:: nni.nas.pytorch.enas.EnasMutator
:members:
```
6 changes: 5 additions & 1 deletion examples/nas/textnas/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,8 @@ By default, 20 sampled architectures will be exported into `checkpoints` directo

## Retrain

Not ready.
```
sh run_retrain.sh
```

By default, the script will retrain the architecture provided by the author on the SST-2 dataset.
212 changes: 212 additions & 0 deletions examples/nas/textnas/arc/final_arc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
{
"LayerChoice1": [
false, false, false, false, false, true, false, false
],
"InputChoice2": [
true
],
"LayerChoice3": [
false, false, false, false, false, false, false, true
],
"InputChoice4": [
false
],
"InputChoice5": [
true, false
],
"LayerChoice6": [
false, false, false, true, false, false, false, false
],
"InputChoice7": [
false, false
],
"InputChoice8": [
false, false, true
],
"LayerChoice9": [
false, false, false, false, false, false, true, false
],
"InputChoice10": [
false, true, true
],
"InputChoice11": [
false, false, true, false
],
"LayerChoice12": [
false, true, false, false, false, false, false, false
],
"InputChoice13": [
false, true, false, false
],
"InputChoice14": [
false, false, false, false, true
],
"LayerChoice15": [
false, true, false, false, false, false, false, false
],
"InputChoice16": [
false, false, true, false, true
],
"InputChoice17": [
false, false, false, false, true
],
"LayerChoice18": [
true, false, false, false, false, false, false, false
],
"InputChoice19": [
false, false, true, true, true, true
],
"InputChoice20": [
true, false, false, false, false
],
"LayerChoice21": [
false, false, false, false, false, false, true, false
],
"InputChoice22": [
false, true, true, false, false, false, false
],
"InputChoice23": [
false, true, false, false, false
],
"LayerChoice24": [
false, false, false, false, false, true, false, false
],
"InputChoice25": [
false, true, false, true, true, false, true, true
],
"InputChoice26": [
false, false, true, false, false
],
"LayerChoice27": [
false, false, false, false, false, true, false, false
],
"InputChoice28": [
false, false, false, false, false, true, false, true, true
],
"InputChoice29": [
true, false, false, false, false
],
"LayerChoice30": [
false, false, false, false, false, false, false, true
],
"InputChoice31": [
true, true, false, false, true, false, false, true, true, false
],
"InputChoice32": [
true, false, false, false, false
],
"LayerChoice33": [
false, false, false, false, true, false, false, false
],
"InputChoice34": [
true, false, false, true, true, true, true, false, false, false, false
],
"InputChoice35": [
false, false, false, true, false
],
"LayerChoice36": [
false, true, false, false, false, false, false, false
],
"InputChoice37": [
true, true, false, true, false, true, false, false, true, false, false, false
],
"InputChoice38": [
false, false, false, true, false
],
"LayerChoice39": [
false, false, true, false, false, false, false, false
],
"InputChoice40": [
true, true, false, false, false, false, true, false, false, true, true, false, true
],
"InputChoice41": [
false, false, false, true, false
],
"LayerChoice42": [
true, false, false, false, false, false, false, false
],
"InputChoice43": [
false, false, true, false, false, false, true, true, true, false, true, true, false, false
],
"InputChoice44": [
false, false, false, false, true
],
"LayerChoice45": [
false, false, false, true, false, false, false, false
],
"InputChoice46": [
true, false, false, false, false, false, true, false, false, false, true, true, false, false, true
],
"InputChoice47": [
false, false, false, true, false
],
"LayerChoice48": [
false, false, true, false, false, false, false, false
],
"InputChoice49": [
false, false, false, false, false, false, false, false, false, true, true, false, true, false, true, false
],
"InputChoice50": [
false, false, false, false, true
],
"LayerChoice51": [
false, false, false, false, true, false, false, false
],
"InputChoice52": [
false, true, true, true, true, false, false, true, false, true, false, false, false, false, true, false, false
],
"InputChoice53": [
false, false, true, false, false
],
"LayerChoice54": [
false, false, false, true, false, false, false, false
],
"InputChoice55": [
false, false, false, false, false, true, false, false, false, false, false, false, false, true, true, true, false, true
],
"InputChoice56": [
false, false, true, false, false
],
"LayerChoice57": [
false, false, false, true, false, false, false, false
],
"InputChoice58": [
false, false, false, true, false, false, false, false, false, false, true, false, false, false, true, false, false, false, false
],
"InputChoice59": [
false, true, false, false, false
],
"LayerChoice60": [
false, false, false, false, false, true, false, false
],
"InputChoice61": [
true, true, false, false, false, false, false, false, false, false, true, true, false, false, true, true, true, true, false, false
],
"InputChoice62": [
true, false, false, false, false
],
"LayerChoice63": [
false, false, false, false, false, false, false, true
],
"InputChoice64": [
false, true, true, true, false, false, false, true, false, true, true, true, true, false, true, false, false, false, false, false, false
],
"InputChoice65": [
false, false, false, false, true
],
"LayerChoice66": [
false, false, false, false, false, false, false, true
],
"InputChoice67": [
false, false, true, true, true, true, false, true, false, true, true, false, false, false, false, true, false, false, false, false, false, true
],
"InputChoice68": [
false, false, false, true, false
],
"LayerChoice69": [
false, false, false, true, false, false, false, false
],
"InputChoice70": [
true, false, false, true, false, false, false, true, false, false, false, false, true, false, false, false, true, false, false, false, false, false, false
]
}
2 changes: 1 addition & 1 deletion examples/nas/textnas/run_retrain.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,6 @@ python -u retrain.py \
--child_lr_T_0=10 \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use python3 instand of python

--child_lr_T_mul=2 \
--multi_path=True \
--child_fixed_arc="./checkpoints/architecture_00.json" \
--child_fixed_arc="./arc/final_arc.json" \
--fixed_seed=True \
"$@"