NNI 2021 Mar~Apr Iteration Planning

[kvartet]

This is the plan for the iteration in Mar~Apr, it's a 4 weeks iteration.

Release Plan for v2.2

• Release manager: @SparkSnail
• Feature freeze date: 3.29
• Code freeze and demo date: 4.6 4.9
• Release date: 4.12 4.19 4.25

---

Let's try to adopt the legend of annotations:

Mark	Description
🏃‍♂️	work in progress
👪	work under review
✋	blocked task
💪	stretch goal for this iteration
🔴	missing issue / pull request reference
🔵	more investigation and survey required
⚫	under discussion within the team
⬛	a large work item, larger than one iteration

Bug bash

nnictl & nni.experiment

• P0 - native config for v2 [tester:@98may] (testing)
• P0 - make nni.experiment complete (view, resume) [tester:@kvartet] (need to update doc @J-shang)

Retiarii & NAS

• P0 - format of output architecture: implementation and doc [tester: @colorjam]
• P0 - nested value choice [tester: @QuanluZhang @linbinskn ]
• P0 - support dict value choice [tester: @QuanluZhang @linbinskn ]
• P0 - Support debug mode (standalone mode) [tester: @ultmaster @linbinskn @colorjam ]

Model compression

• P0 - Quantization improvement: model import/export [tester: @colorjam ]
• P0 - Quantization speedup tool [ tester: @colorjam]
• P1 - Update TF model compression: support checkpoint (deferred from v2.0) [ tester: @J-shang]

Training service & NNI manager

• P0 - Reuse mode envs dir iterate [tester:@acured]
• P0 - Add log information for gpu_collector [tester:@liuzhe-lz]
• P0 - Improve share storage [tester: @acured]
• P1 - Reuse mode remote windows (set "reuse" as default) [tester: @SparkSnail]
• P1 - Support tensorboard [tester: @ultmaster @colorjam ]

WebUI

• minor improvements: [ tester: @J-shang]
  • P0 - Upgrade echarts-for-react
  • P1 - webUI dispatcher log / nnimanager log wrap
• P1 - Support tensorboard [ tester: @ultmaster @colorjam ]
  • UX design DL: 2.3
  • backend

Bug fix

• Fix bug about hyper-parameter Add/Remove axes and table Add/Remove columns button conflict [tester: @SparkSnail]
• Fix bug about monaco editor search text is not display completely [tester: @kvartet]

Features

nnictl & nni.experiment

• P0 - write v2 config files for examples @liuzhe-lz PR: Convert config files in examples to v2 schema #3388 (Depend on pr Add native support for v2 config #3466, ready for review)
  • current items could be merged (fixing IT, 4/7)
• P0 - make nni.experiment complete (view, resume) @J-shang (@nihao backup) (,4/7 PR [Do Not Merge] add resume and view mode in python experiment #3490 )

Retiarii & NAS

• P0 - format of output architecture: implementation and doc (need experiment api design meeting)@ultmaster ([Retiarii] Export topk models #3464 in review)
• P0 - nested value choice @ultmaster ([Retiarii] Nest ValueChoice in LayerChoice and dict/list in ValueChoice #3508 need review)
• P0 - support dict value choice @ultmaster ([Retiarii] Nest ValueChoice in LayerChoice and dict/list in ValueChoice #3508 need view)
• P0 - Support debug mode (standalone mode) @QuanluZhang
• P0 - Refactor NAS examples @QuanluZhang (PR refactor of nas examples #3513)
• P1 - AML & NAS integration demo @kvartet (4/2)
  • PR for notebook example and demo

Model compression

• P0 - Quantization improvement: model import/export @linbinskn (3/19 PR Add model export for QAT #3458)
• P0 - Quantization speedup tool @linbinskn (Combine tensorrt tool with NNI quantization algorithms. #3488 ready review)
• P1 - Update TF model compression: support checkpoint (deferred from v2.0) @liuzhe-lz ([Model Compression / TensorFlow] Support exporting pruned model #3487 need review)
• P1 - doc improvement @colorjam (PR Update readme doc link #3482 need review)

Training service & NNI manager

• P0 - Reuse mode envs dir iterate @SparkSnail (3/22 PR Fix env folder #3472)
• P0 - Add log information for gpu_collector @SparkSnail (Add log information for gpu collector #3506, PR ready for review)
• P0 - add test for shared storage @acured (3/24 Add shared storage integration test #3455 need to fix IT and publish PR, depend on sharedstorage support remote umount and fix bug #3456) ETA 4/6
  • P0 - Improve share storage @J-shang (3/19 sharedstorage support remote umount and fix bug #3456 need review)
• P1 - Reuse mode remote windows (set "reuse" as default) (@acured ask @SparkSnail, 3/31 )
• P1 - Support tensorboard @J-shang (tensorboard backend #3454 need review)

WebUI

• minor improvements: @Lijiaoa (3/19)
  • P0 - Upgrade echarts-for-react (Upgrade echarts version to v5 #3457)
  • P1 - webUI dispatcher log / nnimanager log wrap (WebUI dispatcher log / nnimanager log wrap #3461 need review) @Lijiaoa
• P1 - Support tensorboard @Lijiaoa (4/7 [WebUI] support tensorboard #3361)
  • UX design DL: 2.3
  • backend

Pipelines

• P0 - add aml pipeline test @SparkSnail (Support aml pipeline #3477 draft pr) ETA 4/7
• P0 - add hybrid pipeline test @SparkSnail (Support hybrid pipeline #3478 draft pr) ETA 4/7

Bug fix

• Fix bug about hyper-parameter Add/Remove axes and table Add/Remove columns button conflict PR#3491
• Fix bug about monaco editor search text is not display completely issue The text is not displayed completely when searching in the right column #3484 PR: fix the issue: text is not displayed completely when searching in the search space/config/log monaco editor #3492

Documentation

Next release candidate

• P0 - TensorFlow support exploration @liuzhe-lz (does not follow feature freeze)
• P0 - More high-level APIs for users to easily express model space @ultmaster (4/9 [Retiarii] New API: Repeat and Cell #3481 will PR ready, may delay) ETA 4/9
• P1 - NAS visualization improvements: refactor graph representation (deferred from v2.0) @98may
  • Step1. Data structure design @98may
  • Step2. Visulization @98may
• P1 - high level API design @J-shang (4/2 design, need discussion)
• P1 - customize dafault metric to multiple metrics (how to customize dafault metric to multiple metrics #3504 @kvartet )
• P1 - Bug: Tuner need to exit when experiment done
• P1 - module compression pipeline @colorjam (4/2, will have followup task)
• P1 - Support a fancy NAS algorithm which could leverage architectural information @98may (PR DNGO tuner #3479, need review)
• P0 - Detail Page improvement @Lijiaoa
  • Refine search parameters Refine search by paramters #3024 Improvement of Search Function in Trials Detail page  #3099
  • Put the add/remove button in the last column of the header Improvement of Search Function in Trials Detail page  #3099 (draft PR, put add/remove columns btn to table column endding #3485)
• P1 - [Retiarii] Further improve the logic of graph generation and code generation @QuanluZhang
  • Support control flow in graph/code generation
  • Further integrated with TorchScript
• P1 - Add pure-python execution engine @ultmaster (4/2)
• P1 - Support RL strategy @ultmaster (4/2)
• P1 - NAS visualization
  • UX Design - concept design @QuanluZhang / @scarlett2018
  • UX Design - formal design @yangou1988
  • visualization implementation - @98may
• P1 - set base url for the web report (Is there any option to set base url for the web report? #2771) @acured
• P1 - Refactor readthedocs integration @ultmaster ()
• P0 - (discussion) support experiment resume/view/stop on webUI @Lijiaoa (depending with Quanlu meeting)
• P1 - Pruning model speedup improvement @J-shang