[MetaSchedule][UX] User Interface for Jupyter Notebook #12866

zxybazh · 2022-09-22T05:59:20Z

Previously the tuning process would generate significant amount of logging to screen, which would be troublesome for jupyter notebook usage. This PR proposes several changes to fit in the usage on jupyter:

The output would automatically clean up each time a new tuning result table is generated, i.e., in the beginning of tuning after initializing all tasks, and after each task join.

Slim version of the table is available when ipython environment is used so that the table can fit in output area of the notebook.

Sample slim table:

 ID |                    Name |      FLOP | Weight |   GFLOPS | Latency (us) | Wtd. Latency | Trials | Terminated 
------------------------------------------------------------------------------------------------------------------
  0 |     fused_nn_conv2d_add |  12870144 |      1 | 154.0759 |      83.5312 |      83.5312 |     32 |            
  1 |   fused_nn_conv2d_add_1 |  12895232 |      1 | 186.2171 |      69.2484 |      69.2484 |     32 |            
  2 |   fused_nn_conv2d_add_2 |  12945408 |      1 | 212.0639 |      61.0449 |      61.0449 |     32 |            
  3 |  fused_layout_transform |         1 |      1 |   0.0000 |      23.1081 |      23.1081 |      1 |            
  4 | fused_nn_conv2d_add_... | 237633536 |      1 | 180.0741 |    1319.6433 |    1319.6433 |     32 |            
  5 |     fused_nn_max_pool2d |   1806336 |      1 |  40.6316 |      44.4564 |      44.4564 |     32 |            
  6 | fused_nn_conv2d_add_... | 231612416 |      2 | 276.8975 |     836.4555 |    1672.9111 |     32 |            
  7 | fused_nn_conv2d_add_... | 231813120 |      2 | 100.5202 |    2306.1349 |    4612.2697 |     32 |            
  8 | fused_nn_conv2d_add_... | 115806208 |      1 | 139.2956 |     831.3701 |     831.3701 |     32 |            
  9 | fused_nn_contrib_con... |  93227008 |      1 | 125.3350 |     743.8227 |     743.8227 |     32 |            
 10 | fused_nn_contrib_con... |  93327360 |      2 | 152.6041 |     611.5651 |    1223.1302 |     32 |            
 11 | fused_nn_conv2d_add_... | 115705856 |      1 |  92.3480 |    1252.9333 |    1252.9333 |     32 |            
 12 | fused_nn_contrib_con... |  98600960 |      1 | 151.7883 |     649.5951 |     649.5951 |     32 |            
 13 | fused_nn_contrib_con... |  98651136 |      2 |  21.0946 |    4676.5961 |    9353.1921 |     32 |            
 14 | fused_nn_conv2d_add_... | 115655680 |      1 |  50.2975 |    2299.4334 |    2299.4334 |     32 |            
 15 | fused_nn_conv2d_add_... | 231261184 |      1 |  51.0481 |    4530.2563 |    4530.2563 |     32 |            
 16 | fused_nn_conv2d_add_... | 231286272 |      2 | 123.8502 |    1867.4684 |    3734.9369 |     32 |            
 17 | fused_nn_adaptive_av... |     25600 |      1 |   5.5381 |       4.6225 |       4.6225 |     32 |            
 18 | fused_layout_transfo... |         1 |      1 |   0.0002 |       4.4801 |       4.4801 |      1 |            
 19 |      fused_nn_dense_add |   1025000 |      1 |  46.7726 |      21.9145 |      21.9145 |     32 |            
------------------------------------------------------------------------------------------------------------------

Added progress of tuning shown as percentage.

Example:

Progress: 57.8%
Total Trials: 578 / 1000
Total latency (us): 32535.9

zxybazh · 2022-09-22T17:28:23Z

@tvm-bot rerun

shingjan · 2022-09-22T18:02:14Z

This seems super useful. Do we want to extends this functionality to non-jupyter notebook? Essentially what I have in mind is that we can update the table & process bar and omit all other log info like send xx samples to runner/builder.

zxybazh · 2022-09-22T19:56:46Z

Hi YJ, I think this applies to all the ipython kernel based interface, yet I only tested on jupyter notebook. On the other hand we have progress as percentage available now so I'm not planning to add a progress bar (but definitely doable). IMHO other logs like send xx samples are useful information and won't take a lot of lines on the screen so I tend to keep them as they are, and they would be cleaned once a new table is printed out.

junrushao · 2022-09-23T16:40:20Z

@zxybazh do you have a demo on gist?

src/meta_schedule/task_scheduler/gradient_based.cc

zxybazh · 2022-09-23T17:00:47Z

For testing, I used python tests/python/unittest/test_meta_schedule_tune_relay.py for local cli testing, and I can also upload my notebook to gist for notebook testing.

junrushao · 2022-09-23T20:28:56Z

uploading an jupyter notebook to gist would be great

zxybazh · 2022-09-23T20:41:27Z

Available here: https://gist.github.com/zxybazh/82665e01e94522deadc0646a428155ce

junrushao · 2022-09-23T21:50:50Z

Really cool stuff! I'm happy with merging it in! Just a further question, do you think it's possible that we draw a jupyter table like what's done with Pandas DataFrame?

junrushao

LGTM

MasterJH5574 · 2022-09-24T04:39:22Z

src/meta_schedule/task_scheduler/gradient_based.cc

+      String task_name = record.task->task_name.value();
+      if (using_ipython() && task_name.length() > 23) {
+        std::string temp = task_name.c_str();
+        temp = temp.substr(0, 20) + "...";
+        task_name = String(temp);
+      }


Do we have alternative ways to print the whole task names? Like if a task name is too long, we can try to print the name in multiple lines.

I’m thinking of this because I just noticed that in your example table, there are several lines with exactly the same prefix.

6 | fused_nn_conv2d_add_... | 231612416 | 2 | 276.8975 | 836.4555 | 1672.9111 | 32 | 7 | fused_nn_conv2d_add_... | 231813120 | 2 | 100.5202 | 2306.1349 | 4612.2697 | 32 | 8 | fused_nn_conv2d_add_... | 115806208 | 1 | 139.2956 | 831.3701 | 831.3701 | 32 |

I think this will cause confusion. So perhaps we should try to resolve this problem? What do you think of this.

Also cc @junrushao

This is indeed not ideal. We may have an extra API that allows us to assemble a pandas dataframe so that it’s more organically integrated with jupyter. I have an idea and will communicate with Xiyou next week

zxybazh · 2022-09-26T05:54:44Z

Thanks all for reviewing & we’ll further work on creating dataframe view or other directions to optimize jupyter notebook view in a separate PR.

…e#12866)" This reverts commit 46ea2ed.

* Add features for jupyter notebook. * Fix workload import warnings. * Enable output clearing for cli. * Fix test. * Fix lint. * Change to separate cleaning function.

Add features for jupyter notebook.

f0fe412

zxybazh marked this pull request as ready for review September 22, 2022 06:07

zxybazh added 4 commits September 22, 2022 23:36

Fix workload import warnings.

f76afab

Enable output clearing for cli.

1a3f07c

Fix test.

6ce0e22

Fix lint.

10ed1f4

zxybazh force-pushed the feature/2022-09-21/support-jupyter-notebook-interface branch from 6f74182 to 10ed1f4 Compare September 23, 2022 09:05

junrushao reviewed Sep 23, 2022

View reviewed changes

src/meta_schedule/task_scheduler/gradient_based.cc Outdated Show resolved Hide resolved

junrushao approved these changes Sep 23, 2022

View reviewed changes

shingjan approved these changes Sep 23, 2022

View reviewed changes

MasterJH5574 reviewed Sep 24, 2022

View reviewed changes

Change to separate cleaning function.

1d39bb1

zxybazh merged commit 46ea2ed into apache:main Sep 26, 2022

junrushao added a commit to junrushao/tvm that referenced this pull request Sep 26, 2022

Revert "[MetaSchedule][UX] User Interface for Jupyter Notebook (apach…

fd25eab

…e#12866)" This reverts commit 46ea2ed.

junrushao added a commit to junrushao/tvm that referenced this pull request Sep 26, 2022

Revert "[MetaSchedule][UX] User Interface for Jupyter Notebook (apach…

e0e621a

…e#12866)" This reverts commit 46ea2ed.

junrushao added a commit to junrushao/tvm that referenced this pull request Sep 27, 2022

Revert "[MetaSchedule][UX] User Interface for Jupyter Notebook (apach…

c765e2d

…e#12866)" This reverts commit 46ea2ed.

junrushao added a commit to junrushao/tvm that referenced this pull request Sep 27, 2022

Revert "[MetaSchedule][UX] User Interface for Jupyter Notebook (apach…

8c4e95d

…e#12866)" This reverts commit 46ea2ed.

junrushao added a commit to junrushao/tvm that referenced this pull request Sep 28, 2022

Revert "[MetaSchedule][UX] User Interface for Jupyter Notebook (apach…

ca5ae15

…e#12866)" This reverts commit 46ea2ed.

junrushao added a commit to junrushao/tvm that referenced this pull request Sep 28, 2022

Revert "[MetaSchedule][UX] User Interface for Jupyter Notebook (apach…

898f77f

…e#12866)" This reverts commit 46ea2ed.

junrushao added a commit to junrushao/tvm that referenced this pull request Sep 29, 2022

Revert "[MetaSchedule][UX] User Interface for Jupyter Notebook (apach…

dc271db

…e#12866)" This reverts commit 46ea2ed.

junrushao added a commit to junrushao/tvm that referenced this pull request Sep 29, 2022

Revert "[MetaSchedule][UX] User Interface for Jupyter Notebook (apach…

ab9451f

…e#12866)" This reverts commit 46ea2ed.

zxybazh mentioned this pull request Oct 7, 2022

[MetaSchedule][UX] Support Interactive Performance Table Printing in Notebook #13006

Merged

leandron mentioned this pull request Feb 1, 2023

TVM v0.11.0 Release Candidate Notes #13899

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MetaSchedule][UX] User Interface for Jupyter Notebook #12866

[MetaSchedule][UX] User Interface for Jupyter Notebook #12866

zxybazh commented Sep 22, 2022 •

edited

Loading

zxybazh commented Sep 22, 2022

shingjan commented Sep 22, 2022

zxybazh commented Sep 22, 2022

junrushao commented Sep 23, 2022

zxybazh commented Sep 23, 2022

junrushao commented Sep 23, 2022

zxybazh commented Sep 23, 2022

junrushao commented Sep 23, 2022

junrushao left a comment

MasterJH5574 Sep 24, 2022

junrushao Sep 24, 2022

zxybazh commented Sep 26, 2022

[MetaSchedule][UX] User Interface for Jupyter Notebook #12866

[MetaSchedule][UX] User Interface for Jupyter Notebook #12866

Conversation

zxybazh commented Sep 22, 2022 • edited Loading

zxybazh commented Sep 22, 2022

shingjan commented Sep 22, 2022

zxybazh commented Sep 22, 2022

junrushao commented Sep 23, 2022

zxybazh commented Sep 23, 2022

junrushao commented Sep 23, 2022

zxybazh commented Sep 23, 2022

junrushao commented Sep 23, 2022

junrushao left a comment

Choose a reason for hiding this comment

MasterJH5574 Sep 24, 2022

Choose a reason for hiding this comment

junrushao Sep 24, 2022

Choose a reason for hiding this comment

zxybazh commented Sep 26, 2022

zxybazh commented Sep 22, 2022 •

edited

Loading