Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TVMC] Allow selecting a subset of tasks to be used in tvmc tune #12525

Merged
merged 11 commits into from
Mar 30, 2023

Conversation

PhilippvK
Copy link
Contributor

@PhilippvK PhilippvK commented Aug 22, 2022

This adds a --tasks flag to the tvmc tune command to filter the lists of tasks to be tuned. See examples below.

Motivation

  • As auto-tuning can be quite time consuming, it is often desirable to cut down the number of tuned tasks in a session.
  • If the tuning session was canceled halfway through, it would be a bad idea to start from scratch. Instead continue with the last untuned task
  • Some tasks have more impact on the model performance than others, thus we should be able to train some tasks longer than others

Examples

  1. Use --task list to show which tasks are available for tuning
$ tvmc tune toycar.tflite -o out.txt --task list
Available Tasks for tuning:
  0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), 'int...
  1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), 'int16...
  2. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (128, 128), 'int...
  3. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (128, 128), 'int16...
  4. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (8, 128), 'int16...
  5. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (8, 128), 'int16')...
  6. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 8), 'int16'), ('TENSOR', (128, 8), 'int16')...
  7. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 8), 'int16'), ('TENSOR', (128, 8), 'int16'), ...
  8. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (640, 128), 'int...
  9. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (640, 128), 'int16...
  1. Filter the list of tasks to be tuned:
# Only tune a single task (index 5)
tvmc tune toycar.tflite -o out.txt --tasks 5

# Tunes tasks starting with index 6
tvmc tune toycar.tflite -o out.txt --tasks "6-"

# Tune tasks 1,4,5,6,8,9
tvmc tune toycar.tflite -o out.txt --tasks "1,4-6,8-"

Tests

I added a basic unit test for the filter_tasks utility in tests/python/driver/tvmc/test_autotuner.py.

Open Questions

  • While the (truncated) string representations of AutoTVM tasks are quite helpful to pick the correct tasks, using AutoScheduler the tasks can not really be distinguished from each other (only by index). Is there a way to get similar information from AutoScheduler tasks?

cc @Mousius @gromero

@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch from d249a35 to 7085107 Compare August 22, 2022 06:48
@PhilippvK
Copy link
Contributor Author

PhilippvK commented Aug 22, 2022

CC @leandron @giuseros @areusch @tqchen

Copy link
Contributor

@areusch areusch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @PhilippvK, this looks like a great addition! left a couple comments here and there

python/tvm/driver/tvmc/autotuner.py Outdated Show resolved Hide resolved
python/tvm/driver/tvmc/autotuner.py Outdated Show resolved Hide resolved
python/tvm/driver/tvmc/autotuner.py Outdated Show resolved Hide resolved
tests/python/driver/tvmc/test_autotuner.py Outdated Show resolved Hide resolved
@github-actions github-actions bot requested review from Mousius and gromero September 7, 2022 20:15
@areusch
Copy link
Contributor

areusch commented Sep 7, 2022

@PhilippvK i think this one needs a rebase

@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch 2 times, most recently from e4d85de to c1306f7 Compare September 9, 2022 13:04
@PhilippvK
Copy link
Contributor Author

Short update:

  • With my latest changes I can now print useful task names for AutoScheduler as well (using task.desc instead of str(task))
  • I added a mention of the --task argument in the TVMC CLI tutorial

Copy link
Contributor

@areusch areusch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @PhilippvK, could you address the lint issue?

@PhilippvK
Copy link
Contributor Author

PhilippvK commented Sep 21, 2022

@areusch

As I have to retrigger the CI anyway so I want to propose another small addition:

For AutoTVM, in addition to the task description, I could also print len(task.config_space) to give an idea on how large the tuning search space is for a specific task, which can help a lot to estimate the required number of tuning trials.

Example:

Available Tasks for tuning:
  0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640)... (len=640)
  1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), ... (len=35)

Do you think this would be a good addition to tvmc tune --task list?

@PhilippvK
Copy link
Contributor Author

@areusch Can I have an update on this?

@leandron
Copy link
Contributor

@tvm-bot rerun

@areusch
Copy link
Contributor

areusch commented Oct 17, 2022

sorry @PhilippvK I missed this one. feel free to ping me on discord if that happens again. I retriggered the CI, I'm good for @leandron to merge this if it passes.

Example:

Available Tasks for tuning:
0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640)... (len=640)

  1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), ... (len=35)
    Do you think this would be a good addition to tvmc tune --task list?

Sure that seems like a good addition!

@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch from c4f6894 to 1e7d121 Compare October 18, 2022 07:29
@tvm-bot
Copy link
Collaborator

tvm-bot commented Oct 18, 2022

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

Generated by tvm-bot

@PhilippvK
Copy link
Contributor Author

Rebased and CI should be fixed now @leandron @areusch.

@areusch areusch added needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it and removed needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it labels Oct 19, 2022
@leandron
Copy link
Contributor

Sorry @PhilippvK. It took me a while to catch up with this. Can you rebase and then we move to get this merged?

@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch 4 times, most recently from c66a3e6 to 6c9c16b Compare November 22, 2022 13:05
@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch from 6c9c16b to ff33ca0 Compare November 30, 2022 10:59
@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch from ff33ca0 to 631f97d Compare December 22, 2022 13:02
@PhilippvK
Copy link
Contributor Author

@leandron Rebased on upstream/main

@areusch
Copy link
Contributor

areusch commented Jan 19, 2023

@tvm-bot rerun

@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch from 631f97d to f11bb22 Compare February 9, 2023 08:05
@PhilippvK
Copy link
Contributor Author

Rebased ontop of main and CI errors seem to be fixed. Can we merge this @areusch @leandron ?

@PhilippvK
Copy link
Contributor Author

Any further comments @areusch @leandron?

@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch from f11bb22 to 1aff401 Compare March 21, 2023 19:08
@PhilippvK
Copy link
Contributor Author

Rebased to fix conflict + I added 2 small AutoTVM related improvements:

  1. If #trials is less then, #tasks, tune each task at least once, instead of failing all of them.
  2. If the config space of a single task is smaller than the determined number of trials per task, use the smaller values as bound in the progress bar.

@PhilippvK PhilippvK force-pushed the feature_tvmc_tune_tasks branch from 1aff401 to 858a722 Compare March 22, 2023 10:01
Copy link
Contributor

@lhutton1 lhutton1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed this PR and think it looks very helpful, thanks! I had a couple of nits around the testing, but given this PR has been rebased a few times I'm happy for them to be fixed in a follow-up, let me know :)

tests/python/driver/tvmc/test_autotuner.py Outdated Show resolved Hide resolved
@@ -207,3 +209,27 @@ def test_autotune_pass_context(mock_pc, onnx_mnist, tmpdir_factory):
# AutoTVM overrides the pass context later in the pipeline to disable AlterOpLayout
assert mock_pc.call_count == 2
assert mock_pc.call_args_list[0][1]["opt_level"] == 3


def test_filter_tasks_valid():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: it would be helpful to parameterize these tests similar to below, so failures are reported separately in the CI log

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops I forgot to address this. Can we merge this anyway?

python/tvm/driver/tvmc/autotuner.py Outdated Show resolved Hide resolved
@lhutton1
Copy link
Contributor

@tvm-bot rerun

@PhilippvK
Copy link
Contributor Author

@lhutton1 Hey, I implemented the proposed changes and the CI passed. Feel free to merge after quickly having a look at the refactored unit tests.

Copy link
Contributor

@lhutton1 lhutton1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @PhilippvK, LGTM!

@lhutton1 lhutton1 merged commit ffc1fc0 into apache:main Mar 30, 2023
@lhutton1
Copy link
Contributor

Thanks @PhilippvK @areusch @leandron!

@PhilippvK PhilippvK deleted the feature_tvmc_tune_tasks branch January 23, 2024 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants