[Fix][MetaSchedule] RPCRunner timeout when queueing up #13963
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.
Generated by tvm-bot |
junrushao
approved these changes
Feb 12, 2023
MasterJH5574
force-pushed
the
tvm-dev/2023-02-11-rpc-runner-timeout-fix
branch
from
e2d19eb to
a2b0143
Compare
|
please fix the lint |
MasterJH5574
force-pushed
the
tvm-dev/2023-02-11-rpc-runner-timeout-fix
branch
2 times, most recently
from
a8d490d to
21e3655
Compare
This PR fixes the timeout rule of MetaSchedule RPCRunner. Prior to this PR, the RPCRunner sets a timeout threshold for jobs submitted to popen pool. As a result, the jobs are timed since the time that they are sent to the remote side. Consider the case where there is only a single device for measurement. In this case, all jobs can only be executed serially and jobs must queue up. Therefore, the previous timeout configuration means the time spent on queueing will also be counted. This causes some jobs, in the worst cases, gets timeout without even started to execute, and has negative impacts on RPC MetaSchedule tuning, from the perspectives of both efficiency and result performance. To address the problem, this PR * removes the RPCRunner timeout requirement for PopenExecutor. Since now, the RPCRunner timeout will only be raised from the RPC server side. * changes the default maximum number of RPCRunner workers to 1, which is to avoid the case where too many jobs waiting in line. Co-authored-by: Bohan Hou <[email protected]>
MasterJH5574
force-pushed
the
tvm-dev/2023-02-11-rpc-runner-timeout-fix
branch
from
21e3655 to
5477802
Compare
|
Note: I reverted the removal of |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the timeout rule of MetaSchedule RPCRunner.
Prior to this PR, the RPCRunner sets a timeout threshold for jobs submitted to popen pool. As a result, the jobs are timed since the time that they are sent to the remote side.
Consider the case where there is only a single device for measurement. In this case, all jobs can only be executed serially and jobs must queue up. Therefore, the previous timeout configuration means the time spent on queueing will also be counted. This causes some jobs, in the worst cases, gets timeout without even started to execute, and has negative impacts on RPC MetaSchedule tuning, from the perspectives of both efficiency and result performance.
To address the problem, this PR
Co-authored-by: Bohan Hou [email protected]