Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Group task scheduling optimization #8768

Closed
wants to merge 6 commits into from
Closed

Group task scheduling optimization #8768

wants to merge 6 commits into from

Conversation

peter198
Copy link
Contributor

Introduction:
Group task scheduling optimization, adding a limit on the number of PreCommit1 tasks for workers

Statement of needs:
In the original scheduling, there is no limit to the number of workers running PreCommit1 at the same time, and the maximum task configuration size includes the number of CPU cores, available memory, and hard disk space.
There is no reasonable limit. Although workers can execute when they receive too many tasks, they cannot fully exert their computing performance. The parallel PreCommit1 tasks can be reasonably controlled according to the hardware configuration.
Can improve the overall task completion speed.

improve proposals:
When the worker starts, the --parallel-p1-limit parameter sets the maximum PreCommit1 task it can accept. It will be recorded when the miner schedules the PreCommit1 task to the worker
Quantity allocated to it When the number of tasks already allocated reaches the set maximum number of tasks, it will not be allocated to a new task until 1 task is completed, and the allocation will be submitted in advance.

The default value of the --parallel-p1-limit parameter is -1, which means no limit, and if --parallel-p1-limit--parallel-p1-limit--parallel-p1-limit exceeds all the values ​​​​of the hardware configuration of the on-duty worker maximum reception time of
won't work either.
When viewing worker information through lotus-miner sealed workers, the task number information of PreCommit1 will be displayed, as follows:
p1_limit: 15-7, 15 represents the value set by --parallel-p1-limit, and 7 represents the number of PreCommit1 tasks currently running.

@peter198 peter198 requested a review from a team as a code owner May 31, 2022 08:30
@magik6k
Copy link
Contributor

magik6k commented May 31, 2022

This was already merged - #8688, with some changes in #8725 - the limit is set through an env var like PC1_32G_MAX_CONCURRENT=10, and it supports all task types

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants