optimize trace hang && fix event leak #58707
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gongweibao
reviewed
Nov 10, 2023
| // convert vector to string, concatenate continuous intervals with `:`, | ||
| // concatenate discontinuous intervals with `#` eg: [1,2,3,4,5,7,8,9] => | ||
| // 1:3#4#5#7:9 | ||
| inline std::string VectorToString(const std::vector<int>& vec) { |
| ",seq:" + std::to_string(seq_) + | ||
| ",started:" + std::to_string(IsStarted()) + | ||
| ",completed:" + std::to_string(IsCompleted()) + | ||
| auto global_ranks = |
There was a problem hiding this comment.
Add unit test of it and test the limit of the Msg length.
| @@ -484,6 +484,7 @@ class ProcessGroup { | |||
| } | |||
|
|
|||
| protected: | |||
| int global_rank_; | |||
| @@ -860,6 +862,44 @@ void ProcessGroupNCCL::CreateNCCLEnvCache(const Place& place, | |||
| auto comm_ctx = std::make_unique<phi::GPUContext>(place); | |||
| comm_ctx->set_nccl_comm(nccl_comm); | |||
|
|
|||
| // gather global ranks in current group | |||
| int* gpu_global_rank = nullptr; | |||
There was a problem hiding this comment.
- Use tensor instead of raw data since cuda memory APIs are not efficient?
- If use raw data, check the result of CUDA API.
There was a problem hiding this comment.
- Use tensor instead of raw data since cuda memory APIs are not efficient?
- If use raw data, check the result of CUDA API.
fixed
remove useless log
hitywt
force-pushed
the
fix_trace_event
branch
from
d96f29b to
0d8bf53
Compare
hitywt
force-pushed
the
fix_trace_event
branch
from
0a49939 to
5171f79
Compare
hitywt
force-pushed
the
fix_trace_event
branch
from
5171f79 to
a3a9e47
Compare
hitywt
changed the title
hitywt
pushed a commit
to hitywt/Paddle
that referenced
this pull request
Nov 21, 2023
hitywt
pushed a commit
to hitywt/Paddle
that referenced
this pull request
Nov 24, 2023
ForFishes
added a commit
that referenced
this pull request
Nov 28, 2023
* add comm async trace module, (#56916) * Fix trace hang (#57536) * fix trace hang * fix compile error * fix code style * tinyfix * tiny update * fix code style --------- Co-authored-by: ForFishes <[email protected]> * Fix nccl trace (#58338) * fix nccl_async_trace destruct problem when train finished * update * format code style * optimize trace hang && fix event leak (#58707) * update * fix compile problems * fix code style * fix logging * fix code style * remove useless * add ut && tinyfix * opt cudaMalloc and cudaMemcpy update * tinyfix --------- Co-authored-by: ForFishes <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Description