Skip to content

Commit

Permalink
[CELEBORN-1812] Distinguish sorting-file from sort-tasks waiting to b…
Browse files Browse the repository at this point in the history
…e submitted

### What changes were proposed in this pull request?

Current implementation uses `
shuffleSortTaskDeque.size()` as current sorting file count.This value might be more appropriately described as the sort tasks waiting to be submitted to `fileSorterExecutors`. And the actual current sorting file number ( doing some disk-io operation etc) should be get from `sortingShuffleFiles`.

### Why are the changes needed?

Add metrics to monitor current sorting files which is making disk-io operations.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

![image](https://github.com/user-attachments/assets/6ffed37e-ad12-4d8d-a4aa-2b2695a92168)

Closes #3040 from Z1Wu/fix/sorting_file_metrics.

Authored-by: wuziyi <[email protected]>
Signed-off-by: Shuang <[email protected]>
  • Loading branch information
Z1Wu authored and RexXiong committed Jan 4, 2025
1 parent 4ccb0c7 commit f886751
Show file tree
Hide file tree
Showing 5 changed files with 635 additions and 528 deletions.
Loading

0 comments on commit f886751

Please sign in to comment.