[REVIEW] python 2D shuffling #1133
Conversation
|
Please update the changelog in order to start CI tests. View the gpuCI docs here. |
| def get_2D_div(ngpus): | ||
| pcols = int(math.sqrt(ngpus)) | ||
| while ngpus % pcols != 0: | ||
| pcols = pcols - 1 | ||
| return int(ngpus/pcols), pcols |
There was a problem hiding this comment.
Python code to find prows and pcols
Iroy30
changed the title
python/cugraph/structure/shuffle.py
Outdated
| return partitions | ||
|
|
||
|
|
||
| def shuffle(dg): |
There was a problem hiding this comment.
Suggest adding prows and pcols as parameters to the shuffle function, defaulting to None.
One of the easiest tuning things we want experienced users to be able to do is to configure prows and pcols, and it seems like this would be an easy way to do it.
If prows and pcols are specified then we should use them, otherwise we should compute them. If a user specifies one then the other should be computable (ngpus / 'whichever one is specified'). If user specifies neither then we can compute as the current code does.
python/cugraph/structure/shuffle.py
Outdated
| def shuffle(dg): | ||
| ddf = dg.edgelist.edgelist_df | ||
| ngpus = get_n_workers() | ||
| prows, pcols = get_2D_div(ngpus) |
There was a problem hiding this comment.
Should probably add a check that prows * pcols = ngpus. The implementation of get_2D_div guarantees that, but if the user specifies a value to tune things that isn't guaranteed.
|
|
||
| def get_2D_div(ngpus): | ||
| pcols = int(math.sqrt(ngpus)) | ||
| while ngpus % pcols != 0: |
There was a problem hiding this comment.
Too lazy today to sit down and do math, but we should be able to directly compute this without a while loop.
python/cugraph/structure/shuffle.py
Outdated
| def _set_partitions_pre(df, num_verts, prows, pcols): | ||
| rows_per_div = math.ceil(num_verts/prows) | ||
| cols_per_div = math.ceil(num_verts/pcols) | ||
| partitions = df['src'].floordiv(rows_per_div) * pcols + df['dst'].floordiv(cols_per_div) |
There was a problem hiding this comment.
Can we do this? If I am not mistaken, we apply a hash function to edge source & destination vertices to determine the partition each edge belongs to. So, each partition will have roughly same number of rows or columns but they will not be same (and can't be determined in this way). Am I missing something?
|
rerun tests |
|
rerun tests |
|
rerun tests |
python/cugraph/structure/shuffle.py
Outdated
| return partitions | ||
|
|
||
|
|
||
| def shuffle(dg): |
There was a problem hiding this comment.
A few documentation lines would be a good addition
…into branch-0.16
…into branch-0.16
Codecov Report
@@ Coverage Diff @@
## branch-0.16 #1133 +/- ##
===============================================
- Coverage 73.44% 72.19% -1.26%
===============================================
Files 60 61 +1
Lines 2335 2406 +71
===============================================
+ Hits 1715 1737 +22
- Misses 620 669 +49
Continue to review full report at Codecov.
|
No description provided.