Code

Improve MG PageRank scalability#2038

Code

Merged

rapids-bot[bot] merged 85 commits intorapidsai:branch-22.04rapidsai/cugraph:branch-22.04from seunghwak:enh_mg_pagerank2seunghwak/cugraph:enh_mg_pagerank2Copy head branch name to clipboardMar 14, 2022

+219-203

Commits

Commits on Jan 26, 2022

• enable multi-stream execution and overlapping communication with computation (currently with the temporary mechanism to support stream priorities, eventually, rmm should be updated to support this)

  seunghwak

  committed
  092cf49

  Copy full SHA for 092cf49

Commits on Jan 27, 2022

• update group_by_and_count to not use reduce_by_key (which is expensive and also seems like having an issue with 2^31 or more elements)

  seunghwak

  committed
  6102cd1

  Copy full SHA for 6102cd1

• add time measurements (should be undone)

  seunghwak

  committed
  5146b19

  Copy full SHA for 5146b19

• Merge branch 'upstream_pr2044' into enh_mg_pagerank2

  seunghwak

  committed
  9d31282

  Copy full SHA for 9d31282

Commits on Jan 31, 2022

• cosmetic updates

  seunghwak

  committed
  965f0cd

  Copy full SHA for 965f0cd

Commits on Feb 1, 2022

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into enh_graph_creation

  seunghwak

  committed
  5073304

  Copy full SHA for 5073304

• improve weak scaling behavior of renumber

  seunghwak

  committed
  7c02fcf

  Copy full SHA for 7c02fcf

• move is_first_in_run_t to graph_utils.cuh

  seunghwak

  committed
  0744160

  Copy full SHA for 0744160

• avoid using device lambdas

  seunghwak

  committed
  077008b

  Copy full SHA for 077008b

• Merge branch 'upstream_pr2044' into enh_mg_pagerank2

  seunghwak

  committed
  e0608bb

  Copy full SHA for e0608bb

• fix compile errors

  seunghwak

  committed
  6a0dfa1

  Copy full SHA for 6a0dfa1

• Merge branch 'upstream_pr2044' into enh_mg_pagerank2

  seunghwak

  committed
  f9a9635

  Copy full SHA for f9a9635

Commits on Feb 2, 2022

• code cleanup

  seunghwak

  committed
  41645aa

  Copy full SHA for 41645aa

• Merge branch 'upstream_pr2044' into enh_mg_pagerank2

  seunghwak

  committed
  9bc0284

  Copy full SHA for 9bc0284

• update copy_v_transform_reduce_in_out_nbr to process multiple edge partitions in parallel

  seunghwak

  committed
  6b4b682

  Copy full SHA for 6b4b682

• fix overflow bug with 2^31 or more vertices

  seunghwak

  committed
  a0b009e

  Copy full SHA for a0b009e

• Merge branch 'upstream_pr2044' into enh_mg_pagerank2

  seunghwak

  committed
  c889302

  Copy full SHA for c889302

Commits on Feb 3, 2022

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into enh_graph_creation

  seunghwak

  committed
  b5cb9c4

  Copy full SHA for b5cb9c4

• delete temporary performance measurement code

  seunghwak

  committed
  9a70472

  Copy full SHA for 9a70472

• delete additional temporary performance measurement code

  seunghwak

  committed
  214ada9

  Copy full SHA for 214ada9

• remove temporary performance measurement code

  seunghwak

  committed
  3a605b5

  Copy full SHA for 3a605b5

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into enh_graph_creation

  seunghwak

  committed
  c537e2d

  Copy full SHA for c537e2d

• clang-format & copyright year

  seunghwak

  committed
  4852ce4

  Copy full SHA for 4852ce4

Commits on Feb 4, 2022

• add temporary performance measurement code

  seunghwak

  committed
  96e9693

  Copy full SHA for 96e9693

• Merge branch 'upstream_pr2044' into enh_mg_pagerank2

  seunghwak

  committed
  a31050a

  Copy full SHA for a31050a

• add temporary performance measurement code to PageRank implementation

  seunghwak

  committed
  e8769d3

  Copy full SHA for e8769d3

• add more performance measurements to MG PageRank test

  seunghwak

  committed
  dc6acef

  Copy full SHA for dc6acef

• add more experimental code (should be cleaned-up before merge)

  seunghwak

  committed
  8136b77

  Copy full SHA for 8136b77

Commits on Feb 8, 2022

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into enh_mg_pagerank2

  seunghwak

  committed
  1e74ddb

  Copy full SHA for 1e74ddb

Commits on Feb 9, 2022

• resolve merge conflicts

  seunghwak

  committed
  3b426be

  Copy full SHA for 3b426be

Commits on Feb 10, 2022

• remove some temporary code

  seunghwak

  committed
  f28ccfa

  Copy full SHA for f28ccfa

• undo some temporary fix

  seunghwak

  committed
  1839a61

  Copy full SHA for 1839a61

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into enh_mg_pagerank2

  seunghwak

  committed
  940ead8

  Copy full SHA for 940ead8

• remove host_barrier (no longer used)

  seunghwak

  committed
  7cb1f03

  Copy full SHA for 7cb1f03

Commits on Feb 11, 2022

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into enh_mg_pagerank2

  seunghwak

  committed
  ff5e315

  Copy full SHA for ff5e315

Commits on Feb 12, 2022

• reduce temporary memory requirement in R-mat edge list generation

  seunghwak

  committed
  eefe729

  Copy full SHA for eefe729

• input parameter renaming for clarity

  seunghwak

  committed
  167b5ab

  Copy full SHA for 167b5ab

Commits on Feb 13, 2022

• add a heuristic to cut peak memory footprint

  seunghwak

  committed
  f06f32b

  Copy full SHA for f06f32b

• improve inconsistencies in naming

  seunghwak

  committed
  6dceeab

  Copy full SHA for 6dceeab

Commits on Feb 15, 2022

• split groupby_and_shuffle_edgelist to groupby_and_count and shuffle_edgelist to pass mem_frugal=true to limit the maximum allocation chunk size to avoid malloc failure due to fragmentation with the…

  Show description for febd72d
  seunghwak

  committed
  febd72d

  Copy full SHA for febd72d

Commits on Feb 16, 2022

• use temporary host buffer to concatenate edge list in edge generation

  seunghwak

  committed
  48fa255

  Copy full SHA for 48fa255

• update groupby to take mem_frugal_threshold instead of bool mem_frugal

  seunghwak

  committed
  234a119

  Copy full SHA for 234a119

• move counting unique local edge majors/mionrs inside the graph constructor to limit peak memory usage

  seunghwak

  committed
  95f2295

  Copy full SHA for 95f2295

• clang-format

  seunghwak

  committed
  15aeb4c

  Copy full SHA for 15aeb4c

• copyright year

  seunghwak

  committed
  e7102e0

  Copy full SHA for e7102e0

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into enh_mem_footprint

  seunghwak

  committed
  d8c5178

  Copy full SHA for d8c5178

• clang-format

  seunghwak

  committed
  7b544a6

  Copy full SHA for 7b544a6

Commits on Feb 17, 2022

• cosmetic updates

  seunghwak

  committed
  6b40187

  Copy full SHA for 6b40187

• update louvain tests to take --perf option

  seunghwak

  committed
  85f39fb

  Copy full SHA for 85f39fb

• fix error in comments

  seunghwak

  committed
  48009a6

  Copy full SHA for 48009a6

• add is_first_in_run_pair_t to graph_utils.cuh

  seunghwak

  committed
  59d16a3

  Copy full SHA for 59d16a3

• add R-mat symmetric cases to coarsen_graph tests

  seunghwak

  committed
  b7d9570

  Copy full SHA for b7d9570

• if the input grpah is symmetric, update coarsen_grpah to work on the lower triangular part only and symmetrize to avoid (slightly) asymmetric edge weights due to the limited floating point precisions

  seunghwak

  committed
  0a0c6c1

  Copy full SHA for 0a0c6c1

• clang-format

  seunghwak

  committed
  86f1d11

  Copy full SHA for 86f1d11

• fix comments

  seunghwak

  committed
  75d6f81

  Copy full SHA for 75d6f81

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into bug_shrink_graph

  seunghwak

  committed
  cf48403

  Copy full SHA for cf48403

• Merge branch 'upstream_pr2070' into enh_mg_pagerank2

  seunghwak

  committed
  ef13c32

  Copy full SHA for ef13c32

• fix to make merge with enh_mg_louvain PR easier

  seunghwak

  committed
  29e7b84

  Copy full SHA for 29e7b84

• copyright

  seunghwak

  committed
  57a0843

  Copy full SHA for 57a0843

• resolve merge conflicts

  seunghwak

  committed
  28b21ef

  Copy full SHA for 28b21ef

• Merge branch 'upstream_pr2070' into enh_mem_footprint_part2

  seunghwak

  committed
  6cc6a03

  Copy full SHA for 6cc6a03

• resolve merge conflicts

  seunghwak

  committed
  63397c8

  Copy full SHA for 63397c8

Commits on Feb 18, 2022

• remove one more memory footprint bottleneck in graph object creation

  seunghwak

  committed
  14c47d4

  Copy full SHA for 14c47d4

• add graph constructors taking edge list as R-value (so they can be destroyed as soon as they are no longer necessary to reduce peak memory usage)

  seunghwak

  committed
  550576b

  Copy full SHA for 550576b

• cosmetic fix

  seunghwak

  committed
  09f3072

  Copy full SHA for 09f3072

• update coarsen_graph return type

  seunghwak

  committed
  2b9d37d

  Copy full SHA for 2b9d37d

• refactor graph constructors and implement graph constructors taking edge list in R-values

  seunghwak

  committed
  35fc8d1

  Copy full SHA for 35fc8d1

• update create_grpah_from_edgelist to call graph_t taking R-value edgelists

  seunghwak

  committed
  dc4a213

  Copy full SHA for dc4a213

• update coarsen_graph_impl to call create_graph_from_edgelist instead of directly calling graph_t constructor

  seunghwak

  committed
  5177e86

  Copy full SHA for 5177e86

Commits on Feb 19, 2022

• additional (potential) parallelism vs peak memory trade-off in edge shuffling

  seunghwak

  committed
  3bdb95e

  Copy full SHA for 3bdb95e

• bug fix

  seunghwak

  committed
  edd7cdd

  Copy full SHA for edd7cdd

• Merge branch 'upstream_pr2070' into enh_mem_footprint_part2

  seunghwak

  committed
  ea3736c

  Copy full SHA for ea3736c

Commits on Feb 20, 2022

• fix for a possible hang

  seunghwak

  committed
  ff12636

  Copy full SHA for ff12636

Commits on Feb 22, 2022

• Merge branch 'branch-22.04' of github.com:rapidsai/cugraph into enh_mg_pagerank2

  seunghwak

  committed
  56764c4

  Copy full SHA for 56764c4

• Merge branch 'upstream_pr2081' into enh_mg_pagerank2

  seunghwak

  committed
  c369431

  Copy full SHA for c369431

Commits on Feb 23, 2022

• resolve merge conflicts

  seunghwak

  committed
  1692a46

  Copy full SHA for 1692a46

Commits on Feb 24, 2022

• added temporary code to experiment performance

  seunghwak

  committed
  a34ad17

  Copy full SHA for a34ad17

• additional cut in peak memory and maximum single allocation size (to avoid malloc failure due to fragmentation with the pool allocator)

  seunghwak

  committed
  b1e56e7

  Copy full SHA for b1e56e7

• Merge branch 'upstream_pr2081' into enh_mg_pagerank2

  seunghwak

  committed
  885f28a

  Copy full SHA for 885f28a

Commits on Mar 9, 2022

• resolve merge conflicts

  seunghwak

  committed
  b4bbb9d

  Copy full SHA for b4bbb9d

• remove temporary experimental code

  seunghwak

  committed
  d716f6c

  Copy full SHA for d716f6c

• remove temporary experimental code

  seunghwak

  committed
  99401f3

  Copy full SHA for 99401f3

• fix formatting error

  seunghwak

  committed
  0664216

  Copy full SHA for 0664216

• undo copyright update with the file with no change

  seunghwak

  committed
  c136d3a

  Copy full SHA for c136d3a

• clang-format & copyright year

  seunghwak

  committed
  a4f6528

  Copy full SHA for a4f6528