Code

Add all-to-all benchmark#760

Code

Open

pentschev wants to merge 44 commits intorapidsai:branch-0.21rapidsai/ucx-py:branch-0.21from pentschev:all-to-all-benchmarkpentschev/ucx-py:all-to-all-benchmarkCopy head branch name to clipboard

+1,194

Commits

Commits on Mar 8, 2021

• Add multiprocess test to perform all-to-all ep creation

  pentschev

  committed
  3c0a12a

  Copy full SHA for 3c0a12a

Commits on Mar 10, 2021

• Add support for persistent endpoints in multiprocess all_to_all test

  pentschev

  committed
  35664da

  Copy full SHA for 35664da

• Improve closing of endpoints in multiproc all_to_all test

  pentschev

  committed
  65f1870

  Copy full SHA for 65f1870

• Do more transfers between worker pairs

  pentschev

  committed
  ba66c92

  Copy full SHA for ba66c92

• Support multiple endpoints per worker in multiproc all_to_all test

  pentschev

  committed
  b712ce9

  Copy full SHA for b712ce9

Commits on Jun 9, 2021

• Merge remote-tracking branch 'upstream/branch-0.21' into all-to-all-test

  pentschev

  committed
  e8fee54

  Copy full SHA for e8fee54

• Remove monitor process in favor of multiprocessing shared memory

  pentschev

  committed
  aa06731

  Copy full SHA for aa06731

Commits on Jul 1, 2021

• Merge remote-tracking branch 'upstream/branch-0.21' into all-to-all-test

  pentschev

  committed
  ce98f43

  Copy full SHA for ce98f43

• Mark some multiple processes all-to-all tests as slow

  pentschev

  committed
  68c07a6

  Copy full SHA for 68c07a6

• Store remote EP port in dictionary

  pentschev

  committed
  c91ec3c

  Copy full SHA for c91ec3c

• Mark more tests as slow

  pentschev

  committed
  33b630d

  Copy full SHA for 33b630d

• Benchmark for multiprocess all-to-all

  pentschev

  committed
  4f41ebe

  Copy full SHA for 4f41ebe

Commits on Jul 5, 2021

• Add Tornado-based all-to-all benchmark

  pentschev

  committed
  b6508ed

  Copy full SHA for b6508ed

Commits on Jul 6, 2021

• Change all-to-all worker function into class

  pentschev

  committed
  ac88060

  Copy full SHA for ac88060

• Use BaseWorker class to create TornadoWorker all-to-all benchmark

  pentschev

  committed
  4c485c2

  Copy full SHA for 4c485c2

• Add asyncio all-to-all benchmark

  pentschev

  committed
  6ddcf04

  Copy full SHA for 6ddcf04

• Remove non-persistent endpoints

  pentschev

  committed
  9675844

  Copy full SHA for 9675844

Commits on Jul 7, 2021

• Split UCX transfers in frames to make it similar to Tornado/Asyncio

  pentschev

  committed
  c8fb2a7

  Copy full SHA for c8fb2a7

• Move serialization to comms send method

  pentschev

  committed
  bf48ebd

  Copy full SHA for bf48ebd

Commits on Jul 8, 2021

• Add monitor process to allow syncing workers over network

  pentschev

  committed
  7d2eaff

  Copy full SHA for 7d2eaff

• Split run into multiple methods

  pentschev

  committed
  8ae422a

  Copy full SHA for 8ae422a

• Parametrize communication and monitor enable/disable

  pentschev

  committed
  da2320e

  Copy full SHA for da2320e

• Remove worker_num argument

  pentschev

  committed
  e21a973

  Copy full SHA for e21a973

• Make shm synchronization arguments optional

  pentschev

  committed
  6c7de45

  Copy full SHA for 6c7de45

• Add listener_address argument

  pentschev

  committed
  1bdc4d2

  Copy full SHA for 1bdc4d2

Commits on Jul 9, 2021

• Separate benchmark into multiple files

  pentschev

  committed
  da75bc1

  Copy full SHA for da75bc1

• Improve benchmark result formatting

  pentschev

  committed
  9202310

  Copy full SHA for 9202310

• Reorganize run_worker and run_monitor

  pentschev

  committed
  dba6dbf

  Copy full SHA for dba6dbf

• Fix single-node benchmark

  pentschev

  committed
  26c3712

  Copy full SHA for 26c3712

• Add more all-to-all benchmark CLI arguments

  pentschev

  committed
  199f960

  Copy full SHA for 199f960

• Remove hardcoded enp1s0f0 interface

  pentschev

  committed
  49b194f

  Copy full SHA for 49b194f

• Remove redundant --multi-node argument

  pentschev

  committed
  796880f

  Copy full SHA for 796880f

• Improve documentation for all-to-all benchmark CLI arguments

  pentschev

  committed
  cc7961b

  Copy full SHA for cc7961b

• Fix all-to-all benchmark test

  pentschev

  committed
  18a4557

  Copy full SHA for 18a4557

• Change worker address formatting in benchmark output

  pentschev

  committed
  58a30f8

  Copy full SHA for 58a30f8

• Add --port argument to specify monitor port

  pentschev

  committed
  5a509e3

  Copy full SHA for 5a509e3

• Fix issues with all-to-all pytest

  pentschev

  committed
  75fca3d

  Copy full SHA for 75fca3d

Commits on Jul 12, 2021

• Remove usage of partial

  Show description for b1b0b1b
  pentschev

  committed
  b1b0b1b

  Copy full SHA for b1b0b1b

Commits on Jul 13, 2021

• Add all-to-all benchmark with uvloop

  pentschev

  committed
  d3e835d

  Copy full SHA for d3e835d

• Add uvloop support for multi-node benchmark

  pentschev

  committed
  bd86b11

  Copy full SHA for bd86b11

• Fix bandwidth calculation

  pentschev

  committed
  a96358c

  Copy full SHA for a96358c

• Parse bytes with --size

  pentschev

  committed
  b3bdfb0

  Copy full SHA for b3bdfb0

• Skip uvloop test when not installed

  pentschev

  committed
  29468b4

  Copy full SHA for 29468b4

Commits on Aug 6, 2021

• Raise when number of workers don't match

  pentschev

  committed
  14010ac

  Copy full SHA for 14010ac