Add initial MNMG support, consolidate APIs, minor refactoring, update tests #24

rlratzel · 2022-08-18T13:09:48Z

closes #13
closes #6 (SG was closed by #18)
closes #5
closes #3
closes #2

Added ability to pass in a dask scheduler configuration and have the GaaS server use it to create and work with MG graph data and automatically call MG algos. Extensions which load SG graphs are supported simultaneously.
Removed the get_num_vertices() and get_num_edges() APIs in favor of get_graph_info()
Added get_server_info() which returns meta-data about the GaaS server, useful for test and debug
Added a "server facade" object which is passed to extension functions that can be used to query server state without giving extensions direct access to sensitive server APIs. This is initially useful for allowing extensions to query if the server is MG-capable or not in order to determine how to load data (using dask_cudf or not)
Added utilities for converting to/from Thrift union types
Removed taxpayers test data which is now considered invalid input
Updates to support latest PropertyGraph updates
Added and updated tests accordingly
Removed get_graph_dataframe_rows() and replaced them with get_graph_data()
Added MG and SG tests to cover different ID types (exposed after a typo went unchecked)
Added MG and SG tests to cover get_graph_*_data()

NOTES:

This PR also includes commits from recent 22.08 branch updates, and will bring the 22.10 branch up-to-date with the latest development.
This PR is too large, and future PRs like this should instead be broken into several smaller PRs.
This PR replaces Add ability to use a dask cluster and internally instantiate MG objects when in an MG environment #17

… cluster.

…8-mg_updates

…rghi-nv/gaas into uniform_neighbor_sampling

…' into branch-22.08-mg_updates Added updates for initial support for MGPropertyGraph and additional tests

…s and updates for shape methods for MG.

… consistency, moved graph_id arg to end for uniform_neighbor_sampling for consistency, added tests for info APIs, temporarily disabled vertex shape test while discussing if it is needed or not.

…ensuring it can support the use cases for the old shape() APIs.

…ich are obsoleted by get_graph_info(), removed now invalide taxpayers data, updated tests for these changes.

…' into branch-22.10-mg_updates

Co-authored-by: Rick Ratzel <[email protected]>

…mpling Updates to GaaS for Uniform Neighbor Sampling

…0-mg_updates

…e optional, updated adn added tests accordingly.

alexbarghi-nv

Looks good, just proposed a few minor fixes

python/gaas_server/gaas_handler.py

python/gaas_client/types.py

python/gaas_server/gaas_handler.py

…_*_data(). Added MG and SG tests to cover different ID types (exposed after a typo went unchecked). Added MG and SG tests to cover get_graph_*_data(). Minor cleanup and refactorings.

VibhuJawa

Have given the PR an initial review and asked a bunch of clarifications. Will review again based on the answers to these questions.

python/gaas_client/gaas_thrift.py

scripts/run-dask-process.sh

scripts/functions.sh

python/tests/test_mg_gaas_handler.py

alexbarghi-nv

👍

python/gaas_server/gaas_handler.py

…ts, added FIXMEs.

VibhuJawa

LGTM

…o the cugraph repo (#2661) closes #2652 **_Note to reviewers:_** > The code here is based on the latest 22.10 GaaS sources which include [this last PR to GaaS](rapidsai/GaaS#24), and has only been changed to update names ("gaas" -> "cugraph_service") and pass style checks, but the diffs will obviously show everything as new content. > > In other words, don't feel obligated to review anything beyond name changes if you've already reviewed the GaaS sources. > > Future changes that update functionality related to this move will be done in separate PRs so the diff can clearly show what's being changed. This is the initial PR which only moves source files and changes names in sources, docs, directories, etc. The unit tests were run (SG and MG) after the name change and they all pass. One or more followup PRs will be open to add appropriate support for CI (ie. no MG tests run by default, limit to handler tests to simplify automating server subprocesses if necessary, etc.). The entire GaaS repo is now under `python/cugraph_service` as a peer to `cugraph` and `pylibcugraph`. Authors: - Rick Ratzel (https://github.com/rlratzel) - Alex Barghi (https://github.com/alexbarghi-nv) - Brad Rees (https://github.com/BradReesWork) Approvers: - Brad Rees (https://github.com/BradReesWork) - Vibhu Jawa (https://github.com/VibhuJawa) URL: #2661

rlratzel and others added 23 commits June 16, 2022 18:58

Added initial tests and CLI options for having the handler use a dask…

66ceb0c

… cluster.

Merge remote-tracking branch 'upstream/branch-22.08' into branch-22.0…

42b03e1

…8-mg_updates

uniform neighbor sampling

9779388

Delete .nfs0000000001128f2f00000e4d

51e93c5

add renumber=True to from_cudf_edgelist call

19f0b30

Merge branch 'uniform_neighbor_sampling' of https://github.com/alexba…

61fe30e

…rghi-nv/gaas into uniform_neighbor_sampling

graphsage updates

7bd0927

Merge remote-tracking branch 'alexbarghi-nv/uniform_neighbor_sampling…

a999733

…' into branch-22.08-mg_updates Added updates for initial support for MGPropertyGraph and additional tests

Updates to handle closing dask client connection properly, added test…

46f08da

…s and updates for shape methods for MG.

Removed shape APIs and replaced with info APIs, reordered methods for…

de4dcec

… consistency, moved graph_id arg to end for uniform_neighbor_sampling for consistency, added tests for info APIs, temporarily disabled vertex shape test while discussing if it is needed or not.

Added code and tests for supporting the new get_graph_info() API and …

c1d1327

…ensuring it can support the use cases for the old shape() APIs.

resolve conflict

f0dcc47

fix neighbor sampling to work with current api

76c6087

Updated graph_info API to use new keys consistent with PG updates.

21aa53a

Updates to accommodate latest PG changes, removed get_num_*() APIs wh…

fc280b3

…ich are obsoleted by get_graph_info(), removed now invalide taxpayers data, updated tests for these changes.

Started new test for uniform_neighbor_sampling.

00712af

Merge remote-tracking branch 'alexbarghi-nv/uniform_neighbor_sampling…

fdd5a05

…' into branch-22.10-mg_updates

Update python/gaas_server/gaas_handler.py

94ee72e

Co-authored-by: Rick Ratzel <[email protected]>

remove unused imports

ea7c307

Merge pull request rapidsai#18 from alexbarghi-nv/uniform_neighbor_sa…

a34c43c

…mpling Updates to GaaS for Uniform Neighbor Sampling

Updated SG uniform_neighbor_sampling test to add a passing case.

a156199

Merge remote-tracking branch 'upstream/branch-22.08' into branch-22.1…

55abf5d

…0-mg_updates

Initial working version of MG support for uniform_neighbor_sampling.

f0700aa

rlratzel added breaking Introduces a breaking change improvement Improves an existing functionality labels Aug 18, 2022

rlratzel added this to the 22.10 milestone Aug 18, 2022

rlratzel requested a review from alexbarghi-nv August 18, 2022 13:09

rlratzel self-assigned this Aug 18, 2022

rlratzel mentioned this pull request Aug 18, 2022

Add ability to use a dask cluster and internally instantiate MG objects when in an MG environment #17

Closed

rlratzel linked an issue Aug 18, 2022 that may be closed by this pull request

Initial GaaS working version #2

Closed

9 tasks

This was linked to issues Aug 18, 2022

Need ability to start dask cluster for MG APIs #3

Closed

Add support for using an MG Graph/PropertyGraph #5

Closed

Add Uniform Neighbor Sampling API #6

Closed

[EPIC] GaaS should be able to handle all graph types - even simultaneously #13

Closed

rlratzel requested a review from VibhuJawa August 18, 2022 13:13

Added code to allow the server facade arg to extension functions to b…

6151b58

…e optional, updated adn added tests accordingly.

alexbarghi-nv suggested changes Aug 26, 2022

View reviewed changes

Removed get_graph_*_dataframe_rows() and replaced them with get_graph…

99a1370

…_*_data(). Added MG and SG tests to cover different ID types (exposed after a typo went unchecked). Added MG and SG tests to cover get_graph_*_data(). Minor cleanup and refactorings.

VibhuJawa suggested changes Aug 31, 2022

View reviewed changes

alexbarghi-nv approved these changes Aug 31, 2022

View reviewed changes

python/gaas_server/gaas_handler.py Show resolved Hide resolved

rlratzel added 2 commits August 31, 2022 22:26

Updates from review: copyright update, added args to dask shell scrip…

a550054

…ts, added FIXMEs.

Formatting.

67fcef9

rlratzel requested a review from VibhuJawa September 1, 2022 03:30

VibhuJawa approved these changes Sep 2, 2022

View reviewed changes

rlratzel merged commit ad84042 into rapidsai:branch-22.10 Sep 2, 2022

rlratzel mentioned this pull request Sep 8, 2022

Update MG PG for use with GaaS rapidsai/cugraph-pg#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial MNMG support, consolidate APIs, minor refactoring, update tests #24

Add initial MNMG support, consolidate APIs, minor refactoring, update tests #24

rlratzel commented Aug 18, 2022 •

edited

Loading

alexbarghi-nv left a comment

VibhuJawa left a comment

alexbarghi-nv left a comment

VibhuJawa left a comment

Add initial MNMG support, consolidate APIs, minor refactoring, update tests #24

Add initial MNMG support, consolidate APIs, minor refactoring, update tests #24

Conversation

rlratzel commented Aug 18, 2022 • edited Loading

alexbarghi-nv left a comment

Choose a reason for hiding this comment

VibhuJawa left a comment

Choose a reason for hiding this comment

alexbarghi-nv left a comment

Choose a reason for hiding this comment

VibhuJawa left a comment

Choose a reason for hiding this comment

rlratzel commented Aug 18, 2022 •

edited

Loading