Adds arbitrary server extension support to cugraph-service #2850
Conversation
…to graph creation extensions, changed how unloading extensions work to unload by unique module name, make numpy and cupy optional modules in the client, updated Value and ValueWrapper to handle doubles and lists (recursive), added tests for new extension changes.
rlratzel
added
improvement
…one, added test for extension modules returning results to client GPUs (code not done yet).
…2-cugraph_service_extension_enhancements
…(), updated tests, added test for upcoming ability to load extensions via import paths.
…kage path (eg. import foo.bar.baz), updated test.
…ns, added tests for error conditions, added benchmark for call_extension test.
…2-cugraph_service_extension_enhancements
|
rerun tests reason: gpuCI issue related to conda channel priority in the build image, which may have been resolved by now. |
|
Is this the preferred way of calling cuGraph algorithms, i.e. Basically I'm looking ahead to what #2832 is trying to do, and see if I can use this extension support mechanism to dispatch any cuGraph algorithm. |
I don't think this is the preferred way, but instead a way for clients to just add custom functionality that runs on the server (plugins). I suppose we can use this to easily prototype new functionality too (eg. @VibhuJawa will be using it to add APIs to support heterogeneous graphs more efficiently in the short-term), but for now I'm thinking we'd still want clients to call
You do bring up a good point, and the server could eventually be nothing more than a mechanism to call server-side functions from files on disk which could just make up the entire client API, and the client API could just be hidden behind the Remote classes and algo dispatcher. |
|
Ok, this looks good then, I just want to have one more look-over before I give the final thumbs up. |
…register graphs, added support for returning int8 so graph IDs can be returned easier.
|
rerun tests |
|
rerun tests |
|
@gpucibot merge |
To enable using the same functions on both remote graph and local graph (See PR: #2850) I have reformatted the class to remove dependency from PropertyGraph, cugraph, cudf, dask_cudf CC: @rlratzel , @alexbarghi-nv Authors: - Vibhu Jawa (https://github.com/VibhuJawa) Approvers: - Rick Ratzel (https://github.com/rlratzel) URL: #2855
Add remote CuGraphRemoteStore support to enable using of graph service with DGL . - [x] Transfer via UCX to device - [x] Add node_parquet_data - [x] Add edge_parquet_data - [x] node_storage - [x] edge_storage - [x] node attributes - [x] edge attributes - [x] extract_subgraph - [x] uniform sampling Tests: - [ ] How to run tests in CI !!! - [x] add_node_parquet_data - [x] add_edge_parquet_data - [x] node_storage - [x] edge_storage - [x] node attributes - [x] edge attributes - [x] extract_subgraph - [x] uniform sampling This PR depends upon: #2855 #2850 Authors: - Vibhu Jawa (https://github.com/VibhuJawa) - Rick Ratzel (https://github.com/rlratzel) Approvers: - Alex Barghi (https://github.com/alexbarghi-nv) - Rick Ratzel (https://github.com/rlratzel) URL: #2859
rlratzel
deleted the
branch-22.12-cugraph_service_extension_enhancements
branch
closes #2822
This PR adds the ability for users to add extensions that accept and return arbitrary values that run server-side, in addition to the graph creation extensions already supported.
load_extensions(), andcall_extension()APIsunload_graph_creation_extensions()with the more generalunload_extension_module()call_extension(), and benchmarkdevice=Noneis a host RPC data transfer,device=0is a GPU-GPU data transfer from the server on device 1 to the client on device 0TODO:
Update newcall_extension()API to allow caller to specify a device to directly transfer results toAdd test/example of an extension that accesses a server-side graph