Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk Loading Support for cuGraph-PyG #3170

Merged
merged 509 commits into from
Jan 31, 2023
Merged
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
509 commits
Select commit Hold shift + click to select a range
d47539b
address most PR comments
ChuckHastings Jan 5, 2023
cd07a11
fixes
alexbarghi-nv Jan 6, 2023
035498e
merge
alexbarghi-nv Jan 6, 2023
6d0f7f0
do merge
alexbarghi-nv Jan 6, 2023
87bb91b
merge offsets
alexbarghi-nv Jan 6, 2023
65dc4df
remove test code
alexbarghi-nv Jan 6, 2023
456ab7f
type annotations
alexbarghi-nv Jan 6, 2023
8d6da9c
finished addressing PR comments
ChuckHastings Jan 6, 2023
f4cb88e
remove unused import
alexbarghi-nv Jan 6, 2023
4420751
update copyright
alexbarghi-nv Jan 6, 2023
8b5be47
update copyright year
alexbarghi-nv Jan 6, 2023
4ef2b2b
update copyright year
alexbarghi-nv Jan 6, 2023
d1079ea
update copyright year
alexbarghi-nv Jan 6, 2023
f715d14
update copyright year
alexbarghi-nv Jan 6, 2023
694b23e
update copyright year
alexbarghi-nv Jan 6, 2023
8179b1f
update copyright year
alexbarghi-nv Jan 6, 2023
bc6974e
update copyright year
alexbarghi-nv Jan 6, 2023
79abf44
update copyright year
alexbarghi-nv Jan 6, 2023
7af9b69
remove unused import
alexbarghi-nv Jan 6, 2023
db036ca
do merge
alexbarghi-nv Jan 9, 2023
2646563
update c++ code
alexbarghi-nv Jan 10, 2023
eec3510
Merge branch 'branch-23.02' into update_uniform_sampling_api
ChuckHastings Jan 11, 2023
472f090
fix typing
alexbarghi-nv Jan 11, 2023
f04e00e
resolve merge conflict
alexbarghi-nv Jan 11, 2023
52b2a4d
fix array construction
alexbarghi-nv Jan 11, 2023
cbd5c70
Merge branch 'short-circuit-and-type-indices' of https://github.com/a…
alexbarghi-nv Jan 11, 2023
ebb2014
fix type offsets
alexbarghi-nv Jan 11, 2023
a2120b6
fix index for non-cudf
alexbarghi-nv Jan 11, 2023
6c29705
concatenate fix
alexbarghi-nv Jan 11, 2023
426b529
fix cupy api issue
alexbarghi-nv Jan 11, 2023
3f7b0f6
latest changes, added SG unit test that caught @alexbarghi-nv found, …
ChuckHastings Jan 12, 2023
1886cd1
address a few more PR comments
ChuckHastings Jan 12, 2023
83233f7
add MG testing to the new sampling algorithm
ChuckHastings Jan 12, 2023
6a9ef77
Merge branch 'branch-23.02' into update_uniform_sampling_api
ChuckHastings Jan 12, 2023
3561c61
Merge branch 'branch-23.02' into update_uniform_sampling_api
ChuckHastings Jan 12, 2023
f557750
indexing fix
alexbarghi-nv Jan 13, 2023
3338edf
resolve merge conflicts
alexbarghi-nv Jan 13, 2023
44d5c18
debugging MG issue, shuffling needs to include the edge id/type
ChuckHastings Jan 14, 2023
1c72cf8
merge
alexbarghi-nv Jan 14, 2023
cdf6ae8
update copyright header, update labels logic
ChuckHastings Jan 14, 2023
adf19c7
fix tests, merge conflict
alexbarghi-nv Jan 14, 2023
e3f1d65
latest update
alexbarghi-nv Jan 14, 2023
6c749d1
remove print statement
alexbarghi-nv Jan 14, 2023
f6b97b3
update mg test
alexbarghi-nv Jan 14, 2023
6147cab
Adding first version
VibhuJawa Jan 15, 2023
7f659e3
fix shuffle bug I introduced
ChuckHastings Jan 15, 2023
9e0fb5d
address PR comments
ChuckHastings Jan 15, 2023
dcd5525
fix format
ChuckHastings Jan 15, 2023
bcce588
Merge branch 'branch-23.02' into update_uniform_sampling_api
ChuckHastings Jan 15, 2023
72b8b64
Merge branch 'update_uniform_sampling_api' of https://github.com/chuc…
alexbarghi-nv Jan 16, 2023
e13d443
merge in chuck's changes, remove expensive gather
alexbarghi-nv Jan 16, 2023
28929c7
Working sampling unit-test
VibhuJawa Jan 16, 2023
bdcfbba
Working sampling unit-test
VibhuJawa Jan 16, 2023
e00e40f
Working sampling hetrographs
VibhuJawa Jan 16, 2023
fd1f84a
Working sampling unit-test
VibhuJawa Jan 16, 2023
8a4ff56
Fix minor API access error
VibhuJawa Jan 16, 2023
f50ee91
Working dataloader
VibhuJawa Jan 17, 2023
d6750e0
add self loop tests
alexbarghi-nv Jan 17, 2023
81b4d8b
remove benchmarks
alexbarghi-nv Jan 17, 2023
69ce806
revert changes to dockerfile
alexbarghi-nv Jan 17, 2023
6281cc9
revert pyg changes again
alexbarghi-nv Jan 17, 2023
cc3250e
revert changes to CGS
alexbarghi-nv Jan 17, 2023
2ba8a18
remove debug file
alexbarghi-nv Jan 17, 2023
3a84322
revert script changes
alexbarghi-nv Jan 17, 2023
2f2d32e
update conftest
alexbarghi-nv Jan 17, 2023
bd5d2b8
resolve merge conflict
alexbarghi-nv Jan 17, 2023
d879116
fix style
alexbarghi-nv Jan 17, 2023
71cfe86
fix style
alexbarghi-nv Jan 17, 2023
12ed32f
fix typo
ChuckHastings Jan 17, 2023
ba02995
Add scale testing
VibhuJawa Jan 17, 2023
753d636
Merge branch 'short-circuit-and-type-indices' into bulk-sampling
alexbarghi-nv Jan 17, 2023
463104c
Merge branch 'update_uniform_sampling_api' of https://github.com/chuc…
alexbarghi-nv Jan 17, 2023
0bddd91
bulk the samples
alexbarghi-nv Jan 17, 2023
9b30115
resolve conflicts
alexbarghi-nv Jan 17, 2023
5436c8c
Merge branch 'python-sampling-edge-info' into bulk-sampling
alexbarghi-nv Jan 17, 2023
2baed77
revert file
alexbarghi-nv Jan 17, 2023
559dd98
remove extra files
alexbarghi-nv Jan 17, 2023
687df82
revert file
alexbarghi-nv Jan 17, 2023
a0a863c
remove extra files
alexbarghi-nv Jan 17, 2023
02a6bf4
remove added files
alexbarghi-nv Jan 17, 2023
ebdfa1f
Merge branch 'python-sampling-edge-info' into bulk-sampling
alexbarghi-nv Jan 17, 2023
14aafd4
minor test update
alexbarghi-nv Jan 17, 2023
54bf513
fix illegal deletion
alexbarghi-nv Jan 17, 2023
88501fb
Merge branch 'python-sampling-edge-info' into bulk-sampling
alexbarghi-nv Jan 17, 2023
c4ecb11
Added DLGSampler Tests
VibhuJawa Jan 17, 2023
c5eb2c3
fixed a dask bug
alexbarghi-nv Jan 17, 2023
c6aa9eb
add fix for batch id bug
alexbarghi-nv Jan 17, 2023
9bd7584
Merge branch 'python-sampling-edge-info' into bulk-sampling
alexbarghi-nv Jan 17, 2023
83f531b
allow non-dask inputs
alexbarghi-nv Jan 17, 2023
aab07bf
Merge branch 'python-sampling-edge-info' into bulk-sampling
alexbarghi-nv Jan 17, 2023
3f8fce1
fix style
alexbarghi-nv Jan 17, 2023
64f06e1
rerun style
alexbarghi-nv Jan 17, 2023
1a2b870
switch_to_fs
VibhuJawa Jan 17, 2023
8bb6ede
Merge branch 'branch-23.02' into switch_to_fs
VibhuJawa Jan 17, 2023
095ab72
Fix pytest
VibhuJawa Jan 18, 2023
54d2117
Merge branch 'switch_to_fs' of https://github.com/VibhuJawa/cugraph i…
VibhuJawa Jan 18, 2023
aeee656
drop partition on hop id
alexbarghi-nv Jan 18, 2023
6d551b3
drop partition on hop id
alexbarghi-nv Jan 18, 2023
5093887
intial write
alexbarghi-nv Jan 18, 2023
b901005
style fix
alexbarghi-nv Jan 18, 2023
22ff192
remove create with edge info
alexbarghi-nv Jan 18, 2023
b98f0b3
fix style
alexbarghi-nv Jan 18, 2023
472afe2
fix tests
alexbarghi-nv Jan 18, 2023
ebb2494
update graph test
alexbarghi-nv Jan 18, 2023
086e83c
reformat
alexbarghi-nv Jan 18, 2023
90679a4
update test, fix style
alexbarghi-nv Jan 18, 2023
1969cb5
Update python/cugraph/cugraph/gnn/data_loading/bulk_sampler.py
alexbarghi-nv Jan 18, 2023
5ff6ef0
drop seed arg, add rng state in plc layer
alexbarghi-nv Jan 18, 2023
7612640
Update python/cugraph-dgl/cugraph_dgl/convert.py
VibhuJawa Jan 18, 2023
f43e8f5
Update convert.py
VibhuJawa Jan 18, 2023
dcccc79
Fix Style problems
VibhuJawa Jan 19, 2023
0e3cc82
Update convert.py
VibhuJawa Jan 19, 2023
a9660d7
final improvement to random seeding
alexbarghi-nv Jan 19, 2023
b50196b
check instance of graph instead of df
alexbarghi-nv Jan 19, 2023
fc626cf
fix style
alexbarghi-nv Jan 19, 2023
367ccb2
Merge branch 'branch-23.02' into python-sampling-edge-info
alexbarghi-nv Jan 19, 2023
55acecd
Merge branch 'branch-23.02' into remove-pg
alexbarghi-nv Jan 19, 2023
e534ae9
Merge branch 'switch_to_fs' of https://github.com/vibhujawa/cugraph i…
alexbarghi-nv Jan 19, 2023
498b3fb
fix merge conflicts
alexbarghi-nv Jan 19, 2023
9d7b630
pull in new c++/python api changes
alexbarghi-nv Jan 19, 2023
83b262b
write the refactor
alexbarghi-nv Jan 19, 2023
780ca0d
remove old api
alexbarghi-nv Jan 19, 2023
5e1bd83
cleanup
alexbarghi-nv Jan 19, 2023
5dc521b
fix style
alexbarghi-nv Jan 19, 2023
ac9c603
resolve conflict
alexbarghi-nv Jan 19, 2023
9005918
Merge branch 'branch-23.02' of https://github.com/rapidsai/cugraph in…
alexbarghi-nv Jan 19, 2023
5ad6856
fix conflict
alexbarghi-nv Jan 19, 2023
a2fb8c5
fixes, test updates
alexbarghi-nv Jan 19, 2023
81637f1
remove remote
alexbarghi-nv Jan 19, 2023
8238a2d
reverted unintentional commit of create_with_edge_info test code
alexbarghi-nv Jan 19, 2023
375695d
update graph creation to property check for None
alexbarghi-nv Jan 19, 2023
8ca62fb
set dask num workers to 2
alexbarghi-nv Jan 19, 2023
d0909bf
add comments for random state wrapper object
alexbarghi-nv Jan 19, 2023
9e944af
move cluster creation logic, add teardown
alexbarghi-nv Jan 20, 2023
11a7135
fix style, remove garbage code
alexbarghi-nv Jan 20, 2023
facdbb2
reformat
alexbarghi-nv Jan 20, 2023
95ff288
test updates, fixes to cugraph store
alexbarghi-nv Jan 20, 2023
46377c8
t
alexbarghi-nv Jan 20, 2023
25f835d
Merge branch 'branch-23.02' into python-sampling-edge-info
alexbarghi-nv Jan 20, 2023
e005d6d
test, sampling fixes
alexbarghi-nv Jan 20, 2023
38fa791
Merge branch 'branch-23.02' of https://github.com/rapidsai/cugraph in…
alexbarghi-nv Jan 20, 2023
802ba34
Merge branch 'python-sampling-edge-info' of https://github.com/alexba…
alexbarghi-nv Jan 20, 2023
13c809a
add test for empty start list
alexbarghi-nv Jan 20, 2023
1a7b043
reorganize imports
alexbarghi-nv Jan 20, 2023
c843c87
Merge branch 'branch-23.02' of https://github.com/rapidsai/cugraph in…
alexbarghi-nv Jan 20, 2023
78864ec
redo bulk sampler
alexbarghi-nv Jan 20, 2023
8ce55b8
pull in changes from other pr
alexbarghi-nv Jan 20, 2023
100d70c
fix dask bug
alexbarghi-nv Jan 20, 2023
b905419
skip pyg tests since they are being deprecated
alexbarghi-nv Jan 20, 2023
83a8350
Merge branch 'branch-23.02' of https://github.com/rapidsai/cugraph in…
alexbarghi-nv Jan 20, 2023
ede6863
clean up cugraph import
alexbarghi-nv Jan 20, 2023
00552c4
resolve merge conflicts
alexbarghi-nv Jan 20, 2023
35898e5
add file extension
alexbarghi-nv Jan 23, 2023
0756846
add warning for empty flush
alexbarghi-nv Jan 23, 2023
570c347
remove unnecessary todo
alexbarghi-nv Jan 23, 2023
28b72fa
change behavior of add_batches() and flush() to match Vibhu's expecta…
alexbarghi-nv Jan 23, 2023
4982f83
Merge branch 'bulk-sampling' of https://github.com/alexbarghi-nv/cugr…
alexbarghi-nv Jan 23, 2023
7c13eb5
don't write the index
alexbarghi-nv Jan 23, 2023
26a4147
resolve merge conflict
alexbarghi-nv Jan 23, 2023
96aef19
Merge branch 'bulk-sampling' of https://github.com/alexbarghi-nv/cugr…
alexbarghi-nv Jan 23, 2023
edba7bc
skip empty partitions
alexbarghi-nv Jan 23, 2023
e95208e
update testing code, update random state to be function-based
alexbarghi-nv Jan 23, 2023
055ce50
Merge branch 'python-sampling-edge-info' into bulk-sampling
alexbarghi-nv Jan 23, 2023
df5d0de
Merge branch 'bulk-sampling' of https://github.com/alexbarghi-nv/cugr…
alexbarghi-nv Jan 23, 2023
ff39631
Merge branch 'remove-cgs' into bulk-loading-pyg
alexbarghi-nv Jan 23, 2023
a0e7130
update
alexbarghi-nv Jan 23, 2023
08be16a
style
alexbarghi-nv Jan 23, 2023
d8a0273
fix doctest
alexbarghi-nv Jan 23, 2023
67601e3
Merge branch 'python-sampling-edge-info' into bulk-sampling
alexbarghi-nv Jan 23, 2023
8316130
Merge branch 'python-sampling-edge-info' into bulk-loading-pyg
alexbarghi-nv Jan 23, 2023
7087e1b
bleh
alexbarghi-nv Jan 23, 2023
fa05709
Merge branch 'bulk-sampling' into bulk-loading-pyg
alexbarghi-nv Jan 23, 2023
4f1f585
initial working-ish version
alexbarghi-nv Jan 23, 2023
e6058ba
resolve merge conflict
alexbarghi-nv Jan 23, 2023
f1846e2
resolve merge conflicts
alexbarghi-nv Jan 23, 2023
4a1856e
revert pg changes
alexbarghi-nv Jan 23, 2023
55a9b5a
revert pg changes
alexbarghi-nv Jan 23, 2023
f4bed86
clean up imports
alexbarghi-nv Jan 23, 2023
32f016a
cleanup imports
alexbarghi-nv Jan 23, 2023
43369dd
resolve merge conflict
alexbarghi-nv Jan 23, 2023
28cd54a
revert unintentional change to cugraph store
alexbarghi-nv Jan 24, 2023
0adde85
add additional test for sg input compatibility
alexbarghi-nv Jan 24, 2023
14367ae
Merge branch 'branch-23.02' into bulk-sampling
alexbarghi-nv Jan 24, 2023
6276e8c
pull latest updates
alexbarghi-nv Jan 24, 2023
e3bd50b
some minor fixes, add tests
alexbarghi-nv Jan 24, 2023
fbe557a
testing
alexbarghi-nv Jan 24, 2023
57c4113
don't flush when there are no batches
alexbarghi-nv Jan 24, 2023
89ab491
Merge branch 'bulk-sampling' of https://github.com/alexbarghi-nv/cugr…
alexbarghi-nv Jan 24, 2023
b2f5392
Merge branch 'branch-23.02' of https://github.com/rapidsai/cugraph in…
alexbarghi-nv Jan 24, 2023
f51870e
add additional check just in case
alexbarghi-nv Jan 24, 2023
de2d178
update branch
alexbarghi-nv Jan 25, 2023
1979f6c
update tests to use type property
alexbarghi-nv Jan 25, 2023
29f6d69
mg tests
alexbarghi-nv Jan 25, 2023
f6a5dae
set weights to 0
alexbarghi-nv Jan 25, 2023
b5fc40c
update branch
alexbarghi-nv Jan 25, 2023
6f6e3f8
don't rely on setting batches to None
alexbarghi-nv Jan 25, 2023
9dcfb52
Merge branch 'bulk-sampling' into bulk-loading-pyg
alexbarghi-nv Jan 25, 2023
be43e48
drop empty warning
alexbarghi-nv Jan 25, 2023
cf43c69
Merge branch 'bulk-sampling' into bulk-loading-pyg
alexbarghi-nv Jan 25, 2023
40009a7
remove skip size check arg
alexbarghi-nv Jan 25, 2023
6dafeb4
Merge branch 'bulk-sampling' into bulk-loading-pyg
alexbarghi-nv Jan 25, 2023
7d64c06
fix tests
alexbarghi-nv Jan 25, 2023
51416d8
Merge branch 'bulk-sampling' into bulk-loading-pyg
alexbarghi-nv Jan 25, 2023
f271085
remove debug code
alexbarghi-nv Jan 26, 2023
1ed1f61
change cupy dlpack call
alexbarghi-nv Jan 26, 2023
531f965
Merge branch 'remove-pg' of https://github.com/alexbarghi-nv/cugraph …
alexbarghi-nv Jan 26, 2023
6d92f2c
style fix
alexbarghi-nv Jan 26, 2023
139de54
remove dlpack usage
alexbarghi-nv Jan 26, 2023
b7b574d
correction to asarray
alexbarghi-nv Jan 26, 2023
dd48bcb
update array conversion
alexbarghi-nv Jan 26, 2023
2121058
fix copyright
alexbarghi-nv Jan 27, 2023
6621505
Remove duplicate utils in cugraph-pyg
alexbarghi-nv Jan 27, 2023
31445ec
Merg branch 'bulk-loading-pyg' of https://github.com/alexbarghi-nv/cu…
alexbarghi-nv Jan 27, 2023
a8b380a
move fixtures to conftest
alexbarghi-nv Jan 27, 2023
e0847f8
move bulk sampler to experimental/gnn
alexbarghi-nv Jan 27, 2023
1ba9bea
rename saturation level
alexbarghi-nv Jan 27, 2023
786f6d3
add docstring example
alexbarghi-nv Jan 27, 2023
29e622d
raise notimplementederror
alexbarghi-nv Jan 27, 2023
f05f07e
use public api instead of instance check
alexbarghi-nv Jan 27, 2023
9301622
set npartitions to nworkers
alexbarghi-nv Jan 27, 2023
da3213e
clean up code
alexbarghi-nv Jan 27, 2023
47a7ea7
Merge branch 'bulk-loading-pyg' of https://github.com/alexbarghi-nv/c…
alexbarghi-nv Jan 27, 2023
7e6976a
correct docstring
alexbarghi-nv Jan 27, 2023
b1e86b3
add docstring, additional comment to construct_graph
alexbarghi-nv Jan 27, 2023
978f840
update init files
alexbarghi-nv Jan 27, 2023
95bf526
fix typos
alexbarghi-nv Jan 27, 2023
2b24e4a
Merge branch 'branch-23.02' into bulk-loading-pyg
alexbarghi-nv Jan 27, 2023
ea2c643
fix copyright
alexbarghi-nv Jan 27, 2023
9a436dc
Merge branch 'branch-23.02' into bulk-loading-pyg
alexbarghi-nv Jan 27, 2023
4e99f2e
fix merge conflicts
alexbarghi-nv Jan 27, 2023
e1c7b2e
Merge branch 'bulk-loading-pyg' of https://github.com/alexbarghi-nv/c…
alexbarghi-nv Jan 27, 2023
7fe3122
clean up tests
alexbarghi-nv Jan 27, 2023
e73db39
rename batches_per_call to seeds_per_call
alexbarghi-nv Jan 27, 2023
36b66e2
fix bad type annotation in _sampler_output_from_sampling_results
alexbarghi-nv Jan 28, 2023
80d0f41
Rename duck to tortoise
alexbarghi-nv Jan 28, 2023
77eb191
remove comment about removing cupy, update conda recipe for cupy and pyg
alexbarghi-nv Jan 28, 2023
38a0ee6
fix conda bug, add comment explaining col name issue
alexbarghi-nv Jan 28, 2023
34c3c78
fix merge conflicts
alexbarghi-nv Jan 28, 2023
708ffaf
fix ci configs
alexbarghi-nv Jan 30, 2023
cda4f7c
another ci update
alexbarghi-nv Jan 30, 2023
e8c9537
reset dead files
alexbarghi-nv Jan 30, 2023
77a4e34
remove hardcoded cugraph version
alexbarghi-nv Jan 30, 2023
79e4419
update test_python.sh
alexbarghi-nv Jan 30, 2023
e228861
update pytorch version
alexbarghi-nv Jan 30, 2023
127c345
update pytorch version
alexbarghi-nv Jan 30, 2023
4a1eeca
fix typo
alexbarghi-nv Jan 30, 2023
ac93506
revert changes related to recipe update
alexbarghi-nv Jan 31, 2023
ea8984c
Merge branch 'branch-23.02' into bulk-loading-pyg
alexbarghi-nv Jan 31, 2023
be8dcda
update branch
alexbarghi-nv Jan 31, 2023
3a98356
Merge branch 'bulk-loading-pyg' of https://github.com/alexbarghi-nv/c…
alexbarghi-nv Jan 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions python/cugraph-pyg/cugraph_pyg/data/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2019-2022, NVIDIA CORPORATION.
# Copyright (c) 2022-2023, NVIDIA CORPORATION.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
Expand All @@ -11,10 +11,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

from cugraph_pyg.utilities.api_tools import experimental_warning_wrapper
from cugraph.utilities.api_tools import experimental_warning_wrapper

from cugraph_pyg.data.cugraph_store import EXPERIMENTAL__CuGraphStore
from cugraph_pyg.data.cugraph_store import EXPERIMENTAL__to_pyg

CuGraphStore = experimental_warning_wrapper(EXPERIMENTAL__CuGraphStore)
to_pyg = experimental_warning_wrapper(EXPERIMENTAL__to_pyg)
Loading