[BUG]: MG ego_graph Different from SG
#4190
Comments
nv-rliu
added
bug
nv-rliu
added
breaking
4 tasks
|
What's the minimum scale (i.e. # GPUs) to reproduce this? Can you reproduce this with 2 GPUs? |
It's reproducible with 2-GPUs. This was the result from running on a lab machine. |
|
After looking at a trend of 1 Node 8-GPU runs across multiple days, it appears that the failure is transient. |
|
You mean this is a heisenbug? Sounds worse but let me try to reproduce this first. |
|
How often can you reproduce this? (Say run test_egonet_mg.py 10 times, how many times you see at least one failure?) I am running this on my local system with 2 GPUs, and I can't reproduce the test failure. Let me try this on a DGX node as well. |
|
Never mind, I reproduced this. |
Closed
rapids-bot bot
pushed a commit
that referenced
this issue
Mar 26, 2024
Closes #4190 cc: @jnke2016 This PR adds a function to `comms.py` which returns a mapping of workers to ranks. This is then sorted in `part_utils.py` before being used to submit jobs to `dask`. This should fix a bug in MG `ego_graph` where induced subgraphs were being returned in seemingly random orders (while the results are correct). Authors: - Ralph Liu (https://github.com/nv-rliu) Approvers: - Joseph Nke (https://github.com/jnke2016) - Don Acosta (https://github.com/acostadon) - Rick Ratzel (https://github.com/rlratzel) - Ray Douglass (https://github.com/raydouglass) URL: #4262
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version
24.04
Which installation method(s) does this occur on?
Source
Describe the bug.
Currently, the MG implementation of
ego_graphreturns a value that differs from the SG implementation when passed multiplenvalues, akaseeds.Minimum reproducible example
import cugraph import cugraph.dask as dcg from cugraph.datasets import karate from cugraph.testing.mg_utils import ( start_dask_client, stop_dask_client ) import pytest from cugraph.testing import utils from cudf.testing.testing import assert_frame_equal if __name__ == "__main__": # start dask client, cluster = start_dask_client() # Parameters to re-create failure data = karate directed = True seeds = [0, 2] radius = 1 # prep data_path = data.get_path() ################################################################### # SG g = data.get_graph(create_using=cugraph.Graph(directed=directed)) ################################################################### # MG blocksize = dcg.get_chunksize(data_path) ddf = data.get_dask_edgelist() dg = cugraph.Graph(directed=directed) dg.from_dask_cudf_edgelist( ddf, source="src", destination="dst", edge_attr="wgt", renumber=True, store_transposed=True, ) ################################################################### sg_cugraph_ego_graphs = cugraph.batched_ego_graphs(g, seeds=seeds, radius=radius) mg_cugraph_ego_graphs = dcg.ego_graph(dg, seeds, radius) sg_df, sg_offsets = sg_cugraph_ego_graphs mg_df, mg_offsets = mg_cugraph_ego_graphs mg_df = mg_df.compute() # This Fails assert_frame_equals(mg_df, sg_df) # stop dask stop_dask_client(client, cluster)Relevant log output
Environment details
Other/Misc.
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: