Traverse the DataPipeGraph in in a breadth-first order rather than depth-first order in traverse_dps

[NivekT]

🚀 The feature

Traverse the DataPipeGraph in in a breadth-first order rather than depth-first order in traverse_dps

Motivation, pitch

In cases such as pausing or stopping, a bread-first order will ensure that consumer DataPipes (i.e. DataPipes consuming data from other DataPipes) stop before producer DataPipes.

Alternatives

No response

Additional context

cc: @ejguan

[ejguan]

After thinking deeply on this topic, I think it's not traverse_dps should be breadth-first order because the returned graph is the same no matter it's traversed in BFS or DFS.

I think you are requesting list_dps using BFS rather than DFS. So, the DataPipes with smaller depth should be in the front of returned list. WDYT?

[ejguan]

Will update list_dps in #894

[ejguan]

Closing as #894 has been landed.