[Feature] Unordered parallel graph walk V3 #3474
Conversation
ruseinov
requested review from
jdjaustin and
elmattic
and removed request for
a team
|
Co-authored-by: Josh Jones <[email protected]>
src/ipld/util.rs
Outdated
| @@ -437,3 +438,259 @@ impl<DB: Blockstore, T: Iterator<Item = Tipset> + Unpin> Stream for ChainStream< | |||
| } | |||
| } | |||
| } | |||
|
|
|||
| enum UnorderedTask { | |||
There was a problem hiding this comment.
Any reason to use an enum here?
There was a problem hiding this comment.
It's a leftover, will remove it for good. Thanks for spotting this.
src/ipld/util.rs
Outdated
| tipset_iter: T, | ||
| stateroot_limit: ChainEpoch, | ||
| ) -> UnorderedChainStream<DB, T> { | ||
| let (sender, receiver) = kanal::bounded(2048); |
There was a problem hiding this comment.
Wondering how did you pick this 2048 value? Does the value has any impact on performances or memory usage?
There was a problem hiding this comment.
It's somewhat arbitrary, 2048 worked well on my machine and memory usage for 2048 items is negligible. I'll introduce a constant to be more transparent.
| task::spawn(async move { | ||
| let mut handles = JoinSet::new(); | ||
|
|
||
| for _ in 0..num_cpus::get() { |
There was a problem hiding this comment.
Should I observe some speedup/slowdown if I increase/decrease the number of cores here? For example testing on a calibnet snapshot, I don't see a big difference in performances. Or should I run the benchmark command on a mainnet snapshot maybe?
There was a problem hiding this comment.
Calibnet is too small of a snapshot, yeah. I have tested this and this setup worked better than others.
src/ipld/util.rs
Outdated
| } | ||
| } | ||
|
|
||
| // // Process block messages. |
Summary of changes
Changes introduced in this pull request:
Reference issue to close (if applicable)
Work on #3314
Other information and links
Change checklist