Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: optimize LWG and NWG #2512

Merged
merged 31 commits into from
Aug 2, 2024
Merged
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
ff57b38
Move recursive circuits out of AggregationWrapper struct
0xVolosnikov Jul 26, 2024
64fb5b7
Add comments in prover tests
0xVolosnikov Jul 26, 2024
88e610c
Merge branch 'main' into vv-optimize-lwg-nwg
0xVolosnikov Jul 29, 2024
24ef093
Cargo fmt
0xVolosnikov Jul 29, 2024
cc583c7
Process LWG in batches
0xVolosnikov Jul 29, 2024
7c261a0
Move LWG processing in separate thread
0xVolosnikov Jul 29, 2024
a5105f3
Remove separate thread; cleanup
0xVolosnikov Jul 29, 2024
7b369d5
Fix after merge
0xVolosnikov Jul 30, 2024
69c80f8
Parallelize LWG
0xVolosnikov Jul 30, 2024
9b9c52d
Fix semaphore usage
0xVolosnikov Jul 30, 2024
5a3f20d
Merge branch 'main' into vv-optimize-lwg-nwg
0xVolosnikov Jul 30, 2024
89249e9
Add comments to constants
0xVolosnikov Jul 30, 2024
3ed06aa
Fix todos in basic_test
0xVolosnikov Jul 30, 2024
745461e
Cargo fmt
0xVolosnikov Jul 30, 2024
4b32eb0
Simplify LWG
0xVolosnikov Jul 30, 2024
2dbca20
Parallelize NWG as well
0xVolosnikov Jul 30, 2024
b5b9a04
Update deps
0xVolosnikov Aug 2, 2024
fb75fc2
Address comments
0xVolosnikov Aug 2, 2024
a119262
Cargo.lock
0xVolosnikov Aug 2, 2024
8b6add2
Merge remote-tracking branch 'origin/main' into vv-optimize-lwg-nwg
0xVolosnikov Aug 2, 2024
e7e0235
Hotfix
0xVolosnikov Aug 2, 2024
9899892
Cargo fmt
0xVolosnikov Aug 2, 2024
6468ddd
Use max_circuits_in_flight from config
0xVolosnikov Aug 2, 2024
c2a5e72
Fix
0xVolosnikov Aug 2, 2024
e0694a2
Fix lint issues
0xVolosnikov Aug 2, 2024
0f4db7c
Merge remote-tracking branch 'origin/main' into vv-optimize-lwg-nwg
0xVolosnikov Aug 2, 2024
b61d466
Revert unnessesary Cargo.lock changes
0xVolosnikov Aug 2, 2024
f8c3236
Add comments
0xVolosnikov Aug 2, 2024
1219f7f
Fix lint issues in tests
0xVolosnikov Aug 2, 2024
752c541
Revert blocking spawns
0xVolosnikov Aug 2, 2024
7486f2d
Cleanup
0xVolosnikov Aug 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add comments to constants
0xVolosnikov committed Jul 30, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 89249e95fa3a57e56e45d08dc88ee3d47bc5c7d9
6 changes: 6 additions & 0 deletions prover/crates/bin/witness_generator/src/leaf_aggregation.rs
Original file line number Diff line number Diff line change
@@ -251,7 +251,13 @@ pub async fn prepare_leaf_aggregation_job(
})
}

// The number of recursive circuits processed in one chunk.
// Each circuit requires downloading RECURSION_ARITY (32) proofs, each of which can be roughly estimated at 1 MB.
// So one chunk of recursive circuits with QUEUES_CHUNK_SIZE == 10 should use ~320 MB of RAM + some overhead during serialization.
const QUEUES_CHUNK_SIZE: usize = 10;
// The number of chunks to be processed in parallel.
// Allows us to easily adjust the balance between processing speed and memory usage.
// See the comment on QUEUES_CHUNK_SIZE to estimate memory usage.
const MAX_IN_FLIGHT_QUEUES_CHUNKS: usize = 10;

#[tracing::instrument(