Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: rewrite metrics_unify using Rust #1148

Merged
merged 3 commits into from
Dec 31, 2024
Merged

feat: rewrite metrics_unify using Rust #1148

merged 3 commits into from
Dec 31, 2024

Conversation

keroro520
Copy link
Contributor

Resolve #1030

Note that I did not implement the functionality for 'diff' (--prev), 'sort by labels' (--top-labels), and 'filter' (--excluded-labels) because I observed that the CI does not utilize these features. Please let me know if it is in need.

@keroro520
Copy link
Contributor Author 

cargo run --bin metric_unify -- ci/scripts/metric_unify/example_metrics.json

Output:

Detailed Metrics
group verify_program_compile_ms total_cells_used total_cells stark_proof_time_ms
bench_program_inner 287,313 1,997,968 1,972
inner_verifier 175,728 686,193,568 1,257,701,396 349,444
group air_name rows quotient_deg prep_cols perm_cols main_cols interactions constraints cells
inner_verifier MemoryAuditAir 1,048,576 8 8 19 6 19 28,311,552
inner_verifier FieldArithmeticAir 4,194,304 8 16 31 15 23 197,132,288
bench_program_inner CoreAir 32 4 32 69 19 114 3,232
inner_verifier Poseidon2VmAir 32,768 8 100 502 144 419 19,726,336
inner_verifier ProgramAir 524,288 1 9 8 1 1 4 4,718,592
bench_program_inner FieldArithmeticAir 16 4 24 31 15 25 880
inner_verifier FieldExtensionArithmeticAir 2,097,152 8 40 68 51 38 226,492,416
inner_verifier CoreAir 8,388,608 8 20 73 19 115 780,140,544
inner_verifier VmConnectorAir 2 2 1 8 2 2 4 20
bench_program_inner FieldExtensionArithmeticAir 1 4 72 68 51 46 140
bench_program_inner XorLookupAir 65,536 1 3 8 1 1 4 589,824
inner_verifier VariableRangeCheckerAir 131,072 1 2 8 1 1 4 1,179,648
bench_program_inner VariableRangeCheckerAir 131,072 1 2 8 1 1 4 1,179,648
bench_program_inner MemoryAuditAir 64 4 12 19 6 20 1,984
bench_program_inner VmConnectorAir 2 2 1 8 2 2 4 20
bench_program_inner ProgramAir 64 1 9 8 1 1 4 576
bench_program_inner KeccakVmAir 32 4 3,296 3,631 823 2,866 221,664
group chip_name rows_used
inner_verifier Program 293,076
bench_program_inner Core 28
bench_program_inner Program 37
bench_program_inner Keccak256 24
bench_program_inner RangeChecker 131,072
inner_verifier FieldArithmetic 2,530,790
bench_program_inner Memory 59
inner_verifier Poseidon2 31,058
inner_verifier Core 6,493,962
inner_verifier FieldExtension 1,494,186
inner_verifier Memory 845,260
inner_verifier RangeChecker 131,072
bench_program_inner FieldExtension 1
bench_program_inner ByteXor 65,536
bench_program_inner FieldArithmetic 13
group dsl_ir frequency
inner_verifier ImmE 17,384
inner_verifier ImmV 32,232
bench_program_inner ImmE 8
inner_verifier SubEFI 1,248
inner_verifier SubEI 568
inner_verifier StoreF 195,144
inner_verifier MulF 23,762
inner_verifier Halt 1
inner_verifier CycleTrackerEnd 184,327
inner_verifier AddE 389,586
inner_verifier MulE 728,625
inner_verifier SubV 15,168
inner_verifier SubEF 2,824,776
inner_verifier CycleTrackerStart 184,327
bench_program_inner For 7
inner_verifier MulEFI 2,660
inner_verifier DivE 353,514
inner_verifier StoreHintWord 615,007
bench_program_inner ImmV 3
inner_verifier For 1,823,746
bench_program_inner StoreV 2
inner_verifier IfNeI 1,009
inner_verifier LoadF 554,957
inner_verifier AssertEqE 132
bench_program_inner Keccak256 1
bench_program_inner AddF 1
inner_verifier AddVI 438,550
inner_verifier MulEF 2,200
inner_verifier DivEIN 355
inner_verifier AddFI 13,696
inner_verifier AssertEqEI 4
bench_program_inner AddE 1
inner_verifier LoadE 1,499,980
inner_verifier AssertEqVI 159
inner_verifier Poseidon2PermuteBabyBear 22,874
inner_verifier SubVIN 408
inner_verifier MulV 775
inner_verifier MulEI 35,775
inner_verifier ImmF 15,360
inner_verifier MulVI 9,367
inner_verifier LoadV 83,334
inner_verifier DivFIN 151
inner_verifier Alloc 69,716
inner_verifier SubE 15,235
inner_verifier AddEI 80,776
inner_verifier AssertEqV 1,262
inner_verifier AddEFFI 696
inner_verifier SubVI 1,421
inner_verifier StoreE 25,140
inner_verifier StoreV 27,143
inner_verifier AssertEqF 4,633
bench_program_inner Halt 1
inner_verifier AddEFI 960
inner_verifier NegE 784
inner_verifier IfEqI 211,394
bench_program_inner AddVI 6
inner_verifier Poseidon2CompressBabyBear 8,184
inner_verifier IfNe 7,886
bench_program_inner IfEqI 2
inner_verifier MulFI 9
bench_program_inner ImmF 2
inner_verifier HintBitsF 25
inner_verifier HintInputVec 10,708
bench_program_inner Alloc 6
inner_verifier AddV 6,720
inner_verifier IfEq 6,140
group opcode frequency cells_used
inner_verifier FE4ADD 389,586 29,198,436
inner_verifier FSUB 723,191 22,434,406
inner_verifier COMP_POS2 8,184 4,108,368
bench_program_inner FMUL 2 62
inner_verifier LOADW2 2,045,146 149,300,484
inner_verifier BNE 1,115,747 81,449,531
inner_verifier TERMINATE 1 73
bench_program_inner FADD 11 398
inner_verifier STOREW2 219,301 16,582,469
inner_verifier BBE4MUL 735,780 52,097,884
bench_program_inner LOADW 2 176
inner_verifier LOADW 2,239,721 164,693,384
inner_verifier FADD 1,751,791 55,810,435
inner_verifier HINT_INPUT 10,708 781,684
inner_verifier SHINTW 313,245 28,818,540
inner_verifier CT_START 184,327 13,455,871
inner_verifier FDIV 151 7,493
inner_verifier PERM_POS2 22,874 11,482,748
bench_program_inner FE4ADD 1 144
bench_program_inner KECCAK256 1 87,752
inner_verifier FMUL 55,657 1,817,042
inner_verifier CT_END 184,327 13,455,871
inner_verifier STOREW 142,278 11,264,721
inner_verifier HINT_BITS 25 1,825
bench_program_inner JAL 2 157
bench_program_inner STOREW 16 1,389
bench_program_inner TERMINATE 1 69
inner_verifier FE4SUB 15,235 2,100,816
inner_verifier BEQ 8,874 647,802
inner_verifier JAL 30,262 2,209,145
bench_program_inner BNE 5 345
bench_program_inner STOREW2 2 176
inner_verifier BBE4DIV 353,585 24,050,392
group stark trace_gen_time_ms
inner_verifier vm 27,600
bench_program_inner vm 2

@keroro520
Copy link
Contributor Author 

cargo run --bin metric_unify -- ci/scripts/metric_unify/example_metrics.json --aggregation-json ci/scripts/metric_unify/aggregation.json

Outputs:

group total_cells_used
bench_program_inner 287,313
inner_verifier 686,193,568
Detailed Metrics
group verify_program_compile_ms total_cells_used total_cells stark_proof_time_ms
inner_verifier 175,728 686,193,568 1,257,701,396 349,444
bench_program_inner 287,313 1,997,968 1,972
group air_name rows quotient_deg prep_cols perm_cols main_cols interactions constraints cells
bench_program_inner XorLookupAir 65,536 1 3 8 1 1 4 589,824
inner_verifier CoreAir 8,388,608 8 20 73 19 115 780,140,544
bench_program_inner FieldExtensionArithmeticAir 1 4 72 68 51 46 140
bench_program_inner MemoryAuditAir 64 4 12 19 6 20 1,984
bench_program_inner ProgramAir 64 1 9 8 1 1 4 576
inner_verifier FieldExtensionArithmeticAir 2,097,152 8 40 68 51 38 226,492,416
bench_program_inner CoreAir 32 4 32 69 19 114 3,232
inner_verifier ProgramAir 524,288 1 9 8 1 1 4 4,718,592
bench_program_inner FieldArithmeticAir 16 4 24 31 15 25 880
bench_program_inner VariableRangeCheckerAir 131,072 1 2 8 1 1 4 1,179,648
inner_verifier VariableRangeCheckerAir 131,072 1 2 8 1 1 4 1,179,648
inner_verifier Poseidon2VmAir 32,768 8 100 502 144 419 19,726,336
inner_verifier MemoryAuditAir 1,048,576 8 8 19 6 19 28,311,552
inner_verifier FieldArithmeticAir 4,194,304 8 16 31 15 23 197,132,288
bench_program_inner KeccakVmAir 32 4 3,296 3,631 823 2,866 221,664
inner_verifier VmConnectorAir 2 2 1 8 2 2 4 20
bench_program_inner VmConnectorAir 2 2 1 8 2 2 4 20
group chip_name rows_used
inner_verifier FieldArithmetic 2,530,790
inner_verifier Core 6,493,962
bench_program_inner ByteXor 65,536
bench_program_inner FieldExtension 1
inner_verifier Program 293,076
bench_program_inner Program 37
inner_verifier RangeChecker 131,072
bench_program_inner FieldArithmetic 13
bench_program_inner RangeChecker 131,072
inner_verifier FieldExtension 1,494,186
bench_program_inner Keccak256 24
bench_program_inner Memory 59
inner_verifier Memory 845,260
bench_program_inner Core 28
inner_verifier Poseidon2 31,058
group dsl_ir frequency
inner_verifier AssertEqEI 4
inner_verifier For 1,823,746
inner_verifier StoreF 195,144
inner_verifier SubVI 1,421
inner_verifier ImmE 17,384
bench_program_inner StoreV 2
inner_verifier AddEFFI 696
inner_verifier MulEFI 2,660
inner_verifier LoadE 1,499,980
inner_verifier AddE 389,586
inner_verifier AssertEqV 1,262
inner_verifier NegE 784
inner_verifier DivE 353,514
inner_verifier HintInputVec 10,708
inner_verifier AddV 6,720
inner_verifier AddFI 13,696
inner_verifier LoadF 554,957
inner_verifier StoreV 27,143
inner_verifier IfEq 6,140
inner_verifier DivEIN 355
inner_verifier SubV 15,168
inner_verifier IfNeI 1,009
bench_program_inner Keccak256 1
bench_program_inner For 7
inner_verifier MulFI 9
inner_verifier MulEF 2,200
inner_verifier ImmF 15,360
inner_verifier HintBitsF 25
inner_verifier Poseidon2CompressBabyBear 8,184
inner_verifier AddEFI 960
inner_verifier IfEqI 211,394
inner_verifier CycleTrackerEnd 184,327
inner_verifier Alloc 69,716
inner_verifier AddVI 438,550
inner_verifier DivFIN 151
inner_verifier SubE 15,235
inner_verifier MulE 728,625
bench_program_inner AddF 1
inner_verifier MulV 775
inner_verifier Halt 1
inner_verifier AssertEqF 4,633
inner_verifier SubEFI 1,248
bench_program_inner IfEqI 2
inner_verifier Poseidon2PermuteBabyBear 22,874
inner_verifier MulVI 9,367
bench_program_inner Halt 1
inner_verifier CycleTrackerStart 184,327
inner_verifier MulF 23,762
inner_verifier MulEI 35,775
inner_verifier StoreE 25,140
inner_verifier AddEI 80,776
bench_program_inner ImmE 8
bench_program_inner Alloc 6
inner_verifier AssertEqE 132
bench_program_inner ImmV 3
inner_verifier IfNe 7,886
inner_verifier SubVIN 408
inner_verifier AssertEqVI 159
bench_program_inner ImmF 2
inner_verifier LoadV 83,334
bench_program_inner AddE 1
bench_program_inner AddVI 6
inner_verifier SubEF 2,824,776
inner_verifier StoreHintWord 615,007
inner_verifier SubEI 568
inner_verifier ImmV 32,232
group opcode frequency cells_used
bench_program_inner FADD 11 398
bench_program_inner FMUL 2 62
inner_verifier FE4ADD 389,586 29,198,436
bench_program_inner STOREW 16 1,389
inner_verifier FDIV 151 7,493
inner_verifier COMP_POS2 8,184 4,108,368
bench_program_inner BNE 5 345
inner_verifier BEQ 8,874 647,802
inner_verifier CT_START 184,327 13,455,871
inner_verifier FSUB 723,191 22,434,406
bench_program_inner KECCAK256 1 87,752
inner_verifier STOREW 142,278 11,264,721
inner_verifier PERM_POS2 22,874 11,482,748
inner_verifier LOADW 2,239,721 164,693,384
inner_verifier BNE 1,115,747 81,449,531
inner_verifier HINT_BITS 25 1,825
bench_program_inner LOADW 2 176
inner_verifier TERMINATE 1 73
bench_program_inner FE4ADD 1 144
bench_program_inner JAL 2 157
inner_verifier FE4SUB 15,235 2,100,816
bench_program_inner TERMINATE 1 69
inner_verifier STOREW2 219,301 16,582,469
inner_verifier FADD 1,751,791 55,810,435
inner_verifier FMUL 55,657 1,817,042
bench_program_inner STOREW2 2 176
inner_verifier LOADW2 2,045,146 149,300,484
inner_verifier BBE4DIV 353,585 24,050,392
inner_verifier SHINTW 313,245 28,818,540
inner_verifier CT_END 184,327 13,455,871
inner_verifier HINT_INPUT 10,708 781,684
inner_verifier BBE4MUL 735,780 52,097,884
inner_verifier JAL 30,262 2,209,145
group stark trace_gen_time_ms
bench_program_inner vm 2
inner_verifier vm 27,600

@keroro520 keroro520 changed the title Rust metrics unify feat: rewrite metrics_unify using Rust Dec 31, 2024
@jonathanpwang
Copy link
Contributor

Not sure about the other ones, but we do use --diff

@keroro520
Copy link
Contributor Author

Not sure about the other ones, but we do use --diff

Oh I see.
Could you provide an additional metrics file for me to better test the 'diff' command?

@keroro520
Copy link
Contributor Author

@jonathanpwang I will add diff function in 2 days (a metrics file would be helpful to make real diff). BTW I believe this pr is ready for review. It would be very appreciate if you can take some time to review.

jonathanpwang
jonathanpwang approved these changes Dec 31, 2024
View reviewed changes
Copy link
Contributor

@jonathanpwang jonathanpwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM to first merge.

Some things I'm planning:

  • move this into crates/prof and rename openvm-prof to become a profiling library
  • most of main.rs should be moved into lib.rs
  • we can use serde derive more and have stronger types now that its rust and not python
  • enum to distinguish counter from gauge

@jonathanpwang jonathanpwang merged commit 32e6bbf into openvm-org:main Dec 31, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jonathanpwang jonathanpwang jonathanpwang approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Convert metric analysis scripts into rust crate
2 participants
@keroro520 @jonathanpwang