stylus_benchmark #2827

diegoximenes · 2024-12-11T13:56:49Z

This PR adds the stylus_benchmark binary.
It will mainly be used to fine tune the ink prices that are charged today.

It deterministically creates .wat programs with lots of instructions to be benchmarked.

If this PR is approved then more .wat programs generations, benchmarking other instructions, will be added in a future PR.

This PR introduces two new WASM instruction to jit, start_benchmark and end_benchmark.
Code blocks between start_benchmark and end_benchmark instructions will be benchmarked.

stylus_benchmark uses jit as a library.

This reverts commit 4fd9d10.

arbitrator/tools/stylus_benchmark/src/generate_wats.rs

arbitrator/arbutil/src/benchmark.rs

arbitrator/tools/stylus_benchmark/src/benchmark.rs

gligneul · 2024-12-18T21:15:59Z

arbitrator/tools/stylus_benchmark/src/benchmark.rs

+
+    let exec = &mut WasmEnv::default();
+
+    let module = exec_program(


Is this code benchmarking only the JIT execution? I'm worried about the differences between the JIT and the interpreter, so it would be nice to benchmark non-JIT execution as well.

That is a good point.
By interpreter you mean execution through the prover's binary?

@tsahee, any takes on that?

In the prover the bottleneck is not how long it takes the prover to execute but how many steps it takes - because that affects how long it will take to hash for BOLD.
And in stylus, that's a very different constraint.

diegoximenes · 2024-12-23T16:01:32Z

general approach seems great. would like to get more reviews but I think it's already interesting to add a few benchmarks and start looking at results.

I intend to add more scenarios to be benchmarked in a future PR, together with a more detailed look at results.

But here goes the output of the benchmarks implemented in this PR, that was run in a Apple M3 Pro:

No scenario specified, benchmarking all scenarios

Benchmarking add_i32
Run 0, duration: 120.735917ms, ink_spent: Ink(29298411152)
Run 1, duration: 108.255875ms, ink_spent: Ink(29298411152)
Run 2, duration: 110.482833ms, ink_spent: Ink(29298411152)
Run 3, duration: 95.918417ms, ink_spent: Ink(29298411152)
Run 4, duration: 95.604125ms, ink_spent: Ink(29298411152)
Run 5, duration: 95.353792ms, ink_spent: Ink(29298411152)
Run 6, duration: 95.531375ms, ink_spent: Ink(29298411152)
After discarding top and bottom runs:
avg_duration: 99.926139ms, avg_ink_spent_per_micro_second: 293201
Benchmarking xor_i32
Run 0, duration: 96.054583ms, ink_spent: Ink(29298411152)
Run 1, duration: 96.208625ms, ink_spent: Ink(29298411152)
Run 2, duration: 97.552791ms, ink_spent: Ink(29298411152)
Run 3, duration: 96.658ms, ink_spent: Ink(29298411152)
Run 4, duration: 96.164417ms, ink_spent: Ink(29298411152)
Run 5, duration: 96.1905ms, ink_spent: Ink(29298411152)
Run 6, duration: 96.149083ms, ink_spent: Ink(29298411152)
After discarding top and bottom runs:
avg_duration: 96.187847ms, avg_ink_spent_per_micro_second: 304598

gligneul

LGTM. I think measuring the interpreter (prover) performance is important as well, but you can leave this for a future PR.

tsahee · 2024-12-26T19:57:51Z

arbitrator/jit/src/program.rs

+    exec_program(exec, module, calldata, config, evm_data, gas)
+}
+
+pub fn exec_program(


exec_program vs start_program is very confusing.
Maybe launch_program_thread?

tsahee · 2024-12-26T23:53:34Z

arbitrator/tools/stylus_benchmark/src/main.rs

+    let args = Args::parse();
+
+    match args.scenario {
+        Some(scenario) => handle_scenario(scenario, args.output_wat_dir_path),


should have some option (either general --help or --schenario help) that lists all scenarios

tsahee · 2024-12-27T00:07:26Z

arbitrator/jit/src/lib.rs

+
+#[derive(StructOpt)]
+#[structopt(name = "jit-prover")]
+pub struct Opts {


does that really need to be in lib.rs? I don't see it used elsewhere?

cla-bot bot added the s Automatically added by the CLA bot if the creator of a PR is registered as having signed the CLA. label Dec 11, 2024

diegoximenes requested a review from tsahee December 11, 2024 19:27

diegoximenes added 28 commits December 13, 2024 08:20

wip

fdb42c8

Revert "wip"

6705932

This reverts commit 4fd9d10.

jit as a library and binary

d94665a

Beginning of stylus_benchmark

cd1642d

stylus_benchmark: create_stylus_config and create_evm_data2

25a770b

Refactors jit to have *_with_wasm functions

472a662

Basic CallProgramLoop in stylus_benchmark

f327531

Adds debug.toggle_measurement instruction

2aa59c9

toggle_measurement ink tracking

61620f9

Run benchmark multiple times for the same wat file

5e17f53

toggle_measurement to toggle_benchmark

89eeee8

Adds whole Benchmark object to MessageFromCothread

7d9819c

Stores benchmark as an Option<Benchmark> in WasmEnv

55463ed

reuse compiled module

e9a74d9

Uses ? instead of unwrap when appropriate

1198dfa

Small refactor on stylus_benchmark

d53bcc2

stylus_benchmark: from to_result to check_result

d2668f6

Removes send_response_with_wasm_env

c4fb5df

Removes cycle from Benchmark

f36013f

Renames add_one.wat to add_i32.wat. Uses loop in add_i32.wat

24dac3e

stylus_benchmark: print ink usage

3df9b24

Improves prints in stylus_benchmark

87f7386

fix cargo fmt

654d18a

fix cargo fmt

15ca778

Adds rust fmt to stylus_benchmark in ci

a813433

Improve panics msgs in stylus_benchmark

1f1e5e3

generate-wats subcommand

175e39b

Split benchmark and generate_wats in different files

922fce8

gligneul reviewed Dec 18, 2024

View reviewed changes

diegoximenes added 19 commits December 19, 2024 09:26

Uses strum in stylus_benchmark

ecb5172

Run all scenarios in stylus_benchmark

486cc67

Do not import functions in stylus_benchmark

03e8315

Adds comment to Benchmark struct

6d59589

Uses loop in benchmark programs

babadc9

start_benchmark/end_benchmark instead of toggle_benchmark

7fbd08a

Stores Benchmark instead of Option<Benchmark> in WasmEnv

d3ad3e6

Divide wat generation in multiple functions

1fcab9f

Benchmark XorI32

367f77b

Avoids unnecessary derivative

56b3354

Fixes user-host

0ae7046

Increases number of operations

cc8ba64

Fixes user-test

8af1b4b

Fix clippy check

b0e79e5

Add comment about the strategy of the programs to be benchmarked

c270a85

Improves wat programs to be benchmarked generation

667d6ac

Fix lint

6aa0c91

Fix lint

33a80b3

Fix lint

28c045b

diegoximenes requested review from tsahee and gligneul December 23, 2024 16:01

gligneul approved these changes Dec 23, 2024

View reviewed changes

tsahee approved these changes Dec 27, 2024

View reviewed changes

tsahee added the design-approved label Dec 27, 2024

tsahee enabled auto-merge December 27, 2024 00:07

Merge branch 'master' into stylus_bench

2028101

tsahee merged commit 2409196 into master Dec 27, 2024
17 checks passed

tsahee deleted the stylus_bench branch December 27, 2024 00:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stylus_benchmark #2827

stylus_benchmark #2827

diegoximenes commented Dec 11, 2024 •

edited

Loading

gligneul Dec 18, 2024

diegoximenes Dec 23, 2024

tsahee Dec 27, 2024

diegoximenes commented Dec 23, 2024 •

edited

Loading

gligneul left a comment

tsahee Dec 26, 2024

tsahee Dec 26, 2024

tsahee Dec 27, 2024


		let exec = &mut WasmEnv::default();

		let module = exec_program(

stylus_benchmark #2827

stylus_benchmark #2827

Conversation

diegoximenes commented Dec 11, 2024 • edited Loading

gligneul Dec 18, 2024

Choose a reason for hiding this comment

diegoximenes Dec 23, 2024

Choose a reason for hiding this comment

tsahee Dec 27, 2024

Choose a reason for hiding this comment

diegoximenes commented Dec 23, 2024 • edited Loading

gligneul left a comment

Choose a reason for hiding this comment

tsahee Dec 26, 2024

Choose a reason for hiding this comment

tsahee Dec 26, 2024

Choose a reason for hiding this comment

tsahee Dec 27, 2024

Choose a reason for hiding this comment

diegoximenes commented Dec 11, 2024 •

edited

Loading

diegoximenes commented Dec 23, 2024 •

edited

Loading