Reduce benchmarking time limits

[kwxm]

This is a followup to #6328. Criterion runs each individual benchmark up to some time limit then analyses the times of the different runs. Our benchmark suites currently get run with different time limits: 60s for nofib, 20 for validation, and 15 for lists. There should be scope for reducing these limits without reducing the accuracy of the results too much. I'm going to do some experiments in this PR to see what happens. Using /benchmark, the benchmarks will be run in the base branch with the old time limits and in this branch with the new ones, so we should be able to get a decent comparison.

[kwxm]

/benchmark nofib

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

[Time limit = 30s]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
clausify/formula1	3.115 ms	3.097 ms	-0.6%
clausify/formula2	4.204 ms	4.147 ms	-1.4%
clausify/formula3	11.51 ms	11.37 ms	-1.2%
clausify/formula4	26.51 ms	25.85 ms	-2.5%
clausify/formula5	55.76 ms	54.95 ms	-1.5%
knights/4x4	18.33 ms	17.79 ms	-2.9%
knights/6x6	47.98 ms	46.56 ms	-3.0%
knights/8x8	84.09 ms	81.46 ms	-3.1%
primetest/05digits	10.88 ms	10.19 ms	-6.3%
primetest/10digits	21.24 ms	19.86 ms	-6.5%
primetest/30digits	65.31 ms	61.24 ms	-6.2%
primetest/50digits	108.4 ms	101.7 ms	-6.2%
queens4x4/bt	5.514 ms	5.400 ms	-2.1%
queens4x4/bm	6.975 ms	6.851 ms	-1.8%
queens4x4/bjbt1	6.702 ms	6.547 ms	-2.3%
queens4x4/bjbt2	6.279 ms	6.096 ms	-2.9%
queens4x4/fc	14.13 ms	13.52 ms	-4.3%
queens5x5/bt	74.80 ms	71.40 ms	-4.5%
queens5x5/bm	77.96 ms	75.00 ms	-3.8%
queens5x5/bjbt1	87.51 ms	83.48 ms	-4.6%
queens5x5/bjbt2	84.97 ms	81.94 ms	-3.6%
queens5x5/fc	179.8 ms	174.9 ms	-2.7%
TOTAL	1.002 s	963.3 ms	-3.9%

[kwxm]

/benchmark nofib

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
clausify/formula1	3.033 ms	3.025 ms	-0.3%
clausify/formula2	4.073 ms	4.058 ms	-0.4%
clausify/formula3	11.13 ms	11.15 ms	+0.2%
clausify/formula4	25.34 ms	25.46 ms	+0.5%
clausify/formula5	53.89 ms	54.56 ms	+1.2%
knights/4x4	17.47 ms	17.59 ms	+0.7%
knights/6x6	45.54 ms	45.81 ms	+0.6%
knights/8x8	79.83 ms	80.81 ms	+1.2%
primetest/05digits	10.01 ms	10.26 ms	+2.5%
primetest/10digits	19.60 ms	20.09 ms	+2.5%
primetest/30digits	60.11 ms	62.09 ms	+3.3%
primetest/50digits	100.5 ms	103.5 ms	+3.0%
queens4x4/bt	5.269 ms	5.341 ms	+1.4%
queens4x4/bm	6.721 ms	6.806 ms	+1.3%
queens4x4/bjbt1	6.416 ms	6.508 ms	+1.4%
queens4x4/bjbt2	6.010 ms	6.095 ms	+1.4%
queens4x4/fc	13.56 ms	13.75 ms	+1.4%
queens5x5/bt	71.65 ms	72.85 ms	+1.7%
queens5x5/bm	75.22 ms	76.54 ms	+1.8%
queens5x5/bjbt1	84.21 ms	85.37 ms	+1.4%
queens5x5/bjbt2	81.56 ms	83.05 ms	+1.8%
queens5x5/fc	170.9 ms	174.5 ms	+2.1%
TOTAL	952.0 ms	969.2 ms	+1.8%

[kwxm]

/benchmark nofib

[kwxm]

/benchmark validation

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
clausify/formula1	3.087 ms	3.043 ms	-1.4%
clausify/formula2	4.133 ms	4.075 ms	-1.4%
clausify/formula3	11.29 ms	11.17 ms	-1.1%
clausify/formula4	25.73 ms	25.44 ms	-1.1%
clausify/formula5	54.78 ms	54.08 ms	-1.3%
knights/4x4	17.73 ms	17.69 ms	-0.2%
knights/6x6	46.13 ms	46.08 ms	-0.1%
knights/8x8	80.80 ms	80.50 ms	-0.4%
primetest/05digits	10.13 ms	10.16 ms	+0.3%
primetest/10digits	19.77 ms	19.89 ms	+0.6%
primetest/30digits	61.07 ms	61.33 ms	+0.4%
primetest/50digits	102.0 ms	102.0 ms	0.0%
queens4x4/bt	5.296 ms	5.321 ms	+0.5%
queens4x4/bm	6.752 ms	6.791 ms	+0.6%
queens4x4/bjbt1	6.451 ms	6.480 ms	+0.4%
queens4x4/bjbt2	6.036 ms	6.067 ms	+0.5%
queens4x4/fc	13.62 ms	13.70 ms	+0.6%
queens5x5/bt	71.94 ms	72.45 ms	+0.7%
queens5x5/bm	75.47 ms	76.07 ms	+0.8%
queens5x5/bjbt1	83.95 ms	84.76 ms	+1.0%
queens5x5/bjbt2	81.52 ms	82.12 ms	+0.7%
queens5x5/fc	172.4 ms	173.6 ms	+0.7%
TOTAL	960.1 ms	962.8 ms	+0.3%

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'validation' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
auction_1-1	182.9 μs	182.0 μs	-0.5%
auction_1-2	645.6 μs	642.9 μs	-0.4%
auction_1-3	643.3 μs	639.5 μs	-0.6%
auction_1-4	235.5 μs	233.0 μs	-1.1%
auction_2-1	183.0 μs	182.3 μs	-0.4%
auction_2-2	645.0 μs	642.6 μs	-0.4%
auction_2-3	844.5 μs	838.6 μs	-0.7%
auction_2-4	644.8 μs	633.0 μs	-1.8%
auction_2-5	234.6 μs	233.2 μs	-0.6%
crowdfunding-success-1	214.0 μs	212.2 μs	-0.8%
crowdfunding-success-2	214.2 μs	212.2 μs	-0.9%
crowdfunding-success-3	214.4 μs	212.3 μs	-1.0%
currency-1	250.7 μs	248.6 μs	-0.8%
escrow-redeem_1-1	354.2 μs	350.9 μs	-0.9%
escrow-redeem_1-2	354.4 μs	351.7 μs	-0.8%
escrow-redeem_2-1	411.4 μs	404.5 μs	-1.7%
escrow-redeem_2-2	413.3 μs	409.1 μs	-1.0%
escrow-redeem_2-3	412.8 μs	413.8 μs	+0.2%
escrow-refund-1	158.1 μs	158.7 μs	+0.4%
future-increase-margin-1	251.7 μs	251.7 μs	0.0%
future-increase-margin-2	541.0 μs	544.2 μs	+0.6%
future-increase-margin-3	542.0 μs	546.1 μs	+0.8%
future-increase-margin-4	491.5 μs	496.6 μs	+1.0%
future-increase-margin-5	820.9 μs	826.3 μs	+0.7%
future-pay-out-1	249.1 μs	251.2 μs	+0.8%
future-pay-out-2	541.2 μs	545.9 μs	+0.9%
future-pay-out-3	541.3 μs	545.1 μs	+0.7%
future-pay-out-4	819.9 μs	822.8 μs	+0.4%
future-settle-early-1	249.2 μs	251.2 μs	+0.8%
future-settle-early-2	539.8 μs	543.0 μs	+0.6%
future-settle-early-3	540.6 μs	545.3 μs	+0.9%
future-settle-early-4	614.1 μs	604.0 μs	-1.6%
game-sm-success_1-1	392.1 μs	386.6 μs	-1.4%
game-sm-success_1-2	202.1 μs	198.6 μs	-1.7%
game-sm-success_1-3	642.9 μs	633.3 μs	-1.5%
game-sm-success_1-4	229.7 μs	227.0 μs	-1.2%
game-sm-success_2-1	392.9 μs	386.9 μs	-1.5%
game-sm-success_2-2	202.2 μs	198.8 μs	-1.7%
game-sm-success_2-3	643.2 μs	633.7 μs	-1.5%
game-sm-success_2-4	229.9 μs	227.6 μs	-1.0%
game-sm-success_2-5	642.2 μs	644.2 μs	+0.3%
game-sm-success_2-6	230.2 μs	231.7 μs	+0.7%
multisig-sm-1	400.7 μs	404.5 μs	+0.9%
multisig-sm-2	388.7 μs	392.6 μs	+1.0%
multisig-sm-3	394.2 μs	397.5 μs	+0.8%
multisig-sm-4	397.2 μs	401.7 μs	+1.1%
multisig-sm-5	562.0 μs	568.2 μs	+1.1%
multisig-sm-6	401.5 μs	403.8 μs	+0.6%
multisig-sm-7	389.3 μs	392.8 μs	+0.9%
multisig-sm-8	393.7 μs	397.5 μs	+1.0%
multisig-sm-9	397.7 μs	401.7 μs	+1.0%
multisig-sm-10	562.5 μs	568.9 μs	+1.1%
ping-pong-1	329.9 μs	333.3 μs	+1.0%
ping-pong-2	330.1 μs	333.6 μs	+1.1%
ping-pong_2-1	202.0 μs	203.8 μs	+0.9%
prism-1	171.3 μs	173.2 μs	+1.1%
prism-2	413.8 μs	414.9 μs	+0.3%
prism-3	369.7 μs	372.6 μs	+0.8%
pubkey-1	144.0 μs	144.8 μs	+0.6%
stablecoin_1-1	914.1 μs	918.8 μs	+0.5%
stablecoin_1-2	198.0 μs	199.4 μs	+0.7%
stablecoin_1-3	1.052 ms	1.055 ms	+0.3%
stablecoin_1-4	210.3 μs	211.0 μs	+0.3%
stablecoin_1-5	1.350 ms	1.357 ms	+0.5%
stablecoin_1-6	258.8 μs	261.5 μs	+1.0%
stablecoin_2-1	912.9 μs	919.4 μs	+0.7%
stablecoin_2-2	197.3 μs	199.8 μs	+1.3%
stablecoin_2-3	1.049 ms	1.054 ms	+0.5%
stablecoin_2-4	209.8 μs	211.3 μs	+0.7%
token-account-1	191.3 μs	192.8 μs	+0.8%
token-account-2	338.2 μs	341.3 μs	+0.9%
uniswap-1	412.9 μs	415.8 μs	+0.7%
uniswap-2	222.8 μs	224.4 μs	+0.7%
uniswap-3	1.765 ms	1.781 ms	+0.9%
uniswap-4	338.2 μs	341.0 μs	+0.8%
uniswap-5	1.166 ms	1.173 ms	+0.6%
uniswap-6	325.5 μs	327.2 μs	+0.5%
vesting-1	351.3 μs	352.4 μs	+0.3%
TOTAL	35.52 ms	35.58 ms	+0.2%

[kwxm]

/benchmark nofib

[kwxm]

/benchmark nofib

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
clausify/formula1	3.064 ms	3.148 ms	+2.7%
clausify/formula2	4.060 ms	4.218 ms	+3.9%
clausify/formula3	11.14 ms	11.58 ms	+3.9%
clausify/formula4	25.67 ms	26.81 ms	+4.4%
clausify/formula5	55.38 ms	56.49 ms	+2.0%
knights/4x4	17.90 ms	18.85 ms	+5.3%
knights/6x6	46.56 ms	48.49 ms	+4.1%
knights/8x8	81.68 ms	85.29 ms	+4.4%
primetest/05digits	10.30 ms	10.82 ms	+5.0%
primetest/10digits	20.12 ms	21.07 ms	+4.7%
primetest/30digits	60.53 ms	64.75 ms	+7.0%
primetest/50digits	102.7 ms	107.7 ms	+4.9%
queens4x4/bt	5.403 ms	5.577 ms	+3.2%
queens4x4/bm	6.886 ms	7.106 ms	+3.2%
queens4x4/bjbt1	6.575 ms	6.779 ms	+3.1%
queens4x4/bjbt2	6.212 ms	6.340 ms	+2.1%
queens4x4/fc	13.89 ms	14.46 ms	+4.1%
queens5x5/bt	73.46 ms	76.10 ms	+3.6%
queens5x5/bm	77.07 ms	79.43 ms	+3.1%
queens5x5/bjbt1	85.61 ms	88.73 ms	+3.6%
queens5x5/bjbt2	83.06 ms	86.17 ms	+3.7%
queens5x5/fc	175.8 ms	182.7 ms	+3.9%
TOTAL	973.1 ms	1.013 s	+4.1%

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
clausify/formula1	3.111 ms	3.113 ms	+0.1%
clausify/formula2	4.149 ms	4.167 ms	+0.4%
clausify/formula3	11.38 ms	11.44 ms	+0.5%
clausify/formula4	25.84 ms	25.98 ms	+0.5%
clausify/formula5	54.96 ms	55.40 ms	+0.8%
knights/4x4	17.84 ms	17.89 ms	+0.3%
knights/6x6	46.46 ms	46.70 ms	+0.5%
knights/8x8	81.41 ms	81.60 ms	+0.2%
primetest/05digits	10.20 ms	10.26 ms	+0.6%
primetest/10digits	19.88 ms	20.01 ms	+0.7%
primetest/30digits	61.30 ms	61.64 ms	+0.6%
primetest/50digits	101.8 ms	102.8 ms	+1.0%
queens4x4/bt	5.367 ms	5.395 ms	+0.5%
queens4x4/bm	6.838 ms	6.881 ms	+0.6%
queens4x4/bjbt1	6.536 ms	6.566 ms	+0.5%
queens4x4/bjbt2	6.112 ms	6.146 ms	+0.6%
queens4x4/fc	13.81 ms	14.05 ms	+1.7%
queens5x5/bt	72.91 ms	73.28 ms	+0.5%
queens5x5/bm	76.61 ms	76.95 ms	+0.4%
queens5x5/bjbt1	85.22 ms	85.52 ms	+0.4%
queens5x5/bjbt2	82.66 ms	83.02 ms	+0.4%
queens5x5/fc	175.1 ms	175.9 ms	+0.5%
TOTAL	969.5 ms	974.7 ms	+0.5%

[kwxm]

/benchmark nofib

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
clausify/formula1	3.082 ms	3.033 ms	-1.6%
clausify/formula2	4.117 ms	4.060 ms	-1.4%
clausify/formula3	11.27 ms	11.26 ms	-0.1%
clausify/formula4	25.64 ms	25.62 ms	-0.1%
clausify/formula5	54.65 ms	54.44 ms	-0.4%
knights/4x4	17.72 ms	17.63 ms	-0.5%
knights/6x6	46.12 ms	46.00 ms	-0.3%
knights/8x8	79.55 ms	80.61 ms	+1.3%
primetest/05digits	10.08 ms	10.07 ms	-0.1%
primetest/10digits	19.78 ms	19.64 ms	-0.7%
primetest/30digits	61.54 ms	60.48 ms	-1.7%
primetest/50digits	102.4 ms	100.6 ms	-1.8%
queens4x4/bt	5.341 ms	5.306 ms	-0.7%
queens4x4/bm	6.883 ms	6.771 ms	-1.6%
queens4x4/bjbt1	6.501 ms	6.483 ms	-0.3%
queens4x4/bjbt2	6.081 ms	6.051 ms	-0.5%
queens4x4/fc	13.72 ms	13.69 ms	-0.2%
queens5x5/bt	72.62 ms	72.18 ms	-0.6%
queens5x5/bm	76.18 ms	75.83 ms	-0.5%
queens5x5/bjbt1	84.79 ms	84.45 ms	-0.4%
queens5x5/bjbt2	82.15 ms	81.98 ms	-0.2%
queens5x5/fc	173.7 ms	173.7 ms	0.0%
TOTAL	963.9 ms	959.9 ms	-0.4%

[kwxm]

/benchmark nofib

[kwxm]

/benchmark validation

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
clausify/formula1	3.016 ms	2.998 ms	-0.6%
clausify/formula2	4.048 ms	4.023 ms	-0.6%
clausify/formula3	11.06 ms	11.06 ms	0.0%
clausify/formula4	25.19 ms	25.22 ms	+0.1%
clausify/formula5	53.60 ms	53.51 ms	-0.2%
knights/4x4	17.31 ms	17.32 ms	+0.1%
knights/6x6	45.46 ms	45.35 ms	-0.2%
knights/8x8	80.22 ms	79.47 ms	-0.9%
primetest/05digits	10.15 ms	10.02 ms	-1.3%
primetest/10digits	19.81 ms	19.63 ms	-0.9%
primetest/30digits	61.15 ms	60.52 ms	-1.0%
primetest/50digits	101.3 ms	100.6 ms	-0.7%
queens4x4/bt	5.367 ms	5.336 ms	-0.6%
queens4x4/bm	6.681 ms	6.766 ms	+1.3%
queens4x4/bjbt1	6.399 ms	6.458 ms	+0.9%
queens4x4/bjbt2	5.989 ms	6.053 ms	+1.1%
queens4x4/fc	13.54 ms	13.64 ms	+0.7%
queens5x5/bt	72.26 ms	72.21 ms	-0.1%
queens5x5/bm	75.11 ms	75.92 ms	+1.1%
queens5x5/bjbt1	83.72 ms	84.66 ms	+1.1%
queens5x5/bjbt2	80.75 ms	82.29 ms	+1.9%
queens5x5/fc	171.0 ms	174.1 ms	+1.8%
TOTAL	953.1 ms	957.2 ms	+0.4%

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'validation' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
auction_1-1	178.7 μs	179.0 μs	+0.2%
auction_1-2	631.3 μs	632.1 μs	+0.1%
auction_1-3	628.6 μs	627.9 μs	-0.1%
auction_1-4	230.0 μs	230.6 μs	+0.3%
auction_2-1	178.7 μs	180.2 μs	+0.8%
auction_2-2	631.0 μs	632.0 μs	+0.2%
auction_2-3	823.5 μs	824.5 μs	+0.1%
auction_2-4	627.6 μs	626.5 μs	-0.2%
auction_2-5	229.6 μs	230.1 μs	+0.2%
crowdfunding-success-1	209.1 μs	209.5 μs	+0.2%
crowdfunding-success-2	209.1 μs	209.3 μs	+0.1%
crowdfunding-success-3	209.6 μs	210.8 μs	+0.6%
currency-1	244.6 μs	246.4 μs	+0.7%
escrow-redeem_1-1	345.4 μs	347.3 μs	+0.6%
escrow-redeem_1-2	346.1 μs	347.6 μs	+0.4%
escrow-redeem_2-1	403.4 μs	409.0 μs	+1.4%
escrow-redeem_2-2	404.3 μs	411.6 μs	+1.8%
escrow-redeem_2-3	407.7 μs	411.0 μs	+0.8%
escrow-refund-1	156.2 μs	157.3 μs	+0.7%
future-increase-margin-1	248.6 μs	249.6 μs	+0.4%
future-increase-margin-2	538.6 μs	539.7 μs	+0.2%
future-increase-margin-3	540.1 μs	540.0 μs	-0.0%
future-increase-margin-4	492.3 μs	493.0 μs	+0.1%
future-increase-margin-5	819.5 μs	821.4 μs	+0.2%
future-pay-out-1	248.6 μs	249.4 μs	+0.3%
future-pay-out-2	537.9 μs	537.7 μs	-0.0%
future-pay-out-3	541.4 μs	539.3 μs	-0.4%
future-pay-out-4	818.4 μs	818.0 μs	-0.0%
future-settle-early-1	249.0 μs	248.7 μs	-0.1%
future-settle-early-2	536.9 μs	537.5 μs	+0.1%
future-settle-early-3	538.2 μs	539.5 μs	+0.2%
future-settle-early-4	610.6 μs	608.2 μs	-0.4%
game-sm-success_1-1	390.2 μs	390.0 μs	-0.1%
game-sm-success_1-2	201.1 μs	201.7 μs	+0.3%
game-sm-success_1-3	641.1 μs	640.8 μs	-0.0%
game-sm-success_1-4	229.2 μs	229.6 μs	+0.2%
game-sm-success_2-1	389.7 μs	390.4 μs	+0.2%
game-sm-success_2-2	201.2 μs	201.6 μs	+0.2%
game-sm-success_2-3	640.5 μs	639.6 μs	-0.1%
game-sm-success_2-4	228.9 μs	229.4 μs	+0.2%
game-sm-success_2-5	641.0 μs	641.4 μs	+0.1%
game-sm-success_2-6	229.6 μs	230.0 μs	+0.2%
multisig-sm-1	400.4 μs	399.2 μs	-0.3%
multisig-sm-2	389.2 μs	384.8 μs	-1.1%
multisig-sm-3	391.8 μs	388.6 μs	-0.8%
multisig-sm-4	395.4 μs	390.6 μs	-1.2%
multisig-sm-5	559.9 μs	550.8 μs	-1.6%
multisig-sm-6	399.6 μs	395.2 μs	-1.1%
multisig-sm-7	388.1 μs	382.6 μs	-1.4%
multisig-sm-8	392.5 μs	386.7 μs	-1.5%
multisig-sm-9	398.1 μs	392.0 μs	-1.5%
multisig-sm-10	559.4 μs	553.2 μs	-1.1%
ping-pong-1	330.4 μs	325.2 μs	-1.6%
ping-pong-2	327.9 μs	324.9 μs	-0.9%
ping-pong_2-1	201.5 μs	198.1 μs	-1.7%
prism-1	170.4 μs	168.0 μs	-1.4%
prism-2	410.1 μs	404.2 μs	-1.4%
prism-3	367.6 μs	363.1 μs	-1.2%
pubkey-1	142.9 μs	141.5 μs	-1.0%
stablecoin_1-1	909.0 μs	895.3 μs	-1.5%
stablecoin_1-2	196.5 μs	194.7 μs	-0.9%
stablecoin_1-3	1.043 ms	1.029 ms	-1.3%
stablecoin_1-4	208.4 μs	206.2 μs	-1.1%
stablecoin_1-5	1.340 ms	1.325 ms	-1.1%
stablecoin_1-6	258.1 μs	255.3 μs	-1.1%
stablecoin_2-1	907.1 μs	896.5 μs	-1.2%
stablecoin_2-2	197.0 μs	194.8 μs	-1.1%
stablecoin_2-3	1.041 ms	1.031 ms	-1.0%
stablecoin_2-4	208.0 μs	206.0 μs	-1.0%
token-account-1	190.7 μs	188.2 μs	-1.3%
token-account-2	336.7 μs	331.9 μs	-1.4%
uniswap-1	411.1 μs	405.0 μs	-1.5%
uniswap-2	221.2 μs	219.4 μs	-0.8%
uniswap-3	1.764 ms	1.737 ms	-1.5%
uniswap-4	336.1 μs	333.4 μs	-0.8%
uniswap-5	1.163 ms	1.145 ms	-1.5%
uniswap-6	323.2 μs	319.3 μs	-1.2%
vesting-1	350.9 μs	344.4 μs	-1.9%
TOTAL	35.24 ms	35.05 ms	-0.5%

[kwxm]

/benchmark nofib

[kwxm]

/benchmark validation

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
clausify/formula1	3.110 ms	3.040 ms	-2.3%
clausify/formula2	4.166 ms	4.100 ms	-1.6%
clausify/formula3	11.41 ms	11.15 ms	-2.3%
clausify/formula4	26.00 ms	25.44 ms	-2.2%
clausify/formula5	55.47 ms	54.19 ms	-2.3%
knights/4x4	17.89 ms	17.64 ms	-1.4%
knights/6x6	46.82 ms	45.53 ms	-2.8%
knights/8x8	82.00 ms	79.90 ms	-2.6%
primetest/05digits	10.31 ms	10.08 ms	-2.2%
primetest/10digits	20.19 ms	19.72 ms	-2.3%
primetest/30digits	62.01 ms	60.80 ms	-2.0%
primetest/50digits	103.1 ms	101.2 ms	-1.8%
queens4x4/bt	5.404 ms	5.291 ms	-2.1%
queens4x4/bm	6.890 ms	6.763 ms	-1.8%
queens4x4/bjbt1	6.582 ms	6.452 ms	-2.0%
queens4x4/bjbt2	6.150 ms	6.042 ms	-1.8%
queens4x4/fc	13.91 ms	13.67 ms	-1.7%
queens5x5/bt	73.45 ms	72.28 ms	-1.6%
queens5x5/bm	77.15 ms	75.99 ms	-1.5%
queens5x5/bjbt1	85.80 ms	84.68 ms	-1.3%
queens5x5/bjbt2	83.34 ms	82.17 ms	-1.4%
queens5x5/fc	176.3 ms	174.0 ms	-1.3%
TOTAL	977.5 ms	960.1 ms	-1.8%

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Comparing benchmark results of 'validation' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

Script	ffe41b8	a6d9765	Change
auction_1-1	181.4 μs	180.3 μs	-0.6%
auction_1-2	638.6 μs	632.9 μs	-0.9%
auction_1-3	642.9 μs	633.2 μs	-1.5%
auction_1-4	236.3 μs	231.9 μs	-1.9%
auction_2-1	183.8 μs	180.3 μs	-1.9%
auction_2-2	648.3 μs	637.2 μs	-1.7%
auction_2-3	846.4 μs	832.8 μs	-1.6%
auction_2-4	644.0 μs	632.5 μs	-1.8%
auction_2-5	235.7 μs	231.5 μs	-1.8%
crowdfunding-success-1	214.3 μs	210.7 μs	-1.7%
crowdfunding-success-2	214.9 μs	211.2 μs	-1.7%
crowdfunding-success-3	214.9 μs	211.2 μs	-1.7%
currency-1	251.1 μs	246.8 μs	-1.7%
escrow-redeem_1-1	354.0 μs	348.7 μs	-1.5%
escrow-redeem_1-2	354.9 μs	347.4 μs	-2.1%
escrow-redeem_2-1	414.6 μs	408.9 μs	-1.4%
escrow-redeem_2-2	415.1 μs	409.2 μs	-1.4%
escrow-redeem_2-3	414.9 μs	409.2 μs	-1.4%
escrow-refund-1	158.1 μs	156.3 μs	-1.1%
future-increase-margin-1	251.9 μs	248.6 μs	-1.3%
future-increase-margin-2	546.4 μs	538.8 μs	-1.4%
future-increase-margin-3	545.6 μs	539.0 μs	-1.2%
future-increase-margin-4	498.4 μs	490.7 μs	-1.5%
future-increase-margin-5	826.7 μs	815.5 μs	-1.4%
future-pay-out-1	252.0 μs	248.6 μs	-1.3%
future-pay-out-2	545.9 μs	537.1 μs	-1.6%
future-pay-out-3	547.1 μs	533.9 μs	-2.4%
future-pay-out-4	826.9 μs	808.8 μs	-2.2%
future-settle-early-1	251.2 μs	249.0 μs	-0.9%
future-settle-early-2	544.8 μs	539.6 μs	-1.0%
future-settle-early-3	545.5 μs	536.0 μs	-1.7%
future-settle-early-4	617.8 μs	607.9 μs	-1.6%
game-sm-success_1-1	395.8 μs	391.7 μs	-1.0%
game-sm-success_1-2	203.8 μs	201.6 μs	-1.1%
game-sm-success_1-3	649.2 μs	640.5 μs	-1.3%
game-sm-success_1-4	231.1 μs	228.8 μs	-1.0%
game-sm-success_2-1	393.7 μs	388.3 μs	-1.4%
game-sm-success_2-2	203.2 μs	201.0 μs	-1.1%
game-sm-success_2-3	645.0 μs	635.3 μs	-1.5%
game-sm-success_2-4	231.3 μs	227.0 μs	-1.9%
game-sm-success_2-5	646.5 μs	637.7 μs	-1.4%
game-sm-success_2-6	230.8 μs	230.2 μs	-0.3%
multisig-sm-1	403.4 μs	397.5 μs	-1.5%
multisig-sm-2	391.2 μs	384.9 μs	-1.6%
multisig-sm-3	397.2 μs	390.5 μs	-1.7%
multisig-sm-4	400.7 μs	393.7 μs	-1.7%
multisig-sm-5	565.9 μs	559.8 μs	-1.1%
multisig-sm-6	403.6 μs	397.5 μs	-1.5%
multisig-sm-7	391.8 μs	388.1 μs	-0.9%
multisig-sm-8	395.8 μs	389.3 μs	-1.6%
multisig-sm-9	399.1 μs	397.4 μs	-0.4%
multisig-sm-10	565.1 μs	556.3 μs	-1.6%
ping-pong-1	331.7 μs	327.8 μs	-1.2%
ping-pong-2	328.1 μs	328.2 μs	+0.0%
ping-pong_2-1	199.5 μs	201.1 μs	+0.8%
prism-1	169.5 μs	171.1 μs	+0.9%
prism-2	407.1 μs	412.2 μs	+1.3%
prism-3	365.2 μs	368.0 μs	+0.8%
pubkey-1	142.1 μs	144.0 μs	+1.3%
stablecoin_1-1	901.4 μs	905.9 μs	+0.5%
stablecoin_1-2	195.7 μs	199.1 μs	+1.7%
stablecoin_1-3	1.036 ms	1.058 ms	+2.1%
stablecoin_1-4	206.9 μs	211.9 μs	+2.4%
stablecoin_1-5	1.330 ms	1.357 ms	+2.0%
stablecoin_1-6	256.5 μs	262.6 μs	+2.4%
stablecoin_2-1	907.9 μs	919.8 μs	+1.3%
stablecoin_2-2	196.9 μs	199.7 μs	+1.4%
stablecoin_2-3	1.044 ms	1.054 ms	+1.0%
stablecoin_2-4	208.5 μs	211.6 μs	+1.5%
token-account-1	190.7 μs	193.0 μs	+1.2%
token-account-2	336.8 μs	340.9 μs	+1.2%
uniswap-1	414.3 μs	417.5 μs	+0.8%
uniswap-2	224.6 μs	223.9 μs	-0.3%
uniswap-3	1.783 ms	1.780 ms	-0.2%
uniswap-4	340.6 μs	341.4 μs	+0.2%
uniswap-5	1.175 ms	1.176 ms	+0.1%
uniswap-6	328.3 μs	328.6 μs	+0.1%
vesting-1	355.5 μs	354.5 μs	-0.3%
TOTAL	35.58 ms	35.37 ms	-0.6%

[kwxm]

/benchmark nofib

[kwxm]

/benchmark validation

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Click here to check the status of your benchmark.

[kwxm]

/benchmark validation

[kwxm]

/benchmark nofib

[github-actions]

Click here to check the status of your benchmark.

[github-actions]

Click here to check the status of your benchmark.

[kwxm]

This is inconclusive. There's a lot of variation in the results but it's not clear if that's due to the reduction in benchmarking times or inconsistency in the machine.

[kwxm]

Also the benchmarks have stopped working because the base commit has vanished, so I'll close this and try again some other time.