Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: add more sims tests #23421

Merged
merged 2 commits into from
Jan 17, 2025
Merged

test: add more sims tests #23421

merged 2 commits into from
Jan 17, 2025

Conversation

alpe
Copy link
Contributor

@alpe alpe commented Jan 16, 2025

Description

See #23265

Adds sims fuzzer and benchmark tests


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • included the correct type prefix in the PR title, you can find examples of the prefixes below:
  • confirmed ! in the type prefix if API or client breaking change
  • targeted the correct branch (see PR Targeting)
  • provided a link to the relevant issue or specification
  • reviewed "Files changed" and left comments if necessary
  • included the necessary unit and integration tests
  • added a changelog entry to CHANGELOG.md
  • updated the relevant documentation or specification, including comments for documenting Go code
  • confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

Please see Pull Request Reviewer section in the contributing guide for more information on how to review a pull request.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

Summary by CodeRabbit

  • New Features

    • Added new simulation testing framework with enhanced random source generation
    • Introduced fuzz testing for full application simulation
    • Added benchmark testing for full application simulation
  • Improvements

    • Updated simulation testing infrastructure to support more flexible random source handling
    • Transitioned testing environment to use simapp/v2 directory
  • Testing

    • Implemented comprehensive test coverage for random number generation sources
    • Added new test methods for app state determinism and simulation scenarios
  • Chores

    • Removed test-sim-benchmark-invariants testing target
    • Updated build and testing configuration files

@alpe alpe requested review from julienrbrt, tac0turtle and a team as code owners January 16, 2025 16:42
Copy link
Contributor

coderabbitai bot commented Jan 16, 2025

📝 Walkthrough

Walkthrough

This pull request introduces significant changes to the simulation testing framework, primarily focusing on the transition to simapp/v2. The modifications include updating build scripts, adding new test files, and implementing a more flexible random source generation mechanism. The changes affect how simulation tests are run, with a shift towards using a new directory structure and enhanced random source handling for more consistent and reproducible testing.

Changes

File Change Summary
scripts/build/simulations.mk - Removed test-sim-benchmark-invariants target
- Updated test targets to use simapp/v2 directory
- Uncommented and simplified benchmark commands
simapp/v2/.gitignore - Added entry to ignore simapp.test file
simapp/v2/sim_bench_test.go - Added BenchmarkFullAppSimulation function
- Included build constraint for sims tag
simapp/v2/sim_fuzz_test.go - New file with FuzzFullAppSimulation function
- Implemented fuzz testing for full app simulation
simapp/v2/sim_runner.go - Replaced Seed with RandSource in TestInstance
- Updated function signatures for random source handling
- Added RunWithRandSource function
simapp/v2/sim_test.go - Updated seed handling in TestAppStateDeterminism
- Modified test instance setup
simsx/v2/rand_source.go - Added RandSource interface
- Implemented SeededRandomSource and ByteSource
- Added methods for random source generation
simsx/v2/rand_source_test.go - Added tests for SeededRandSource and ByteSource

Possibly related PRs

Suggested Labels

C:x/nft, C:CLI

Suggested Reviewers

  • tac0turtle
  • alpe

Finishing Touches

  • 📝 Generate Docstrings (Beta)

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@julienrbrt julienrbrt added the backport/v0.52.x PR scheduled for inclusion in the v0.52's next stable release label Jan 16, 2025
@alpe alpe changed the title tests: add more sims tests test: add more sims tests Jan 16, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (7)
simapp/v2/sim_bench_test.go (1)

10-17: Add documentation for the benchmark function.

Following Uber's Go style guide, exported functions should be documented. Add a comment explaining the purpose, parameters, and expected behavior of this benchmark.

+// BenchmarkFullAppSimulation benchmarks the full application simulation process.
+// It measures the performance of running the simulation with different random seeds.
 func BenchmarkFullAppSimulation(b *testing.B) {
simapp/v2/sim_fuzz_test.go (1)

11-23: Add documentation and error handling for the fuzz test.

The function needs documentation and should handle potential errors from RunWithRandSource.

+// FuzzFullAppSimulation implements fuzz testing for the full application simulation.
+// It uses the provided raw seed data to generate random simulation scenarios.
 func FuzzFullAppSimulation(f *testing.F) {
     cfg := simcli.NewConfigFromFlags()
     cfg.ChainID = SimAppChainID

     f.Fuzz(func(t *testing.T, rawSeed []byte) {
         if len(rawSeed) < 8 {
             t.Skip()
             return
         }
         randSource := simsxv2.NewByteSource(cfg.FuzzSeed, cfg.Seed)
-        RunWithRandSource[Tx](t, NewSimApp[Tx], AppConfig, cfg, randSource)
+        if err := RunWithRandSource[Tx](t, NewSimApp[Tx], AppConfig, cfg, randSource); err != nil {
+            t.Fatal(err)
+        }
     })
 }
simsx/v2/rand_source_test.go (2)

9-20: Add test cases for edge cases in SeededRandSource.

The test should verify behavior with zero and negative seeds, and include benchmarks for performance comparison.

 func TestSeededRandSource(t *testing.T) {
     const (
         seed1              int64 = 1
         firstValFromSeed1  int64 = 0x4d65822107fcfd52
         secondValFromSeed1 int64 = 0x78629a0f5f3f164f
     )
-    src := NewSeededRandSource(seed1)
-    for _, v := range []int64{firstValFromSeed1, secondValFromSeed1} {
-        assert.Equal(t, v, src.Int63())
+    tests := []struct {
+        name     string
+        seed     int64
+        expected []int64
+    }{
+        {
+            name: "positive seed",
+            seed: seed1,
+            expected: []int64{firstValFromSeed1, secondValFromSeed1},
+        },
+        {
+            name: "zero seed",
+            seed: 0,
+            expected: []int64{0x0, 0x0}, // Add actual expected values
+        },
+        {
+            name: "negative seed",
+            seed: -1,
+            expected: []int64{0x0, 0x0}, // Add actual expected values
+        },
     }
-    assert.Equal(t, seed1, src.GetSeed())
+    for _, tt := range tests {
+        t.Run(tt.name, func(t *testing.T) {
+            src := NewSeededRandSource(tt.seed)
+            for i, expected := range tt.expected {
+                assert.Equal(t, expected, src.Int63(), "iteration %d", i)
+            }
+            assert.Equal(t, tt.seed, src.GetSeed())
+        })
+    }
 }

22-53: Add test cases for panic conditions in ByteSource.

The test should verify that Seed() and GetSeed() methods panic as expected.

 func TestByteSource(t *testing.T) {
+    t.Run("seed method panics", func(t *testing.T) {
+        src := NewByteSource([]byte{}, 1)
+        assert.Panics(t, func() {
+            src.Seed(1)
+        })
+    })
+
+    t.Run("getSeed method panics", func(t *testing.T) {
+        src := NewByteSource([]byte{}, 1)
+        assert.Panics(t, func() {
+            src.GetSeed()
+        })
+    })
simsx/v2/rand_source.go (1)

49-55: Enhance documentation for ByteSource struct.

The documentation should include more details about the fallback behavior and thread safety considerations.

-// ByteSource offers deterministic pseudo-random numbers for math.Rand with fuzzer support.
-// The 'seed' data is read in big endian to uint64. When exhausted,
-// it falls back to a standard random number generator initialized with a specific 'seed' value.
+// ByteSource provides deterministic pseudo-random numbers for math.Rand with fuzzer support.
+// It implements the RandSource interface but does not support seed manipulation after creation.
+//
+// The source reads seed data in big-endian format to generate uint64 values. When the seed data
+// is exhausted, it automatically falls back to a standard random number generator initialized
+// with the provided seed value.
+//
+// Thread Safety: ByteSource is not safe for concurrent use. Callers must ensure
+// proper synchronization when using ByteSource across multiple goroutines.
 type ByteSource struct {
scripts/build/simulations.mk (1)

71-71: Consider parameterizing the fuzz test configuration.

The hardcoded values for numBlocks=2 and blockSize=20 could be made configurable through variables, similar to how SIM_NUM_BLOCKS and SIM_BLOCK_SIZE are used in the benchmark target.

+SIM_FUZZ_NUM_BLOCKS ?= 2
+SIM_FUZZ_BLOCK_SIZE ?= 20
 test-sim-fuzz:
 	@echo "Running application fuzz for numBlocks=2, blockSize=20. This may take awhile!"
-	@cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -json -tags='sims' -timeout=60m -fuzztime=60m -run=^$$ -fuzz=FuzzFullAppSimulation -GenesisTime=1714720615 -NumBlocks=2 -BlockSize=20
+	@cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -json -tags='sims' -timeout=60m -fuzztime=60m -run=^$$ -fuzz=FuzzFullAppSimulation -GenesisTime=1714720615 -NumBlocks=$(SIM_FUZZ_NUM_BLOCKS) -BlockSize=$(SIM_FUZZ_BLOCK_SIZE)

Also applies to: 73-73

simapp/v2/sim_runner.go (1)

Line range hint 268-293: Consider adding validation for randSource.

The RunWithRandSourceX function should validate that the provided randSource is not nil before using it.

 func RunWithRandSourceX[T Tx](
 	tb testing.TB,
 	tCfg simtypes.Config,
 	setupChainStateFn func(ctx context.Context, r *rand.Rand) (TestInstance[T], ChainState[T], []simtypes.Account),
 	randSource rand.Source,
 	postRunActions ...func(t testing.TB, cs ChainState[T], app TestInstance[T], accs []simtypes.Account),
 ) {
 	tb.Helper()
+	if randSource == nil {
+		tb.Fatal("randSource cannot be nil")
+	}
 	r := rand.New(randSource)
📜 Review details

Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 85682ac and 15e32df.

📒 Files selected for processing (8)
  • scripts/build/simulations.mk (1 hunks)
  • simapp/v2/.gitignore (1 hunks)
  • simapp/v2/sim_bench_test.go (1 hunks)
  • simapp/v2/sim_fuzz_test.go (1 hunks)
  • simapp/v2/sim_runner.go (7 hunks)
  • simapp/v2/sim_test.go (4 hunks)
  • simsx/v2/rand_source.go (1 hunks)
  • simsx/v2/rand_source_test.go (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • simapp/v2/.gitignore
🧰 Additional context used
📓 Path-based instructions (6)
simapp/v2/sim_bench_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simapp/v2/sim_fuzz_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simsx/v2/rand_source_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simapp/v2/sim_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simsx/v2/rand_source.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

simapp/v2/sim_runner.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: test-system-v2
  • GitHub Check: Analyze
  • GitHub Check: golangci-lint
  • GitHub Check: Summary
🔇 Additional comments (10)
simapp/v2/sim_bench_test.go (1)

15-15: Add error handling for RunWithSeed.

The RunWithSeed function might return errors that should be handled appropriately in the benchmark.

-		RunWithSeed[Tx](b, NewSimApp[Tx], AppConfig, cfg, 1)
+		if err := RunWithSeed[Tx](b, NewSimApp[Tx], AppConfig, cfg, 1); err != nil {
+			b.Fatal(err)
+		}
✅ Verification successful

Add error handling for RunWithSeed.

The RunWithSeed function returns an error that should be handled. The suggested fix to use b.Fatal(err) follows the established error handling pattern for benchmarks in the codebase.

🏁 Scripts executed

The following scripts were executed for the analysis:

Error: No shell scripts found


Script:

#!/bin/bash
# Find RunWithSeed implementation
ast-grep --pattern 'func RunWithSeed[$_]($_) $$$'

# Look for similar benchmark patterns
rg 'func.*Benchmark.*\(' -A 5 --type go

Length of output: 42685


Script:

#!/bin/bash
# Find RunWithSeed implementation with a more specific pattern
ast-grep --pattern 'func RunWithSeed[$_]($_, $_, $_, $_, $_) $_'

# Search for RunWithSeed using ripgrep
rg 'RunWithSeed' -A 5 --type go

Length of output: 5874

simapp/v2/sim_test.go (3)

28-31: Documentation looks good!

Clear and concise documentation explaining the test scenario.


93-93: LGTM: Consistent use of random source.

The changes correctly propagate the random source usage in the import/export simulation test.

Also applies to: 105-105


51-52: Verify the determinism with the new random source.

The transition from seed to random source looks correct. However, we should ensure that using RandSource.GetSeed() maintains the same deterministic behavior as before.

Run this script to verify determinism:

Also applies to: 54-54, 58-58

✅ Verification successful

Implementation maintains deterministic behavior

The RandSource implementation properly maintains determinism through:

  • Consistent seed initialization and storage
  • Standard Go random source usage
  • Reliable seed retrieval via GetSeed()

The test structure correctly verifies this determinism by running multiple simulations with identical seeds and comparing results.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify that multiple runs with the same random source produce identical app hashes
# Test: Run the determinism test multiple times and compare outputs
cd simapp/v2 && for i in {1..3}; do 
  go test -v -failfast -timeout=30m -tags='sims' -run TestAppStateDeterminism -NumBlocks=10 -BlockSize=50 -Seed=1 | grep "app hash"
done

Length of output: 737


Script:

#!/bin/bash
# Find RandSource implementation and related files
rg -g '!{vendor,node_modules}/**' -l "type RandSource" --type go

# Find GetSeed implementation
ast-grep --pattern 'func $_($_ RandSource) GetSeed() $_'

# Look for TestAppStateDeterminism implementation
rg -g '!{vendor,node_modules}/**' "func TestAppStateDeterminism" -A 5

Length of output: 597


Script:

#!/bin/bash
# Check RandSource implementation
cat simsx/v2/rand_source.go

# Look for RandSource usage in test
rg -g '!{vendor,node_modules}/**' "RandSource" simapp/v2/sim_test.go -B 2 -A 2

Length of output: 2996

scripts/build/simulations.mk (2)

77-79: LGTM: Benchmark configuration is well structured.

The benchmark target correctly uses the configurable SIM_NUM_BLOCKS and SIM_BLOCK_SIZE variables.


83-85: LGTM: Profile target matches benchmark configuration.

The profile target consistently uses the same configuration variables as the benchmark target.

simapp/v2/sim_runner.go (4)

109-109: LGTM: Clean transition to random source.

The TestInstance struct correctly replaces the Seed field with RandSource, aligning with the new randomization strategy.


227-230: LGTM: Clean backwards compatibility.

The RunWithSeed function maintains backward compatibility by wrapping RunWithRandSource, making the transition smoother.


Line range hint 232-265: LGTM: Well-structured random source implementation.

The new RunWithRandSource function properly handles the simulation setup and execution with the provided random source.


129-129: Verify random source initialization.

The SetupTestInstance function now accepts a randSource parameter instead of a seed. Ensure that all callers have been updated to provide a valid random source.

Also applies to: 156-156

✅ Verification successful

All callers properly updated to use RandSource

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for any remaining direct seed usage in test setup
# Test: Look for potential missed conversions from seed to random source
rg -A 3 "SetupTestInstance.*seed" simapp/v2/

Length of output: 46


Script:

#!/bin/bash
# Find all occurrences of SetupTestInstance with context
rg -A 5 "SetupTestInstance" simapp/v2/

# Also search for test files specifically
fd ".*_test\.go" simapp/v2/ --exec rg -A 5 "SetupTestInstance" {}

Length of output: 1392

simsx/v2/rand_source.go Show resolved Hide resolved
simsx/v2/rand_source.go Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (3)
simapp/v2/sim_fuzz_test.go (1)

16-17: Document the minimum seed length requirement.

The magic number 8 should be documented to explain why this minimum length is required.

+    // Skip if seed length is less than 8 bytes (size of uint64)
     if len(rawSeed) < 8 {
simsx/v2/rand_source_test.go (1)

22-53: Add tests for panic scenarios in ByteSource.

The test suite should verify that Seed() and GetSeed() methods panic as expected for ByteSource.

Add these test cases:

func TestByteSourcePanics(t *testing.T) {
    byteSource := NewByteSource([]byte{}, 1)
    
    assert.Panics(t, func() {
        byteSource.Seed(1)
    }, "Seed() should panic")
    
    assert.Panics(t, func() {
        byteSource.GetSeed()
    }, "GetSeed() should panic")
}
simsx/v2/rand_source.go (1)

49-52: Improve ByteSource documentation.

The documentation should clarify the behavior when methods are not supported and explain the fallback mechanism in more detail.

 // ByteSource offers deterministic pseudo-random numbers for math.Rand with fuzzer support.
 // The 'seed' data is read in big endian to uint64. When exhausted,
 // it falls back to a standard random number generator initialized with a specific 'seed' value.
+// Note: Seed() and GetSeed() operations are not supported and will return errors.
+// The fallback mechanism ensures continuous random number generation even after the initial
+// byte source is exhausted.
📜 Review details

Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 85682ac and 15e32df.

📒 Files selected for processing (8)
  • scripts/build/simulations.mk (1 hunks)
  • simapp/v2/.gitignore (1 hunks)
  • simapp/v2/sim_bench_test.go (1 hunks)
  • simapp/v2/sim_fuzz_test.go (1 hunks)
  • simapp/v2/sim_runner.go (7 hunks)
  • simapp/v2/sim_test.go (4 hunks)
  • simsx/v2/rand_source.go (1 hunks)
  • simsx/v2/rand_source_test.go (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • simapp/v2/.gitignore
🧰 Additional context used
📓 Path-based instructions (6)
simapp/v2/sim_bench_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simapp/v2/sim_fuzz_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simsx/v2/rand_source_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simsx/v2/rand_source.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

simapp/v2/sim_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simapp/v2/sim_runner.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

⏰ Context from checks skipped due to timeout of 90000ms (1)
  • GitHub Check: Summary
🔇 Additional comments (8)
simapp/v2/sim_test.go (2)

51-52: LGTM! The seed retrieval has been updated correctly.

The changes properly integrate with the new RandSource abstraction while maintaining the determinism check functionality.

Also applies to: 54-54, 58-58


93-93: LGTM! Proper integration with the new random source API.

The changes correctly propagate the random source through the test setup and simulation execution.

Also applies to: 105-105

scripts/build/simulations.mk (2)

77-79: LGTM! Benchmark configuration is properly structured.

The benchmark target is correctly configured with appropriate parameters and flags.


83-85: LGTM! Profiling configuration is properly set up.

The profile target correctly configures both CPU and memory profiling with appropriate parameters.

simapp/v2/sim_runner.go (4)

109-109: LGTM! Good abstraction of random source.

The change from Seed to RandSource improves flexibility by allowing different random source implementations.


129-129: LGTM! Proper integration of random source in test setup.

The function correctly integrates the new random source parameter while maintaining its core setup functionality.

Also applies to: 156-156


232-239: LGTM! Well-structured simulation setup with proper benchmark handling.

The function correctly integrates random source initialization and properly handles benchmark timer reset.

Also applies to: 290-292


268-271: LGTM! Well-documented and properly implemented custom chain setup.

The function provides clear documentation and correctly handles random source initialization for custom chain setups.

Also applies to: 275-275, 279-279

simapp/v2/sim_bench_test.go Show resolved Hide resolved
simapp/v2/sim_fuzz_test.go Show resolved Hide resolved
simsx/v2/rand_source.go Show resolved Hide resolved
simsx/v2/rand_source.go Show resolved Hide resolved
scripts/build/simulations.mk Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (7)
simapp/v2/sim_bench_test.go (1)

14-16: Consider parameterizing the seed value for more comprehensive benchmarking.

The benchmark uses a fixed seed value of 1, which may not provide a representative sample of simulation scenarios.

-		RunWithSeed[Tx](b, NewSimApp[Tx], AppConfig, cfg, 1)
+		RunWithSeed[Tx](b, NewSimApp[Tx], AppConfig, cfg, int64(i))
simsx/v2/rand_source_test.go (2)

11-14: Improve constant naming for better readability.

The magic numbers would be more maintainable with descriptive names explaining their significance.

-		seed1              int64 = 1
-		firstValFromSeed1  int64 = 0x4d65822107fcfd52
-		secondValFromSeed1 int64 = 0x78629a0f5f3f164f
+		testSeed          int64 = 1
+		expectedFirstVal  int64 = 0x4d65822107fcfd52  // First value from Go's math/rand with seed 1
+		expectedSecondVal int64 = 0x78629a0f5f3f164f  // Second value from Go's math/rand with seed 1

22-53: Add edge case tests for ByteSource.

The test suite should include additional cases:

  • Invalid byte lengths (e.g., 7 bytes)
  • Maximum uint64 values
  • Zero values
simsx/v2/rand_source.go (3)

82-84: Improve panic messages in ByteSource methods.

The panic messages should provide more context about why the operations are not supported.

-	panic("not supported")
+	panic("ByteSource.Seed() is not supported: use NewByteSource() to initialize with desired seed")

71-73: Consider handling EOF error case separately.

The current implementation combines EOF with other errors in the panic condition. It would be clearer to handle EOF separately.

-	if _, err := s.seed.Read(b[:]); err != nil && err != io.EOF {
-		panic(err) // Should not happen.
+	n, err := s.seed.Read(b[:])
+	if err == io.EOF {
+		return s.fallback.Uint64()
+	}
+	if err != nil {
+		panic(fmt.Sprintf("unexpected error reading from byte source: %v", err))

66-75: Consider adding buffer pooling for performance optimization.

For high-frequency random number generation, using a sync.Pool for the byte buffer could reduce allocations.

simapp/v2/sim_runner.go (1)

Line range hint 268-292: Consider adding error handling for timer reset.

The timer reset block should include error handling for cases where the type assertion fails.

-	if b, ok := tb.(interface{ ResetTimer() }); ok {
-		b.ResetTimer()
-	}
+	if b, ok := tb.(interface{ ResetTimer() }); ok {
+		b.ResetTimer()
+	} else {
+		tb.Log("warning: benchmark timer could not be reset")
+	}
📜 Review details

Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 85682ac and 15e32df.

📒 Files selected for processing (8)
  • scripts/build/simulations.mk (1 hunks)
  • simapp/v2/.gitignore (1 hunks)
  • simapp/v2/sim_bench_test.go (1 hunks)
  • simapp/v2/sim_fuzz_test.go (1 hunks)
  • simapp/v2/sim_runner.go (7 hunks)
  • simapp/v2/sim_test.go (4 hunks)
  • simsx/v2/rand_source.go (1 hunks)
  • simsx/v2/rand_source_test.go (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • simapp/v2/.gitignore
🧰 Additional context used
📓 Path-based instructions (6)
simapp/v2/sim_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simapp/v2/sim_bench_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simapp/v2/sim_fuzz_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

simapp/v2/sim_runner.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

simsx/v2/rand_source.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

simsx/v2/rand_source_test.go (2)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.


Pattern **/*_test.go: "Assess the unit test code assessing sufficient code coverage for the changes associated in the pull request"

⏰ Context from checks skipped due to timeout of 90000ms (1)
  • GitHub Check: Summary
🔇 Additional comments (10)
simapp/v2/sim_fuzz_test.go (1)

15-22: LGTM! Well-structured fuzz test implementation.

The implementation correctly handles edge cases and integrates well with the new random source system.

simapp/v2/sim_test.go (3)

28-31: LGTM! Clear and descriptive documentation.

The documentation block clearly explains the test scenario and its purpose.


51-52: LGTM! Improved seed handling.

The change from direct seed usage to RandSource.GetSeed() enhances consistency in random number generation across test runs.

Also applies to: 54-54, 58-58


93-93: LGTM! Consistent random source usage.

The update to use RandSource in test instance setup aligns with the new randomization framework.

Also applies to: 105-105

scripts/build/simulations.mk (2)

77-79: LGTM! Clean benchmark configuration.

The benchmark test command is properly configured with parameterized block size and count.


83-85: LGTM! Comprehensive profiling setup.

The profile test command correctly includes both CPU and memory profiling.

simapp/v2/sim_runner.go (4)

109-109: LGTM! Enhanced randomization handling.

The change from Seed to RandSource in TestInstance improves flexibility and encapsulation of random number generation.


129-129: LGTM! Consistent random source initialization.

The SetupTestInstance function now properly initializes the test instance with the provided random source.

Also applies to: 156-156


227-230: LGTM! Clean delegation to new implementation.

The RunWithSeed function now correctly delegates to RunWithRandSource while maintaining backward compatibility.


232-239: LGTM! Well-documented new function.

The RunWithRandSource function is properly documented and follows the same pattern as the existing functions.

scripts/build/simulations.mk Show resolved Hide resolved
@tac0turtle tac0turtle added this pull request to the merge queue Jan 17, 2025
Merged via the queue into main with commit b8638f8 Jan 17, 2025
86 of 88 checks passed
@tac0turtle tac0turtle deleted the alex/simsv2_benchmarks branch January 17, 2025 10:01
mergify bot pushed a commit that referenced this pull request Jan 17, 2025
(cherry picked from commit b8638f8)

# Conflicts:
#	scripts/build/simulations.mk
julienrbrt added a commit that referenced this pull request Jan 17, 2025
Co-authored-by: Alexander Peters <[email protected]>
Co-authored-by: Julien Robert <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/v0.52.x PR scheduled for inclusion in the v0.52's next stable release C:Simulations
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants