Move e2e tests to independent folder (#297)

* wip * provider * rename test suites * two more * consumer * final cleans and readme * Update README.md * Update README.md * mas formatting in readme * Update README.md * Update README.md * pr changes * Update README.md Co-authored-by: Marius Poke <[email protected]> * remove debug code Co-authored-by: Marius Poke <[email protected]>
cosmos · Aug 26, 2022 · 469fc56 · 469fc56
1 parent 3416e4c
commit 469fc56
Show file tree

Hide file tree

Showing 21 changed files with 2,493 additions and 2,461 deletions.
diff --git a/.github/workflows/automated-tests.yml b/.github/workflows/automated-tests.yml
@@ -11,10 +11,7 @@ jobs:
         with:
           go-version: "1.18.0" # The Go version to download (if necessary) and use.
 
-      - name: Figure out where we are
-        run: ls
-
-      - name: Unit tests
+      - name: Unit and e2e tests
         run: go test ./...
 
       - name: Integration tests

diff --git a/README.md b/README.md
@@ -13,8 +13,6 @@ CCV stands for cross chain validation and refers to the subset of Interchain Sec
 
 The code for CCV is housed under [x/ccv](./x/ccv). The `types` folder contains types and related functions that are used by both provider and consumer chains, while the `consumer` module contains the code run by consumer chains and the `provider` module contains the code run by provider chain.
 
-NOTE: At the moment the testing app may not be functional, please rely on the IBC testing suite to write unit tests for the moment.
-
 ## Instructions
 
 **Prerequisites**
@@ -45,30 +43,56 @@ export PATH=$PATH:$(go env GOPATH)/bin
 
 Inspect the [Makefile](./Makefile) if curious.
 
-**Running tests**
+## Testing
+
+### Unit Tests
+
+Unit tests are useful for simple standalone functionality, and CRUD operations. Unit tests should use golang's standard testing package, and be defined in files formatted as ```<file being tested>_test.go``` in the same directory as the file being tested, following standard conventions. 
+
+[Mocked external keepers](./testutil/keeper/mocks.go) (implemented with [gomock](https://github.com/golang/mock)) are available for testing more complex functionality, but still only relevant to execution within a single node. Ie. no internode or interchain communication. 
+
+### End to End (e2e) Tests
+
+[e2e-tests](./e2e-tests/) utilize the [IBC Testing Package](https://github.com/cosmos/ibc-go/tree/main/testing), and test functionality that is wider in scope than a unit test, but still able to be validated in-memory. Ie. code where advancing blocks would be useful, simulated handshakes, simulated packet relays, etc. 
+
+### Differential Tests (WIP)
+
+Similar to e2e tests, but they compare the system state to an expected state generated from a model implementation.
+
+### Integration Tests 
+
+[Integration tests](./integration-tests/) run true consumer and provider chain binaries within a docker container and are relevant to the highest level of functionality. Integration tests use queries/transactions invoked from CLI to drive and validate the code.
+
+### Running Tests
 
 ```bash
-# run all unit tests using make
+# run all static analysis, unit, e2e, and integration tests using make
+TODO
+# run all unit and e2e tests using make
 make test
-# run all unit tests using go
+# run all unit and e2e tests using go
 go test ./...
-# run all unit tests with verbose output
+# run all unit and e2e tests with verbose output
 go test -v ./..
-# run all unit tests with coverage stats
+# run all unit and e2e tests with coverage stats
 go test -cover ./..
 # run a single unit test
-go test -run <test-suite-name>/<test-name> ./...
+go test -run <unit-test-name> path/to/package
 # example: run a single unit test
+go test -run TestSlashAcks ./x/ccv/provider/keeper
+# run a single e2e test
+go test -run <test-suite-name>/<test-name> ./...
+# example: run a single e2e test
 go test -run TestProviderTestSuite/TestPacketRoundtrip ./...
-# run the integration tests
+# run all integration tests
 go run ./integration-tests/...
-# run integration tests with a local cosmos sdk
+# run all integration tests with a local cosmos sdk
 go run ./integration-tests/... --local-sdk-path "/Users/bob/Documents/cosmos-sdk/"
 # run golang native fuzz tests (https://go.dev/doc/tutorial/fuzz)
 go test -fuzz=<regex-to-match-test-name>
 ```
 
-**Linters and static analysis**
+### Linters and Static Analysis
 
 Several analyzers are used on the code including [CodeQL](https://codeql.github.com/), [SonarCloud](https://sonarcloud.io/), [golangci-lint](https://golangci-lint.run/) and [gosec](https://github.com/securego/gosec). Some of these are run on github when committing to PRs ect, but some tools are also applicable locally, and are built into golang.
 
@@ -98,11 +122,11 @@ go install github.com/go-critic/go-critic/cmd/gocritic@latest
 pre-commit run --all-files
 ```
 
-**Debugging**
+### Debugging
 
 If using VSCode, see [vscode-go/wiki/debugging](https://github.com/golang/vscode-go/wiki/debugging) to debug unit tests or go binaries.
 
-**More**
+### More
 
 More instructions will be added soon, in time for the testnet.
 

diff --git a/e2e-tests/README.md b/e2e-tests/README.md
@@ -0,0 +1,13 @@
+## End To End Testing
+
+E2e tests are categorized into files as follows:
+
+- `setup_test.go` - setup for the e2e tests
+- `common_test.go` - helper functions
+- `channel_init_test.go` - e2e tests for the _Channel Initialization_ sub-protocol
+- `valset_update_test.go` - e2e tests for the _Validator Set Update_ sub-protocol
+- `unbonding_test.go` - e2e tests for the _Completion of Unbonding Operations_
+- `slashing_test.go` - e2e tests for the _Consumer Initiated Slashing_ sub-protocol
+- `distribution_test.go` - e2e tests for the _Reward Distribution_ sub-protocol
+- `stop_consumer_test.go` - e2e tests for the _Consumer Chain Removal_ sub-protocol
+- `normal_operations_test.go` - e2e tests for _normal operations_ of ICS enabled chains