Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestHarness Enhanced Documentation #285

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

TestHarness Enhanced Documentation #285

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8d01b57
perf docs update http error code, example trx_generator
ericpassmore Jun 12, 2024
ff60336
rm some test keys
ericpassmore Jun 12, 2024
a98de1d
updated TestHarness docs
ericpassmore Jun 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 20 additions & 4 deletions tests/PerformanceHarness/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,21 @@
# Performance Harness

## Notes and Overview

### Protocol
HTTP connection recommended as we can get back more information. P2P will not have the same robust information. P2P will handle more troughput.

#### HTTPS Support
Currently no support for HTTPS.

### Generating Load
Use the `TestHarness/TransactionGeneratorLancher.py` to spin up many transactions generators to generate lots of load.

### Responsibilities
PerformanceHarness generate loads against an existing network, and generate reports on that network.
PerformanceHarness delegates setup of nodoes instances and cluster to the TestHarness
PerformanceHarness can toggle and tweak config settings in test harness

The Performance Harness is a module which provides the framework and utilities to run performance load tests on node infrastructure.

`PerformanceHarnessScenarioRunner.py` is currently the main entry point and provides the utility to configure and run such tests. It also serves as an example of how one can import the `PerformanceHarness` module to configure and run performance tests. The `PerformanceHarnessScenarioRunner` currently provides two options for running performance tests. The first `findMax` uses the `PerformanceTest` class to run a suite of `PerformanceTestBasic` test run to zero in on a max performance (see section on `PerformanceTest` following.). The second is `singleTest` which allows a user to run a single `PerformanceTestBasic` and see the results of a single configuration (see `PerformanceTestBasic` section following).
Expand Down Expand Up @@ -569,11 +585,11 @@ Performance Test Basic Base:
--read-only-read-window-time-us READ_ONLY_READ_WINDOW_TIME_US
Time in microseconds the read window lasts.
--http-max-in-flight-requests HTTP_MAX_IN_FLIGHT_REQUESTS
Maximum number of requests http_plugin should use for processing http requests. 429 error response when exceeded. -1 for unlimited
Maximum number of requests http_plugin should use for processing http requests. 503 error response when exceeded. -1 for unlimited
--http-max-response-time-ms HTTP_MAX_RESPONSE_TIME_MS
Maximum time for processing a request, -1 for unlimited
--http-max-bytes-in-flight-mb HTTP_MAX_BYTES_IN_FLIGHT_MB
Maximum size in megabytes http_plugin should use for processing http requests. -1 for unlimited. 429 error response when exceeded.
Maximum size in megabytes http_plugin should use for processing http requests. -1 for unlimited. 503 error response when exceeded.
--del-perf-logs Whether to delete performance test specific logs.
--del-report Whether to delete overarching performance run report.
--save-state Whether to save node state. (Warning: large disk usage)
Expand Down Expand Up @@ -726,11 +742,11 @@ Performance Test Basic Base:
The number of milliseconds early the last block of a production round should
be produced.
--http-max-in-flight-requests HTTP_MAX_IN_FLIGHT_REQUESTS
Maximum number of requests http_plugin should use for processing http requests. 429 error response when exceeded. -1 for unlimited (default: -1)
Maximum number of requests http_plugin should use for processing http requests. 503 error response when exceeded. -1 for unlimited (default: -1)
--http-max-response-time-ms HTTP_MAX_RESPONSE_TIME_MS
Maximum time for processing a request, -1 for unlimited (default: -1)
--http-max-bytes-in-flight-mb HTTP_MAX_BYTES_IN_FLIGHT_MB
Maximum size in megabytes http_plugin should use for processing http requests. -1 for unlimited. 429 error response when exceeded. (default: -1)
Maximum size in megabytes http_plugin should use for processing http requests. -1 for unlimited. 503 error response when exceeded. (default: -1)
--del-perf-logs Whether to delete performance test specific logs. (default: False)
--del-report Whether to delete overarching performance run report. (default: False)
--save-state Whether to save node state. (Warning: large disk usage)
Expand Down
97 changes: 97 additions & 0 deletions tests/PerformanceHarness/example_trx_generator_run.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@

## Command
Sleep is needed otherwise subjective transaction may reject transactions as too soon.
```
CHAIN_ID=$(cleos --url $HTTP_URL get info | grep chain_id | cut -d: -f2 | sed 's/[ ",]//g')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not suggesting that you need to change this, but if you're open to using jq you can do this shorter via cleos get info | jq -r .chain_id

LIB_ID=$(cleos --url $HTTP_URL get info | grep last_irreversible_block_id | cut -d: -f2 | sed 's/[ ",]//g')
sleep 3
${BUILD_TEST_DIR}/trx_generator/trx_generator --generator-id $GENERATORID \
--chain-id $CHAIN_ID \
--contract-owner-account eosio \
--accounts $COMMA_SEP_ACCOUNTS \
--priv-keys $COMMA_SEP_KEYS \
--last-irreversible-block-id $LIB_ID \
--log-dir $TRX_LOG_DIR \
--peer-endpoint-type p2p \
--peer-endpoint $P2P_HOST \
--port $PEER2PEERPORT
```

## Output
Why is this 0us
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is 0 because this is logging _total_us which is never updated. Just initialized to 0. Seems you found a bug. I don't guess anyone ever noticed.

Copy link
Contributor Author

@ericpassmore ericpassmore Jun 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forgot I had that question, converted PR to draft. I'll continue to add documentation updates, and better document the why 0us.

This example is from Jungle Spring Beta2 test net. I believe an intermediate node is filtering the submitted transactions to the P2P endpoint, and they are not making onto the blockchain. Need some additional tests to verify.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time is zero, because _total_us is never updated.

```
info 2024-06-12T21:14:31.379 trx_gener main.cpp:128 main ] Initializing accounts. Attempt to create name for purplepurple
info 2024-06-12T21:14:31.379 trx_gener main.cpp:128 main ] Initializing accounts. Attempt to create name for orangeorange
info 2024-06-12T21:14:31.379 trx_gener main.cpp:128 main ] Initializing accounts. Attempt to create name for pinkpink1111
info 2024-06-12T21:14:31.379 trx_gener main.cpp:128 main ] Initializing accounts. Attempt to create name for blueblue1111
info 2024-06-12T21:14:31.379 trx_gener main.cpp:128 main ] Initializing accounts. Attempt to create name for yellowyellow
info 2024-06-12T21:14:31.379 trx_gener main.cpp:128 main ] Initializing accounts. Attempt to create name for greengreen11
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.379 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:141 main ] Initializing private keys. Attempt to create private_key for 5JX...ZZZ : gen key 5JX...ZZZ
info 2024-06-12T21:14:31.380 trx_gener main.cpp:189 main ] Initial Trx Generator config: generator id: 0 chain id: b836a453e919f3dc7be91cd60cb949c625af7dc0f0a1f129f84763238d8bbc50 contract owner account: eosio trx expiration seconds: 3600 lib id: 0010f26923a1c9b094a778c2e58a19808af8ebaaa4346d72e1d3d19842fa5d2c log dir: /bigata1/log/trx_generator stop on trx failed: 1
info 2024-06-12T21:14:31.380 trx_gener main.cpp:190 main ] Initial Provider config: Provider base config endpoint type: p2p peer_endpoint: p2p.spring-beta2.jungletestnet.io port: 9898 api endpoint: /v1/chain/send_transaction2
info 2024-06-12T21:14:31.380 trx_gener main.cpp:191 main ] Initial Accounts config: Accounts Specified: accounts: [ purplepurple, orangeorange, pinkpink1111, blueblue1111, yellowyellow, greengreen11 ] keys: [ 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ, 5JX...ZZZ ]
info 2024-06-12T21:14:31.380 trx_gener main.cpp:192 main ] Transaction TPS Tester config: Trx Tps Tester Config: duration: 60 target tps: 1
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:103 setup ] Stop Generation (form potential ongoing generation in preparation for starting new generation run).
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:342 stop_generation ] Stopping transaction generation
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:106 setup ] Create All Initial Transfer Action/Reaction Pairs (acct 1 -> acct 2, acct 2 -> acct 1) between all provided accounts.
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from purplepurple to orangeorange
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from orangeorange to purplepurple
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from purplepurple to pinkpink1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from pinkpink1111 to purplepurple
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from purplepurple to blueblue1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from blueblue1111 to purplepurple
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from purplepurple to yellowyellow
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from yellowyellow to purplepurple
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from purplepurple to greengreen11
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from greengreen11 to purplepurple
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from orangeorange to pinkpink1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from pinkpink1111 to orangeorange
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from orangeorange to blueblue1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from blueblue1111 to orangeorange
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from orangeorange to yellowyellow
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from yellowyellow to orangeorange
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from orangeorange to greengreen11
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from greengreen11 to orangeorange
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from pinkpink1111 to blueblue1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from blueblue1111 to pinkpink1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from pinkpink1111 to yellowyellow
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from yellowyellow to pinkpink1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from pinkpink1111 to greengreen11
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from greengreen11 to pinkpink1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from blueblue1111 to yellowyellow
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from yellowyellow to blueblue1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from blueblue1111 to greengreen11
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from greengreen11 to blueblue1111
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:74 create_initial_trans ] create_initial_transfer_actions: creating transfer from yellowyellow to greengreen11
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:79 create_initial_trans ] create_initial_transfer_actions: creating transfer from greengreen11 to yellowyellow
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:87 create_initial_trans ] create_initial_transfer_actions: total action pairs created: 15
info 2024-06-12T21:14:31.380 trx_gener trx_generator.cpp:109 setup ] Create All Initial Transfer Transactions (one for each created action).
info 2024-06-12T21:14:31.382 trx_gener trx_generator.cpp:112 setup ] Setup p2p transaction provider
info 2024-06-12T21:14:31.382 trx_gener trx_generator.cpp:114 setup ] Update each trx to qualify as unique and fresh timestamps, re-sign trx, and send each updated transactions via p2p transaction provider
info 2024-06-12T21:14:31.382 trx_gener trx_provider.cpp:71 connect ] Attempting P2P connection to p2p.spring-beta2.jungletestnet.io:9898.
info 2024-06-12T21:14:31.750 trx_gener trx_provider.cpp:75 connect ] Connected to p2p.spring-beta2.jungletestnet.io:9898.
info 2024-06-12T21:15:30.751 trx_gener trx_provider.cpp:84 disconnect ] disconnect waiting on ack - sent 60 | acked 59 | waited 0
info 2024-06-12T21:15:31.751 trx_gener trx_generator.cpp:292 tear_down ] Sent transactions: 60
info 2024-06-12T21:15:31.751 trx_gener trx_generator.cpp:293 tear_down ] Tear down p2p transaction provider
info 2024-06-12T21:15:31.751 trx_gener trx_generator.cpp:296 tear_down ] Stop Generation.
info 2024-06-12T21:15:31.751 trx_gener trx_generator.cpp:342 stop_generation ] Stopping transaction generation
info 2024-06-12T21:15:31.751 trx_gener trx_generator.cpp:345 stop_generation ] 60 transactions executed, 0.00000000000000000us / transaction
info 2024-06-12T21:15:31.751 trx_gener main.cpp:226 main ] Exiting main SUCCESS
```
Loading
Loading