[performance] Introduction to performance testing #9907
Conversation
|
/packit test --labels performance |
4 similar comments
|
/packit test --labels performance |
|
/packit test --labels performance |
|
/packit test --labels performance |
|
/packit test --labels performance |
|
@strimzi-ci run tests --cluster-type=ocp --cluster-version=4.15 --install-type=bundle --profile=performance |
|
|
|
❗ Systemtests Failed (no tests results are present) ❗ |
|
/packit test --labels performance |
|
@strimzi-ci run tests --cluster-type=ocp --cluster-version=4.15 --install-type=bundle --profile=performance |
|
|
|
/packit test --labels performance |
|
❗ Systemtests Failed (no tests results are present) ❗ |
|
@strimzi-ci run tests --cluster-type=ocp --cluster-version=4.15 --install-type=bundle --profile=performance --parallel-count=1 --env=WORKER_NODES_COUNT=5,WORKER_NODES_FLAVOR=g.standard.xl |
|
|
|
❗ Systemtests Failed (no tests results are present) ❗ |
|
@strimzi-ci run tests --cluster-type=ocp --cluster-version=4.15 --install-type=bundle --profile=performance --parallel-count=1 --env=WORKER_NODES_COUNT=5,WORKER_NODES_FLAVOR=g.standard.xl |
|
|
|
❗ Systemtests Failed (no tests results are present) ❗ |
|
@strimzi-ci run tests --cluster-type=ocp --cluster-version=4.15 --install-type=bundle --profile=performance --parallel-count=1 --env=WORKER_NODES_COUNT=5,WORKER_NODES_FLAVOR=g.standard.xl |
|
|
|
/packit test --labels performance |
1 similar comment
|
/packit test --labels performance |
|
❗ Systemtests Failed (no tests results are present) ❗ |
|
/packit test --labels performance |
|
@strimzi-ci run tests --cluster-type=ocp --cluster-version=4.15 --install-type=bundle --profile=performance --parallel-count=1 --env=WORKER_NODES_COUNT=5,WORKER_NODES_FLAVOR=g.standard.xl |
|
|
Signed-off-by: see-quick <[email protected]>
Signed-off-by: see-quick <[email protected]>
Signed-off-by: see-quick <[email protected]>
Signed-off-by: see-quick <[email protected]>
Signed-off-by: see-quick <[email protected]>
Signed-off-by: see-quick <[email protected]>
|
/packit test --labels performance |
There was a problem hiding this comment.
@see-quick really nice work. Left some suggestions.
Running bobStreamingUseCase took 45 minutes on my laptop.
On the output format: it would be great to have this in the target/performance folder, instead of printing it out on the console; when you have time or size, please always mention the unit as header suffix; add total/average as header prefix when needed; looks like some columns do not have an header.
+------------+------------------+----------------------------+--------------------+---------------+--------------------+---------------+-----------------+----------------+---------------------+-------------------------+--------------------------+--------------------------+--------------------------+-------------------------------------+------------------------------------------------+--------------------------------------------+--------------------------------------------+--------------------------------------------------+------------------------------------------+------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------+
| Experiment | Number of Topics | Number of Client Instances | Number of Messages | Creation Time | Send And Recv Time | Deletion Time | Total Test Time | Max Batch Size | Max Batch Linger Ms | Update Number of Topics | Bob Update Times Round 1 | Bob Update Times Round 2 | Bob Update Times Round 3 | jvm_memory_used_megabytes_total.txt | strimzi_update_status_duration_seconds_max.txt | strimzi_reconciliations_max_queue_size.txt | strimzi_reconciliations_max_batch_size.txt | strimzi_reconciliations_duration_seconds_max.txt | system_load_average_per_core_percent.txt | strimzi_create_topics_duration_seconds_max.txt | strimzi_describe_configs_duration_seconds_max.txt | strimzi_total_time_spend_on_uto_event_queue_duration_seconds.txt |
| 1 | 750 | 0 | 10000 | 117912 | 0 | 153802 | 452056 | 250 | 1000 | 10 | 125 ms | 131 ms | 85 ms | 156.695176 | 0.061445074 | 5098.0 | 250.0 | 49.309339858 | 1.9508333333333334 | 46.394964991 | 0.348587152 | 44922.722765665 |
| 2 | 750 | 0 | 10000 | 114168 | 0 | 154148 | 448706 | 500 | 1000 | 10 | 97 ms | 155 ms | 137 ms | 303.528072 | 0.049932045 | 6726.0 | 500.0 | 66.646814578 | 1.7116666666666667 | 60.072250268 | 0.440632293 | 77831.588177951 |
| 3 | 750 | 0 | 10000 | 102879 | 0 | 153932 | 437108 | 250 | 5000 | 10 | 82 ms | 103 ms | 111 ms | 302.370232 | 0.068045472 | 3900.0 | 250.0 | 38.657464634 | 1.5525 | 35.595952363 | 0.18384528 | 41521.039881013996 |
| 4 | 750 | 0 | 10000 | 99746 | 0 | 153308 | 433412 | 500 | 5000 | 10 | 76 ms | 79 ms | 201 ms | 284.80548 | 0.033743837 | 4484.0 | 500.0 | 61.404692625 | 1.7074999999999998 | 56.214431372 | 0.365710384 | 68669.328255135 |
+------------+------------------+----------------------------+--------------------+---------------+--------------------+---------------+-----------------+----------------+---------------------+-------------------------+--------------------------+--------------------------+--------------------------+-------------------------------------+------------------------------------------------+--------------------------------------------+--------------------------------------------+--------------------------------------------------+------------------------------------------+------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------+
systemtest/src/main/java/io/strimzi/systemtest/performance/PerformanceConstants.java
Outdated
Show resolved
Hide resolved
systemtest/src/main/java/io/strimzi/systemtest/performance/PerformanceConstants.java
Outdated
Show resolved
Hide resolved
...est/src/main/java/io/strimzi/systemtest/performance/gather/TopicOperatorMetricsGatherer.java
Outdated
Show resolved
Hide resolved
systemtest/src/main/java/io/strimzi/systemtest/performance/gather/BaseMetricsGatherer.java
Outdated
Show resolved
Hide resolved
systemtest/src/main/java/io/strimzi/systemtest/performance/gather/BaseMetricsGatherer.java
Outdated
Show resolved
Hide resolved
...src/main/java/io/strimzi/systemtest/performance/report/TopicOperatorPerformanceReporter.java
Outdated
Show resolved
Hide resolved
...est/src/main/java/io/strimzi/systemtest/performance/gather/TopicOperatorMetricsGatherer.java
Outdated
Show resolved
Hide resolved
...st/src/main/java/io/strimzi/systemtest/performance/gather/TopicOperatorMetricsCollector.java
Outdated
Show resolved
Hide resolved
...st/src/main/java/io/strimzi/systemtest/performance/gather/TopicOperatorMetricsCollector.java
Outdated
Show resolved
Hide resolved
systemtest/src/main/java/io/strimzi/systemtest/performance/gather/BaseMetricsGatherer.java
Outdated
Show resolved
Hide resolved
|
@see-quick I think it would be useful to have time units in the result tables. Also the INFO log from the STs is quite polluted. Wouldn't be better to switch all things regarding creation of the topics to DEBUG level and keep INFO a little bit more cleaner? Of course just in scenarios like this. |
Yeah, this is what it look like now (with my local changes). Yeah KafkaConfiguration should be ignored: Use Case: aliceBulkUseCase
+------------+----------------------+---------------------------------+------------------------+-------------------------+------------------------------+-------------------------+---------------------------+-------------------------+---------------------------+---------------------+------------------------------------------------------------------+------------------------------------------------+----------------------------------+-------------------------------------+--------------------------------------------------+--------------------------------------------+--------------------------------------------+------------------------------------------------+---------------------------------------------------+
| Experiment | IN: NUMBER OF TOPICS | IN: NUMBER OF CLIENTS INSTANCES | IN: NUMBER OF MESSAGES | OUT: Creation Time (ms) | OUT: Send And Recv Time (ms) | OUT: Deletion Time (ms) | OUT: Total Test Time (ms) | IN: MAX BATCH SIZE (ms) | IN: MAX BATCH LINGER (ms) | Kafka Configuration | strimzi_total_time_spend_on_uto_event_queue_duration_seconds.txt | strimzi_create_topics_duration_seconds_max.txt | system_load_average_per_core.txt | jvm_memory_used_megabytes_total.txt | strimzi_reconciliations_duration_seconds_max.txt | strimzi_reconciliations_max_batch_size.txt | strimzi_reconciliations_max_queue_size.txt | strimzi_update_status_duration_seconds_max.txt | strimzi_describe_configs_duration_seconds_max.txt |
| 1 | 1 | 0 | 10000 | 1454 | 0 | 1168 | 2624 | 100 | 10 | | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
+------------+----------------------+---------------------------------+------------------------+-------------------------+------------------------------+-------------------------+---------------------------+-------------------------+---------------------------+---------------------+------------------------------------------------------------------+------------------------------------------------+----------------------------------+-------------------------------------+--------------------------------------------------+--------------------------------------------+--------------------------------------------+------------------------------------------------+---------------------------------------------------+ |
I think having both approaches is okay. Should I address it in this PR or it is okay to leave that on the next one? @fvaleri |
Signed-off-by: see-quick <[email protected]>
Sure, we can have both at the same time. If it is not complicated I would prefer to have it here, otherwise I'm fine doing this in a follow up PR. The reason I think it's useful is that one may lose the shell for multiple reasons. In that case, all results would be lost. |
Signed-off-by: see-quick <[email protected]>
Signed-off-by: see-quick <[email protected]>
Much better, thanks. |
Signed-off-by: see-quick <[email protected]>
It was a quite a journey but I have updated also a support to export such table into directory under |
|
/packit test --labels performance |
Signed-off-by: see-quick <[email protected]>
|
/packit test --labels performance |
see-quick
added
ready for merge
Type of change
Description
This PR addresses a strimzi/proposals#113 proposal. It adds support for adding performance tests in our ST framework. Moreover, I have implemented two scenarios for the Topic Operator (i.e., Alice's bulk batch use case and Bob's streaming use case -> more info about that in test suites).
Infrastructures
Azure Pipelines
Testing farm
Strimzi-ci
Q1: I am thinking of making implementation more generic in terms of when one would add different performance test cases for other components such as CO or UO. Or let it be for now...
Q2: I am not sure we want to include a dashboard that I have created, specifically if one wants to check also in Grafana metrics during performance testing.
Currently when one runs:
Testing farm:
Jenkins:
Jenkins with 2k
KafkaTopics:Latest update:
Checklist