Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Performance] microbenchmark framework #3141

Closed
ZhiHanZ opened this issue Nov 28, 2021 · 0 comments
Closed

[Performance] microbenchmark framework #3141

ZhiHanZ opened this issue Nov 28, 2021 · 0 comments
Labels
A-query Area: databend query

Comments

@ZhiHanZ
Copy link
Collaborator

ZhiHanZ commented Nov 28, 2021

Summary
In order to summarize on query engine performance and get avoid from performance rollback, we should have some generic microbenchmark framework to run during each nightly release.
Microbenchmark is designed as regular approach to get avoid of performance rollback, which perform as supplementary role in performance engineering. Comparing to e2e benchmark, microbenchmark rely on static data and trying to resolve performance rollback issue during each release(or commit) by running same query to different storage backend.
each performance test should support to target on different storage backend given number of iteration and should be scalable to cluster level tests.
performance benchmark should behave like the following Pseudo code

def bench_s3(i : iteration, r: reference) {
    c = init_query_cluster(s3)
    execute_query(i)
   collect_metrics(c)
   report
   compare_with(r)
}

Output:
Query metrics for each benchmark query

Metric Average
planner 1s
optimizer 2s
... ...

generic summarizations

benchmark name branch name time compare
memory sum 12e34 10s 20 % +

Possible implementation
Adopting some CI tool such as cirrus CI which provide consistent container env run on k8s to run benchmarks in uniform way.
Critirion.rs(https://bheisler.github.io/criterion.rs/book/criterion_rs.html) maybe a good library for microbenchmark implementation

Should fix:
#3084

Reference:
http://www.vldb.org/pvldb/vol13/p3285-gruenheid.pdf

@BohuTANG BohuTANG added the A-query Area: databend query label Nov 29, 2021
@ZhiHanZ ZhiHanZ closed this as completed May 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-query Area: databend query
Projects
None yet
Development

No branches or pull requests

2 participants