Reproducible benchmarks

[cortwave]

Which environment uses for benchmarks (OS, memory, cpu, java version)? What about moving benchmarks execution inside vm/containers with constant OS, resources for +- reproducible benchmarks?

[stevehu]

I am doing testing on my desktop (Ubuntu 16.04, i5 CPU with 32GB memory on Java 8). I am totally with you on moving to docker containers. Currently thinking to dockerize each of them into individual containers with wrk inside. Do you have any better idea?

[IRus]

We need better explanation of results, at least define what every digit in result means.

Also i found that some people create wrk2 https://github.com/giltene/wrk2

Currently thinking to dockerize each of them into individual containers with wrk inside.

We can try docker-compose, so then wrk would live in separate container. But we should test impact of docker to our tests.

[stevehu]

@IRus I totally agree. I will write up something when time is permitted. The test is to gauge the raw throughput and latency of each framework on a very simple response ("Hello World!") without network limitation involves. I am guessing that docker-compose might impact the performance number a little bit as traffic goes through docker network although on the same docker host. Need to test it out on both approaches.

The wrk2 looks pretty good. Thanks for the link.

[IRus]

I just create docker image for wkr2

it can be used this way:

docker run --net=host irus/wrk2 -t 4 -c 128 -d 30 --rate 1000 http://localhost:8080

[IRus]

Dockerfile

[IRus]

Other tools:
Apache Bench
Apache JMeter
httperf

Personally i used JMeter and ab in past. But i can't compate they with each other and wrk/wrk2.

[stevehu]

I have used both AB and JMeter. They are not designed to work with high performance microservices as they can only generate less than 100K request per second on a commodity hardware. wrk is the most efficient tool to generate enough load without hogging all CPUs.

[cortwave]

@IRus I think that memory and cpu limits should be specified

[IRus]

@cortwave they can be specified via arguments.

--memory="1g" --cpuset="0-3" 

Something like this for 1GB Ram and 4 CPUs

[IRus]

Docker vs host

yoda@ux32vd:18:27:~/dev/GitHub_IRus/wrk2 (master)$ ./wrk -t 4 -c 128 -d 30 --rate 1000000 http://localhost:8080
Running 30s test @ http://localhost:8080
  4 threads and 128 connections
  Thread calibration: mean lat.: 5077.301ms, rate sampling interval: 17072ms
  Thread calibration: mean lat.: 3912.209ms, rate sampling interval: 16326ms
  Thread calibration: mean lat.: 4107.606ms, rate sampling interval: 16482ms
  Thread calibration: mean lat.: 4988.354ms, rate sampling interval: 16990ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.42s     6.93s   28.44s    65.53%
    Req/Sec    22.21k     1.71k   24.42k    50.00%
  2574749 requests in 30.00s, 260.28MB read
Requests/sec:  85831.26
Transfer/sec:      8.68MB
yoda@ux32vd:18:28:~/dev/GitHub_IRus/wrk2 (master)$ docker run --net=host irus/wrk2 -t 4 -c 128 -d 30 --rate 1000000 http://localhost:8080
Running 30s test @ http://localhost:8080
  4 threads and 128 connections
  Thread calibration: mean lat.: 4568.203ms, rate sampling interval: 17104ms
  Thread calibration: mean lat.: 4686.661ms, rate sampling interval: 16449ms
  Thread calibration: mean lat.: 3811.947ms, rate sampling interval: 15990ms
  Thread calibration: mean lat.: 4236.410ms, rate sampling interval: 15024ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.36s     6.18s   29.15s    64.90%
    Req/Sec    18.06k     2.60k   21.60k    50.00%
  2163230 requests in 30.00s, 218.68MB read
Requests/sec:  72109.14
Transfer/sec:      7.29MB

[stevehu]

This is expected as requests have to go though docker network which added another layer. Is your first run using wrk2?

[IRus]

Sure, i compiled wrk2 (there are few warning, but looks good).
I also will try wrk in containers, and without it.

[IRus]

I trying to limit container with wrk2 (server running in host), but it works fine(i mean it still pretty fast, so result doesn't changes) even with 50mb of RAM, and one CPU. I don't know how to limit CPU performance in container, and i think that this is impossible actually :) Maybe virtual machines can help for limiting CPU. So my conclusion that limiting wrk2 doesn't make much sense.

Upd. Why we actually want limit wrk2? I think it doesn't make sense at all, we can limit server, but because of every machine have different CPU it doesn't help too much anyway.

[stevehu]

These limitations will only work in certain scenarios on certain OS. For Java it is very complicated. I was trying to gauge memory usage and couldn't find any reliable way to do so.

[stevehu]

This task is still in our pipeline but as the benchmark has been moved to its own repo, we are going to trace it there.