4 New Benchmarking Tools (akash-network#335)

* Benchmarking suite

- [Speedtest by Ookla](speedtest-cli)
- [Fast.com by Netflix](fast)
- [Geekbench 5](geekbench)
- [Speedtest Tracker](speedtest-tracker)
- [Librespeed](librespeed)
- [OpenSpeedTest](openspeedtest)

* Cleanup

* Add speedtest-logo

* Fast logo

* Bug fixes and cleanup README 

Cleanup after more testing in Akashlytics

* Fixes images in Akashlytics and various README issues

* Add multiple benchmarks

MonkeyTest
Cross Platform Disk Test
Serverbench
Flexible IO Tester (FIO)
Phoronix

* Serverbench and Phoronix good to go

Co-authored-by: Crypto and Coffee <[email protected]>

README.md

@@ -78,12 +78,16 @@ Awesome DeFi apps you can deploy on Akash
 
 ### Benchmarking
 
-- [Speedtest by Ookla](speedtest-cli)
 - [Fast.com by Netflix](fast)
+- [Flexible IO Tester](fio)
 - [Geekbench 5](geekbench)
-- [Speedtest Tracker](speedtest-tracker)
 - [LibreSpeed](librespeed)
+- [MonkeyTest](monkeytest)
 - [OpenSpeedTest](openspeedtest)
+- [Phoronix](phoronix)
+- [Serverbench](serverbench)
+- [Speedtest by Ookla](speedtest-cli)
+- [Speedtest Tracker](speedtest-tracker)
 
 ### Blockchain
 

cross-platform-disk-test/Dockerfile

@@ -0,0 +1,6 @@
+FROM debian:bookworm
+RUN apt-get update
+RUN apt-get install -y wget unzip libgssapi-krb5-2 libicu-dev
+RUN wget https://github.com/maxim-saplin/NetCoreStorageSpeedTest/releases/download/2.0.2-u/CPDT_Linux.zip
+RUN unzip CPDT_Linux.zip
+ENTRYPOINT ["/CPDT_Console_x64"]

cross-platform-disk-test/README.md

@@ -0,0 +1,22 @@
+# Cross Platform Disk Test (CPDT)
+
+Measuring storage performance (SSD, HDD, USB Flash etc.) and RAM speed.  Random and sequential throughput (read/write operations) is calculted in MB/s and can be compared in consistent and reliable manner between mobile and desktop platfotms and devices.
+
+![UI](https://raw.githubusercontent.com/maxim-saplin/CrossPlatformDiskTest/master/Img%203.png)
+
+## Benchmark chart
+
+https://maxim-saplin.github.io/cpdt_results/
+
+![Results](https://raw.githubusercontent.com/maxim-saplin/CrossPlatformDiskTest/master/Results.jpg)
+
+## How it works
+
+The tests measure time it takes to read/write each block (RAM -> Disk, Disk -> RAM, RAM ->), let you choose read/write modes (e.g. turning on/off write buffering and file cache in memory), conduct sereies operations in sequential and random manner and show the average throughput (total traffic over total time) in MB/s for each test. The tests let you bencmark how same storage operations (FileStream.Write and FileStrem.Read) are handled by different OS across different devices and compare the results.
+CPDT is single-threaded, no IO queues or parallel execution of reads/writes.
+
+![Concept](https://raw.githubusercontent.com/maxim-saplin/CrossPlatformDiskTest/master/EnBlack.png)
+
+## Technology
+
+Xamarin.Forms, .NET Framework and Xamarin.WPF on Windows, Mono and Xamarin.Mac on macOS, Mono and Xamarin.Android on Android. Quite a few custom controls (Stack and Grid repeaters, clickable label) and renderers (changing mouse cursor when hovering over button and label on desktop)

fio/Dockerfile

@@ -0,0 +1,8 @@
+FROM debian:bookworm
+RUN apt-get update ; apt-get install -y fio
+
+ENV COMMAND=
+ENV TEST=
+COPY run.sh /
+
+ENTRYPOINT ["bash","/run.sh"]

fio/README.md

@@ -0,0 +1,8 @@
+# Fio
+
+Flexible IO Tester (Fio) is a benchmarking and workload simulation tool for Linux/Unix created by Jens Axboe, who also maintains the block layer of the Linux kernel. Fio is highly tunable and widely used for storage performance benchmarking. It is also used in the official performance benchmarking tool created by Nutanix that is called X-Ray. X-Ray automates the deployment of VMs on which it runs Fio and then provides the results in a user-friendly way with graphs and reports.
+There are also ways to run Fio on Windows, but generally other tools that are better suited for Windows OS, such as IOmeter or CrystalDiskMark are recommended.
+
+More documentation : https://fio.readthedocs.io/en/latest/
+
+Version 3.30-r0

fio/deploy.yaml

@@ -0,0 +1,45 @@
+---
+version: "2.0"
+
+services:
+  fio:
+    image: cryptoandcoffee/akash-fio:4
+    expose:
+      - port: 80
+        to:
+          - global: true
+    #Replace COMMAND with the test you want to run.
+    env:
+      - TEST=standard #Run 3 tests recommended by ArtsTechnica / set to custom and uncomment the COMMAND variable to run custom test
+      #- COMMAND="--name=write_throughput --numjobs=8 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write --group_reporting=1"
+        #Test write throughput by performing sequential writes with multiple parallel streams (8+), using an I/O block size of 1 MB and an I/O depth of at least 64
+#########
+        #COMMAND: "--name=write_iops --directory=$TEST_DIR --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1"
+        #Test write IOPS by performing random writes, using an I/O block size of 4 KB and an I/O depth of at least 64:
+        #COMMAND: "--name=read_throughput --directory=$TEST_DIR --numjobs=8 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read --group_reporting=1"
+        #Test read throughput by performing sequential reads with multiple parallel streams (8+), using an I/O block size of 1 MB and an I/O depth of at least 64:
+        #COMMAND: "--name=read_iops --directory=$TEST_DIR --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=64 --rw=randread --group_reporting=1
+        #Test read IOPS by performing random reads, using an I/O block size of 4 KB and an I/O depth of at least 64:
+#########
+profiles:
+  compute:
+    fio:
+      resources:
+        cpu:
+          units: 4.0
+        memory:
+          size: 2Gi
+        storage:
+          size: 32Gi #32Gi min for the default test.
+  placement:
+    akash:
+      pricing:
+        fio:
+          denom: uakt
+          amount: 10000
+deployment:
+  fio:
+    akash:
+      profile: fio
+      count: 1
+

fio/run.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env sh
+if [[ $TEST == "custom" ]]; then
+fio $COMMAND
+else
+echo "Starting to test with defaults / ArsTechnica recommended tests"
+echo "https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/"
+echo "---------------------------------------"
+echo "Single 4KiB random write process"
+echo "This is a single process doing random 4K writes. This is where the pain really, really lives; it\'s basically the worst possible thing you can ask a disk to do. Where this happens most frequently in real life: copying home directories and dotfiles, manipulating email stuff, some database operations, source code trees."
+fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
+echo "---------------------------------------"
+echo "16 parallel 64KiB random write processes"
+echo "This is a pretty decent approximation of a significantly busy system. It\'s not doing any one particularly nasty thing—like running a database engine or copying tons of dotfiles from a user\'s home directory—but it is coping with a bunch of applications doing moderately demanding stuff all at once."
+fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1
+echo "---------------------------------------"
+echo "Single 1MiB random write process"
+echo "This is pretty close to the best-case scenario for a real-world system doing real-world things. No, it\'s not quite as fast as a single, truly contiguous write... but the 1MiB blocksize is large enough that it\'s quite close. Besides, if literally any other disk activity is requested simultaneously with a contiguous write, the "contiguous" write devolves to this level of performance pretty much instantly, so this is a much more realistic test of the upper end of storage performance on a typical system."
+fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
+echo "---------------------------------------"
+echo "All Tests Complete - sleeping for 15 minutes before restarting the test."
+sleep 900
+exit
+fi

monkeytest/Dockerfile

@@ -0,0 +1,5 @@
+FROM python
+RUN apt-get update ; apt-get install -y git
+RUN git clone https://github.com/thodnev/MonkeyTest
+COPY run.sh /
+ENTRYPOINT ["sh","./run.sh"]

monkeytest/README.md

@@ -0,0 +1,19 @@
+# MonkeyTest
+
+![monkeytest](https://cloud.githubusercontent.com/assets/16870636/12601547/7a3a4f14-c4aa-11e5-8b2e-48a20d7f7c17.png)
+
+A disk benchmark to test your hard drive read-write speed in Python.  A simplistic script to show that such system programming tasks are possible and convenient to be solved in Python
+
+The file is being created, then written with random data, randomly read
+and deleted, so the script doesn't waste your drive
+
+# Usage
+```
+usage: monkeytest.py [-h] [-f FILE] [-s SIZE] [-w WRITE_BLOCK_SIZE]
+                     [-r READ_BLOCK_SIZE] [-j JSON]
+```
+
+If you want to contribute, be sure to see our TODO list first
+  https://github.com/thodnev/MonkeyTest/milestones
+
+

monkeytest/deploy.yaml

@@ -0,0 +1,33 @@
+---
+version: "2.0"
+
+services:
+  monkeytest:
+    image: cryptoandcoffee/akash-monkeytest:1
+    expose:
+      - port: 80
+        to:
+          - global: true
+profiles:
+  compute:
+    monkeytest:
+      resources:
+        cpu:
+          units: 4.0
+        memory:
+          size: 2Gi
+        storage:
+          size: 4Gi #8Gi min for the default test.
+  placement:
+    akash:
+      pricing:
+        monkeytest:
+          denom: uakt
+          amount: 10000
+deployment:
+  monkeytest:
+    akash:
+      profile: monkeytest
+      count: 1
+
+

monkeytest/run.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env sh
+python3 /MonkeyTest/monkeytest.py
+echo "All Tests Complete - sleeping for 15 minutes before restarting the test."
+sleep 900

phoronix/Dockerfile

@@ -0,0 +1,13 @@
+FROM phoronix/pts
+RUN uname -a
+
+RUN apt-get update ; apt-get dist-upgrade -yqq ; apt-get install -y python3-pip
+RUN pip3 install apt-select ; apt-select --country US ; cp /sources.list /etc/apt/source.list ; apt-get update
+RUN apt-get install -y build-essential autoconf python3-pip python3-yaml libopenmpi-dev openmpi-bin libmpich-dev libpopt-dev libaio-dev python2-minimal npm cmake cmake-data
+
+ENV TEST="ramspeed"
+ENV COMMAND="batch-benchmark"
+
+COPY run.sh /
+
+ENTRYPOINT ["sh","/run.sh"]

phoronix/README.md

@@ -0,0 +1,45 @@
+# Phoronix Test Suite
+
+![Phoronix Test Suite](pts-core/static/images/pts-308x160.png)
+
+[Phoronix Test Suite
+Documentation](https://github.com/phoronix-test-suite/phoronix-test-suite/blob/master/documentation/phoronix-test-suite.md)
+
+The Phoronix Test Suite itself is an open-source framework for conducting
+automated tests along with reporting of test results, detection of installed
+system software/hardware, and other features. This framework is designed to be
+an extensible architecture so that new test profiles and suites can be easily
+added to represent performance benchmarks, unit tests, and other quantitative
+and qualitative (e.g. image quality comparison and pass/fail) measurements.
+Available through OpenBenchmarking.org, a collaborative storage platform
+developed in conjunction with the Phoronix Test Suite, are more than 600
+individual test profiles and more than 200 test suites available by default from
+the Phoronix Test Suite. Independent users are also able to upload their test
+results, profiles, and suites to OpenBenchmarking.org.
+
+A test profile is a single test that can be executed by the Phoronix Test Suite
+-- with a series of options possible within every test -- and a test suite is a
+seamless collection of test profiles and/or additional test suites. A test
+profile consists of a set of Bash/shell scripts and XML files while a test suite
+is a single XML file. Modules for the Phoronix Test Suite also allow for
+integration with git-bisect and other revision control systems for per-commit
+regression testing, system sensor monitoring, and other extras.
+
+[OpenBenchmarking.org](https://www.openbenchmarking.org/) also allows for
+conducting side-by-side result comparisons, a central location for storing and
+sharing test results, and collaborating over test data.
+[Phoromatic](https://www.phoromatic.com/) is a complementary platform to
+OpenBenchmarking.org and the Phoronix Test Suite for interfacing with Phoronix
+Test Suite client(s) to automatically execute test runs on a timed, per-commit,
+or other trigger-driven basis. Phoromatic is designed for enterprise and allows
+for the easy management of multiple networked systems running Phoronix Test
+Suite clients via a single web-based interface.
+
+Professional support and custom engineering for the Phoronix Test Suite,
+Phoromatic, and OpenBenchmarking.org is available by contacting
+<https://www.phoronix-test-suite.com/>.
+
+Full details on the Phoronix Test Suite setup and usage is available from the
+included HTML/PDF documentation within the phoronix-test-suite package and from
+the [Phoronix Test Suite documentation directory](documentation/).
+

phoronix/deploy.yaml

@@ -0,0 +1,35 @@
+---
+version: "2.0"
+
+services:
+  phoronix:
+    image: cryptoandcoffee/akash-phoronix:2
+    expose:
+      - port: 80
+        to:
+          - global: true
+    environment:
+      - TEST="ramspeed fio"
+      - COMMAND="batch-benchmark"
+profiles:
+  compute:
+    phoronix:
+      resources:
+        cpu:
+          units: 4.0
+        memory:
+          size: 6Gi
+        storage:
+          size: 32Gi #32Gi min for the default test.
+  placement:
+    akash:
+      pricing:
+        phoronix:
+          denom: uakt
+          amount: 10000
+deployment:
+  phoronix:
+    akash:
+      profile: phoronix
+      count: 1
+

phoronix/phoronix-test-suite.xml

@@ -0,0 +1,11 @@
+<BatchMode>
+<SaveResults>FALSE</SaveResults>
+<OpenBrowser>FALSE</OpenBrowser>
+<UploadResults>FALSE</UploadResults>
+<PromptForTestIdentifier>FALSE</PromptForTestIdentifier>
+<PromptForTestDescription>FALSE</PromptForTestDescription>
+<PromptSaveName>FALSE</PromptSaveName>
+<RunAllTestCombinations>TRUE</RunAllTestCombinations>
+<Configured>TRUE</Configured>
+</BatchMode>
+

phoronix/run.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env sh
+./phoronix-test-suite/phoronix-test-suite batch-setup << EOF
+n
+Y
+EOF
+/phoronix-test-suite/phoronix-test-suite install-dependencies $TEST
+./phoronix-test-suite/phoronix-test-suite $COMMAND $TEST

serverbench/Dockerfile

@@ -0,0 +1,6 @@
+FROM debian:bookworm
+RUN apt-get update ; apt-get install -y wget iputils-ping procps
+COPY run.sh /
+COPY serverbench.sh /
+ENTRYPOINT ["sh","./run.sh"]
+

serverbench/README.md

@@ -0,0 +1,5 @@
+# ServerBench
+
+View basic system information, disk speed along with network performance - tested from multiple locations across the world!
+
+https://github.com/K4Y5/ServerBench

serverbench/deploy.yaml

@@ -0,0 +1,32 @@
+---
+version: "2.0"
+
+services:
+  serverbench:
+    image: cryptoandcoffee/akash-serverbench:9
+    expose:
+      - port: 80
+        to:
+          - global: true
+profiles:
+  compute:
+    serverbench:
+      resources:
+        cpu:
+          units: 4.0
+        memory:
+          size: 2Gi
+        storage:
+          size: 4Gi #4Gi min for the default test.
+  placement:
+    akash:
+      pricing:
+        serverbench:
+          denom: uakt
+          amount: 10000
+deployment:
+  serverbench:
+    akash:
+      profile: serverbench
+      count: 1
+

serverbench/run.sh

@@ -0,0 +1,7 @@
+#! /usr/bin/env bash
+export TERM=linux
+set -eu
+exec ./serverbench.sh --logfile /var/log/my-application.log
+tail --pid $$ -F /var/log/my-application.log &
+echo "All Tests Complete - sleeping for 15 minutes before restarting the test."
+sleep 900