apps.mellanox.connectx: benchmark matrix

[eugeneia]

This adds some scripts and scaffolding for running benchmark matrices on our Connect-X driver. At its heart is a Snabb script that, given a system with two connected Connect-X ports, let’s you run a benchmark like so:

./snabb snsh apps/mellanox/benchmark.snabb \
  --pci-a 81:00.0 --pci-b 81:00.1 --cores-a 6-11 --cores-b 12-17 \
  --mode source|source-sink|source-fwd
  --nworkers <int> --nqueues <int> --npackets <int> --pktsize <int> --nmacs <int> --nvlans <int> \
  --opt <string> # Additional options for apps.mellanox.connectx

If you wrap this in a shell script like so you can generate matrix CSV data to then investigate with R...

qsizes="512
1024
2048
4096"

echo i,workers,queues,qsize,pktsize,rate,drop,error
for i in `seq 1 3`; do
    for w in `seq 1 1`; do
        for q in `seq 1 16`; do
            for qsize in $qsizes; do
                for s in "64"; do
                    out=$(./snabb snsh apps/mellanox/benchmark.snabb -a 81:00.0 -b 81:00.1 -A 6-11 \
                            -m source -w $w -q $q -s $s -n 100e6 -o "{sendq_size=$qsize}")
                    rate=$(echo "$out" | grep "Tx Rate is" | cut -d " " -f 4)
                    drop=$(echo "$out" | grep "Tx Drop Rate is" | cut -d " " -f 5)
                    error=$(echo "$out" | grep "Tx Error Rate is" | cut -d " " -f 5)
                    echo "$i,$w,$q,$qsize,$s,$rate,$drop,$error"
                    #echo $out
                done
            done
        done
    done
done

…and maybe end up with a plot like this:

We already used this in an attempt to reproduce #1007 and debug a performance issue on EPYC systems. A report on the results can be found here: https://rpubs.com/eugeneia/860725

[eugeneia]

I forgot to mention, but 7f967e4 and 9adad52 are interesting commits in their own right. They (re)introduce a packetblaster mode for the Connect-X driver that should be on-par with the trusty 52899 packetblaster in transmitting packets with low CPU overhead. I’ve double checked it to use the equivalent transmit routine @lukego used in the 2016 expirements/measurements.

This specialized transmit routine should be able to tease out hardware limits as it has negligible CPU overhead. It doesn’t however necessarily reflect “real” application transmit performance (as its sending the same buffer worth of packets over and over).