How to benchmark adapters?

[antedeguemon]

Hey, folks! 🖖

I want to start a discussion and get your opinions on how to benchmark wireless network adapters.

I think it would be beneficial to have easily reproducible benchmark setups:

• That way, anyone can contribute with benchmark results for exotic adapters and different usage conditions
• And most important, it would help people to be aware of adapters' advantages and limitations

The performance comparison document is a good kickstart for this discussion.

There are a lot of variables involved and many of them can't be controlled in a testing setup. And even more, there are different usage scenarios where adapters can perform differently depending on what is measured.

If you folks agree, we can use this thread to:

• Elaborate on benchmarking metrics and usage scenarios that we think are important and describe them in detail so they can be reproduced
• Establish baselines that will allow determining which adapter performs better than another - always taking into consideration the metric and scenario
• Come up with scripts that would make it easier for the community to contribute with their results

[morrownr]

Hi @antedeguemon

Interesting topic.

I think it would be beneficial to have easily reproducible benchmark setups

You are asking for the impossible.

A good goal for either speed or range tests would be to design a test and determine the adapters that we want to test and then have someone do it over a short period of time. Users submitting their own tests at their own locations would make for useless comparison tests. I know it sounds good but this technology is design to share so unless we have a rock solid way to duplicate local conditions, it can't be done and we have no practical way to duplicate local conditions.

I recently posted the following:

#139

A lot of thought went into making sure this was a fair test and the test was executed over short period of time. I have many wifi adapters. Design a test or two and I'll see what I can do.

Nick

[antedeguemon]

Hi @morrownr, thank you very much for your reply!!

You are 100% correct. Adapter benchmarks depend too much on the environment and uncontrollable variables - too much to be reproducible by different people. It is all electromagnetic waves afterall.

it can't be done and we have no practical way to duplicate local conditions.

Yes!! Even the simplest experiment I could come up with is impossible to be reproduced even when done by the same person at different times.

For example, the experiment of measuring the rate of lost packets for a given adapter, with:

1. A baseline adapter sending NULL data frame packets
2. A subject adapter receiving these packets

Below is a diagram containing some of the experiment problems and for sure there are many more:

Anyway, if all the problems are mitigated with unpractical strategies (a Faraday cage, custom schedulers, ...), the obtained results will still be useless because there is no way to compare them with other results. 😿

[antedeguemon]

As much as I like the idea of comparing adapter experiments between the community, I agree it is a dead-end.

• There is no practical way for different people to run an experiment with the same conditions
• The results won't be useful as they have many unknowns involved

I will keep the discussion open if it is okay. Perhaps someone has a different perspective.

[fhteagle]

One thing I can add to this is maybe standardizing around the use of iperf3 for throughout figures. I see speedtests out to the general Internet being used as metrics and I wouldn't call those reliable. One gotcha with iperf3 if not the ending and receiving devices also need to have enough CPU to create and revive/validate the flood of packet data. I see artificially low results either sending from or receiving to old routers, my Engenius APs, etc due to this factor.

I also really like wavemon as a tool to see what's going on with signal strength (especially signal strength transients), link rates, etc. Much faster update than "watch iwconfig", for example.

Finally, hostapd_cli gives deeper reporting on what the AP thinks of the capabilities of the STAs that are connected to it.