Test zmq with faulty network

[xjules]

We should stress test zmq with "faulty" network before zmq is provide to stable version. The testing should include the following scenarios:

• dispatchers lose connection, which gets reestablished after some time
• the messages from dispatchers get a delay

Or combinations of the above ideally.

@EJahren suggested a nice library that could be exploited https://vaurien.readthedocs.io/en/1.8/ This library only supports python 2.7
There are other alternatives:

• https://github.com/douglas/toxiproxy-python
• https://github.com/locustio/locust

[xjules]

Tested with socat script

#!/bin/bash

# Default ports
DEFAULT_INBOUND_PORT="51823"
DEFAULT_OUTBOUND_PORT="51822"

# Parse input arguments
INBOUND_PORT=${1:-$DEFAULT_INBOUND_PORT}  # First argument: Inbound port, defaults to 51823
OUTBOUND_PORT=${2:-$DEFAULT_OUTBOUND_PORT}  # Second argument: Outbound port, defaults to 51822

# Start socat forwarding
echo "Starting socat to forward inbound port $INBOUND_PORT to outbound port $OUTBOUND_PORT"
socat TCP-LISTEN:$INBOUND_PORT,reuseaddr,fork TCP:127.0.0.1:$OUTBOUND_PORT

inbound port is the one that dealers (client and monitor) connect to and outbound is the one that is bind by router.

The following was tested:
[x] Lose connection

All was able to recover. Nevertheless this impacted monitor too and thus the events to the GUI were not sent, while all the job dispatchers events were resent when re-enabling connection again.