functional-tests: Provide more advanced network failure injections

[chancez]

Someone in the kubernetes cluster-sig-ops group wrote the following set of scripts to do black box fauult injection against etcd: https://github.com/jdumars/etcdeath

I see we have https://github.com/coreos/etcd/blob/master/pkg/netutil/isolate_linux.go today which does total packet loss, and latency, but etcdeath provides some interesting additional failures that would be good to add to our functional tests.

In particular the following:

packet corruption:

        sudo tc qdisc add dev eth0 root handle 1:1 netem corrupt 15%
        sudo tc qdisc add dev eth0 parent 1:1 handle 10:1 netem corrupt 15%

partial packet loss (instead of complete packet loss):

        sudo tc qdisc add dev eth0 root netem loss 15%

packet reordering:

        sudo tc qdisc add dev eth0 root handle 1:1 netem delay 10ms reorder 25% 50%
        sudo tc qdisc add dev eth0 parent 1:1 handle 10:1 delay 10ms reorder 25% 50%

partitioning:

        sudo tc qdisc add dev eth0 root netem delay 30000ms 20ms distribution normal

Those are just the ones I see we currently don't have, but they seem to be genuinely good tests to run, so I'm just suggesting we add these as alternative ways to stress etcd.

[xiang90]

@chancez For the ordering, is it actually testing the TCP+HTTP stack?

[heyitsanthony]

@xiang90 I think ordering makes sense among multiple connections. For example, the bridge could buffer data on each connection for a bit, then control the order that it drains the buffers into the receiver.

[xiang90]

@heyitsanthony Oh. Sure. You are right.

[xiang90]

/cc @fanminshi You might be interested in this one.

[fanminshi]

@xiang90 cool!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.