e2e: add pure go stateful fuzz test generator

[askervin]

Purpose of this patch is to drop dependency to fMBT in e2e fuzz test generation. This patch implements parts that were needed from fMBT in go and includes an example for generating tests: a model and a logic what to cover and how to cover it.

[klihub]

There is a lint error (unused unexported member in model.go:Model.transitions) and a bunch of indentation white space/member indentation formatting violations. These should be the easy part to fix.

Now the harder part is that, although there is an example in there, I suspect that this could use a it more hand-holding instructions about how it is supposed to be used. I don't think it should be a show-stopper for merging. But do you think that it would eventually be possible to add, with subsequent PRs, a bit more explanation, sort of a crash course for dummies who don't know fMBT at all ?

[klihub]

There are a few CI linting and formatting verification errors that need to be fixed before this can be merged.

Ideally it probably would be nice to get a bit more instructions about how this is supposed to be used. The existing example is very useful, but it might be a bit steep as a crash course for dummies, at least for me it is. But, if you agree that it is first of all possible and additionally usefuly, it can be address later with further PRs IMO.

[askervin]

Thanks @klihub!

Converted to draft, because I wouldn't like to get this merged yet. My purpose was to get your opinion on switching from fMBT into pure-go model-based fuzzing.

As you sound generally positive about the idea of getting rid of very hard-to-get-and-use fmbt docker images and the modeling language that is alien to 100-epsilon % of developers, I'd be happy to put everything in place to this PR. That is: licenses, documentation, examples, integration to generate.sh, and removal of fMBT.

[klihub]

I think this does not work correctly yet.

I gave this a test with

• the above suggested command line options added,
• generate.sh updated to the one listed at the end of this comment, and
• a minimal test run of STEPS=10 ./test/e2e/policies.test-suite/topology-aware/n4c16/test06-fuzz/generate.sh,
• which produced a generated0.sh as listed below

All in all, the test failed as it tried to get allocations up to a total request of 900+200+2000+6000+4200+100+100+100+250+200+100+50+1900+1900 18000 with kube-system reserved CPUs (900m) included, but the node only has 16000. So pod bu1 from
the generated set stayed in pending state forever.

klitkey1-mobl1 mbtfuzz $ less test/e2e/policies.test-suite/topology-aware/n4c16/test06-fuzz/generated0.sh

# step 1, coverage: 0, state: [cpu:14050 mem:7500 pods:map[]]
NAME=gu0 CONTCOUNT=1 CPU=200m MEM=1500M create guaranteed

# step 2, coverage: 1, state: [cpu:13850 mem:6000 pods:map[gu0:[200 1500]]]
NAME=gu1 CONTCOUNT=2 CPU=1000m MEM=500M create guaranteed

# step 3, coverage: 3, state: [cpu:11850 mem:5000 pods:map[gu0:[200 1500] gu1:[2000 1000]]]
NAME=gu3 CONTCOUNT=3 CPU=2000m MEM=500M create guaranteed

# step 4, coverage: 6, state: [cpu:5850 mem:3500 pods:map[gu0:[200 1500] gu1:[2000 1000] gu3:[6000 1500]]]
NAME=gu4 CONTCOUNT=1 CPU=4200m MEM=100M create guaranteed

# step 5, coverage: 9, state: [cpu:1650 mem:3400 pods:map[gu0:[200 1500] gu1:[2000 1000] gu3:[6000 1500] gu4:[4200 100]]]
NAME=bu0 CONTCOUNT=1 CPU=1200m MEM=50M CPUREQ=900m MEMREQ=49M CPULIM=1200m MEMLIM=50M create burstable

# step 6, coverage: 12, state: [cpu:450 mem:3350 pods:map[bu0:[1200 50] gu0:[200 1500] gu1:[2000 1000] gu3:[6000 1500] gu4:[4200 100]]]
NAME=be0 CONTCOUNT=1 CPU=0 MEM=0 create besteffort

# step 7, coverage: 15, state: [cpu:450 mem:3350 pods:map[be0:[0 0] bu0:[1200 50] gu0:[200 1500] gu1:[2000 1000] gu3:[6000 1500] gu4:[4200 100]]]
NAME=be1 CONTCOUNT=3 CPU=0 MEM=0 create besteffort

# step 8, coverage: 18, state: [cpu:450 mem:3350 pods:map[be0:[0 0] be1:[0 0] bu0:[1200 50] gu0:[200 1500] gu1:[2000 1000] gu3:[6000 1500] gu4:[4200 100]]]
NAME=gu0 kubectl delete pod gu0 --now

# step 9, coverage: 21, state: [cpu:650 mem:4850 pods:map[be0:[0 0] be1:[0 0] bu0:[1200 50] gu1:[2000 1000] gu3:[6000 1500] gu4:[4200 100]]]
NAME=gu1 kubectl delete pod gu1 --now

# step 10, coverage: 24, state: [cpu:2650 mem:5850 pods:map[be0:[0 0] be1:[0 0] bu0:[1200 50] gu3:[6000 1500] gu4:[4200 100]]]
NAME=gu3 kubectl delete pod gu3 --now

# step 11, coverage: 27, state: [cpu:8650 mem:7350 pods:map[be0:[0 0] be1:[0 0] bu0:[1200 50] gu4:[4200 100]]]
NAME=bu1 CONTCOUNT=2 CPU=1900m MEM=300M CPUREQ=1800m MEMREQ=299M CPULIM=1900m MEMLIM=300M create burstable

# step 12, coverage: 30, state: [cpu:4850 mem:6750 pods:map[be0:[0 0] be1:[0 0] bu0:[1200 50] bu1:[3800 600] gu4:[4200 100]]]
NAME=gu4 kubectl delete pod gu4 --now

klitkey1-mobl1 mbtfuzz $ less test/e2e/policies.test-suite/topology-aware/n4c16/test06-fuzz/generate.sh

#!/bin/bash

usage() {
    cat <<EOF
generate.sh - generate fuzz tests.

Configuring test generation with environment variables:
  TESTCOUNT=<NUM>       Number of generated test scripts than run in parallel.
  MEM=<NUM>             Memory [MB] available for test pods in the system.
  CPU=<NUM>             Non-reserved CPU [mCPU] available for test pods in the system.
  RESERVED_CPU=<NUM>    Reserved CPU [mCPU] available for test pods in the system.
  STEPS=<NUM>           Total number of test steps in all parallel tests.

EOF
    exit 0
}

if [ -n "$1" ]; then
    usage
fi

TESTCOUNT=${TESTCOUNT:-1}
MEM=${MEM:-7500}
# 950 mCPU taken by the control plane, split the remaining 15050 mCPU
# available for test pods to CPU and RESERVED_CPU pods.
CPU=${CPU:-14050}
RESERVED_CPU=${RESERVED_CPU:-1000}
STEPS=${STEPS:-100}

mem_per_test=$(( MEM / TESTCOUNT ))
cpu_per_test=$(( CPU / TESTCOUNT ))
reserved_cpu_per_test=$(( RESERVED_CPU / TESTCOUNT ))
steps_per_test=$(( STEPS / TESTCOUNT ))

cd "$(dirname "$0")" || {
    echo "cannot cd to the directory of $0"
    exit 1
}

for testnum in $(seq 1 "$TESTCOUNT"); do
    testid=$(( testnum - 1))
    OUTFILE=generated${testid}.sh
    echo "generating $OUTFILE..."
    go run ./generate.go \
       --mem $mem_per_test \
       --cpu $cpu_per_test \
       --reserved-cpu $reserved_cpu_per_test \
       --test-steps $steps_per_test > "$OUTFILE"
done

[askervin]

This is now ready for review.

Generating and running multiple fuzz tests (up to 10, I think) in parallel works.

Based on earlier discussion, I still left out the evilness of crashing and restarting containers. These tests, even 4 parallel ones, seem to be passing. Haven't run really long ones yet.

Thanks for the doing the ground work on generate.sh!

PR updated.

[klihub]

These tests, even 4 parallel ones, seem to be passing.

This is probably just due to me not understanding or misunderstanding something, but... looking at this latest version I find it surprising that parallel versions really work/pass.

I was also doing parallel testing with the earlier version I hacked on a bit, as a final test. And when I did that, I also had to introduce an --instance $int option for generate.go, otherwise parallel instances were causing false positives. And if I recall correctly the reason was simply that they used identical pod names without understanding that another instance might have already created the same pod (although I have the vague recollection that I would have seen also a failure due to this name conflict and deletion then happening from the instance which did not create the pod but failed to detect it so the creating instance then failed the deletion, but it is entirely possible that I remember this wrong). Anyway, either way/eventually my workaround was to podname-'namespace' the pods per instance. IOW, I added an inst$instance- prefix to each pod name and with that I got parallel tests going and passing.

[klihub]

Thanks for taking the time and effort for writing this explanation ! ❤️

[askervin]

I was also doing parallel testing with the earlier version I hacked on a bit, as a final test. And when I did that, I also had to introduce an --instance $int option for generate.go

Yeah, something like that is absolutely necessary. In the current version of this patch this is done in generate.sh with ... | sed -e "$prefix_pods_with_testid"

[klihub]

I was also doing parallel testing with the earlier version I hacked on a bit, as a final test. And when I did that, I also had to introduce an --instance $int option for generate.go

Yeah, something like that is absolutely necessary. In the current version of this patch this is done in generate.sh with ... | sed -e "$prefix_pods_with_testid"

Totally missed that. 🙈