Test harness improvements (Was: Use parallel test runner)

[lkarsten]

Implementation in https://github.com/varnish/hitch/tree/parallel_tests branch works to some extent, but has issues when run on travis-ci.org.

Output seen:

PASS: tests/test01-start-and-stop
./tests/test02-simple-request: line 5: common.sh: No such file or directory
Assert error in front_arg_add(), configuration.c line 565:
  Condition((fa->port) != 0) not true.
./tests/test02-simple-request: line 8:  8570 Aborted                 (core dumped) hitch $HITCH_ARGS --backend=[hitch-tls.org]:80 "--frontend=[${LISTENADDR}]:$LISTENPORT" certs/site1.example.com
FAIL: tests/test02-simple-request
./tests/test03-multiple-listen: line 6: common.sh: No such file or directory
Assert error in front_arg_add(), configuration.c line 565:
  Condition((fa->port) != 0) not true.
./tests/test03-multiple-listen: line 13:  8577 Aborted                 (core dumped) hitch $HITCH_ARGS --backend=[hitch-tls.org]:80 "--frontend=[${LISTENADDR}]:$LISTENPORT" "--frontend=[${LISTENADDR}]:$PORT2" certs/site1.example.com
./tests/test03-multiple-listen: line 14: die: command not found
./tests/test03-multiple-listen: line 17: die: command not found

It does not look like the TESTDIR set in AM_TESTS_ENVIRONMENT in src/Makefile.am is being applied, since the include path doesn't contain any sign of it. It may be that this has changed somehow since ubuntu precise that travis runs on. (not checked)

Also make distcheck is inoperable. The current TESTS content seem to refer to the sourcedir tests (../../src/tests/) and writing output there, and as such tests/ in _build (DESTDIR?) isn't written. This leads to a build error. Last paragraph of https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html#Parallel-Test-Harness talks about this magic, but unclear if this is the problem at hand.

Note to self: there is no winning with automake. you just have to live with the pain.

[dridi]

Hello,

This is not specifically about parallel tests, but rather a global refresh of the test suite that encompasses parallel testing.

I have worked a bit on making the test suite more readable, or less painful to write, while extending its capabilities. I have a working PoC that would make T2 look like this:

#/bin/bash
# Test basic argument handling.
# This implements T2 in the original test plan.

. common.sh

set -e # XXX: this should eventually belong to common.sh

backend="$(start_http_server)"

try "start hitch" $HITCH $HITCH_ARGS \
        --backend="$backend" \
        --frontend="[localhost]:0" \
        certs/site1.example.com

hitch_pid="$(cat "$PIDFILE")"
hitch_sock="$(get_sock "$hitch_pid")"

echo |
try "s_client" openssl s_client -prexit -connect "$hitch_sock" |
try "check certificate" grep -q -c "subject=/CN=site1.example.com"

try "http request" curl --silent --insecure https://$hitch_sock/ >/dev/null

Instead of the "or die" logic, instead I introduced a non-intrusive function (notice the pipeline) that you use like this:

try msg cmd args...

The function's logic itself is relatively simple:

try() {
        msg="$1"
        shift
        logdump="$(mktemp)"
        errdump="$(mktemp)"
        if ! "$@" >"$logdump" 2>"$errdump"
        then
                echo >&2 "FAILED: $msg"
                echo >&2 "Stdout:"
                cat  >&2 "$logdump"
                echo >&2 "Stderr:"
                cat  >&2 "$errdump"
                exit 255
        fi
        cat "$logdump"
        cat "$errdump" >&2
}

Now, for parallel testing, or even CI-friendliness (eg. more than one hitch build in parallel) we need hitch to be able to bind randomly. Truly randomly, not the current racy randomness.

It looks like hitch only supports numeric ports, it might be a good idea not trying to be to smart about it and let it resolve whatever the user puts. Less logic in hitch and bonus points, we can bind things like [localhost]:https. 😃

In my PoC I only did this:

diff --git i/src/configuration.c w/src/configuration.c
index f9b0ac4..be188e6 100644
--- i/src/configuration.c
+++ w/src/configuration.c
@@ -331,7 +331,7 @@ config_param_host_port_wildcard(const char *str, char **addr,
        // printf("PARSED ADDR '%s', PORT '%s'\n", addr_buf, port_buf);

        int p = atoi(port_buf);
-       if (p < 1 || p > 65536) {
+       if (p < 0 || p > 65536) {
                config_error_set("Invalid port number '%s'", port_buf);
                return 0;
        }

After that I have a utility function to extract the socket information, but need to be tested on OtherPlatforms(tm):

get_sock() {
        lsof -a -p "$1" -i TCP -F |
        awk '/^n/ { print substr($1,2) }'
}

Finally, I have also written a small python HTTP server (54 LoC) that can listen to either IPv4 or IPv6 on random ports. It's based on SimpleHTTPServer, and allows insecure tests to be run offline.

FYI I use ShellCheck and follow its recommendations most of the time, so the code style leans towards paranoia. I would also change other things about the test suite but I'll consider this a milestone.

Is this an acceptable direction?

If yes, once cleaned up, I'll only need to make it an automake test suite and you'll get parallelism with make -j.

[fgsch]

I've done this before I knew there was a branch and it's working on travis CI.
See PR #81

[daghf]

Parallel tests no in place after merging #81

I'll leave this open for now to address @dridi's points later

[daghf]

Hi @dridi

Sorry for the delay here. I do think the changes you've outlined would be very welcome.

Would you like to take a stab at it, if time permits? (no rush)

[dridi]

Yes, and I suppose I will start afresh from current master when time permits. I'd like to squash other things on the varnish side first, but I should come back to hitch soon for #154 too.

[dridi]

#209 takes care of most of this.