GnuTLS failures in compat.sh on Travis

[mpg]

In compat.sh cases using GnuTLS, we started to see failures on multiple Travis runs in multiple PRs recently which don't seem to be related to the PR being tested, but more likely to a change on the Travis platform. All the failures are for cases using GnuTLS, but not all cases using GnuTLS fail.

Example of failing jobs:

• https://travis-ci.org/github/ARMmbed/mbedtls/jobs/711464150
• https://travis-ci.com/github/mpg/mbedtls/jobs/364562033

[mpg]

My first thought was perhaps they updated their GnuTLS and the new version removed support for some old things (this happened in the past), but looking at what fails and what passe doesn't seem consistent with that hypothesis, for example:

m->G tls1_2,yes TLS-ECDHE-PSK-WITH-AES-256-CBC-SHA ..................... PASS
m->G tls1_2,yes TLS-ECDHE-PSK-WITH-AES-128-CBC-SHA ..................... PASS
m->G tls1_2,yes TLS-RSA-PSK-WITH-AES-256-CBC-SHA ....................... PASS
m->G tls1_2,yes TLS-RSA-PSK-WITH-AES-128-CBC-SHA ....................... PASS
m->G tls1_2,yes TLS-ECDHE-PSK-WITH-AES-256-CBC-SHA384 .................. FAIL
  ! outputs saved to c-srv-1503.log, c-cli-1503.log

TLS-ECDHE-PSK-WITH-AES-256-CBC-SHA passes but not the same ciphersuite with just SHA-384 instead of SHA-1...

[mpg]

On Slack, @gilles wrote:

and it isn't an expired certificate or clock skew since PSK is randomly failing as well

@AndrzejKurek wrote:

I can see that there are problems with the certificate - according to gnutls it uses an insecure algorithm

- subject `C=NL,O=PolarSSL,CN=localhost', issuer `C=NL,O=PolarSSL,CN=PolarSSL Test CA', RSA key 2048 bits, signed using RSA-SHA1 (broken!), activated `2019-02-10 14:44:06 UTC', expires `2029-02-10 14:44:06 UTC', SHA-1 fingerprint `7d2245a4d39839d696df0566100cb3043ad53634'
[...]
 Status: The certificate is NOT trusted. The certificate chain uses insecure algorithm. 

@gilles-peskine-arm wrote:

The travis image was updated on Wednesday. Jobs started failing when
Build image provisioning date and time
Mon Mar 25 16:43:24 UTC 2019
changed to
Build image provisioning date and time
Wed Jun 24 13:36:52 UTC 2020

Poking further, I noticed that while the version of gnutls-bin is 3.4.10-4ubuntu1.8 both on Travis and on my machine, the version of libgnutls30 is 3.4.10-4ubuntu1.7. The discrepancy on Travis is because we install gnutls-bin explicitly (it's in the default image now, but it didn't used to be).

But I tried to upgrade libgnutls30. And this job https://travis-ci.org/github/ARMmbed/mbedtls/builds/711538485 with an up-to-date libgnutls30 still fails.

I tried upgrading everything (https://travis-ci.org/github/ARMmbed/mbedtls/builds/711554498), and it's no better.

[mpg]

@gilles-peskine-arm also wrote an awk script to check which suites fail always, never, etc. Here's a slightly reformatted version, wrapped in a shell loop to iterate over multiple log files:

#!/bin/sh

for i in travis-*-log.txt; do
    printf "\n*** %s ***\n" $i
    < $i \
    awk '$1 != "m->G" && $1 != "G->m" {next} \
        {pass = /PASS/; p[$3] = p[$3] pass} \
        END {for (c in p) {print p[c], c}}' |
        awk '$1~/01/ {++c01} \
            $1~/10/ {++c10} \
            $1~/^0+$/ {++c11}
            $1~/^1+$/ {++c00} {++c} \
            END {print "Total: " c; \
                 print "Always fail: " c00; \
                 print "Always pass: " c11; \
                 print "Fail after passing: " c10; \
                 print "Pass after failing: " c01}'
done

Output of running that script over a couple of logs:

*** travis-364562033-log.txt ***
Total: 200
Always fail: 86
Always pass: 32
Fail after passing: 82
Pass after failing: 

*** travis-711464150-log.txt ***
Total: 200
Always fail: 86
Always pass: 32
Fail after passing: 82
Pass after failing: 

*** travis-711538485-log.txt ***
Total: 200
Always fail: 168
Always pass: 32
Fail after passing: 
Pass after failing: 

*** travis-711554498-log.txt ***
Total: 200
Always fail: 168
Always pass: 32
Fail after passing: 
Pass after failing: 

*** travis-71210599-log.txt ***
Total: 200
Always fail: 86
Always pass: 32
Fail after passing: 82
Pass after failing: 

As Gilles observes, the number of "always pass" is constant, and the number of "pass after failing" is always zero.

However it looks like updating libgnutls30 did change something, as the two corresponding logs (711538485 and 711554498) also have "Fail after passing" equal to zero, which is perhaps a bit more sensible.

[gilles-peskine-arm]

However it looks like updating libgnutls30 did change something

Aha, I hadn't noticed that! So there must be two issues. In chat you wrote

it seems upgrading libgnutls does make things a bit better, in the sense that after the upgrade, each ciphersuite either always fails or always passes.

which makes sense given the changelog for the upgrade of libgnutls30 (3.4.10-4ubuntu1.7 to 3.4.10-4ubuntu1.8):

  * d/p/50_Update-session_ticket.c-to-add-support-for-zero-leng.patch:
    - add support for zero length  session tickets returned from the server,
      thanks Rod for the backport and testing! (lp: #1876286)

So whatever issue there is on Travis is compounded with 3.4.10-4ubuntu1.7 by a bug with session tickets whose effect is that once a server has failed, it tries to use a session ticket and fails again.

[mpg]

Nope, sorry I don't think that's it: I failed to notice that compat.sh is just not called the same way in those logs: tests/compat.sh -m tls1_2 -p GnuTLS -V NO vs the usual from all.sh... So it doesn't make sense to compare those logs with the others.

Back to square one.