Tests should either match cluster version, or use latest patch version of same minor release

[aaronlevy]

The instructions currently point to using sonobuoy/e2e tests pinned at v1.7.3 for all v1.7 clusters https://github.com/cncf/k8s-conformance/blob/master/sonobuoy-conformance-1.7.yaml#L95

I would expect that the e2e tests should either match the same version as the cluster, or use the highest patch version for that minor release.

[luxas]

cc @timothysc

[timothysc]

As mentioned in the other issue, feel free to submit a PR to update the instructions.

[aaronlevy]

@timothysc no newer v1.7.x images exist. If we're going to use this as the recommended way to run conformance tests - is it possible to get it built for every release? Otherwise maybe we could document building conformance test images (or auto-build conformance-only images) on their own?

I've just been using the go docker image with something like:

apt-get update && apt-get install -y rsync && go get -u github.com/jteeuwen/go-bindata/go-bindata

cd /go/src/k8s.io/kubernetes && \
 git checkout ${CONFORMANCE_VERSION} && \
 make all WHAT=cmd/kubectl && \
 make all WHAT=vendor/github.com/onsi/ginkgo/ginkgo && \
 make all WHAT=test/e2e/e2e.test"

[timothysc]

I would expect that the e2e tests should either match the same version as the cluster, or use the highest patch version for that minor release.

Highest version is the answer b/c we now patch tests to be version aware and not every fix gets pulled back to the other series on a release. So 1.8.x works properly on 1.7.y and has been validated on across a number of distros and envs.

Here is the Sonobuoy 0.9.0 example which will work for both 1.8 and 1.7 - https://github.com/heptio/sonobuoy/blob/release-0.9/examples/conformance.yaml

[timothysc]

I'll PR the instructions to make it easier and to point to the other docs.