Skip to content

Commit

Permalink
[CI][Windows] Support connectivity tests & enable e2e Jenkins jobs in…
Browse files Browse the repository at this point in the history 
… Windows (#2018)

* [CI][Windows] Support connectivity test & enable e2e Jenkins jobs in Windows
* Enable some connectivity tests
* Added windowsNodes, windowsNodeIdxes fields in ClusterInfo
* Involved at least a Windows Node when createBusyboxPodOnNode
* Refactore some e2e test code
* Removed unnecessary numWorkerNodes field
* Updated test.sh to support Windows e2e tests
* Updated CI doc
* Used provider remote

Co-authored-by: Antonin Bas <[email protected]>
  • Loading branch information
@lzhecheng @antoninbas
lzhecheng and antoninbas authored May 12, 2021
1 parent 429f230 commit beee659
Show file tree
Hide file tree
Showing 24 changed files with 538 additions and 152 deletions.
67 changes: 67 additions & 0 deletions ci/jenkins/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,24 +33,86 @@ should be deleted. This ensures that all tests are run on a clean testbed.

* ipv6-ds-e2e: e2e tests in a dual-stack setup.

```shell
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase e2e --registry ${DOCKER_REGISTRY}
```

* ipv6-ds-conformance: conformance tests in a dual-stack setup.

```shell
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase conformance --registry ${DOCKER_REGISTRY}
```

* ipv6-ds-networkpolicy: NetworkPolicy tests in a dual-stack setup.

```shell
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase networkpolicy --registry ${DOCKER_REGISTRY}
```

* ipv6-only-e2e: e2e tests in an IPv6 only setup.

```shell
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase e2e --registry ${DOCKER_REGISTRY}
```

* ipv6-only-conformance: conformance tests in an IPv6 only setup.

```shell
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase conformance --registry ${DOCKER_REGISTRY}
```

* ipv6-only-networkpolicy: NetworkPolicy tests in an IPv6 only setup.

```shell
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase networkpolicy --registry ${DOCKER_REGISTRY}
```

* windows e2e: e2e tests in a Windows setup.

```shell
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase windows-e2e --registry ${DOCKER_REGISTRY}
```

* windows conformance: community tests on Windows cluster, focusing on "Conformance|sig-windows" and "sig-network",
and skipping "LinuxOnly", "Slow", "Serial", "Disruptive", "Flaky", "Feature", "sig-cli", "sig-storage", "sig-auth",
"sig-api-machinery", "sig-apps", "sig-node", "Privileged", "should be able to change the type from", "[sig-network]
Services should be able to create a functioning NodePort service [Conformance]", "Service endpoints latency should not
be very high".

```shell
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase windows-conformance --registry ${DOCKER_REGISTRY}
```

* windows network policy: community tests on Windows cluster, focusing on "Feature:NetworkPolicy".

```shell
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase windows-networkpolicy --registry ${DOCKER_REGISTRY}
```

* [whole-conformance [daily]](https://jenkins.antrea-ci.rocks/job/antrea-whole-conformance-for-pull-request/):
community tests using sonobuoy, with certified-conformance mode.

Expand All @@ -69,6 +131,11 @@ should be deleted. This ensures that all tests are run on a clean testbed.
* Jenkins Windows OVS validator: this job only executes for PRs that include changes to [hack/windows/Install-OVS.ps1](hack/windows/Install-OVS.ps1). It validates
if Windows OVS can be installed correctly.

```shell
#!/bin/bash
./ci/jenkins/test.sh --testcase windows-install-ovs
```

* [EKS conformance/network policy [bi-daily]](http://jenkins.antrea-ci.rocks/view/cloud/job/cloud-antrea-eks-conformance-net-policy/)
community tests on EKS cluster using sonobuoy, focusing on "Conformance" and "Feature:NetworkPolicy", skipping the same regexes as in job __conformance__ above, as well as "NodePort" (See [#690](https://github.com/vmware-tanzu/antrea/issues/690)).\
Current test environment matrix:
Expand Down
124 changes: 84 additions & 40 deletions ci/jenkins/test.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ NETWORKPOLICY_SKIP="should allow egress access to server in CIDR block|should en
CONTROL_PLANE_NODE_ROLE="master"

_usage="Usage: $0 [--kubeconfig <KubeconfigSavePath>] [--workdir <HomePath>]
[--testcase <windows-install-ovs|windows-conformance|windows-networkpolicy|e2e|conformance|networkpolicy>]
[--testcase <windows-install-ovs|windows-conformance|windows-networkpolicy|windows-e2e|e2e|conformance|networkpolicy>]
Run K8s e2e community tests (Conformance & Network Policy) or Antrea e2e tests on a remote (Jenkins) Windows or Linux cluster.
Expand Down Expand Up @@ -130,7 +130,7 @@ function clean_for_windows_install_cni {
kubectl get nodes -o wide --no-headers=true | awk -v role="$CONTROL_PLANE_NODE_ROLE" '$3 != role && $1 ~ /win/ {print $6}' | while read IP; do
CLEAN_LIST=("/cygdrive/c/opt/cni/bin/antrea.exe" "/cygdrive/c/opt/cni/bin/host-local.exe" "/cygdrive/c/k/antrea/etc/antrea-agent.conf" "/cygdrive/c/etc/cni/net.d/10-antrea.conflist" "/cygdrive/c/k/antrea/bin/antrea-agent.exe")
for file in "${CLEAN_LIST[@]}"; do
ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "rm ${file}" || true
ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "rm -f ${file}"
done
done
}
Expand Down Expand Up @@ -163,6 +163,43 @@ function collect_windows_network_info_and_logs {
tar zcf antrea_agent_log.tar.gz antrea_agent_log
}

function wait_for_antrea_windows_pods_ready {
kubectl apply -f "${WORKDIR}/antrea.yml"
kubectl apply -f "${WORKDIR}/kube-proxy-windows.yml"
kubectl apply -f "${WORKDIR}/antrea-windows.yml"
kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system
kubectl rollout status deployment.apps/antrea-controller -n kube-system
kubectl rollout status daemonset/antrea-agent -n kube-system
kubectl rollout status daemonset.apps/antrea-agent-windows -n kube-system
kubectl rollout status daemonset/kube-proxy-windows -n kube-system
kubectl get nodes -o wide --no-headers=true | awk -v role="$CONTROL_PLANE_NODE_ROLE" '$3 != role && $1 ~ /win/ {print $6}' | while read IP; do
for i in `seq 5`; do
sleep 5
timeout 5s ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "powershell Get-NetAdapter -Name br-int -ErrorAction SilentlyContinue" && break
done
sleep 10
done
}

function wait_for_antrea_windows_processes_ready {
kubectl apply -f "${WORKDIR}/antrea.yml"
kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system
kubectl rollout status deployment.apps/antrea-controller -n kube-system
kubectl rollout status daemonset/antrea-agent -n kube-system
kubectl get nodes -o wide --no-headers=true | awk -v role="$CONTROL_PLANE_NODE_ROLE" '$3 != role && $1 ~ /win/ {print $6}' | while read IP; do
echo "===== Run script to startup Antrea agent ====="
ANTREA_VERSION=$(ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "/cygdrive/c/k/antrea/bin/antrea-agent.exe --version" | awk '{print $3}')
ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "chmod +x /cygdrive/c/k/antrea/Start.ps1 && powershell 'c:\k\antrea\Start.ps1 -AntreaVersion ${ANTREA_VERSION}'"
for i in `seq 5`; do
sleep 5
timeout 5s ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "powershell Get-NetAdapter -Name br-int -ErrorAction SilentlyContinue" && break
done
sleep 10
done
}

function deliver_antrea_windows {
echo "====== Cleanup Antrea Installation ======"
export KUBECONFIG=$KUBECONFIG_PATH
Expand Down Expand Up @@ -235,16 +272,17 @@ function deliver_antrea_windows {
scp -o StrictHostKeyChecking=no -T hack/windows/Helper.psm1 Administrator@${IP}:/cygdrive/c/k/antrea/
scp -o StrictHostKeyChecking=no -T build/yamls/windows/base/conf/antrea-cni.conflist Administrator@${IP}:/cygdrive/c/etc/cni/net.d/10-antrea.conflist
scp -o StrictHostKeyChecking=no -T build/yamls/windows/base/conf/antrea-agent.conf Administrator@${IP}:/cygdrive/c/k/antrea/etc
elif [ "$TESTCASE" == "windows-conformance" ]; then
else
if ! (test -f antrea-windows.tar.gz); then
# Compress antrea repo and copy it to a Windows node
mkdir -p jenkins
tar --exclude='./jenkins' -czvf jenkins/antrea_repo.tar.gz -C "$(pwd)" .
tar --exclude='./jenkins' -czf jenkins/antrea_repo.tar.gz -C "$(pwd)" .
for i in `seq 2`; do
timeout 2m scp -o StrictHostKeyChecking=no -T jenkins/antrea_repo.tar.gz Administrator@${IP}: && break
done
echo "=== Build Windows on Windows Node==="
ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "docker pull ${DOCKER_REGISTRY}/antrea/golang:1.15 && docker tag ${DOCKER_REGISTRY}/antrea/golang:1.15 golang:1.15"
ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "rm -rf antrea && mkdir antrea && cd antrea && tar -xzvf ../antrea_repo.tar.gz && sed -i \"s|build/images/Dockerfile.build.windows .|build/images/Dockerfile.build.windows . --network host|g\" Makefile && NO_PULL=${NO_PULL} make build-windows && docker save -o antrea-windows.tar projects.registry.vmware.com/antrea/antrea-windows:latest && gzip -f antrea-windows.tar" || true
ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "rm -rf antrea && mkdir antrea && cd antrea && tar -xzf ../antrea_repo.tar.gz && sed -i \"s|build/images/Dockerfile.build.windows .|build/images/Dockerfile.build.windows . --network host|g\" Makefile && NO_PULL=${NO_PULL} make build-windows && docker save -o antrea-windows.tar ${DOCKER_REGISTRY}/antrea/antrea-windows:latest && gzip -f antrea-windows.tar" || true
for i in `seq 2`; do
timeout 2m scp -o StrictHostKeyChecking=no -T Administrator@${IP}:antrea/antrea-windows.tar.gz . && break
done
Expand Down Expand Up @@ -330,9 +368,10 @@ function run_e2e {
export PATH=$GOROOT/bin:$PATH
export KUBECONFIG=$KUBECONFIG_PATH

mkdir -p test/e2e/infra/vagrant/playbook/kube
cp -f "${WORKDIR}/kube.conf" test/e2e/infra/vagrant/playbook/kube/config
cp -f "${WORKDIR}/ssh-config" test/e2e/infra/vagrant/ssh-config
mkdir -p "${WORKDIR}/.kube"
mkdir -p "${WORKDIR}/.ssh"
cp -f "${WORKDIR}/kube.conf" "${WORKDIR}/.kube/config"
cp -f "${WORKDIR}/ssh-config" "${WORKDIR}/.ssh/config"

set +e
mkdir -p `pwd`/antrea-test-logs
Expand Down Expand Up @@ -375,6 +414,34 @@ function run_conformance {
fi
}

function run_e2e_windows {
echo "====== Running Antrea e2e Tests ======"
export GO111MODULE=on
export GOPATH=${WORKDIR}/go
export GOROOT=/usr/local/go
export GOCACHE=${WORKDIR}/.cache/go-build
export PATH=$GOROOT/bin:$PATH
export KUBECONFIG=$KUBECONFIG_PATH

clean_for_windows_install_cni
wait_for_antrea_windows_pods_ready

mkdir -p "${WORKDIR}/.kube"
mkdir -p "${WORKDIR}/.ssh"
cp -f "${WORKDIR}/kube.conf" "${WORKDIR}/.kube/config"
cp -f "${WORKDIR}/ssh-config" "${WORKDIR}/.ssh/config"

set +e
mkdir -p `pwd`/antrea-test-logs
go test -v github.com/vmware-tanzu/antrea/test/e2e --logs-export-dir `pwd`/antrea-test-logs --provider remote -timeout=50m --prometheus
if [[ "$?" != "0" ]]; then
TEST_FAILURE=true
fi
set -e

tar -zcf antrea-test-logs.tar.gz antrea-test-logs
}

function run_conformance_windows {
echo "====== Running Antrea Conformance Tests ======"
export GO111MODULE=on
Expand All @@ -385,39 +452,12 @@ function run_conformance_windows {
export KUBECONFIG=$KUBECONFIG_PATH

if [[ "$TESTCASE" == "windows-conformance" ]]; then
# Antrea Windows agent Pods are deployed for Windows Conformance test
clean_for_windows_install_cni
kubectl apply -f build/yamls/antrea.yml
kubectl apply -f ${WORKDIR}/kube-proxy-windows.yml
kubectl apply -f build/yamls/antrea-windows.yml
kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system
kubectl rollout status deployment.apps/antrea-controller -n kube-system
kubectl rollout status daemonset/antrea-agent -n kube-system
kubectl rollout status daemonset.apps/antrea-agent-windows -n kube-system
kubectl rollout status daemonset/kube-proxy-windows -n kube-system
kubectl get nodes -o wide --no-headers=true | awk -v role="$CONTROL_PLANE_NODE_ROLE" '$3 != role && $1 ~ /win/ {print $6}' | while read IP; do
for i in `seq 5`; do
sleep 5
timeout 5s ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "powershell Get-NetAdapter -Name br-int -ErrorAction SilentlyContinue" && break
done
sleep 10
done
wait_for_antrea_windows_pods_ready
else
kubectl apply -f build/yamls/antrea.yml
kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system
kubectl rollout status deployment.apps/antrea-controller -n kube-system
kubectl rollout status daemonset/antrea-agent -n kube-system
kubectl get nodes -o wide --no-headers=true | awk -v role="$CONTROL_PLANE_NODE_ROLE" '$3 != role && $1 ~ /win/ {print $6}' | while read IP; do
echo "===== Run script to startup antrea agent ====="
ANTREA_VERSION=$(ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "/cygdrive/c/k/antrea/bin/antrea-agent.exe --version" | awk '{print $3}')
ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "chmod +x /cygdrive/c/k/antrea/Start.ps1 && powershell 'c:\k\antrea\Start.ps1 -AntreaVersion ${ANTREA_VERSION}'"
for i in `seq 5`; do
sleep 5
timeout 5s ssh -o StrictHostKeyChecking=no -n Administrator@${IP} "powershell Get-NetAdapter -Name br-int -ErrorAction SilentlyContinue" && break
done
sleep 10
done
# Antrea Windows agents are deployed with scripts as processes on host for Windows NetworkPolicy test
wait_for_antrea_windows_processes_ready
fi

echo "====== Run test with e2e.test ======"
Expand Down Expand Up @@ -464,7 +504,11 @@ if [[ ${TESTCASE} == "windows-install-ovs" ]]; then
run_install_windows_ovs
elif [[ ${TESTCASE} =~ "windows" ]]; then
deliver_antrea_windows
run_conformance_windows
if [[ ${TESTCASE} =~ "e2e" ]]; then
run_e2e_windows
else
run_conformance_windows
fi
clean_antrea
elif [[ ${TESTCASE} =~ "e2e" ]]; then
deliver_antrea
Expand Down
8 changes: 8 additions & 0 deletions test/e2e/antctl_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ func antctlCoverageArgs(antctlPath string) []string {

// TestAntctlAgentLocalAccess ensures antctl is accessible in a agent Pod.
func TestAntctlAgentLocalAccess(t *testing.T) {
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down Expand Up @@ -126,6 +128,8 @@ func copyAntctlToNode(data *TestData, nodeName string, antctlName string, nodeAn
// the kubernetes cluster. It uses the antctl client binary copied from the controller
// Pod.
func TestAntctlControllerRemoteAccess(t *testing.T) {
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down Expand Up @@ -185,6 +189,8 @@ func TestAntctlControllerRemoteAccess(t *testing.T) {
// TestAntctlVerboseMode ensures no unexpected outputs during the execution of
// the antctl client.
func TestAntctlVerboseMode(t *testing.T) {
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down Expand Up @@ -262,6 +268,8 @@ func runAntctProxy(nodeName string, antctlName string, nodeAntctlPath string, pr
func TestAntctlProxy(t *testing.T) {
const proxyPort = 8001

skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down
8 changes: 8 additions & 0 deletions test/e2e/antreapolicy_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2565,6 +2565,8 @@ func waitForResourceDelete(namespace, name string, resource string, timeout time
}

func TestAntreaPolicy(t *testing.T) {
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down Expand Up @@ -2664,6 +2666,8 @@ func TestAntreaPolicy(t *testing.T) {
}

func TestAntreaPolicyStatus(t *testing.T) {
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down Expand Up @@ -2727,6 +2731,8 @@ func TestAntreaPolicyStatus(t *testing.T) {
// TestANPNetworkPolicyStatsWithDropAction tests antreanetworkpolicystats can correctly collect dropped packets stats from ANP if
// networkpolicystats feature is enabled
func TestANPNetworkPolicyStatsWithDropAction(t *testing.T) {
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down Expand Up @@ -2878,6 +2884,8 @@ func TestANPNetworkPolicyStatsWithDropAction(t *testing.T) {
}

func TestAntreaClusterNetworkPolicyStats(t *testing.T) {
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down
6 changes: 6 additions & 0 deletions test/e2e/bandwidth_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ const iperfPort = 5201
func TestBenchmarkBandwidthIntraNode(t *testing.T) {
skipIfNotBenchmarkTest(t)
skipIfNotIPv4Cluster(t)
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down Expand Up @@ -58,6 +60,8 @@ func TestBenchmarkBandwidthIntraNode(t *testing.T) {
}

func benchmarkBandwidthService(t *testing.T, endpointNode, clientNode string) {
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down Expand Up @@ -111,6 +115,8 @@ func TestPodTrafficShaping(t *testing.T) {
skipIfIPv6Cluster(t)
nodeName := controlPlaneNodeName()
skipIfMissingKernelModule(t, nodeName, []string{"ifb", "sch_tbf", "sch_ingress"})
skipIfHasWindowsNodes(t)

data, err := setupTest(t)
if err != nil {
t.Fatalf("Error when setting up test: %v", err)
Expand Down
Loading

0 comments on commit beee659

Please sign in to comment.