Integrate Docker for E2E testing with Makefile and Azure pipeline orc…

…hestration fixed the vpn issue Replaced all instances of [ ] with [[ ]] for conditional checks, as per the discussion about how [[ ]] handles variables better Improve error handling and logging in e2e setup to capture detailed context for failures Improve networking Improve networking-with network- aro-dev: Improve networking-with network- aro-dev: ci pipeline cleanup ci pipeline cleanup ci pipeline cleanup ci pipeline cleanup ci pipeline cleanup ci pipeline cleanup ci pipeline cleanup ci pipeline cleanup solved the veriables issue solved the veriables issue using old way as e2e uses aro image shubhadapaithankar/ARO-9501-RP-E2E-Test - Adding image build step for E2E image. - Removed docker compose build from ci-rp target. - Added RP_IMAGE_ACR_URL env var to test overriding ARO operator image pull URL. shubhadapaithankar/ARO-9501-RP-E2E-Test - Reset to 8988bd2. - Changed image names in ci.yml to use ACR image URLs. - Added RP_IMAGE_ACR env var after sourcing secrets to debug ARO operator image pull error. - Removed localhost image check from ACR push script. shubhadapaithankar/ARO-9501-RP-E2E-Test - Fixed type in RP_IMAGE_ACR_URL env var name. changed to to match the name used in your CI code Modified some template veriables Fixed the env as per the latest changres Added ACR pull step to fix image pull error Update e2e-pipeline template to align with push template and ci pipeline Fix image path construction in e2e-pipeline template change environment variable format that Azure DevOps recognizes change environment variable format that Azure DevOps recognizes change environment variable format shubhadapaithankar/ARO-9501-RP-E2E-Test - Added az acr login before docker compose up. shubhadapaithankar/ARO-9501-RP-E2E-Test - Removed --target=builder for RP and E2E image builds so all stages are built. shubhadapaithankar/ARO-9501-RP-E2E-Test - Added ARO_IMAGE env var to set ARO operator image URL. shubhadapaithankar/ARO-9501-RP-E2E-Test - Added version to the ARO_IMAGE env var and set to Build.BuildId. shubhadapaithankar/ARO-9501-RP-E2E-Test - Moved Build.BuildId variable inside quotes. change makefile target to fix ci-rp issue change makefile target to fix ci-rp issue change makefile target to fix ci-rp issue change makefile target to fix ci-rp issue change makefile target to fix ci-rp issue change makefile target to fix ci-rp issue change makefile target to fix ci-rp issue Added logs Reverting forced push to the yestardays changes changed the indentation changed the indentation removed cluster version as its causing issue Updated Dockerfile to ensure test report and coverage files are generated Updated Dockerfile to ensure test report and coverage files are generated shubhadapaithankar/ARO-9501-RP-E2E-Test - Added WORKDIR setting and set to /usr/local/bin to fix portalauth command invocation. shubhadapaithankar/ARO-9501-RP-E2E-Test - Updated ARO_IMAGE env var to use new arosvcdev registry. Updated the ARO_IMAGE to arointsvc Updated the missing veriable Updated the ARO_IMAGE to arosvcdev shubhadapaithankar/ARO-9501-RP-E2E-Test - Disabling E2E cluster delete to debug API connectivity error. shubhadapaithankar/ARO-9501-RP-E2E-Test - Added AZURE_PORTAL_ELEVATED_GROUP_IDS env var that is required for portalauth cmd args. shubhadapaithankar/ARO-9501-RP-E2E-Test - Set path to portalauth binary to /usr/local/bin/portalauth. - Added info log to debug portal auth args. Removed the unwanted comment from the code shubhadapaithankar/ARO-9501-RP-E2E-Test - Adding KEYVAULT_PREFIX env var for portalauth command. Added E2E_LABEL Added the E2E_LABEL variable to the env file, as it's used to run E2E tests. shubhadapaithankar/ARO-9501-RP-E2E-Test - Cleanup: move duplicate variable exports to top level variables definition, remove unnecessary exports, etc. Added WaitWithTimeout in adminPortalSessionSetup to ensure <ul> element is present before proceeding fix(e2e): add extended wait and capture debug information for missing <ul> element removed the extended wait and capture debug information for missing <ul> element Modified must gather code Replaced host + /healthz/ready to explicitly set the path to check readiness. revert back the changes as its not useful Add enhanced logging for /portal access in adminPortalSessionSetup - Log attempt to access the main portal path (/portal) before making the request. - Add success log message upon successful access to /portal for better visibility. - Improves debugging by providing clear feedback on portal accessibility. Added must-gather logic Add error handling for E2E test step in pipeline Capture exit code immediately after E2E test command to prevent alteration by subsequent commands Capture exit code immediately after E2E test command to prevent alteration by subsequent commands Added Session Retry Mechanism in adminPortalSessionSetup Removed the last commit code and added skip for hive Removed the last commit code and added skip for hive Removed the unwanted code Removed the unwanted space and modified the stage name Removed the unwanted space from the code Removed the unwanted space from the code Removed unwanted code Removed unwanted code Documented docker-compose stuff in doc Documented docker-compose stuff in doc Documented docker-compose stuff in doc shubhadapaithankar/ARO-9501-RP-E2E-Test - Removed log statement used for debuging portal auth command and args. Removed the unwanted code removed ./ prefix from the portalauth command and WORKDIR /usr/local/bin from the Dockerfile.aro-e2e Added WORKDIR /usr/local/bin again as its mendetory removed ./ prefix from the portalauth command and WORKDIR /usr/local/bin from the Dockerfile.aro-e2e Added copy command to Dockerfile.aro-e2e Modified copy command to Dockerfile.aro-e2e Added multi-path support for portalAuthCmd in e2e tests Removed WORKING DIR and else Added the commit message to showcase e2e pipeline working stuff
Azure · Nov 20, 2024 · f58549a · f58549a
1 parent fde486f
commit f58549a
Show file tree

Hide file tree

Showing 13 changed files with 483 additions and 116 deletions.
diff --git a/.pipelines/ci.yml b/.pipelines/ci.yml
@@ -1,4 +1,4 @@
-# Azure DevOps Pipeline running CI
+# Azure DevOps Pipeline running CI and E2E
 
 trigger:
   branches:
@@ -13,70 +13,216 @@ trigger:
 
 variables:
   - template: vars.yml
+  - name: CI
+    value: true
   - name: REGISTRY
     value: registry.access.redhat.com
   - name: LOCAL_ARO_RP_IMAGE
-    value: "arosvcdev.azurecr.io/aro"
+    value: arosvcdev.azurecr.io/aro
   - name: LOCAL_ARO_AZEXT_IMAGE
-    value: "arosvcdev.azurecr.io/azext-aro"
+    value: arosvcdev.azurecr.io/azext-aro
   - name: LOCAL_VPN_IMAGE
-    value: "arosvcdev.azurecr.io/vpn"
+    value: arosvcdev.azurecr.io/vpn
+  - name: LOCAL_E2E_IMAGE
+    value: arosvcdev.azurecr.io/e2e
   - name: TAG
     value: $(Build.BuildId)
   - name: VERSION
     value: $(Build.BuildId)
+  - name: ARO_IMAGE
+    value: arosvcdev.azurecr.io/aro:$(Build.BuildId)
+  - name: ARO_SELENIUM_HOSTNAME
+    value: localhost
+  - name: E2E_LABEL
+    value: "!smoke&&!regressiontest"
+  - name: KUBECONFIG
+    value: aks.kubeconfig
 
-jobs:
-  - job: Build_Test_And_Push_Az_ARO_Extension
-    pool:
-      name: 1es-aro-ci-pool
-    steps:
-      - template: ./templates/template-checkout.yml
-
-      # Build and test the Az ARO Extension
-      - script: |
-          set -xe
-          DOCKER_BUILD_CI_ARGS="--load" make ci-azext-aro VERSION=$(VERSION)
-        displayName: 🛠 Build & Test Az ARO Extension
-
-      # Push the image to ACR
-      - template: ./templates/template-acr-push.yml
-        parameters:
-          acrFQDN: 'arosvcdev.azurecr.io'
-          repository: 'azext-aro'
-          pushLatest: true
-
-  - job: Build_And_Test_RP_And_Portal
-    pool:
-      name: 1es-aro-ci-pool
-    steps:
-      - template: ./templates/template-checkout.yml
-
-      # Build and test RP and Portal
-      - script: |
-          set -xe
-          DOCKER_BUILD_CI_ARGS="--load" make ci-rp VERSION=$(VERSION)
-        displayName: 🛠 Build & Test RP and Portal
-
-      # Publish test results
-      - task: PublishTestResults@2
-        displayName: 📊 Publish tests results
-        inputs:
-          testResultsFiles: $(System.DefaultWorkingDirectory)/report.xml
-        condition: succeededOrFailed()
-
-      # Publish code coverage results
-      - task: PublishCodeCoverageResults@2
-        displayName: 📈 Publish code coverage
-        inputs:
-          codeCoverageTool: Cobertura
-          summaryFileLocation: $(System.DefaultWorkingDirectory)/coverage.xml
-          failIfCoverageEmpty: false
-        condition: succeededOrFailed()
-
-      # Push the image to ACR
-      - template: ./templates/template-acr-push.yml
-        parameters:
-          acrFQDN: 'arosvcdev.azurecr.io'
-          repository: 'aro'
-          pushLatest: true
+stages:
+  - stage: Containerized_CI
+    jobs:
+      - job: Build_Test_And_Push_Az_ARO_Extension
+        pool:
+          name: 1es-aro-ci-pool
+        steps:
+          - template: ./templates/template-checkout.yml
+
+          # Build and test the Az ARO Extension
+          - script: |
+              set -xe
+              DOCKER_BUILD_CI_ARGS="--load" make ci-azext-aro VERSION=$(VERSION)
+            displayName: 🛠 Build & Test Az ARO Extension
+
+          # Push the image to ACR
+          - template: ./templates/template-acr-push.yml
+            parameters:
+              acrFQDN: 'arosvcdev.azurecr.io'
+              repository: 'azext-aro'
+              pushLatest: true
+
+      - job: Build_And_Test_RP_And_Portal
+        pool:
+          name: 1es-aro-ci-pool
+        steps:
+          - template: ./templates/template-checkout.yml
+
+          # Build and test RP and Portal
+          - script: |
+              set -xe
+              DOCKER_BUILD_CI_ARGS="--load" make ci-rp VERSION=$(VERSION)
+            displayName: 🛠 Build & Test RP and Portal
+
+          # Publish test results
+          - task: PublishTestResults@2
+            displayName: 📊 Publish tests results
+            inputs:
+              testResultsFiles: $(System.DefaultWorkingDirectory)/report.xml
+            condition: succeededOrFailed()
+
+          # Publish code coverage results
+          - task: PublishCodeCoverageResults@2
+            displayName: 📈 Publish code coverage
+            inputs:
+              codeCoverageTool: Cobertura
+              summaryFileLocation: $(System.DefaultWorkingDirectory)/coverage.xml
+              failIfCoverageEmpty: false
+            condition: succeededOrFailed()
+
+          # Push the RP image to ACR
+          - template: ./templates/template-acr-push.yml
+            parameters:
+              acrFQDN: 'arosvcdev.azurecr.io'
+              repository: 'aro'
+              pushLatest: true
+
+      - job: Build_And_Push_E2E_Image
+        pool:
+          name: 1es-aro-ci-pool
+        steps:
+          - template: ./templates/template-checkout.yml
+
+          # Build the E2E image
+          - script: |
+              set -xe
+              DOCKER_BUILD_CI_ARGS="--load" make aro-e2e VERSION=$(VERSION)
+            displayName: 🛠 Build the E2E image
+
+          # Push the E2E image to ACR
+          - template: ./templates/template-acr-push.yml
+            parameters:
+              acrFQDN: 'arosvcdev.azurecr.io'
+              repository: 'e2e'
+              pushLatest: true
+
+  - stage: E2E  # E2E Stage using Docker Compose
+    dependsOn: Containerized_CI
+    jobs:
+      - job: Run_E2E_Tests
+        timeoutInMinutes: 0
+        pool:
+          name: 1es-aro-ci-pool
+        steps:
+          # Checkout the code
+          - template: ./templates/template-checkout.yml
+
+          # Install Docker, Docker Compose, and dependencies
+          - bash: |
+              . ./hack/e2e/utils.sh
+              install_docker_dependencies
+            displayName: Install Docker and Docker Compose
+
+          # AZ CLI Login
+          - template: ./templates/template-az-cli-login.yml
+            parameters:
+              azureDevOpsJSONSPN: $(aro-v4-e2e-devops-spn)
+
+          # Get Kubeconfig for AKS Cluster with corrected Key Vault configuration
+          - bash: |
+              az account set -s $AZURE_SUBSCRIPTION_ID
+              SECRET_SA_ACCOUNT_NAME=$(SECRET_SA_ACCOUNT_NAME) make secrets
+              . secrets/env
+              export KEYVAULT_PREFIX="e2e-classic-eastus-cls"
+
+              # Retrieve the kubeconfig
+              hack/get-admin-aks-kubeconfig.sh > aks.kubeconfig
+
+              if [ -f aks.kubeconfig ]; then
+                echo "Kubeconfig retrieved successfully."
+              else
+                echo "Failed to retrieve Kubeconfig."
+                exit 1
+              fi
+            displayName: Get Kubeconfig for AKS Cluster
+
+          # Deploy Hive Operator
+          - bash: |
+              az account set -s $AZURE_SUBSCRIPTION_ID
+              SECRET_SA_ACCOUNT_NAME=$(SECRET_SA_ACCOUNT_NAME) make secrets
+              . secrets/env
+              docker compose -f docker-compose.yml up -d vpn
+              while [ "$(docker inspect --format '{{.State.Health.Status}}' vpn)" != "healthy" ]; do
+                echo "Waiting for VPN to be healthy..."
+                sleep 10
+              done
+              docker ps
+              kubectl get nodes
+              ./hack/hive/hive-dev-install.sh
+            displayName: Deploy Hive Operator
+
+          # Run the E2E test suite
+          - bash: |
+              az account set -s $AZURE_SUBSCRIPTION_ID
+              az acr login --name arosvcdev
+              SECRET_SA_ACCOUNT_NAME=$(SECRET_SA_ACCOUNT_NAME) make secrets
+              . ./hack/e2e/run-rp-and-e2e.sh
+              deploy_e2e_db
+              register_sub
+              docker compose up e2e
+            displayName: ⚙️  Run E2E Test Suite
+
+          # Log the output from the services in case of failure
+          - bash: |
+              docker compose logs vpn
+              docker compose logs selenium
+              docker compose logs rp
+              docker compose logs portal
+              docker compose logs e2e
+            displayName: Log Service Output
+            condition: always()
+
+          # Collect must-gather logs
+          - bash: |
+              wget -nv https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/$(OpenShiftCLIVersion)/openshift-client-linux-$(OpenShiftCLIVersion).tar.gz
+              tar xf openshift-client-linux-$(OpenShiftCLIVersion).tar.gz
+              ./oc adm must-gather --image quay.io/cmarches/aro-must-gather:20231030.00
+              tar cf must-gather.tar.gz must-gather.local.*
+            displayName: Collect must-gather
+            condition: Failed()
+          # Publish the must-gather result to the pipeline
+          - publish: must-gather.tar.gz
+            artifact: must-gather
+            displayName: Append must-gather to Pipeline
+            condition: Failed()
+
+          # Clean up Docker Compose services
+          - bash: |
+              docker compose down
+              rm -f aks.kubeconfig
+            displayName: Cleanup Docker Compose Services and Kubeconfig
+            condition: always()
+
+          # Clean Up Database
+          - bash: |
+              az cosmosdb sql database delete --name "$DATABASE_NAME" --yes --account-name "$DATABASE_ACCOUNT_NAME" --resource-group "$RESOURCEGROUP"
+            displayName: Clean Up Database
+            condition: always()
+
+          # Cleanup Hive Operator
+          - bash: |
+              echo "Cleaning up Hive Operator..."
+              kubectl delete namespace hive || echo "Namespace already deleted or does not exist."
+            displayName: Cleanup Hive Operator
+            condition: always()
+
+          # AZ CLI Logout
+          - template: ./templates/template-az-cli-logout.yml
diff --git a/.pipelines/templates/e2e-pipeline-template.yml b/.pipelines/templates/e2e-pipeline-template.yml
@@ -0,0 +1,79 @@
+# ./templates/e2e-pipeline-template.yml
+parameters:
+  - name: rpImageACR
+    type: string
+  - name: acrCredentialsJSON
+    type: string
+
+steps:
+  # Step 1: Authenticate to ACR and Install Docker Compose
+  - task: AzureCLI@2
+    displayName: 'Authenticate to ACR and Install Docker Compose'
+    inputs:
+      azureSubscription: 'ado-pipeline-dev-image-push'  # service connection
+      scriptType: bash
+      scriptLocation: 'inlineScript'
+      inlineScript: |
+        set -xe
+        # Ensure RP_IMAGE_ACR is correctly passed as a parameter
+        if [ -z "${{ parameters.rpImageACR }}" ]; then
+          echo "Error: RP_IMAGE_ACR is not set"
+          exit 1
+        fi
+
+        ACR_FQDN="${{ parameters.rpImageACR }}"
+        REGISTRY_NAME=$(echo $ACR_FQDN | cut -d'.' -f1)
+
+        # Install Docker Compose
+        echo "Installing Docker and Docker Compose Plugin..."
+        sudo apt-get update
+        sudo apt-get install -y ca-certificates curl gnupg
+        sudo install -m 0755 -d /etc/apt/keyrings
+        curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc
+        sudo chmod a+r /etc/apt/keyrings/docker.asc
+        echo \
+          "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
+          $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable" | \
+          sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+        sudo apt-get update
+        sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+        sudo systemctl start docker
+        sudo systemctl enable docker
+
+        # Login to ACR
+        az acr login --name $REGISTRY_NAME
+
+  # Step 2: Pull the RP Docker image
+  - script: |
+      if [ -z "${{ parameters.rpImageACR }}" ]; then
+          echo "Error: RP_IMAGE_ACR is not set"
+          exit 1
+      fi
+
+      export RP_IMAGE_ACR=${{ parameters.rpImageACR }}
+      export VERSION=$(Build.BuildId)
+
+      # Construct the RP image path correctly
+      if [[ "${RP_IMAGE_ACR}" == */aro ]]; then
+          RP_IMAGE="${RP_IMAGE_ACR}:${VERSION}"
+      else
+          RP_IMAGE="${RP_IMAGE_ACR}/aro:${VERSION}"
+      fi
+
+      echo "Pulling RP image: $RP_IMAGE"
+      docker pull $RP_IMAGE || { echo "Error: RP image $RP_IMAGE not found."; exit 1; }
+    displayName: Pull RP Docker Image
+
+  # Step 3: Pull the E2E Docker image
+  - script: |
+      if [ -z "${{ parameters.rpImageACR }}" ]; then
+          echo "Error: RP_IMAGE_ACR is not set"
+          exit 1
+      fi
+
+      export VERSION=$(Build.BuildId)
+      export E2E_IMAGE=${{ parameters.rpImageACR }}/e2e:${VERSION}
+
+      echo "Pulling E2E image: $E2E_IMAGE"
+      docker pull $E2E_IMAGE || { echo "Error: E2E image $E2E_IMAGE not found."; exit 1; }
+    displayName: Pull E2E Docker Image
diff --git a/.pipelines/templates/template-az-cli-login.yml b/.pipelines/templates/template-az-cli-login.yml
@@ -7,7 +7,13 @@ steps:
       set -e
 
       trap 'rm -f devops-spn.json' EXIT
-      base64 -d >devops-spn.json <<<${{ parameters.azureDevOpsJSONSPN }}
+      echo "${{ parameters.azureDevOpsJSONSPN }}" | base64 -d > devops-spn.json
 
-      az login --service-principal -u "$(jq -r .clientId <devops-spn.json)" -p "$(jq -r .clientSecret <devops-spn.json)" -t "$(jq -r .tenantId <devops-spn.json)" --allow-no-subscriptions >/dev/null
+      az login --service-principal \
+        -u "$(jq -r .clientId <devops-spn.json)" \
+        -p "$(jq -r .clientSecret <devops-spn.json)" \
+        -t "$(jq -r .tenantId <devops-spn.json)" --allow-no-subscriptions >/dev/null
+
+      # Cleanup
+      rm -f devops-spn.json
     displayName: 🗝 AZ Login
diff --git a/Dockerfile.aro-e2e b/Dockerfile.aro-e2e
@@ -17,6 +17,9 @@ FROM ${REGISTRY}/ubi8/ubi-minimal
 RUN microdnf update && microdnf clean all
 COPY --from=builder /root/go/bin/gojq /usr/local/bin/jq
 COPY --from=builder /app/aro /app/e2e.test /app/db /app/cluster /app/portalauth /usr/local/bin/
+# Setting ENV HOME=/tmp does not change the user’s default home directory of /
+# This setting is required to keep the existing e2e pipeline working without any code changes
+COPY --from=builder /app/portalauth /
 ENTRYPOINT ["aro"]
 EXPOSE 2222/tcp 8080/tcp 8443/tcp 8444/tcp 8445/tcp
 USER 1000

diff --git a/Dockerfile.ci-rp b/Dockerfile.ci-rp
@@ -72,6 +72,7 @@ FROM ${REGISTRY}/ubi8/ubi-minimal AS final
 LABEL aro-final=true
 RUN microdnf update && microdnf clean all
 COPY --from=builder /app/aro /app/e2e.test /usr/local/bin/
+COPY --from=builder /app/report.xml /app/coverage.xml /app/
 ENTRYPOINT ["aro"]
 EXPOSE 2222/tcp 8080/tcp 8443/tcp 8444/tcp
 USER 1000