Skip to content

Commit

Permalink
[QT-304] Add enos ui scenario (hashicorp#18518)
Browse files Browse the repository at this point in the history
* Add enos ui scenario
* Add github action for running the UI scenario
  • Loading branch information
Mike Baum authored Feb 3, 2023
1 parent 7f1e021 commit 6b7787c
Show file tree
Hide file tree
Showing 11 changed files with 507 additions and 4 deletions.
147 changes: 147 additions & 0 deletions .github/workflows/test-enos-scenario-ui.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
---
name: Vault UI Tests

on:
workflow_call:
inputs:
test_filter:
type: string
description: "A filter to limit the ui tests to. Will be appended to the ember test command as '-f=<filter>'"
required: false
storage_backend:
type: string
description: "The storage backend to use, either 'raft' or 'consul'"
default: raft
workflow_dispatch:
inputs:
test_filter:
type: string
description: "A filter to limit the ui tests to. Will be appended to the ember test command as '-f=<filter>'"
required: false
storage_backend:
description: "The storage backend to use, either 'raft' or 'consul'"
required: true
default: raft
type: choice
options:
- raft
- consul

jobs:
get-metadata:
name: Get metadata
runs-on: ubuntu-latest
outputs:
go-version: ${{ steps.get-metadata.outputs.go-version }}
node-version: ${{ steps.get-metadata.outputs.node-version }}
runs-on: ${{ steps.get-metadata.outputs.runs-on }}
vault_edition: ${{ steps.get-metadata.outputs.vault_edition }}
steps:
- uses: actions/checkout@v3
- id: get-metadata
env:
IS_ENT: ${{ startsWith(github.event.repository.name, 'vault-enterprise' ) }}
run: |
echo "go-version=$(cat ./.go-version)" >> $GITHUB_OUTPUT
echo "node-version=$(cat ./ui/.nvmrc)" >> $GITHUB_OUTPUT
echo "chrome-installed=$(chrome --version && echo true || echo false)" >> $GITHUB_OUTPUT
if ${IS_ENT} == true; then
echo "detected vault_edition=ent"
echo "runs-on=['self-hosted', 'ondemand', 'os=linux', 'type=m5d.4xlarge']" >> $GITHUB_OUTPUT
echo "vault_edition=ent" >> $GITHUB_OUTPUT
else
echo "detected vault_edition=oss"
echo "runs-on=custom-linux-xl-vault-latest" >> $GITHUB_OUTPUT
echo "vault_edition=oss" >> $GITHUB_OUTPUT
fi
run-ui-tests:
name: Run UI Tests
needs: get-metadata
runs-on: ${{ fromJSON(needs.get-metadata.outputs.runs-on) }}
timeout-minutes: 90
env:
GITHUB_TOKEN: ${{ secrets.ELEVATED_GITHUB_TOKEN }}
# Pass in enos variables
ENOS_VAR_aws_region: us-east-1
ENOS_VAR_aws_ssh_keypair_name: ${{ github.event.repository.name }}-ci-ssh-key
ENOS_VAR_aws_ssh_private_key_path: ./support/private_key.pem
ENOS_VAR_tfc_api_token: ${{ secrets.TF_API_TOKEN }}
ENOS_VAR_terraform_plugin_cache_dir: ./support/terraform-plugin-cache
ENOS_VAR_vault_license_path: ./support/vault.hclic
GOPRIVATE: github.com/hashicorp
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Set Up Go
uses: actions/setup-go@v3
with:
go-version: ${{ needs.get-metadata.outputs.go-version }}
- uses: hashicorp/action-setup-enos@v1
with:
github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }}
- name: Set Up Git
run: git config --global url."https://${{ secrets.elevated_github_token }}:@github.com".insteadOf "https://github.com"
- name: Set Up Node
uses: actions/setup-node@v3
with:
node-version: ${{ needs.get-metadata.outputs.node-version }}
- name: Set Up Terraform
uses: hashicorp/setup-terraform@v2
with:
cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
terraform_wrapper: false
- name: Prepare scenario dependencies
run: |
mkdir -p ./enos/support/terraform-plugin-cache
echo "${{ secrets.SSH_KEY_PRIVATE_CI }}" > ./enos/support/private_key.pem
chmod 600 ./enos/support/private_key.pem
- name: Set Up Vault Enterprise License
if: contains(${{ github.event.repository.name }}, 'ent')
run: echo "${{ secrets.VAULT_LICENSE }}" > ./enos/support/vault.hclic || true
- name: Install Chrome Dependencies
if: ${{ !needs.get-metadata.outputs.chrome-installed }}
run: |
sudo apt update
sudo apt install -y libnss3-dev libgdk-pixbuf2.0-dev libgtk-3-dev libxss-dev libasound2
- name: Install Chrome
if: ${{ !needs.get-metadata.outputs.chrome-installed }}
uses: browser-actions/setup-chrome@v1
- run: |
chrome --version
- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v1-node16
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID_CI }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY_CI }}
aws-region: us-east-1
role-to-assume: ${{ secrets.AWS_ROLE_ARN_CI }}
role-skip-session-tagging: true
role-duration-seconds: 3600
- name: Set Up Cluster
id: setup_cluster
env:
ENOS_VAR_ui_run_tests: false
# Continue once and retry to handle occasional blips when creating infrastructure.
continue-on-error: true
run: enos scenario launch --timeout 60m0s --chdir ./enos ui edition:${{ needs.get-metadata.outputs.vault_edition }} backend:${{ inputs.storage_backend }}
- name: Retry Set Up Cluster
id: setup_cluster_retry
if: steps.setup_cluster.outcome == 'failure'
env:
ENOS_VAR_ui_run_tests: false
run: enos scenario launch --timeout 60m0s --chdir ./enos ui edition:${{ needs.get-metadata.outputs.vault_edition }} backend:${{ inputs.storage_backend }}
- name: Run UI Tests
id: run_ui_tests
env:
ENOS_VAR_ui_test_filter: "${{ inputs.test_filter }}"
run: enos scenario run --timeout 60m0s --chdir ./enos ui edition:${{ needs.get-metadata.outputs.vault_edition }} backend:${{ inputs.storage_backend }}
- name: Ensure scenario has been destroyed
if: ${{ always() }}
run: enos scenario destroy --timeout 60m0s --chdir ./enos ui edition:${{ needs.get-metadata.outputs.vault_edition }} backend:${{ inputs.storage_backend }}
- name: Clean up Enos runtime directories
if: ${{ always() }}
run: |
rm -rf /tmp/enos*
rm -rf ./enos/support
rm -rf ./enos/.enos
50 changes: 50 additions & 0 deletions enos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,56 @@ The [`replication` scenario](./enos-scenario-replication.hcl) creates two 3-node

This scenario verifies the performance replication status on both clusters to have their connection_status as "connected" and that the secondary cluster has known_primaries cluster addresses updated to the active nodes IP addresses of the primary Vault cluster. This scenario currently works around issues VAULT-12311 and VAULT-12309. The scenario fails when the primary storage backend is Consul due to issue VAULT-12332

## UI Tests
The [`ui` scenario](./enos-scenario-ui.hcl) creates a Vault cluster (deployed to AWS) using a version
built from the current checkout of the project. Once the cluster is available the UI acceptance tests
are run in a headless browser.
### Variables
In addition to the required variables that must be set, as described in the [Scenario Variables](#Scenario Variables),
the `ui` scenario has two optional variables:

**ui_test_filter** - An optional test filter to limit the tests that are run, i.e. `'!enterprise'`.
To set a filter export the variable as follows:
```shell
> export ENOS_VAR_ui_test_filter="some filter"
```
**ui_run_tests** - An optional boolean variable to run or not run the tests. The default value is true.
Setting this value to false is useful in the case where you want to create a cluster, but run the tests
manually. The section [Running the Tests](#Running the Tests) describes the different ways to run the
'UI' acceptance tests.

### Running the Tests
The UI tests can be run fully automated or manually.
#### Fully Automated
The following will deploy the cluster, run the tests, and subsequently tear down the cluster:
```shell
> export ENOS_VAR_ui_test_filter="some filter" # <-- optional
> cd enos
> enos scenario ui run edition:oss
```
#### Manually
The UI tests can be run manually as follows:
```shell
> export ENOS_VAR_ui_test_filter="some filter" # <-- optional
> export ENOS_VAR_ui_run_tests=false
> cd enos
> enos scenario ui launch edition:oss
# once complete the scenario will output a set of environment variables that must be exported. The
# output will look as follows:
export TEST_FILTER='some filter>' \
export VAULT_ADDR='http://<some ip address>:8200' \
export VAULT_TOKEN='<some token>' \
export VAULT_UNSEAL_KEYS='["<some key>","<some key>","<some key>"]'
# copy and paste the above into the terminal to export the values
> cd ../ui
> yarn test:enos # run headless
# or
> yarn test:enos -s # run manually in a web browser
# once testing is complete
> cd ../enos
> enos scenario ui destroy edition:oss
```

# Variants
Both scenarios support a matrix of variants. In order to achieve broad coverage while
keeping test run time reasonable, the variants executed by the `enos-run` Github
Expand Down
6 changes: 6 additions & 0 deletions enos/enos-modules.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -197,3 +197,9 @@ module "vault_raft_remove_peer" {
source = "./modules/vault_raft_remove_peer"
vault_install_dir = var.vault_install_dir
}

module "vault_test_ui" {
source = "./modules/vault_test_ui"

ui_run_tests = var.ui_run_tests
}
2 changes: 1 addition & 1 deletion enos/enos-scenario-autopilot.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ scenario "autopilot" {
}

step "verify_undo_logs_status" {
skip_step = semverconstraint(var.vault_product_version, "<1.13.0-0")
skip_step = try(semverconstraint(var.vault_product_version, "<1.13.0-0"), true)
module = module.vault_verify_undo_logs
depends_on = [
step.remove_old_nodes,
Expand Down
Loading

0 comments on commit 6b7787c

Please sign in to comment.