Skip to content

Commit

Permalink
Merge branch 'florianduros/encryption-tab' into dbkr/e2e_toasts
Browse files Browse the repository at this point in the history
  • Loading branch information
dbkr authored Jan 10, 2025
2 parents cb43672 + e5dea48 commit b7b7a98
Show file tree
Hide file tree
Showing 115 changed files with 3,816 additions and 2,935 deletions.
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,5 @@
/src/i18n/strings
/src/i18n/strings/en_EN.json @element-hq/element-web-reviewers
# Ignore the synapse plugin as this is updated by GHA for docker image updating
/playwright/plugins/homeserver/synapse/index.ts
/playwright/testcontainers/synapse.ts

3 changes: 3 additions & 0 deletions .github/labels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,9 @@
- name: "X-Upcoming-Release-Blocker"
description: "This does not affect the current release cycle but will affect the next one"
color: "e99695"
- name: "X-Run-All-Tests"
description: "When applied to PRs, it'll run the full gamut of end-to-end tests on the PR"
color: "ff7979"
- name: "Z-Actions"
color: "ededed"
- name: "Z-Cache-Confusion"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/dockerhub.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
uses: sigstore/cosign-installer@dc72c7d5c4d10cd6bcb8cf6e3fd625a9e5e537da # v3

- name: Set up QEMU
uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf # v3
uses: docker/setup-qemu-action@53851d14592bedcffcf25ea515637cff71ef929a # v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@6524bf65af31da8d45b59e8c27de4bd072b392f5 # v3
Expand All @@ -51,7 +51,7 @@ jobs:
- name: Build and push
id: build-and-push
uses: docker/build-push-action@48aba3b46d1b1fec4febb7c5d0c644b249a11355 # v6
uses: docker/build-push-action@b32b51a8eda65d6793cd0494a773d4f6bcef32dc # v6
with:
context: .
push: true
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/end-to-end-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -114,13 +114,13 @@ jobs:
- Chrome
- Firefox
- WebKit
isCron:
- ${{ github.event_name == 'schedule' }}
# Skip the Firefox & Safari runs unless this was a cron trigger
runAllTests:
- ${{ github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'X-Run-All-Tests') }}
# Skip the Firefox & Safari runs unless this was a cron trigger or PR has X-Run-All-Tests label
exclude:
- isCron: false
- runAllTests: false
project: Firefox
- isCron: false
- runAllTests: false
project: WebKit
steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -170,7 +170,7 @@ jobs:
yarn playwright test \
--shard "${{ matrix.runner }}/${{ needs.build.outputs.num-runners }}" \
--project="${{ matrix.project }}" \
${{ github.event_name == 'pull_request' && '--grep-invert @mergequeue' || '' }}
${{ (github.event_name == 'pull_request' && matrix.runAllTests == false ) && '--grep-invert @mergequeue' || '' }}
- name: Upload blob report to GitHub Actions Artifacts
if: always()
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/playwright-image-updates.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
docker pull "$IMAGE"
INSPECT=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE")
DIGEST=${INSPECT#*@}
sed -i "s/const DOCKER_TAG.*/const DOCKER_TAG = \"develop@$DIGEST\";/" playwright/plugins/homeserver/synapse/index.ts
sed -i "s/const TAG.*/const TAG = \"develop@$DIGEST\";/" playwright/testcontainers/synapse.ts
env:
IMAGE: ghcr.io/element-hq/synapse:develop

Expand Down
61 changes: 19 additions & 42 deletions docs/playwright.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,21 +23,19 @@ element-web project is fine: leave it running it a different terminal as you wou
when developing. Alternatively if you followed the development set up from element-web then
Playwright will be capable of running the webserver on its own if it isn't already running.

The tests use Docker to launch Homeserver (Synapse or Dendrite) instances to test against, so you'll also
need to have Docker installed and working in order to run the Playwright tests.
The tests use [testcontainers](https://node.testcontainers.org/) to launch Homeserver (Synapse or Dendrite)
instances to test against, so you'll also need to one of the
[supported container runtimes](#supporter-container-runtimes)
installed and working in order to run the Playwright tests.

There are a few different ways to run the tests yourself. The simplest is to run:

```shell
docker pull ghcr.io/element-hq/synapse:develop
yarn run test:playwright
```

This will run the Playwright tests once, non-interactively.

Note: you don't need to run the `docker pull` command every time, but you should
do it regularly to ensure you are running against an up-to-date Synapse.

You can also run individual tests this way too, as you'd expect:

```shell
Expand All @@ -61,29 +59,25 @@ Some tests are excluded from running on certain browsers due to incompatibilitie

## How the Tests Work

Everything Playwright-related lives in the `playwright/` subdirectory of react-sdk
Everything Playwright-related lives in the `playwright/` subdirectory
as is typical for Playwright tests. Likewise, tests live in `playwright/e2e`.

`playwright/plugins/homeservers` contains Playwright plugins that starts instances
of Synapse/Dendrite in Docker containers. These servers are what Element-web runs
against in the tests.
`playwright/testcontainers` contains the testcontainers which start instances
of Synapse/Dendrite. These servers are what Element-web runs against in the tests.

Synapse can be launched with different configurations in order to test element
in different configurations. `playwright/plugins/homeserver/synapse/templates`
contains template configuration files for each different configuration.

Each test suite can then launch whatever Synapse instances it needs in whatever
configurations.
in different configurations. You can specify `synapseConfigOptions` as such:

Note that although tests should stop the Homeserver instances after running and the
plugin also stop any remaining instances after all tests have run, it is possible
to be left with some stray containers if, for example, you terminate a test such
that the `after()` does not run and also exit Playwright uncleanly. All the containers
it starts are prefixed, so they are easy to recognise. They can be removed safely.
```typescript
test.use({
synapseConfigOptions: {
// The config options to pass to the Synapse instance
},
});
```

After each test run, logs from the Synapse instances are saved in `playwright/logs/synapse`
with each instance in a separate directory named after its ID. These logs are removed
at the start of each test run.
The appropriate homeserver will be launched by the Playwright worker and reused for all tests which match the worker configuration.
The logs from testcontainers will be attached to any reports output from Playwright.

## Writing Tests

Expand Down Expand Up @@ -113,25 +107,6 @@ Homeserver instances should be reasonably cheap to start (you may see the first
while as it pulls the Docker image).
You do not need to explicitly clean up the instance as it will be cleaned up by the fixture.

### Synapse Config Templates

When a Synapse instance is started, it's given a config generated from one of the config
templates in `playwright/plugins/homeserver/synapse/templates`. There are a couple of special files
in these templates:

- `homeserver.yaml`:
Template substitution happens in this file. Template variables are:
- `REGISTRATION_SECRET`: The secret used to register users via the REST API.
- `MACAROON_SECRET_KEY`: Generated each time for security
- `FORM_SECRET`: Generated each time for security
- `PUBLIC_BASEURL`: The localhost url + port combination the synapse is accessible at
- `localhost.signing.key`: A signing key is auto-generated and saved to this file.
Config templates should not contain a signing key and instead assume that one will exist
in this file.

All other files in the template are copied recursively to `/data/`, so the file `foo.html`
in a template can be referenced in the config as `/data/foo.html`.

### Logging In

We again heavily leverage the magic of [Playwright fixtures](https://playwright.dev/docs/test-fixtures).
Expand Down Expand Up @@ -227,6 +202,8 @@ has to be disabled in Playwright on Firefox & Webkit to retain routing functiona
Anything testing VoIP/microphone will need to have `@no-webkit` as fake microphone functionality is not available
there at this time.

If you wish to run all tests in a PR, you can give it the label `X-Run-All-Tests`.

## Supporter container runtimes

We use testcontainers to spin up various instances of Synapse, Matrix Authentication Service, and more.
Expand Down
10 changes: 6 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@
"@matrix-org/spec": "^1.7.0",
"@sentry/browser": "^8.0.0",
"@types/png-chunks-extract": "^1.0.2",
"@vector-im/compound-design-tokens": "^2.0.1",
"@vector-im/compound-design-tokens": "^2.1.0",
"@vector-im/compound-web": "^7.5.0",
"@vector-im/matrix-wysiwyg": "2.38.0",
"@zxcvbn-ts/core": "^3.0.4",
Expand Down Expand Up @@ -151,7 +151,9 @@
"temporal-polyfill": "^0.2.5",
"ua-parser-js": "^1.0.2",
"uuid": "^11.0.0",
"what-input": "^5.2.10"
"what-input": "^5.2.10",
"@types/react-virtualized": "^9.21.30",
"react-virtualized": "^9.22.5"
},
"devDependencies": {
"@action-validator/cli": "^0.6.0",
Expand Down Expand Up @@ -256,12 +258,12 @@
"lint-staged": "^15.0.2",
"mailhog": "^4.16.0",
"matrix-web-i18n": "^3.2.1",
"mini-css-extract-plugin": "2.9.0",
"mini-css-extract-plugin": "2.9.2",
"minimist": "^1.2.6",
"modernizr": "^3.12.0",
"node-fetch": "^2.6.7",
"playwright-core": "^1.45.1",
"postcss": "8.4.38",
"postcss": "8.4.46",
"postcss-easings": "^4.0.0",
"postcss-hexrgba": "2.1.0",
"postcss-import": "16.1.0",
Expand Down
91 changes: 91 additions & 0 deletions playwright/e2e/crypto/backups-mas.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
/*

Check failure on line 1 in playwright/e2e/crypto/backups-mas.spec.ts

View workflow job for this annotation

GitHub Actions / Run Tests [Chrome] 6/6

[Chrome] › crypto/backups-mas.spec.ts:31:9 › Encryption state after registration › user is prompted to set up recovery

1) [Chrome] › crypto/backups-mas.spec.ts:31:9 › Encryption state after registration › user is prompted to set up recovery Test timeout of 30000ms exceeded.

Check failure on line 1 in playwright/e2e/crypto/backups-mas.spec.ts

View workflow job for this annotation

GitHub Actions / Run Tests [Chrome] 6/6

[Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere

2) [Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere Test timeout of 30000ms exceeded.
Copyright 2024 New Vector Ltd.
Copyright 2023 The Matrix.org Foundation C.I.C.
SPDX-License-Identifier: AGPL-3.0-only OR GPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE files in the repository root for full details.
*/

import { test, expect } from "../../element-web-test";
import { registerAccountMas } from "../oidc";
import { isDendrite } from "../../plugins/homeserver/dendrite";
import { TestClientServerAPI } from "../csAPI";
import { masHomeserver } from "../../plugins/homeserver/synapse/masHomeserver.ts";

// These tests register an account with MAS because then we go through the "normal" registration flow
// and crypto gets set up. Using the 'user' fixture create a a user an synthesizes an existing login,
// which is faster but leaves us without crypto set up.
test.use(masHomeserver);
test.describe("Encryption state after registration", () => {
test.skip(isDendrite, "does not yet support MAS");

test("Key backup is enabled by default", async ({ page, mailhogClient, app }) => {
await page.goto("/#/login");
await page.getByRole("button", { name: "Continue" }).click();
await registerAccountMas(page, mailhogClient, "alice", "[email protected]", "Pa$sW0rD!");

await app.settings.openUserSettings("Security & Privacy");
await expect(page.getByText("This session is backing up your keys.")).toBeVisible();
});

test("user is prompted to set up recovery", async ({ page, mailhogClient, app }) => {
await page.goto("/#/login");
await page.getByRole("button", { name: "Continue" }).click();
await registerAccountMas(page, mailhogClient, "alice", "[email protected]", "Pa$sW0rD!");

await page.getByRole("button", { name: "Add room" }).click();
await page.getByRole("menuitem", { name: "New room" }).click();
await page.getByRole("textbox", { name: "Name" }).fill("test room");
await page.getByRole("button", { name: "Create room" }).click();

await expect(page.getByRole("heading", { name: "Set up recovery" })).toBeVisible();
});
});

test.describe("Key backup reset from elsewhere", () => {
test.skip(isDendrite, "does not yet support MAS");

test("Key backup is disabled when reset from elsewhere", async ({ page, mailhogClient, request, homeserver }) => {
const testUsername = "alice";
const testPassword = "Pa$sW0rD!";

// there's a delay before keys are uploaded so the error doesn't appear immediately: use a fake
// clock so we can skip the delay
await page.clock.install();

await page.goto("/#/login");
await page.getByRole("button", { name: "Continue" }).click();
await registerAccountMas(page, mailhogClient, testUsername, "[email protected]", testPassword);

await page.getByRole("button", { name: "Add room" }).click();
await page.getByRole("menuitem", { name: "New room" }).click();
await page.getByRole("textbox", { name: "Name" }).fill("test room");
await page.getByRole("button", { name: "Create room" }).click();

// @ts-ignore - this runs in the browser scope where mxMatrixClientPeg is a thing. Here, it is not.
const accessToken = await page.evaluate(() => mxMatrixClientPeg.get().getAccessToken());

const csAPI = new TestClientServerAPI(request, homeserver, accessToken);

const backupInfo = await csAPI.getCurrentBackupInfo();

await csAPI.deleteBackupVersion(backupInfo.version);

await page.getByRole("textbox", { name: "Send an encrypted message…" }).fill("/discardsession");
await page.getByRole("button", { name: "Send message" }).click();

await page.getByRole("textbox", { name: "Send an encrypted message…" }).fill("Message with broken key backup");
await page.getByRole("button", { name: "Send message" }).click();

// Should be the message we sent plus the room creation event
await expect(page.locator(".mx_EventTile")).toHaveCount(2);
await expect(
page.locator(".mx_RoomView_MessageList > .mx_EventTile_last .mx_EventTile_receiptSent"),
).toBeVisible();

// Wait for it to try uploading the key
await page.clock.fastForward(20000);

await expect(page.getByRole("heading", { level: 1, name: "New Recovery Method" })).toBeVisible();
});
});
83 changes: 0 additions & 83 deletions playwright/e2e/crypto/backups.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,6 @@ Please see LICENSE files in the repository root for full details.
import { type Page } from "@playwright/test";

import { test, expect } from "../../element-web-test";
import { registerAccountMas } from "../oidc";
import { isDendrite } from "../../plugins/homeserver/dendrite";
import { TestClientServerAPI } from "../csAPI";
import { masHomeserver } from "../../plugins/homeserver/synapse/masHomeserver.ts";

async function expectBackupVersionToBe(page: Page, version: string) {
await expect(page.locator(".mx_SecureBackupPanel_statusList tr:nth-child(5) td")).toHaveText(
Expand All @@ -22,85 +18,6 @@ async function expectBackupVersionToBe(page: Page, version: string) {
await expect(page.locator(".mx_SecureBackupPanel_statusList tr:nth-child(6) td")).toHaveText(version);
}

// These tests register an account with MAS because then we go through the "normal" registration flow
// and crypto gets set up. Using the 'user' fixture create a a user an synthesizes an existing login,
// which is faster but leaves us without crypto set up.
test.describe("Encryption state after registration", () => {
test.use(masHomeserver);
test.skip(isDendrite, "does not yet support MAS");

test("Key backup is enabled by default", async ({ page, mailhogClient, app }) => {
await page.goto("/#/login");
await page.getByRole("button", { name: "Continue" }).click();
await registerAccountMas(page, mailhogClient, "alice", "[email protected]", "Pa$sW0rD!");

await app.settings.openUserSettings("Security & Privacy");
await expect(page.getByText("This session is backing up your keys.")).toBeVisible();
});

test("user is prompted to set up recovery", async ({ page, mailhogClient, app }) => {
await page.goto("/#/login");
await page.getByRole("button", { name: "Continue" }).click();
await registerAccountMas(page, mailhogClient, "alice", "[email protected]", "Pa$sW0rD!");

await page.getByRole("button", { name: "Add room" }).click();
await page.getByRole("menuitem", { name: "New room" }).click();
await page.getByRole("textbox", { name: "Name" }).fill("test room");
await page.getByRole("button", { name: "Create room" }).click();

await expect(page.getByRole("heading", { name: "Set up recovery" })).toBeVisible();
});
});

test.describe("Key backup reset from elsewhere", () => {
test.use(masHomeserver);
test.skip(isDendrite, "does not yet support MAS");

test("Key backup is disabled when reset from elsewhere", async ({ page, mailhogClient, request, homeserver }) => {
const testUsername = "alice";
const testPassword = "Pa$sW0rD!";

// there's a delay before keys are uploaded so the error doesn't appear immediately: use a fake
// clock so we can skip the delay
await page.clock.install();

await page.goto("/#/login");
await page.getByRole("button", { name: "Continue" }).click();
await registerAccountMas(page, mailhogClient, testUsername, "[email protected]", testPassword);

await page.getByRole("button", { name: "Add room" }).click();
await page.getByRole("menuitem", { name: "New room" }).click();
await page.getByRole("textbox", { name: "Name" }).fill("test room");
await page.getByRole("button", { name: "Create room" }).click();

// @ts-ignore - this runs in the browser scope where mxMatrixClientPeg is a thing. Here, it is not.
const accessToken = await page.evaluate(() => mxMatrixClientPeg.get().getAccessToken());

const csAPI = new TestClientServerAPI(request, homeserver, accessToken);

const backupInfo = await csAPI.getCurrentBackupInfo();

await csAPI.deleteBackupVersion(backupInfo.version);

await page.getByRole("textbox", { name: "Send an encrypted message…" }).fill("/discardsession");
await page.getByRole("button", { name: "Send message" }).click();

await page.getByRole("textbox", { name: "Send an encrypted message…" }).fill("Message with broken key backup");
await page.getByRole("button", { name: "Send message" }).click();

// Should be the message we sent plus the room creation event
await expect(page.locator(".mx_EventTile")).toHaveCount(2);
await expect(
page.locator(".mx_RoomView_MessageList > .mx_EventTile_last .mx_EventTile_receiptSent"),
).toBeVisible();

// Wait for it to try uploading the key
await page.clock.fastForward(20000);

await expect(page.getByRole("heading", { level: 1, name: "New Recovery Method" })).toBeVisible();
});
});

test.describe("Backups", () => {
test.use({
displayName: "Hanako",
Expand Down
Loading

0 comments on commit b7b7a98

Please sign in to comment.