Fix e2e test flakiness #9249

fungairino · 2024-10-07T19:31:35Z

What does this PR do?

Closes Fix flaky Playwright Tests #9193
I verified flakiness was fixed by temporarily forcing each test to run twice (which I then removed in the last commit)
Including more documentation on how debug flakiness
Add a new option for opening the e2e report for any specific run
Re-enable e2e test that we had been skipping before, working around a new(ish) msedge bug. https://pixiebrix.slack.com/archives/C0436P48QHY/p1728414907563279

Discussion

Follow up with @johnnymetz on latency identified in this PR

Future Work

Tighten up global timeout when we observe backend performance in a better state.

Checklist

Added jest or playwright tests and/or storybook stories

For more information on our expectations for the PR process, see the
code review principles doc

github-actions · 2024-10-07T20:23:16Z

Playwright test results

138 passed
2 flaky
2 skipped

Details

Open report ↗︎
142 tests across 46 suites
13 minutes, 2 seconds
7c6331e
For more information on how to debug and view this report, see our readme

Flaky tests

msedge-setup › setup/unaffiliated.setup.ts › authenticate with unaffiliated user
msedge › tests/pageEditor/addStarterBrick.spec.ts › Add new mod with different starter brick components

Skipped tests

chrome › tests/runtime/googleSheetsIntegration.spec.ts › can activate a google spreadsheet mod with config options
msedge › tests/runtime/googleSheetsIntegration.spec.ts › can activate a google spreadsheet mod with config options

codecov · 2024-10-07T21:34:18Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.09%. Comparing base (8318d74) to head (7c6331e).
Report is 360 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    pixiebrix/pixiebrix-extension#9249      +/-   ##
==========================================
+ Coverage   74.24%   75.09%   +0.84%     
==========================================
  Files        1332     1369      +37     
  Lines       40817    42257    +1440     
  Branches     7634     7882     +248     
==========================================
+ Hits        30306    31733    +1427     
- Misses      10511    10524      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ight-tests

fungairino · 2024-10-09T21:07:22Z

end-to-end-tests/pageObjects/extensionConsole/modsPage.ts

@@ -74,7 +81,7 @@ export class ModsPage extends BasePageObject {
    const contentLoadedLocator = this.getByText("Welcome to PixieBrix!").or(
      this.modTableItems.nth(0),
    );
-    await expect(contentLoadedLocator).toBeVisible({ timeout: 15_000 });
+    await expect(contentLoadedLocator).toBeVisible();


A lot of these changes were to remove the timeout since we prefer using the global timeout unless we need to specify a longer time to wait.

fungairino · 2024-10-09T21:08:09Z

end-to-end-tests/fixtures/authentication.ts

@@ -85,7 +85,7 @@ export const test = mergeTests(
      );

      // The admin console automatically opens a new tab to log in and link the newly installed extension to the user's account.
-      const page = await context.waitForEvent("page", { timeout: 10_000 });
+      const page = await context.waitForEvent("page", { timeout: 20_000 });


A lot of these timeout bumps are just to align with the new default global assertion timeout of 20s, but in the future we could go through and see which ones we can tweak.

I imagine you already looked into a global setting for these? Alternatively, let's introduce a global constant so that we don't have to continually update all of these

mnholtz · 2024-10-10T17:12:00Z

.github/workflows/ci.yml

+  #      - .editorconfig
+  #      - LICENSE
+  #      - "**.iml"
+  #      - .idea/**


This this come up in this PR? Or somewhere else?

Yea, it came up at the start of this pr when I just made a simple readme change.

mnholtz · 2024-10-10T17:12:48Z

end-to-end-tests/README.md

+You can also pass in the run id instead if you want to view a specific run's report.
+
+```bash
+./scripts/show-pr-e2e-report.sh -r <run-id>


mnholtz · 2024-10-10T17:16:13Z

end-to-end-tests/pageObjects/pageEditor/pageEditorPage.ts

    const saveNewModModal = this.page.locator(".modal-content");
-    await expect(saveNewModModal).toBeVisible();
-    await expect(saveNewModModal.getByText("Save new mod")).toBeVisible();
+    // The save button re-mounts several times so we need to retry clicking the saveButton until the modal is visible


I've actually run into this as a user before (i.e. it's not just a testing papercut); do we have a ticket for this issue? Let's open one before merging

~~I re-opened https://github.com/pixiebrix/pixiebrix-extension/issues/8458~~

Actually this is a separate issue. I can open one.

mnholtz · 2024-10-10T17:21:58Z

end-to-end-tests/pageObjects/extensionConsole/modsPage.ts

+    //     predicate: (request) =>
+    //       request.url().includes(API_PATHS.REGISTRY_BRICKS),
+    //     timeout: 30_000,
+    //   });


Did you mean to keep these commented out?

In the latest change I wanted to see if I could remove this. Seems like currently we still need them for a few other tests.

mnholtz · 2024-10-10T17:30:21Z

playwright.config.ts

-    trace: CI ? "on-first-retry" : "retain-on-failure",
+    // trace: CI ? "on-first-retry" : "retain-on-failure",
+    // temporarily always collect trace on failure to debug flaky tests
+    trace: "retain-on-failure",


Just to clarify, will we be retaining this setting on main for now? Or was this just to facilitate your debugging?

No, it's just for this PR. I'll revert back to the previous setting once I have things in a good state before merging

…ight-tests

github-actions · 2024-10-10T20:00:14Z

No loom links were found in the first post. Please add one there if you'd like to it to appear on Slack.

Do not edit this comment manually.

…ight-tests

update readme

28cab22

fungairino linked an issue Oct 7, 2024 that may be closed by this pull request

Fix flaky Playwright Tests #9193

Closed

3 tasks

vercel bot deployed to Preview October 7, 2024 19:32 View deployment

twschiller added testing developer experience labels Oct 7, 2024

update CI.yml

b3efa35

vercel bot deployed to Preview October 7, 2024 20:00 View deployment

collect traces on error always

023b848

vercel bot deployed to Preview October 7, 2024 20:04 View deployment

twschiller assigned fungairino Oct 7, 2024

Try bumping global assertion timeout to 20s

e8a66ba

vercel bot deployed to Preview October 7, 2024 21:28 View deployment

Add a visibility check for mod action and retry if failed

4795c1e

vercel bot deployed to Preview October 7, 2024 21:36 View deployment

fungairino added 2 commits October 8, 2024 15:15

re-enabling sidebar close e2e spec

89a2723

Merge remote-tracking branch 'origin/main' into 9193-fix-flaky-playwr…

010d281

…ight-tests

vercel bot deployed to Preview October 8, 2024 19:19 View deployment

fixing flaky liveEditing test

279f03a

vercel bot deployed to Preview October 8, 2024 20:50 View deployment

wip

c3a0773

vercel bot deployed to Preview October 8, 2024 21:37 View deployment

Merge remote-tracking branch 'origin/main' into 9193-fix-flaky-playwr…

a0ddc75

…ight-tests

vercel bot deployed to Preview October 9, 2024 20:28 View deployment

adjust timeouts to use global val

d71b747

vercel bot deployed to Preview October 9, 2024 21:00 View deployment

fungairino commented Oct 9, 2024

View reviewed changes

config back to normal

8d8b12b

vercel bot deployed to Preview October 10, 2024 15:51 View deployment

fix assignment

3555742

vercel bot deployed to Preview October 10, 2024 16:08 View deployment

mnholtz reviewed Oct 10, 2024

View reviewed changes

pr feedback

18f7009

vercel bot deployed to Preview October 10, 2024 17:39 View deployment

fungairino added 2 commits October 10, 2024 14:13

config back to normal

5ef88f9

Merge remote-tracking branch 'origin/main' into 9193-fix-flaky-playwr…

4aa1d16

…ight-tests

vercel bot deployed to Preview October 10, 2024 18:17 View deployment

fungairino requested a review from mnholtz October 10, 2024 18:17

fix welcome starter bricks flakiness

05f4df2

vercel bot deployed to Preview October 10, 2024 18:38 View deployment

more flakiness fixes

b48bb91

vercel bot deployed to Preview October 10, 2024 18:46 View deployment

fix setup flakiness

67b4143

vercel bot deployed to Preview October 10, 2024 19:11 View deployment

try to fix setup flakiness

0d481c1

vercel bot deployed to Preview October 10, 2024 19:57 View deployment

mnholtz approved these changes Oct 10, 2024

View reviewed changes

fungairino added 2 commits October 10, 2024 16:12

sidebar open flakiness

19d00ef

Merge remote-tracking branch 'origin/main' into 9193-fix-flaky-playwr…

7c6331e

…ight-tests

vercel bot deployed to Preview October 10, 2024 20:15 View deployment

fungairino merged commit bf34e41 into main Oct 10, 2024
20 checks passed

fungairino deleted the 9193-fix-flaky-playwright-tests branch October 10, 2024 20:36

twschiller added this to the 2.1.5 milestone Oct 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix e2e test flakiness #9249

Fix e2e test flakiness #9249

fungairino commented Oct 7, 2024 •

edited

Loading

github-actions bot commented Oct 7, 2024 •

edited

Loading

codecov bot commented Oct 7, 2024 •

edited

Loading

fungairino Oct 9, 2024

fungairino Oct 9, 2024

mnholtz Oct 10, 2024 •

edited

Loading

mnholtz Oct 10, 2024

fungairino Oct 10, 2024

mnholtz Oct 10, 2024

mnholtz Oct 10, 2024 •

edited

Loading

fungairino Oct 10, 2024 •

edited

Loading

mnholtz Oct 10, 2024

fungairino Oct 10, 2024

mnholtz Oct 10, 2024

fungairino Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024

Fix e2e test flakiness #9249

Fix e2e test flakiness #9249

Conversation

fungairino commented Oct 7, 2024 • edited Loading

What does this PR do?

Discussion

Future Work

Checklist

github-actions bot commented Oct 7, 2024 • edited Loading

Playwright test results

Details

codecov bot commented Oct 7, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mnholtz Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mnholtz Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

fungairino Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fungairino Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Oct 10, 2024

fungairino commented Oct 7, 2024 •

edited

Loading

github-actions bot commented Oct 7, 2024 •

edited

Loading

codecov bot commented Oct 7, 2024 •

edited

Loading

mnholtz Oct 10, 2024 •

edited

Loading

mnholtz Oct 10, 2024 •

edited

Loading

fungairino Oct 10, 2024 •

edited

Loading

fungairino Oct 10, 2024 •

edited

Loading