Tests: Disable randomly failing e2e tests

[gziolo]

Description

I got several pings about failing e2e tests on Travis in the past weeks. We don't know why some tests fail randomly so I went through Travis reports for PRs merged to master branch in last week and through newly opened PRs to identify them. This PR disables 3 tests that were listed the most often.

Disabled tests

https://travis-ci.com/WordPress/gutenberg/jobs/195867138#L1098
https://travis-ci.com/WordPress/gutenberg/jobs/195696939#L1097
https://travis-ci.com/WordPress/gutenberg/jobs/195391880#L1097
https://travis-ci.com/WordPress/gutenberg/jobs/195627327#L1101

FAIL packages/e2e-tests/specs/taxonomies.test.js (5.824s)
  ● Taxonomies › should be able to open the tags panel and create a new tag if the user has the right capabilities
    expect(received).toHaveLength(expected)
    Expected length: 1
    Received length: 0
    Received array:  []
      153 | 
      154 | 		// The post should only contain the tag we added.
    > 155 | 		expect( tags ).toHaveLength( 1 );
          | 		               ^
      156 | 		expect( tags[ 0 ] ).toEqual( 'tag1' );
      157 | 
      158 | 		// Type something in the title so we can publish the post.
      at Object.toHaveLength (specs/taxonomies.test.js:155:18)
      at tryCatch (../../node_modules/regenerator-runtime/runtime.js:62:40)
      at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:288:22)
      at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:114:21)
      at asyncGeneratorStep (specs/taxonomies.test.js:13:103)
      at _next (specs/taxonomies.test.js:15:194)

https://travis-ci.com/WordPress/gutenberg/jobs/195887677#L1101
https://travis-ci.com/WordPress/gutenberg/jobs/195790721#L1100
https://travis-ci.com/WordPress/gutenberg/jobs/195293180#L1099
https://travis-ci.com/WordPress/gutenberg/jobs/195237225#L1102

FAIL packages/e2e-tests/specs/plugins/wp-editor-meta-box.test.js (6.002s)
  ● WP Editor Meta Boxes › Should save the changes
    expect(received).toMatchSnapshot()
    Snapshot name: `WP Editor Meta Boxes Should save the changes 1`
    - Snapshot
    + Received
    - "<p>Typing in a metabox</p>"
    + ""
      38 | 		);
      39 | 
    > 40 | 		expect( content ).toMatchSnapshot();
         | 		                  ^
      41 | 	} );
      42 | } );
      43 | 
      at Object.toMatchSnapshot (specs/plugins/wp-editor-meta-box.test.js:40:21)
      at tryCatch (../../node_modules/regenerator-runtime/runtime.js:62:40)
      at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:288:22)
      at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:114:21)
      at asyncGeneratorStep (specs/plugins/wp-editor-meta-box.test.js:9:103)
      at _next (specs/plugins/wp-editor-meta-box.test.js:11:194)
 › 1 snapshot failed.

https://travis-ci.com/WordPress/gutenberg/jobs/195586217#L1080

FAIL packages/e2e-tests/specs/links.test.js (50.467s)
  ● Links › allows autocomplete suggestions to be navigated with the keyboard
    expect(received).toBe(expected) // Object.is equality
    Expected: "true"
    Received: "false"
      336 | 		await page.keyboard.press( 'ArrowDown' );
      337 | 		const isSelected = await page.evaluate( () => document.querySelector( '.block-editor-url-input__suggestion' ).getAttribute( 'aria-selected' ) );
    > 338 | 		expect( isSelected ).toBe( 'true' );
          | 		                     ^
      339 | 
      340 | 		// Expect the link to apply correctly when pressing Enter.
      341 | 		// Note - have avoided using snapshots here since the link url can't be determined ahead of time.
      at Object.toBe (specs/links.test.js:338:24)
      at tryCatch (../../node_modules/regenerator-runtime/runtime.js:62:40)
      at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:288:22)
      at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:114:21)
      at asyncGeneratorStep (specs/links.test.js:15:103)
      at _next (specs/links.test.js:17:194)

[aduth]

For future debugging information:

• The first of these sounds quite similar to issues we'd had recently with the need for this permissions information to be fetched from the REST API: Reusable Blocks: Preload user permissions #15061 . It may similarly benefit from either preloading this data, or ensuring that the network request is allowed to finish.
• The second sounds familiar to issues I'd seen in the past where sometimes TinyMCE takes some time to initialize, so "click" in the test to activate the field might be delayed. It could possibly be improved by ensuring that whatever DOM changes might occur to indicate that the initialization is complete are accounted for in e.g. page.waitForSelector.

[gziolo]

Travis failed on the node with JS unit test which seems to be surprising:

The command "npm run check-local-changes" exited with 1.

Both npm i and npm run docs:build don't produce changes locally ...

[aduth]

Travis failed on the node with JS unit test which seems to be surprising:

The command "npm run check-local-changes" exited with 1.

Both npm i and npm run docs:build don't produce changes locally ...

🤔 From the changes, it clearly shouldn't be needed.

I have vague recollections of @nosolosw experiencing something similar recently and tracing it back to Travis cache wrongly affecting other builds ? I may be mis-remembering.

[gziolo]

I resolved this issue by rebasing with master

[oandregal]

The command "npm run check-local-changes" exited with 1. Both npm i and npm run docs:build don't produce changes locally ...

I couldn't find the PR where this happened to investigate, but it smells to the same problem I discovered a few weeks ago: Travis is executed upon the merge between the PR and master. This is problematic and deeply saddenning. It means that you can't reliable test your PR locally as it is, and builds are not reproducible (depend on the state of master at the time of build).

Two tips to solve this:

Rebase your PR from master

When you do this, errors will become evident or will be solved by the rebase mechanism.

Instead of testing on your local branch, you have to test on the merge commit

This is more involved and not obvious at all. Steps are:

1. Go to the Travis job that failed and check the commit it used to run the build. Job example.

2. If it isn't the latest PR commit, it is a merge commit. To reproduce and test the same code Travis executed, you have to do the merge locally. Click the link to the commit to identify which one was master at the time of build (first is the PR and last is master at the time of build). Merge commit example.

3. Pull master locally to make sure you have that commit available. Then:

git checkout -b <master-at-the-time-of-build> <master-at-the-time-of-build>
git merge <latest-commit-in-PR>
# you have now the same code than Travis

[gziolo]

There is another test to disable:
#15241 (comment)

Edit: I have found an easy fix with #15247.