Safari failing infrastructure tests #12621

gsnedders · 2018-08-22T14:08:06Z

With Safari TP 63:

/infrastructure/assumptions/html-elements.html
  FAIL Compare CSS div definitions (only valid if pre-reqs pass) - assert_equals: Different value for -webkit-mask-box-image-slice expected "0" but got "0 fill"
http://web-platform.test:8000/infrastructure/assumptions/html-elements.html:71:20
step@http://web-platform.test:8000/resources/testharness.js:1553:30
test@http://web-platform.test:8000/resources/testharness.js:544:22
global code@http://web-platform.test:8000/infrastructure/assumptions/html-elements.html:58:5
  FAIL Compare CSS span definitions (only valid if pre-reqs pass) - assert_equals: Different value for -webkit-mask-box-image-slice expected "0" but got "0 fill"
http://web-platform.test:8000/infrastructure/assumptions/html-elements.html:88:18
step@http://web-platform.test:8000/resources/testharness.js:1553:30
test@http://web-platform.test:8000/resources/testharness.js:544:22
global code@http://web-platform.test:8000/infrastructure/assumptions/html-elements.html:76:5
/infrastructure/server/context.any.sharedworker.html
  FAIL context - ReferenceError: Can't find variable: SharedWorker
global code@http://web-platform.test:8000/infrastructure/server/context.any.sharedworker.html:8:41
/infrastructure/server/context.any.serviceworker.html
  FAIL context - assert_true: expected true got undefined
https://web-platform.test:8443/infrastructure/server/context.any.js:5:18
step@https://web-platform.test:8443/resources/testharness.js:1553:30
test@https://web-platform.test:8443/resources/testharness.js:544:22
global code@https://web-platform.test:8443/infrastructure/server/context.any.js:2:5
importScripts@[native code]
global code@https://web-platform.test:8443/infrastructure/server/context.any.worker.js:8:14
/infrastructure/server/order-of-metas.any.sharedworker.html
  FAIL foo - ReferenceError: Can't find variable: SharedWorker
global code@http://web-platform.test:8000/infrastructure/server/order-of-metas.any.sharedworker.html:9:41
/infrastructure/server/secure-context.https.any.serviceworker.html
  FAIL Use of .https file name flag implies secure context - assert_true: expected true got undefined
https://web-platform.test:8443/infrastructure/server/secure-context.https.any.js:3:14
step@https://web-platform.test:8443/resources/testharness.js:1553:30
test@https://web-platform.test:8443/resources/testharness.js:544:22
global code@https://web-platform.test:8443/infrastructure/server/secure-context.https.any.js:2:5
importScripts@[native code]
global code@https://web-platform.test:8443/infrastructure/server/secure-context.https.any.worker.js:8:14
/infrastructure/server/secure-context.https.any.sharedworker.html
  FAIL secure-context - ReferenceError: Can't find variable: SharedWorker
global code@https://web-platform.test:8443/infrastructure/server/secure-context.https.any.sharedworker.html:8:41
/infrastructure/server/title.any.sharedworker.html
  FAIL foobar - ReferenceError: Can't find variable: SharedWorker
global code@http://web-platform.test:8000/infrastructure/server/title.any.sharedworker.html:8:41
/infrastructure/testdriver/bless.html
  FAIL user activation - promise_test: Unhandled rejection with value: object "NotSupportedError: The operation is not supported."
promiseReactionJob@[native code]

/infrastructure/assumptions/html-elements.html — https://bugs.webkit.org/show_bug.cgi?id=187052
/infrastructure/server/context.any.sharedworker.html, /infrastructure/server/order-of-metas.any.sharedworker.html, /infrastructure/server/secure-context.https.any.sharedworker.html, /infrastructure/server/title.any.sharedworker.html — SharedWorker unsupported (https://bugs.webkit.org/show_bug.cgi?id=149850)
/infrastructure/server/context.any.serviceworker.html, /infrastructure/server/secure-context.https.any.serviceworker.html — https://bugs.webkit.org/show_bug.cgi?id=188842
/infrastructure/testdriver/bless.html — Minimize <video> dependency in test_driver.bless test #13403

The text was updated successfully, but these errors were encountered:

gsnedders · 2018-08-24T15:47:50Z

Do we want to workaround these bugs, because they aren't per-se Safari running tests incorrectly? Except maybe the bless.html one?

foolip · 2018-10-05T22:23:24Z

@cdumez @youennf could either of you help triage https://bugs.webkit.org/show_bug.cgi?id=187052?

youennf · 2018-10-05T22:44:17Z

I cced some folks on bugs.webkit.org.
What is the impact of failing these infra tests?

foolip · 2018-10-05T22:49:18Z

@youennf it's not a big problem, but in order to run infrastructure/ tests in CI (foolip#5) to ensure that Safari doesn't regress, we'll need to check in expectations about every failure in infrastructure/. That in turn is in service of #13299, i.e., preventing regressions of non-infrastructure tests in Safari before they happen.

Understanding why the tests are failing would be great even if it's not fixed soon.

foolip · 2018-10-05T23:33:46Z

For bless.html, this is the failing test:

promise_test(() => {
  const video = document.createElement('video');
  video.setAttribute('src', '/media/counting.ogv');
  document.body.appendChild(video);
  return test_driver.bless('start video playback', () => video.play())
}, 'user activation');

And the failure is Unhandled rejection with value: object "NotSupportedError: The operation is not supported." That's because of /media/counting.ogv, Safari doesn't support Ogg Theora, so #9354.

Switching to /media/counting.mp4 seems to fix the problem.

However, with the same fix, I've also seen flaky timeout or failure due to Unhandled rejection with value: object "AbortError: The operation was aborted." without any extra changes.

Running the tests flips back and forth a lot between the two tabs, which is #10398. My best guess is that media playback is being paused or prevented to start in background tabs.

Anyway, this test doesn't need a media resource as well. Browsers that don't allow media playback without a user gesture will leave video.paused as true, so I'll try to change the test.

See diagnosis of Safari Technology Preview failure: #12621 (comment) This test does not need to load any media resource or consider the state of the returned promise.

foolip · 2018-10-05T23:53:25Z

I've sent #13403 to fix bless.html.

See diagnosis of Safari Technology Preview failure: #12621 (comment) This test does not need to load any media resource or consider the state of the returned promise.

gsnedders · 2018-10-08T10:56:40Z

@youennf probably nothing; https://bugs.webkit.org/show_bug.cgi?id=187052 is the most annoying given it makes it hard to verify behaviour is correct.

…ver.bless` test, a=testonly Automatic update from web-platform-testsMinimize <video> dependency in `test_driver.bless` test (#13403) See diagnosis of Safari Technology Preview failure: web-platform-tests/wpt#12621 (comment) This test does not need to load any media resource or consider the state of the returned promise. -- wpt-commits: 21c33fe4cc3963faa4b38ab19e47660fa72c122f wpt-pr: 13403

foolip · 2018-10-20T17:52:46Z

The test infrastructure/assumptions/allowed-to-play.html test I added in #13622 is also failing in Safari Technology Preview 67.

foolip · 2018-10-20T17:56:45Z

I've filed https://bugs.webkit.org/show_bug.cgi?id=190775 for that.

foolip · 2018-12-25T10:25:24Z

Downgrading the priority, this isn't causing any widespread trouble it seems.

…ver.bless` test, a=testonly Automatic update from web-platform-testsMinimize <video> dependency in `test_driver.bless` test (#13403) See diagnosis of Safari Technology Preview failure: web-platform-tests/wpt#12621 (comment) This test does not need to load any media resource or consider the state of the returned promise. -- wpt-commits: 21c33fe4cc3963faa4b38ab19e47660fa72c122f wpt-pr: 13403 UltraBlame original commit: 113102643432c772f228e02246a2f7d606abe725

gsnedders · 2021-03-16T22:18:12Z

So current issues:

infrastructure/assumptions/ahem.html since we stopped installing Ahem on macOS
infrastructure/assumptions/allowed-to-play.html due to the inability to automatically change the default autoplay setting (https://bugs.webkit.org/show_bug.cgi?id=190775)
All print reftests due to lack of the WebDriver print command (https://bugs.webkit.org/show_bug.cgi?id=217173)
All sharedworker tests due to its continued disablement (https://bugs.webkit.org/show_bug.cgi?id=218944)
infrastructure/server/http2-websocket.sub.h2.any.js due to lack of support for WebSockets over HTTP/2 (rdar://19558310)
Various infrastructure/testdriver/actions failures (rdar://75494624)

With that being the status now, I don't know if it makes sense to keep the meta issue open?

gsnedders added infra priority:roadmap labels Aug 22, 2018

foolip mentioned this issue Oct 5, 2018

Testdriver: tabs switch frequently when invokes multiple actions #10398

Closed

foolip added a commit that referenced this issue Oct 5, 2018

Minimize <video> dependency in test_driver.bless test

2d494a5

See diagnosis of Safari Technology Preview failure: #12621 (comment) This test does not need to load any media resource or consider the state of the returned promise.

foolip mentioned this issue Oct 5, 2018

Minimize <video> dependency in test_driver.bless test #13403

Merged

foolip added priority:backlog and removed priority:roadmap labels Dec 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Safari failing infrastructure tests #12621

Safari failing infrastructure tests #12621

gsnedders commented Aug 22, 2018 •

edited

Loading

gsnedders commented Aug 24, 2018

foolip commented Oct 5, 2018

youennf commented Oct 5, 2018

foolip commented Oct 5, 2018

foolip commented Oct 5, 2018

foolip commented Oct 5, 2018

gsnedders commented Oct 8, 2018

foolip commented Oct 20, 2018

foolip commented Oct 20, 2018

foolip commented Dec 25, 2018

gsnedders commented Mar 16, 2021 •

edited

Loading

Safari failing infrastructure tests #12621

Safari failing infrastructure tests #12621

Comments

gsnedders commented Aug 22, 2018 • edited Loading

gsnedders commented Aug 24, 2018

foolip commented Oct 5, 2018

youennf commented Oct 5, 2018

foolip commented Oct 5, 2018

foolip commented Oct 5, 2018

foolip commented Oct 5, 2018

gsnedders commented Oct 8, 2018

foolip commented Oct 20, 2018

foolip commented Oct 20, 2018

foolip commented Dec 25, 2018

gsnedders commented Mar 16, 2021 • edited Loading

gsnedders commented Aug 22, 2018 •

edited

Loading

gsnedders commented Mar 16, 2021 •

edited

Loading