Tests with non-OK/PASS/FAIL results #48
Comments
The load event never fires, so the harness behaves like it doesn't know if there are more tests to be registered. See https://bugs.webkit.org/show_bug.cgi?id=235407. |
This is the fact wptrunner doesn't support the top-level test page being navigated during the test, and html/browsers/browsing-the-web/navigating-across-documents/replace-before-load/resources/helpers.js is calling
https://bugs.webkit.org/show_bug.cgi?id=235947, though clearly the tests could be fixed to not timeout. |
|
This investigation is done and there's a list of tests with non-OK harness statuses in the metrics code. If any new tests are found, metrics generation fails. |
In the Jan 13 meeting I took an action item to look into tests that fail in some other way than the usual, which usually means ERROR or TIMEOUT. When querying on wpt.fyi, it also includes
status:missing, and I also agreed to look into tests that have different numbers of subtests in different browsers.I've looked through the 36 tests found by the
!(status:ok or status:pass or status:fail) and (label:interop-2021-aspect-ratio or label:interop-2021-flexbox or label:interop-2021-grid or label:interop-2021-position-sticky or label:interop-2021-transforms or label:interop-2022-cascade or label:interop-2022-color or label:interop-2022-contain or label:interop-2022-dialog or label:interop-2022-forms or label:interop-2022-scrolling or label:interop-2022-subgrid or label:interop-2022-text or label:interop-2022-viewport or label:interop-2022-webcompat)query.<dialog>and::backdrop/html/semantics/interactive-elements/the-dialog-element/backdrop-receives-element-events.html is TIMEOUT in Safari with the single subtest being NOTRUN. It's linked to https://webkit.org/b/233072 which was recently fixed, confirmed with Safari + WebKitGTK.
Flexbox
/css/css-flexbox/break-nested-float-in-flex-item-001-print.html and -002-print.html aren't run in Safari. Already discussed in #2 (comment) and we should drop these tests.
Form Controls
/dom/events/Event-dispatch-click.html is TIMEOUT in Chrome, but that's the expected failure mode when the event doesn't fire.
The 6 form-submit* tests in /html/browsers/browsing-the-web/navigating-across-documents/replace-before-load/ are ERROR in Safari. There's a
NoSuchFrameExceptionraised in Python so probably there's some WebDriver issue underneath this.The 2 form-requestsubmit-during-* tests in the same directory are TIMEOUT in Safari. Needs investigation.
5 form-validation-* tests in /html/semantics/forms/constraints/ have MISSING tests because two different tests are created (and fail) in Safari than in Chrome and Firefox. These tests are about week/month and like web-platform-tests/wpt-metadata#2401 should be excluded, but the tests need to be fixed to run the same subtests in all browsers first to see if there are other failing tests that aren't about week/month.
/html/semantics/forms/constraints/infinite_backtracking.html is TIMEOUT in Firefox, presumably because it has the bug the test is probing for.
/html/semantics/forms/form-submission-0/form-double-submit-multiple-targets.html is TIMEOUT in Safari.
/html/semantics/forms/form-submission-0/form-double-submit-to-different-origin-frame.html is TIMEOUT in Firefox and Safari, but also fails in Chrome, tracked by https://crbug.com/1087077. I would guess one of the events never fire in Firefox/Safari and that the timeout is legit.
The rel-* tests in /html/semantics/forms/form-submission-target/ are TIMEOUT in Safari, but all of the subtests pass. This very unusual and needs investigation.
/html/semantics/forms/textfieldselection/select-event.html is TIMEOUT in Safari after the first subtest times out. Looks like the "select" event never fires and the test is OK.
/html/semantics/forms/textfieldselection/selection-not-application.html has fewer subtests in Safari, the week/month ones aren't created. This actually seems fine from a metrics point of view if we score each browser individually.
/html/semantics/forms/textfieldselection/selection-start-end.html is an overall TIMEOUT in Safari, with a bunch of NOTRUN subtests. What's odd is that no subtest has timed out. Needs investigation.
/html/semantics/forms/textfieldselection/textfieldselection-setRangeText.html is an overall TIMEOUT in Safari after one of the subtests is TIMEOUT due to an event not being fired. The test seems OK, but it's not obvious how to score this.
/html/semantics/forms/textfieldselection/textfieldselection-setSelectionRange.html is TIMEOUT in Safari because one subtest is TIMEOUT. Also tricky for scoring.
/html/semantics/forms/the-input-element/range-restore-oninput-onchange-event.html is TIMEOUT in Safari, both harness and subtests. An event isn't fired, test seems OK.
Grid
/css/css-grid/grid-definition/grid-limits-001.html is TIMEOUT in Safari. I've tried running it locally and it just shows a white page, but no console errors.
Scrolling
There are 3 tests that use
new test_driver.Actions().scroll(...)and either ERROR or TIMEOUT in Firefox/Safari:https://bugzil.la/1746601 is the Gecko bug about this testdriver API. The harness errors can probably be fixed with the proper
t.step_funcwrapping or similar./css/css-scroll-snap/input/keyboard.html is TIMEOUT in Firefox and Safari, but the first test runs and fails in Firefox. I don't think this is due to a missing testdriver.js API, but it needs investigation.
/css/cssom-view/scroll-behavior-smooth.html is ERROR in Chrome and TIMEOUT in Safari. Needs investigation.
The text was updated successfully, but these errors were encountered: