CSSWG specs need usable test suite docs

[tantek]

Problem: while CSSWG drafts link to their test suites, it is far too hard to figure out how to review & report problems with existing tests, and how to contribute new tests. We need usable test suite documentation, and should consider at a minimum having our specs directly link to where to file test suite issues.

The WG wiki page on tests is horribly out of date, as are nearly all its subpages. And links to the "Web Platform Tests" site land you at pages so opaque that it is unclear if you’ll have to guess correctly and click 3 or 30 nested links to find out where to give feedback or contribute tests. I made some bandaid fixes to that WG wiki test page with a direct link to issues and also to Rachel Andrew's excellent post Christmas Gifts for Your Future Self: Testing the Web Platform, but that wiki page as a whole needs to be completely rethought and rewritten.

Since having valid (reviewed) tests for a feature is required for that feature to exit CR, and missing such tests (or being unsure of them) is often something that blocks CSSWG drafts from progressing, this issue should remain open here (perhaps as a meta issue) until sufficient solutions are found / deployed that demonstrate increased feedback and contributions to test suites for CSSWG specifications. (When people wonder why aren’t more web developers contributing or giving feedback on CSS tests, the poor usability barriers of these docs alone are a likely sufficient explanation, and thus a good place to start fixing.)

This issue directly affects all current CSSWG drafts at CR or earlier stages that are being developed in this repo and which we want to eventually (preferably efficiently) advance to PR. Thus please leave this issue open here in this repo for the CSS Working Group to resolve (rather than say, attempting to lump it or transfer it into some broader "Web Platform Tests" issue). Some more background: issue 2378

cc: @gsnedders @liamquin @atanassov @astearns

Labels: wg-css, PR-blocker

(Originally published at: http://tantek.com/2018/073/b1/csswg-specs-need-usable-test-docs)

[dbaron]

There's another piece that I think is worth considering: are there ways we can make the process easier?

For example, a number of browser-developers have messed up issues related to the built test suite by creating tests that worked in the web-platform-tests repo, but not in the post-build directory structure. Do we still need the built test suites, or can we just point to tests in the web-platform-tests repo?

[rachelandrew]

I'd love to help fix this, unfortunately I don't have a huge amount of free time to devote to doing so but writing dev-friendly documentation is a big part of what I do so happy to get involved in some discussion or help to map out a good approach for tackling it.

[gsnedders]

Notwithstanding the web-platform-tests documentation issues (see web-platform-tests/wpt#10177), I think there's a number of CSS WG specific issues:

• The test build system makes it hard to find what file(s!) you need to modify to fix a test if you're looking at http://test.csswg.org/harness/ or http://test.csswg.org/suites/ (which is a large part of [css-cascade][css-cascade-3] Test suite appears orphaned, not in web-platform-tests, nor any other discoverable repo #2378 AFAICT)
• The report issue button on tests at http://test.csswg.org/harness/ points at Shepherd which is read-only (and there's nothing on Shepherd to make that clear).

We could do with decent documentation on the build system (or, as @dbaron says, get rid of it) and how to deal with various things (like the growing list of warnings it produces when being built, and how you should know whether any warnings were caused by you, etc.).

[mrego]

For example, a number of browser-developers have messed up issues related to the built test suite by creating tests that worked in the web-platform-tests repo, but not in the post-build directory structure. Do we still need the built test suites, or can we just point to tests in the web-platform-tests repo?

BTW, this is being discussed in web-platform-tests/wpt#10053.

[gsnedders]

There is also https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md when it comes to docs for WPT.

[fantasai]

For reference, a pile of links I've sent over to various people about Mozilla's QA documentation from a long time ago. It was very effective at creating a community of QA volunteers.

This is the Mozilla website around when I joined. It has a lot of personality. ;)
https://web.archive.org/web/20000125153750/http://www.mozilla.org:80/
Important docs besides the home page:
https://web.archive.org/web/20000229165445/http://www.mozilla.org:80/mozorg.html
https://web.archive.org/web/20000301043132/http://www.mozilla.org:80/get-involved.html

---

https://web.archive.org/web/20000302035824/http://www.mozilla.org:80/quality/bug-writing-guidelines.html
https://web.archive.org/web/20000304015940/http://www.mozilla.org:80/newlayout/bugathon.html

You can also browse the website circa 2009 at
http://www-archive.mozilla.org/quality/
It's a little more polished and informative at this era, but still managed by the Mozilla technical community rather than the marketing team. The overall landing page for getting involved is
http://www-archive.mozilla.org/developer/
which gives a nice overview of the Mozilla development ecosystem. It was linked from the www.mozilla.org home page, which opened up people curious about Mozilla to
the possibilities of getting involved.

The common themes here are

• www.mozilla.org is up-front about being the home page of an open source organization and allows people curious to learn more about Mozilla development
• The development pages are written by developers for potential developers to be useful and welcoming
• QA for Gecko is encouraged and potential contributors are given the sense that there's a team of real people working on problems we all care about that could really use their help!

[css-meeting-bot]

The Working Group just discussed CSSWG specs need usable test suite docs.

The full IRC log of that discussion

<dael_> Topic: CSSWG specs need usable test suite docs
<dael_> github: https://github.com//issues/2438
<rego> so we keep duplicating files all over the place, or use absolute paths :(
<dael_> fantasai: I 100% agree testing documentation sucks.
<dael_> tantek: Problem highlighted is our testing documentation makes it very hard for community of people helping us to grow.
<dael_> florian: Shouldn't we delete what's in the wiki and point to WPT's documentation and if that's insufficient file issues on that.
<dael_> gsnedders: The build system and test harness should be there.
<dael_> fantasai: And meta
<dael_> florian: Meta is on WPT.
<dael_> florian: I'd like to rpopose we delete our test documentation and any time we need to reference instructions we point to WPT and then we file issues if there's a bug.
<dael_> tantek: Their tests are completely opaque. I disagree. Here's a bunch of things that look identical and if you guess right after 3 levels you'll find information.
<dael_> florian: The wiki is old information, though.
<dael_> tantek: First thing I did is go through the wiki and it has rudementary information on how to do it. If you want to help witht he problem look at stuff.
<dael_> florian: I added a bunch of documentation to WPT.org for stuff we had.
<dael_> tantek: WPT is opaque.
<dael_> florian: The information is mostly complete, discoverability is horrendous.
<fantasai> s/bunch of things/bunch of links/
<dael_> gsnedders: My objection to putting everything on css wiki...WPT documentation is bad and the solution here isn't to write new docuemtnation, it's to fix it.
<dael_> florian: It's not bad for containing information.
<dael_> gsnedders: It's hard to nav and not at the level for browser vendors or new people. Too verbose for borwser vendors.
<dael_> TabAtkins: florian do you remember how often browser vendors look at old WD instead of what's in TR? They'll look at wrong things.
<dael_> gsnedders: And again it doesn't start basic enough level.
<dael_> tantek: You start at the spec, it links to the test suite, you try a test and it fails. Look at test and it fails. now you want to fix it. Now you're in trouble.
<dael_> tantek: If we fix that on csswg wiki that's a huge improvement. We're not going to move all documentation here. But we want an avg web dev to have a prayer.
<dael_> gsnedders: Some of the problem may be test harness links to sheppard for bugs. Where should we file bug on the test harness?
<dael_> [silence]
<dael_> tantek: First scenario is make it easy to report bugs on test. Second thing we hsould make easy is for people to contribute tests. Both of those can be incrementally fixed without fixing everything.
<dael_> fantasai: I've sent prob 7 people a list of links to Moz documentation from 2001 which was very straight forward. That'st he documentation we need. It's clear these are the people they need help.
<dael_> tantek: Can you link to those in this issue?
<dael_> fantasai: Yeah.
<dael_> tantek: We won't solve it here but I'm hoping we can align on objectives.
<dael_> florian: Rephasing my position, I don't mean delete everything from the wiki. Do not attempt to duplicate what's in WPT in the wiki. Additional guidance to help people find the things or help people know what to do.
<dael_> tantek: Agree with the caveat that undiscoverable or unlinkable is the eq. to not there.
<dael_> florian: Yeah.
<dael_> florian: WPT.org needs a search button too. That would be useful.
<dael_> astearns: File an issue for that?
<dael_> florian: Yes.
<dael_> florian: WPT.org is done with Jeckal, right?
<dael_> gsnedders: Yes.
<gsnedders> s/Jeckal/Jekyll/
<dael_> florian: There are plug ins for search.
<dael_> [talking about how to add search]
<dael_> astearns: Anything more?

[svgeesus]

So, given that:

• the primary source of CSS WG tests is now WPT not the ancient and un-maintained tests in Shepherd
• CSS WG specs increasingly use the <wpt> annotations to link from specs to individual tests
• Bikeshed complains about missing tests (in WPT but not in the bikeshed source) and unfindable tests (in bikeshed but not in WPT

I believe this issue is in practice solved for many years now. Closing.