Write a cssharness.js

[gsnedders]

We should have something that can take a CSS property definition (in some format, probably JSON?) and generate a lot of tests based upon it, similar to what we have with idlharness.js.

We have a lot of tests that could be auto-generated from the property definition, especially in the CSS2 testsuite (and these are mostly reftests, which probably costs us a fair bit of time when we could test behaviour through the CSSOM much more quickly en-mass).

Note the one awkward thing here is going to be computing what the computed value will be for any given specified value, but I hope we can do that with relative sanity!

• @tabatkins: Bikeshed uploads propdef info to Shepherd, right? Is this documented anywhere, as to how to get it? What about typedef info?
• @foolip: WDYT?

[emilio]

FWIW Gecko has a property_database.js doing something of the sort:

https://searchfox.org/mozilla-central/source/layout/style/test/property_database.js

Not perfect of course, but maybe a good starting point?

[emilio]

Automatic syntax tests from Bikeshed would be awesome, btw.

[gsnedders]

@emilio Yeah, I was looking over property_database.js, because for some reason I thought it generated the values, rather than having its long explicit lists.

[gsnedders]

FWIW: given the number of CSS2 tests we have that essentially test property parsing, I think the MVP here is something that can replace those, and that avoids some of the complexity from more recent specs.

[foolip]

I've wanted this too, if it's not too hard we should just do it. Like IDL, probably split per spec. @tabatkins, if one wants to extract all CSS syntax stuff (properties, values, pseudo-things, etc.) would one start from the Bikeshed source, or get it from shepherd?

[guest271314]

Interestingly just encountered a bug (unless misinterpreting the specification) for CSS url() function implementation at both Chromium and Firefox. According to the specification

3.4.2. Empty URLs

If the value of the url() is the empty string (like url("") or url()), the url must resolve to an invalid resource (similar to what the url about:invalid does).

where at HTML

<div id="test-empty-string-at-css-url-function" style="background-image:url('')">div</div>

results in location.href being set as the argument to the function

console.assert(
  !new RegExp(location.href)
  .test(
    window.getComputedStyle(
      document.getElementById("test-empty-string-at-css-url-function")
    ).backgroundImage)
, [window.getComputedStyle(
    document.getElementById("test-empty-string-at-css-url-function")).backgroundImage
  ]
)

[gsnedders]

The other thing that strikes me here is that a lot of what we want to test would be much easier with CSS Typed OM, but obviously we can't rely on that any time soon. :(

[darrnshn]

+1 css-typed-om tests would benefit a lot from something like this.

[frivoal]

I am not convinced this is actually a good idea. Yes, it has obvious benefits, but it would also have downsides. CSS, unlike JS, is not a general purpose language, and there are not that many ways you can detect and recover from lack of support. The ones we do have (using the cascade, @supports) work well, but they totally rely on things working when they parse.

Adding tests (especially large amount of tests) that can pass even though the behavior is totally absent seems like something that could backfire and cause tons of problems.

See #10700 (review) or #8899 (comment)

[dbaron]

I've been meaning to upstream the Gecko test_{value,inherit,initial}_{storage,computation}.html tests for ages -- probably by merging them all into a single JS library that could be applied to a property database, so that we could have separate per-spec property databases. Those tests were very valuable in catching common errors in parsing, computation, and serialization, and while they don't cover everything, I think they're useful to have.

[foolip]

@dbaron, that would be great! I didn't know something like this existed already.

[foolip]

I've asked in w3c/reffy#113 if this information is possible to extract from specs and put into a machine-readable form.

[foolip]

In https://github.com/web-platform-tests/wpt/compare/master...foolip:cssharness?expand=1 I have an experiment I did back in August around this. It boils down to CSS.supports(`${propertyName}:initial`) for every known propertyName.

@Elchi3 @samthor FYI, this the experiment I mentioned.

[AmeliaBR]

Adding tests (especially large amount of tests) that can pass even though the behavior is totally absent seems like something that could backfire

We still need the tests for parsing at some point, so I take it that your concern (@frivoal) was that parsing tests will get added easily and automatically, and function/rendering tests will lag far behind?

I agree that this is a concern, we have far too many examples of properties that parse but are badly implemented, so that @supports tests are useless (display: contents breaking accessibility, gap only being implemented for grid not flex, all the visual effects properties and values that were implemented at different times in different browsers for SVG vs HTML elements.) But those examples prove that the concern exists whether the parsing tests are generated automatically or not.

A more comprehensive WPT approach would require a way to declare dependencies between tests. But in this case, the dependency wouldn't merely be about filtering for false positives/negatives (test results that are invalid because the prerequisite test doesn't pass), but would trigger a change in expected outcome of this test depending on the results of the dependencies: A user agent should not parse a given property/value combo unless it can also pass the function/rendering tests associated with the same spec section.

Essentially, this would be quantifying the CSS rule about Partial Implementations:

So that authors can exploit the forward-compatible parsing rules to assign fallback values, CSS renderers must treat as invalid (and ignore as appropriate) any at-rules, properties, property values, keywords, and other syntactic constructs for which they have no usable level of support. In particular, user agents must not selectively ignore unsupported property values and honor supported values in a single multi-value property declaration: if any value is considered invalid (as unsupported values must be), CSS requires that the entire declaration be ignored.

(with "usable level of support" implying in this context that it passes the functional/rendering tests.)

But again, that's a bit of a separate issue from whether parsing tests should be automatically generated or not.

Until such a test-dependency structure exists, I suppose a reasonable compromise for this issue is to say that automatic parsing tests shouldn't be added to WPT until there are also functional tests associated with the same section (cross-referenced in the bikeshed spec). That way a user agent that adds parsing without function is guaranteed to at least fail one test.