report: add fractional category scores

[adamraine]

This moves the category ratio indicator from the flow report down to the legacy report renderer. There are still some style details to work out, but I wanted to get some initial impressions on the structure.

[adamraine]

I'm starting to think that we should use a score gauge for timespan performance and snapshot accessibility. Quick mock:

Snapshot accessibility makes the most sense because it's the same 54 audits that are found in navigation mode, so it makes sense to represent it the same way.

For timespan performance, it could be confusing when the category is graded as "average" even though the metrics are all passing (example). The unscored diagnostics and opportunities heavily impact how the category is scored, even though the metrics are the only audits scored. I think a score gauge makes more sense than a ratio in this case.

[patrickhulce]

Back in April, we had this to say as a group.

We can’t give 0-100 Performance in these, we can for other categories (a11y) but should we?
We like consistency, if we can’t do it for performance, we probably shouldn’t for the others.
Perhaps we just do green/yellow/red for the categories.

I think that largely holds true for me today, details below.

Snapshot accessibility makes the most sense because it's the same 54 audits that are found in navigation mode, so it makes sense to represent it the same way.

I agree, and this is the one I'm closest to using a gauge for. In addition to the above, I see two more arguments against this.

1. Reinforcing "this isn't the same navigation environment" is more important than conveying the a11y score out of 0-100.
2. There's been a lot of pushback on a11y in particular for mistakenly conveying "100" when there's a lot not covered. Switching to a fraction helps alleviate that and remind folks it's not comprehensive.

I think with the consistency argument plus these two points, I still lean toward the fraction.

I think a score gauge makes more sense than a ratio in this [timespan] case.

I'd feel really uncomfortable with a timespan score given it's just TBT and CLS. Long-term I have hope for more metric-driven scores on timespans but we really need to let the space mature and establish some benchmarks for success before we can confidently provide a holistic score in this area, IMO. A suite of metrics that doesn't consider network latency or feedback at all can't reasonably be considered a 0-100 quality, balanced performance score.

it could be confusing when the category is graded as "average" even though the metrics are all passing

If more people knew that the metrics alone are what matter for the score in navigation, I might agree here, but with a switch to the fraction I think it's reasonable that the number of red/yellow items you see matches the number of failures reported.

[adamraine]

It would be a simple change if we do decide to make it, so I'm willing to defer to our previous position and finish this PR as is. I had a couple more thoughts if we decide to revisit:

Long-term I have hope for more metric-driven scores on timespans but we really need to let the space mature and establish some benchmarks for success before we can confidently provide a holistic score in this area, IMO.

I agree that a 0-100 score for just TBT and CLS isn't ideal, I just thought it might be better than the fraction. I guess I'm hoping for a more metric-driven score as well :)

1. Reinforcing "this isn't the same navigation environment" is more important than conveying the a11y score out of 0-100.
2. There's been a lot of pushback on a11y in particular for mistakenly conveying "100" when there's a lot not covered. Switching to a fraction helps alleviate that and remind folks it's not comprehensive.

I think with the consistency argument plus these two points, I still lean toward the fraction.

The problem is that a11y is the same between the two environments, so I'm still worried about the consistency within the category.

[adamraine]

I've been calling this "ratio", but I've seen others refer to it as "fraction". Any strong opinions about the name?

[patrickhulce]

I typically refer to a representation like this as a "fraction" when the first number is strictly less than or equal to the second number, and a "ratio" when the two numbers have a more arbitrary relationship.

By that lens, I would call these a fraction.

[patrickhulce]

this seems like a bit of a misleading path where we've asked for a gauge but receive a ratio

new method renderCategoryScore that does this delegation like you've done in the flow report?

[adamraine]

I started doing this but backed out once I reached

lighthouse/report/renderer/report-renderer.js

Line 167 in ee53af3

} else if (renderer.renderScoreGauge === categoryRenderer.renderScoreGauge) {

We would just keep that line the same right?

[patrickhulce]

My preference would be to replace this line with renderCategoryScore === renderCategoryScore and rename the PWA one to renderCategoryScore. In fact that name makes more sense anyway because it doesn't have a gauge :)

We can alias renderScoreGauge to renderCategoryScore in PWA category for backcompat until v9 in case any partners are using it.

[brendankenny]

thinking of the API proposal in #12772 (where prepareReportResult won't necessarily be used) and the possibility there might be more changes needed to category rendering besides just gauge vs fraction, what do you think about an options object passed into renderCategoryScore?

It seems like that's ultimately what we're going to want from renderReport()/renderReportPartials() and it would be nice to not alter ReportResult further from the LHR.

[adamraine]

what do you think about an options object passed into renderCategoryScore?

That would mean passing the options through multiple layers of function calls, in multiple different locations. We could do it, but it seems like ReportResult was designed so we don't have to make these kinds of structural changes.

[brendankenny]

ReportResult wasn't so much designed :)

lighthouse/report/renderer/util.js

Line 55 in 5cbad24

* TODO(team): we all agree the LHR shape change is technical debt we should fix

It is a structural change and it would have to go through a few layers (though not so many). I think it may be worth it, though:

• I want to call renderGaugeForCategory(category, groups, options), I don't want to have to call prepareReportResult first
• if there are other per-mode changes (e.g. if we go with Use fraction score for non-performance categories #13022, perf is going to be different than the other categories in navigation reports but not in timespan/snapshot), we're going to have to expand displayMode or generalize it anyways

basically if we're going to parameterize on gatherMode, let's just do it outright rather than roundabout through ReportResult :)