-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inconsistent end-point for Interop pages #286
Comments
When we've talked about this in the past I've been skeptical of stopping the graphs at the end of the year, because that would make any improvements after that invisible. But now that we have "previous focus areas" it's not totally invisible, and I think it's reasonable that the graphs eventually stop updating. To make https://wpt.fyi/interop-2021 consistent by lowering the scores now presented would be not-great I think, so I wonder if we could let that run to 2022 instead, in the graph as well? cc @DanielRyanSmith who I've discussed this with in the past. |
After making the changes to the results-analysis scoring, it seems that we can pretty easily delimit the any year's scores by defining the end date in which we collect runs to score (the Basically, we can define whenever we'd like to stop accumulating the score by setting this flag for specific years that we would like to freeze. |
@gsnedders can you file an issue in the interop repo and agenda+ it so we can decide what we want to do? |
@DanielRyanSmith there is one more complication that specifying an end date doesn't deal with, and that's changes in the test lists that happen after the year has ended. I think what we'll need to do is go back and see what the test lists were and the end of 2021 and 2022, and freeze those lists somewhere. Then we could keep running the scripts because they should always produce the same output, or just take a frozen copy of the output and put in a storage bucket. |
We have the same problem for Interop 2023 as well now, where the graph ends in December but we kept updating the scores until Jan 31. |
This is resolved now with how we're archiving the dashboard for each year: |
https://wpt.fyi/interop-2022 shows a graph ending at the end of 2022, but the scores shown above are for the latest run (currently 2023). These should probably be consistent.
This difference is more stark if you look at https://wpt.fyi/interop-2021, where the scores are notably higher than the graph shows.
Per https://twitter.com/olwinfr/status/1618721146802950144, this does appear to be causing some actual confusion among people looking at the dashboard, so we should ensure it's clear what data is being presented.
cc @web-platform-tests/interop
The text was updated successfully, but these errors were encountered: