Changes in the 95th percentile aren't always visible in Glam when they are visible in Telemetry

[smaug----]

The following links show the difference on 95th percentile. It is hard to see in the Glam UI if there was basically any change to 95th, yet 95th is often the most interesting value.

https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=Median!5th%2520percentile!25th%2520percentile!75th%2520percentile!95th%2520percentile&cumulative=0&end_date=2021-10-04&include_spill=0&keys=!none!none&max_channel_version=nightly%252F95&measure=IMAGE_DECODE_ON_DRAW_LATENCY&min_channel_version=nightly%252F95&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2021-09-06&trim=1&use_submission_date=0

https://glam.telemetry.mozilla.org/firefox/probe/image_decode_on_draw_latency/explore?ref=20211002213629&timeHorizon=WEEK

[chutten]

I have two thoughts on why this might occur (and thoughts about what GLAM could do):

1. TMO's underpinning library telemetry.js v2 performs interpolation for its percentiles. This isn't really correct, but if we assume that the interpolation's being done on a continuous distribution that's just being approximated as a discrete distribution, then it's correct enough. And it's more sensitive to changes, this being just one of them.
   • GLAM could adopt interpolation (perhaps optionally) for numeric metric types that are likely discrete approximations of continuous phenomena
2. If the population is, like it might be for this case, predominated by frequent users having a lovely time and infrequent users having absolutely awful image decode latency, GLAM might let that minority actually speak up sufficiently to be heard.
   • This would explain why GLAM's 95ths are so much higher (200k to 130k) than TMO's.

[edugfilho]

I believe this has been fixed with bucket interpolation (#2995) and non-normalized histograms (mozilla/bigquery-etl#3873)