Web Vitals are captured incorrectly

[lforst]

Description

It seems like the SDK is reporting completely wrong webvitals: Trace

• TTFB is completely off
• FCP after LCP

Looking through some traces in Sentry this did not seem to be an issue in v7.

[Lms24]

Hmm I'm not sure if both are related. Some observations so far:

• TTFB:
  • We changed ttfb reporting in #fix(metrics): use web-vitals ttfb calculation #11185 and fix(tracing): [v7] use web-vitals ttfb calculation #11231. Maybe this change is related
  • In

    sentry-javascript/packages/browser-utils/src/metrics/browserMetrics.ts

    Lines 344 to 361 in 5a709df

    	['fcp', 'fp', 'lcp'].forEach(name => {
    	const measurement = _measurements[name];
    	if (!measurement || !transactionStartTime || timeOrigin >= transactionStartTime) {
    	return;
    	}
    	// The web vitals, fcp, fp, lcp, and ttfb, all measure relative to timeOrigin.
    	// Unfortunately, timeOrigin is not captured within the span span data, so these web vitals will need
    	// to be adjusted to be relative to span.startTimestamp.
    	const oldValue = measurement.value;
    	const measurementTimestamp = timeOrigin + msToSec(oldValue);
    	// normalizedValue should be in milliseconds
    	const normalizedValue = Math.abs((measurementTimestamp - transactionStartTime) * 1000);
    	const delta = normalizedValue - oldValue;
    	DEBUG_BUILD && logger.log(`[Measurements] Normalized ${name} from ${oldValue} to ${normalizedValue} (${delta})`);
    	measurement.value = normalizedValue;
    	});

    , we do not iterate over ttfb although the comment mentions it explciitly. Not sure yet if oversight or on purpose. I asked @AbhiPrasad if he knows more about this. Could just be that the comment is outdated and this is a false flag 😅
• LCP:
  • In the given trace the values are fairly close to each other. Maybe this is one of these cases where we'd report an intermediate/initial LCP value which would change later after the transaction ended. I'm really not sure though. Do we have examples where there is a larger gap between LCP and FCP and LCP < FCP?

[md384]

I have also found that web-vitals data for my site are seemingly incorrect. We have long recorded web vitals data using the web-vitals package, and we have noticed that the values from that do not line up with ones reported by Sentry (with v7 too) - we're currently on 8.22.0.

Below shows the difference of LCP, TTFB and FCP between sentry reported values (left) and ones from the web-vitals (right) package (webvitalscomp_XXX metric).

p75

p50

LCP seems to be the most off - significantly different in many cases.

Individual examples (self-hosted so can't link examples)

On individual traces, here is an example of the sentry values being different to our custom metric and also not lining up with the spans in the trace

Here can see the difference of start time of the request and the start time of the first paint span is 1723035794.5266 - 1723035793.3029 = 1.22s which is still lower than our custom metric but closer than the sentry reported one.

This also shows LCP > FCP which incorrect as mentioned in the original post.

We have many examples of individual traces like this.

[Lms24]

Hey @md384 thanks for providing this data!

I can see a couple of things that might explain the discrepancies:

• we collect all web vitals at the time when we end our pageload span. This might be too early, resulting in only capturing an intermediate value for LCP. This could be the reason why our reported LCP values tend to be shorter than your recorded LCP values.
  • it doesn't really explain why we get FCP > LCP values though. I've seen it before, leading me to believe that:
• Our start time + vital offset calculation must be off somehow. This also might explain the TTFB issue that was mentioned in this issue.

I'm curious: How do you use the web-vitals library? Specifically, do you set reportAllChanges: true in the onFCP and related callbacks.

[Lms24]

Unrelated to my comment above but adding it here:

A further example of incorrect LCP/FP/FCP values: https://sentry-sdks.sentry.io/performance/trace/10e33722186ea42a792504e313d192e5/?field=title&field=event.type&field=project&field=user.display&field=timestamp&name=All+Events&node=txn-e507fc916a5f4d3f8bba7a1a9fb1032d&node=txn-b7ddd0be9dc04146be202f3865bf3cc6&project=4504796902588416&query=&sort=-timestamp&source=discover&statsPeriod=10m&timestamp=1723115339&yAxis=count%28%29

[md384]

Hey @md384 thanks for providing this data!

I can see a couple of things that might explain the discrepancies:

• we collect all web vitals at the time when we end our pageload span. This might be too early, resulting in only capturing an intermediate value for LCP. This could be the reason why our reported LCP values tend to be shorter than your recorded LCP values.

  • it doesn't really explain why we get FCP > LCP values though. I've seen it before, leading me to believe that:

• Our start time + vital offset calculation must be off somehow. This also might explain the TTFB issue that was mentioned in this issue.

I'm curious: How do you use the web-vitals library? Specifically, do you set reportAllChanges: true in the onFCP and related callbacks.

We do not set reportAllChanges. FWIW, I have seen some instances where web-vitals reported values with FCP > LCP but seems less often than sentry reports.

One other thing, is that the metric counts for my custom metric is consistently lower than the sentry count

We do have a browser guard (typeof window !== 'undefined' && Boolean(window?.navigator?.platform)) to only report the custom metric for browsers, but perhaps there is something there.

[Lms24]

@md384 which version of the web-vitals library are you using?

[md384]

@md384 which version of the web-vitals library are you using?

Currently 3.5.0. I will try upgrading to the latest and seeing if we get different results.

[euijinkk]

I'm having the same issue right now. LCP < FCP.

I'm using nextjs (server side rendering) and measuring pageload fcp and lcp.
In my case, most of data give a higher fcp.

I think FCP timing is weired. It is after Next.js hydration.

FCP, LCP, which vitals I can trust?

FYI :

• sentry v7.64

[euijinkk]

I got to know that LCP log are taken right after nextjs-hydration.
And after that, fcp are taken.

But originally in ssr, FCP and LCP can be stamped before hydration.
How can I collect the correct fcp, lcp logs that consider SSR.

[Lms24]

I got to know that LCP log are taken right after nextjs-hydration.
And after that, fcp are taken.
But originally in ssr, FCP and LCP can be stamped before hydration.
How can I collect the correct fcp, lcp logs that consider SSR.

We collect the LCP value when the pageload span ends. This is likely a bit later than when hydration completes. The web-vitals library doesn't differentiate between pre or post hydration but we can only start tracking the vitals once the SDK is initialized. This happens a bit before hydration completes.

In a pageload trace, you should find a mark span for when hydration of a nextJS app completes and another one called something like "sentry tracing init". Could you share the link to such a trace if possible?

[md384]

@Lms24 confirmed the same results with web-vitals 4.2.3

For LCP which has the largest discrepancy

TTFB and FCP looks the same as before too - typically TTFB is a little higher in sentry and FCP a little lower in sentry vs the web-vitals measurements

[mydea]

Thank you for the added detail, we will continue to look into this & investigate this!

[Lms24]

Hello everyone, after investigating, I'm fairly sure we found the root cause of the LCP/FCP/FP values being off:

We normalize LCP/FCP/FP values in a very weird way in the SDK, where it looks like we try to re-calculate LCP from the beginning of the pageload transaction instead of what the browser gives us. I think the reason for this is that we need to draw the vertical line for LCP in the trace view but there are a couple of fundamental problems:

• Before our normalization logic, the LCP value we'd send is perfectly aligned with the raw value from the PerformanceObserver and the web-vitals chrome extension. So at some point we actually collect the correct one.
• I understand that we might need to shift the value somewhat to align it properly in the trace view (actually, do we really need to?) but for this we cannot change the measured LCP value itself
• From what I could reproduce and observe, it looks like we always decrease the LCP value, which correlates well with the reports in this issue.
  • Example: Before normalization, we measure TTFB 10ms, LCP/FCP 180m. After normalization: TTFB 10ms, LCP/FCP 2ms. This is completely incorrect.

I will check what removing the LCP/FCP/FP value normalization does to the values shown in the Sentry UI. My hunch is that they'll be much more accurate. We need to check the impact on the trace view especially though.

[keradus]

after #13502 is merged, shall this issue be still open ?
cc @Lms24

[Lms24]

Hey @keradus thanks for the callout! Yeah, let's close this for now. We haven't received as many reports about incorrect web vitals recently. With all the changes made recently, I think it's fair to close.

To anyone reading this issue: If you observe web vitals that appear incorrect, please open a new issue with a detailed description and reproduction of the problem. Thank you!