Add ability to prime URL metrics across a site upon installation of Optimization Detective

[westonruter]

Feature Description

When activating Optimization Detective, it starts gathering URL metrics from visitors to the site. However, it has a storage locking mechanism to prevent a user (by hashed IP address) from submitting URL metrics once every 60 seconds. Additionally, URL metrics are not collected for administrator users since pages often include additional elements for admins which will make the URL metrics not applicable to non-admins. All this means that if a site does not get much traffic it can take a few days (or longer) to gather enough URL metrics to apply the optimizations. We could speed up this process by providing a way for an administrator to initiate a site-wide collection of URL metrics. A new query parameter could be introduced which (1) loads the page as a non-admin and (2) disables any storage locking for URL metrics. Then given a list of URLs on the site (e.g. from the sitemaps), each one could be loaded up in an iframe for each breakpoint. A progress bar could also be shown. The process could be interrupted at any time, and if the user wants to start again they could either initiate collection only for URLs for which there are no URL metrics gathered or else to start again from scratch.

[westonruter]

This would be cool to implement as a CLI tool as well, using Puppeteer.

[westonruter]

This would make sense to do in a hidden iframe in the post editor screen whenever a post has been updated. This would not require any new UI (which we've avoided until now) and it would ensure that posts get fresh URL metrics immediately.

[westonruter]

This would address the issue where someone blocks the REST API for authenticated requests.

[westonruter]

For very high traffic sites, this could be an alternative to collecting URL Metrics from visitors. See #1655.

[b1ink0]

@westonruter

I've been working on a proof of concept (POC) for priming URL metrics and wanted to share an update and gather your thoughts on it.

Current Implementation

The POC uses URLs from WordPress sitemaps, fetched in batches. Breakpoints are calculated on the backend with simple logic: the height for each breakpoint is based on its width and the min/max aspect ratios.

• The minimum aspect ratio is capped at 1 to avoid very small heights for large breakpoints.
• The maximum is capped at 2 to prevent overly tall small breakpoints.
• To accommodate n+1 breakpoints, an additional one is added by increasing the last breakpoint’s width by 400.

These are placeholder settings for now and can be adjusted as needed.

The detect.js is also modified to address two limitations:

1. Data is now sent immediately, regardless of tab visibility.
2. Requests are made using fetch instead of navigator.beacon when od_prime=1 is present, as fetch provides better feedback on request success.

Parent-iframe communication uses postMessage. When the iframe completes a fetch request, it notifies the parent to move to the next breakpoint or URL.

Handling URLs

The od_prime=1 parameter has been added to iframe URLs. This flag:

• Allows detect.js in Optimization Detective to skip the storage lock and other restrictions, enabling bulk requests.
• Is also passed to the REST endpoint to bypass its storage lock.

While this works for now, maybe a nonce would be a more secure approach.

On the backend, the od_prime=1 parameter is removed from URLs before hashing.

Demo

od-demo.mp4

Note: To aid debugging, the current status of the processing URL, breakpoint dimensions, and the iframe are currently visible. These could be replaced with a progress bar later.

This feature can be accessed through a new settings page in the tools. However, we need to discuss the UI design further. @ShyamGadde is also looking into adding a resume feature to skip URLs with completed metrics after a page refresh.

[westonruter]

@b1ink0 awesome!

The POC uses URLs from WordPress sitemaps, fetched in batches.

For initial collection of URL Metrics this makes sense, but for incremental updates in the course of normal operations of a site, it would probably make more sense to prime URL metrics in the background on the post editor screen just for the current post's URL.

1. Data is now sent immediately, regardless of tab visibility. [///]
Parent-iframe communication uses postMessage. When the iframe completes a fetch request, it notifies the parent to move to the next breakpoint or URL.

Is that necessary? Wouldn't the data be sent when navigating away anyway? For example, once the iframe's load event fires, you could navigate it to about:blank. Or rather, when in this priming mode, the JS in the page could do postMessage right here:

performance/plugins/optimization-detective/detect.js

Line 568 in 6ca5c4b

// Wait for the page to be hidden.

At that point, the parent frame would know it is free to continue on to the next URL, allowing it to proceed with sending the metrics.

2. Requests are made using fetch instead of navigator.beacon when od_prime=1 is present, as fetch provides better feedback on request success.

What feedback is needed?

While this works for now, maybe a nonce would be a more secure approach.

Yes, a nonce will be important as otherwise malicious actors could bypass the safeguards.

Note: To aid debugging, the current status of the processing URL, breakpoint dimensions, and the iframe are currently visible. These could be replaced with a progress bar later.

Since desktop viewport size can be too large to display in the admin, you can consider using a transform to shrink it down to fit. I implemented this in https://googlechromelabs.github.io/layout-shift-terminator/

Related: ampproject/amp-wp#4729 and the work that @thelovekesh did in ampproject/amp-wp#4729 (comment) (note the post editor integration, although for the actual implementation no button should be required as it should happen automatically when saving a post)

Note also that Optimization Detective is a spiritual descendant of Layout Shift Terminator!