Offline support #1483

bgrgicak · 2024-06-02T18:01:01Z

Motivation for the change, related issues

Most of Playgrounds load time is waiting for files to download.
If the user reloads Playground they are waiting for the same files to download.

This PR improves Playground load time by caching WP, PHP, and the SQLite plugin in the browser.

Implementation details

The PR uses browser cache storage to cache fetch responses and uses the request URL as the cache key.
Because WP, PHP, and the SQLite plugin change URLs when they are rebuilt the cache will be busted and the correct version will be downloaded.

Testing Instructions (or ideally a Blueprint)

Checkout this branch
Open Playground in Chrome
In dev tools > Application clear all cache storage
See that wp-6.5.zip was downloade
Reload and see that it wasn't downloaded

packages/playground/remote/src/lib/worker-thread.ts

bgrgicak · 2024-06-05T09:13:13Z

We will need to rebuild this prototype to support:

WP zips
PHP wasm and JS files
SQLite plugin

We won't cache WP assets that are currently loaded from the server. Instead, we will load them all during boot.
This will be implemented in a separate PR. The files will be stored in a zip and downloaded async so that they don't block the boot process.

packages/php-wasm/web-service-worker/src/initialize-service-worker.ts

…rker.ts

adamziel · 2024-06-11T10:39:19Z

@bgrgicak I meant this file:

https://github.com/WordPress/wordpress-playground/blob/172d78fd63e1502deda034934e0323256869d327/packages/playground/remote/service-worker.ts

All the fetch requests to playground.wordpress.net go through that code path. The fetch event handler is registered here and returns early for non-scoped URLs – it needs a refactor to cache WordPress and PHP assets. One way to do it is to remove the initializeServiceWorker function entirely and just register a fetch listener in remote/service-worker.ts:

wordpress-playground/packages/php-wasm/web-service-worker/src/initialize-service-worker.ts

Line 20 in 172d78f

self.addEventListener('fetch', (event) => {

The cachedFetch function is in the remote package, so we will be able to reuse it for handleRequest.

A downside of cachedFetch is that we must remember to use it everywhere. The service worker can blanket-cache all the relevant static files while only requiring very lightweight maintenance. Also, the service worker needs to support caching anyway and that change could trivially cover also the PHP and WP assets.

brandonpayton · 2024-06-12T00:10:15Z

The service worker can blanket-cache all the relevant static files while only requiring very lightweight maintenance. Also, the service worker needs to support caching anyway and that change could trivially cover also the PHP and WP assets.

This. From the documented motivations in the Service Worker spec:

Events that correspond to network requests are dispatched to the worker and the responses generated by the worker may override default network stack behavior. This puts the service worker, conceptually, between the network and a document renderer, allowing the service worker to provide content for documents, even while offline.

Service Workers are intentionally positioned to provide such a caching layer.

I often wish the web platform was not so complicated and generally resent the complexity introduced by Service Workers. But this seems like a place we can appreciate them. :) (Actually, Playground as a whole has been a wonderful demonstration of the merits of Service Workers)

bgrgicak · 2024-06-18T04:48:10Z

@brandonpayton @adamziel offline support should work now in this PR (without preloading and busting).

I have an issue locally where Playground crashes on reload because the Vite server isn't available while offline, but I'm not sure how to resolve it. If you have some time, please take a look at it.

To recreate the issue:

checkout branch
open Playground
in Chrome go to Dev tools > Application
See that there are files in the Cache storage section
In the Service worker section click the offline checkbox
reload
see the error in the browser console

If you don't see the error enable Persistent log in the JS console.

brandonpayton · 2024-06-19T00:46:07Z

packages/playground/remote/src/test/fetch-caching.spec.ts

@@ -0,0 +1,68 @@
+import { areCacheKeysForSameFile, isValidFile } from '../lib/fetch-caching';


Where is isValidFile() defined?

brandonpayton

Thanks for working on this! I took a look and left some questions. There are likely things I don't properly understand yet.

brandonpayton · 2024-06-19T00:46:28Z

packages/playground/remote/src/test/fetch-caching.spec.ts

+		).toBe(true);
+	});
+
+	it('Shoud return true if the cache keys are for the same file', () => {


typo: "Shoud"

brandonpayton · 2024-06-19T00:52:27Z

packages/php-wasm/web-service-worker/src/fetch-caching.ts

+	return await cache.then((c) => c.match(key));
+};
+
+const validHostnames = ['playground.wordpress.net', '127.0.0.1', 'localhost'];


Do we need to validate hostnames? What if people want to host their own Playground?

We don't but I didn't want to cache their party scripts like Google Analytics.
But we could just cache everything.

Wow, I had no idea Service Workers also receive fetch events for requests to third party resources. But it appears to be true.

It looks like fetchEvent.currentTarget.registration.scope is available, so we could filter based on that rather than a specific set of hostnames.

fetchEvent.currentTarget points to the ServiceWorkerGlobalScope. ServiceWorkerGlobalScope exposes the ServiceWorkerRegistration as its registration property, and ServiceWorkerRegistration has the scope property.

brandonpayton · 2024-06-19T00:57:52Z

packages/php-wasm/web-service-worker/src/fetch-caching.ts

+
+const precachedResources: RequestInfo[] = [
+	'/website-server/',
+	'/website-server/index.html',


Why are we precaching these paths specifically? How do we adjust in production where there is no /website-server/ directory?

These files are required for Playground to load when offline. They aren't fetched by the service worker so they need to be preached in this way.

These files are required for Playground to load when offline.

These files don't appear to exist at all on the production site.

They aren't fetched by the service worker so they need to be preached in this way.

Do you mean on initial load before the service worker can be registered?

brandonpayton · 2024-06-19T01:02:29Z

packages/php-wasm/web-service-worker/src/initialize-service-worker.ts

@@ -22,28 +33,28 @@ export function initializeServiceWorker(config: ServiceWorkerConfiguration) {

 		// Don't handle requests to the service worker script itself.
 		if (url.pathname.startsWith(self.location.pathname)) {
-			return;
+			// return;


Why is this commented out?

This return breaks caching. I'm still working on the PR so I didn't do anything with it.

Do we want to cache the service worker script?

This return breaks caching. I'm still working on the PR so I didn't do anything with it.

Do we want to cache the service worker script?

To clarify the question: It looks like this only stops caching of the service worker script. Is that correct, and if so, why do we want to cache the service worker script?

brandonpayton · 2024-06-19T01:10:27Z

packages/php-wasm/web-service-worker/src/initialize-service-worker.ts

+			)
+		) {
+			// Add caching for non-scoped requests
+			event.respondWith(cachedFetch(event.request));


Are non-scoped requests typically requests for Playground web UI assets?

Would it be good for us to use a completely separate cache for app resources so we can easily flush the cache for those when we detect a new version of the Playground web UI?

We could consider things like that more in a future iteration, but if we do that, I think it will be easy to forget our previous caching scheme(s) and leave cruft in the cache from earlier versions. Maybe that is OK... I'm not sure.

Are non-scoped requests typically requests for Playground web UI assets?

They request Playground web UI assets, WP remote assets, WP zip, PHP wasm, and JS.

Would it be good for us to use a completely separate cache for app resources so we can easily flush the cache for those when we detect a new version of the Playground web UI?

Yes, but I would prefer to get this out first and improve later.

Yes, but I would prefer to get this out first and improve later.

👍 works for me.

brandonpayton · 2024-06-19T01:15:21Z

packages/php-wasm/web-service-worker/src/initialize-service-worker.ts

-			event.respondWith(responsePromise);
+		const response = handleRequest(event);
+		if (response) {
+			event.respondWith(response);


Are these scoped requests for things like WP assets? Does that mean we are not caching them? Or do the de-scoped requests made by handleRequest() become new fetch events that are handled above.

In my current mental model, fetches by the Service Worker do not trigger additional fetch events, but that might just be a poor assumption on my part.

I'm not sure if this helpes, but here is how requests work.

All requests in the app go to this listener, no matter if it's a remote request or an internal Playground request.

Remote requests should be cached to add offline support. These are files like the WP zip and remote static assets.
There are also third-party remote requests like Google analytics, that we don't cache for now.

Internal Playground requests are the same as scoped requests. This is where we reroute the request to go to PHP-wasm instead of going to a fetch. This would be a WordPress page load, an asset that exists inside the Playground filesystem like style.css, or a PHP script execution request.

Thank you for explaining a bit. In my mental model yesterday, I'd forgotten that many WordPress files would be available locally.

Note: The following question may be irrelevant if you are planning to proactively cache remote WP assets separately.

For scoped requests for remote static assets that were eliminated from the WordPress build, won't those skip caching because:

scoped requests are not cached

scoped requests for remote WP assets are fetched by the Service Worker

fetch() by the Service Worker does not trigger subsequent fetch events (in my understanding and testing)
?

bgrgicak · 2024-06-19T11:25:22Z

I made some progress on caching static assets today, but need to get back to this PR tomorrow. The Vite server is preventing caching from working.

When online it messes with MIME types , when online it prevents Playground from running because the socket connection fails.

brandonpayton · 2024-06-19T19:13:08Z

I made some progress on caching static assets today, but need to get back to this PR tomorrow. The Vite server is preventing caching from working.

If needed, we can just skip the Vite server for now to test single website builds:

npm run build:website
php -S localhost:9999 -t dist/packages/playground/wasm-wordpress-net

The site is then served from:
http://localhost:9999

There are some errors when building the website under this offline branch, but I am able to test a trunk website build with the above approach. It requires rebuilding the website after each change, but it is at least an alternative to the Vite server.

brandonpayton · 2024-06-20T01:23:50Z

@bgrgicak I pushed c9ed3e4 which fixed the website build for me. Please feel free to drop, modify, or whatever.

bgrgicak · 2024-06-27T05:26:16Z

I'm closing this PR, all of the changes are moved to #1535

## Motivation for the change, related issues To achieve [offline support](#1483) we need to download all WordPress files into the Playground filesystem. Today Playground downloads WordPress partially on boot and downloads assets as they are needed using fetch requests. This PR downloads all assets into the Playground filesystem without blocking the boot process. ## Implementation details Zip files are generated using the WordPress build script and are accessible in the `/wp-VERSION/` folder like any other assets. The download is triggered after WordPress is installed. The fetch is async and won't block the boot process. In case some of the assets are needed immediately on boot they will still be downloaded using fetch to ensure they are available to WordPress. Once the assets are downloaded they are stored in the Playground filesystem and will be served from there. ## Testing Instructions (or ideally a Blueprint) - Run `npx nx bundle-wordpress:nightly playground-wordpress-builds` - Confirm that a `packages/playground/wordpress-builds/public/wp-nightly/wordpress-static.zip` file was created - [Load Playground using this blueprint](http://127.0.0.1:5400/website-server/?php=8.0&wp=nightly) - In network tools see that the `wordpress-static.zip` file was downloaded - Open `/wp-admin/` - In network tools confirm that static assets like _thickbox.js_ were loaded from the service worker instead of a fetch

## Motivation for the change, related issues To achieve [offline support](#1483) we need to download all WordPress files into the Playground filesystem. Today Playground downloads WordPress partially on boot and downloads assets as they are needed using fetch requests. This PR downloads all assets into the Playground filesystem without blocking the boot process. ## Implementation details Zip files are generated using the WordPress build script and are accessible in the `/wp-VERSION/` folder like any other assets. The download is triggered after WordPress is installed. The fetch is async and won't block the boot process. In case some of the assets are needed immediately on boot they will still be downloaded using fetch to ensure they are available to WordPress. Once the assets are downloaded they are stored in the Playground filesystem and will be served from there. ## Testing Instructions (or ideally a Blueprint) - Run `npx nx bundle-wordpress:nightly playground-wordpress-builds` - Confirm that a `packages/playground/wordpress-builds/public/wp-nightly/wordpress-static.zip` file was created - [Load Playground using this blueprint](http://127.0.0.1:5400/website-server/?php=8.0&wp=nightly) - In network tools see that the `wordpress-static.zip` file was downloaded - Open `/wp-admin/` - In network tools confirm that static assets like _thickbox.js_ were loaded from the service worker instead of a fetch --------- Co-authored-by: Adam Zieliński <[email protected]>

bgrgicak added 3 commits June 2, 2024 17:44

Add browser directory mounting support

f315461

Merge branch 'trunk' into add/php-opfs-mount-trigger

c75c033

Add basic caching TODO test limits and busting

8b964ac

bgrgicak self-assigned this Jun 2, 2024

bgrgicak changed the base branch from trunk to add/php-opfs-mount-trigger June 2, 2024 18:01

Remove test code

12eb27c

bgrgicak requested a review from brandonpayton June 2, 2024 18:09

bgrgicak commented Jun 2, 2024

View reviewed changes

packages/playground/remote/src/lib/worker-thread.ts Outdated Show resolved Hide resolved

Add note

3b2959a

Base automatically changed from add/php-opfs-mount-trigger to trunk June 2, 2024 19:47

bgrgicak changed the base branch from trunk to wp-now-standalone June 5, 2024 08:02

bgrgicak changed the base branch from wp-now-standalone to trunk June 5, 2024 08:02

Merge branch 'trunk' into add/wp-version-caching

6569a69

bgrgicak added 2 commits June 5, 2024 11:26

Remove cache from worker thread

5593d9f

Cache boot files

b40da6c

bgrgicak commented Jun 5, 2024

View reviewed changes

packages/php-wasm/web-service-worker/src/initialize-service-worker.ts Outdated Show resolved Hide resolved

Update packages/php-wasm/web-service-worker/src/initialize-service-wo…

ddbe6e8

…rker.ts

bgrgicak changed the title ~~Add WordPress ZIP caching~~ Cache WP, PHP, and SQLites in browser cache Jun 5, 2024

bgrgicak changed the title ~~Cache WP, PHP, and SQLites in browser cache~~ Cache WP, PHP, and SQLite in browser cache Jun 5, 2024

bgrgicak changed the title ~~Cache WP, PHP, and SQLite in browser cache~~ Cache WP, PHP, and SQLite plugin in browser cache Jun 5, 2024

bgrgicak marked this pull request as ready for review June 5, 2024 10:08

bgrgicak requested a review from adamziel June 5, 2024 10:08

bgrgicak added [Type] Enhancement New feature or request [Aspect] Browser [Aspect] Networking labels Jun 5, 2024

bgrgicak added 4 commits June 5, 2024 13:12

Add outdated cache cleanup

980592a

Clear only current version cache

1702e0d

Merge branch 'trunk' into add/wp-version-caching

027eeba

More tests

f03b3d4

Merge branch 'trunk' into add/wp-version-caching

76fbbb0

bgrgicak added 6 commits June 12, 2024 17:00

Merge branch 'trunk' into add/wp-version-caching

e0c28ab

Merge branch 'trunk' into add/wp-version-caching

2a1ebaf

Move cache to web-service-worker

e34381b

Cache all unscoped urls

b1182bf

Add preloading TODO preload all files

dfa9763

Load index.html when offline

dc1f71c

bgrgicak changed the title ~~Cache WP, PHP, and SQLite plugin in browser cache~~ Offline support Jun 18, 2024

bgrgicak marked this pull request as draft June 18, 2024 04:44

bgrgicak mentioned this pull request Jun 18, 2024

Add remote asset listing to build #1502

Merged

Catch fetch error

c2fd2f9

brandonpayton reviewed Jun 19, 2024

View reviewed changes

Fix web-service-worker build

c9ed3e4

Merge branch 'trunk' into add/wp-version-caching

096c0bc

bgrgicak mentioned this pull request Jun 21, 2024

Download all WordPress assets on boot #1532

Merged

bgrgicak closed this Jun 27, 2024

adamziel linked an issue Jul 3, 2024 that may be closed by this pull request

Offline mode (PWA) #133

Closed

adamziel removed a link to an issue Jul 3, 2024

Offline mode (PWA) #133

Closed

bgrgicak mentioned this pull request Jul 12, 2024

Backfill the assets removed from minified WordPress bundles #1604

Merged

adamziel mentioned this pull request Sep 28, 2024

[Website] 100% reliable deployments #1821

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Offline support #1483

Offline support #1483

bgrgicak commented Jun 2, 2024 •

edited

Loading

bgrgicak commented Jun 5, 2024

adamziel commented Jun 11, 2024 •

edited

Loading

brandonpayton commented Jun 12, 2024

bgrgicak commented Jun 18, 2024

brandonpayton Jun 19, 2024

brandonpayton left a comment

brandonpayton Jun 19, 2024

brandonpayton Jun 19, 2024

bgrgicak Jun 19, 2024

brandonpayton Jun 19, 2024

brandonpayton Jun 19, 2024

bgrgicak Jun 19, 2024

brandonpayton Jun 19, 2024

brandonpayton Jun 19, 2024

bgrgicak Jun 19, 2024

brandonpayton Jun 19, 2024

brandonpayton Jun 19, 2024

brandonpayton Jun 19, 2024

bgrgicak Jun 19, 2024

brandonpayton Jun 19, 2024

brandonpayton Jun 19, 2024

bgrgicak Jun 19, 2024

brandonpayton Jun 19, 2024 •

edited

Loading

bgrgicak commented Jun 19, 2024

brandonpayton commented Jun 19, 2024

brandonpayton commented Jun 20, 2024

bgrgicak commented Jun 27, 2024

		@@ -0,0 +1,68 @@
		import { areCacheKeysForSameFile, isValidFile } from '../lib/fetch-caching';

Offline support #1483

Offline support #1483

Conversation

bgrgicak commented Jun 2, 2024 • edited Loading

Motivation for the change, related issues

Implementation details

Testing Instructions (or ideally a Blueprint)

bgrgicak commented Jun 5, 2024

adamziel commented Jun 11, 2024 • edited Loading

brandonpayton commented Jun 12, 2024

bgrgicak commented Jun 18, 2024

Choose a reason for hiding this comment

brandonpayton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brandonpayton Jun 19, 2024 • edited Loading

Choose a reason for hiding this comment

bgrgicak commented Jun 19, 2024

brandonpayton commented Jun 19, 2024

brandonpayton commented Jun 20, 2024

bgrgicak commented Jun 27, 2024

bgrgicak commented Jun 2, 2024 •

edited

Loading

adamziel commented Jun 11, 2024 •

edited

Loading

brandonpayton Jun 19, 2024 •

edited

Loading