Revisit Chrome throttling support

[Rarst]

I've been looking if there is a way to run SiteSpeed with Chrome's network/CPU throttling.

I found a mention in #895 that network wasn't possible at the time, but from some looking around that had been added to Selenium since, see this SO answer (not sure if that extends to CPU).

So could possibly be revisited? Thank you for considering. :)

[soulgalore]

Hey @Rarst there two things I think: When Selenium 4 is released (the current alpha is broken) we can start to use the devtools protocol and that opens up to a lot of new things. But maybe the throttling is already implemented I haven't checked. :)

About the Chrome throttling: doing it in the browser doesn't work so good. I mean doing once when you are using Chrome is perfectly fine but using it in a tool it will make the metrics flawed. For example the Chrome team themselves knows it doesn't work read: https://github.com/GoogleChrome/lighthouse/blob/master/docs/throttling.md

For network throttling I always use https://github.com/sitespeedio/throttle on my local machine, but you can use other tools too.

For CPU throttling, I haven't looked into that yet. I know WebPageTest throttles the CPU on a OS level and that seems smarter. I think testing on real devices is best though :) For example at work we use a c5.large AWS machines and run our testing and time spent in CPU for JavaScript is something like 600 ms and testing on a real device (Moto G5) gives us 2000 ms. I think testing CPU throttle in Chrome would be cool and then compare it on a OS level.

Best
Peter

[Rarst]

I am on Windows so my throttling options seem a little more limited.

For the current task I am working on Lighthouse metrics, which uses Chrome throttling when run locally. So it would be useful if I could produce SiteSpeed profile with same limitations.

I get the downside is less realistic simulated environment, but the upside is ease of use and consistency.

[soulgalore]

Yeah but Lighthouse uses guesstimates as first alternative right? If you don't use it the metrics are consistent right?

Patrick Meenan has written about winshaper that you can use on Windows: https://calendar.perfplanet.com/2016/testing-with-realistic-networking-conditions/

Best
Peter

[Rarst]

I am not sure what do you mean by guesstimate/alternative.

If I run Lighthouse from command line, this is the defaults I see in generated report:

Device: Emulated Nexus 5X
Network throttling: 150 ms TCP RTT, 1,638.4 Kbps throughput (Simulated)
CPU throttling: 4x slowdown (Simulated)

I don't think online version (PageSpeed Insights) reports these, but I would guess similar settings.

I could achieve consistency by telling it not to use throttling, but I am specifically interested in using it with at the moment.

[soulgalore]

Ah sorry for being unclear:

I am not sure what do you mean by guesstimate/alternative.

When you run Simulated there's no throttling going on, it is done after the test manipulating the trace log from Chrome if I understand https://github.com/GoogleChrome/lighthouse/blob/master/docs/throttling.md correctly. So they guess what performance would be IF you use throttling.

I think the outcome is that you shouldn't compare metrics between tools. Compare metrics between runs within the same tool to find regressions/improvement. We have some docs on why it can differs when you run WebPageTest vs sitespeed.io: https://www.sitespeed.io/documentation/sitespeed.io/best-practice/#difference-in-metrics-between-webpagetest-and-sitespeedio

[MasonM]

FYI: With Sitespeed.io 8.11.0, it's now possible to enable Chrome throttling using the DevTools protocol in a script. Example:

module.exports = async function(context, commands) {
	// Emulate fast 3G
    await commands.cdp.send('Network.emulateNetworkConditions', {
      'offline': false,
      'downloadThroughput': 1.5 * 1024 * 1024 / 8, // 1.5 Mbps
      'uploadThroughput': 750 * 1024 / 8, // 750 Kbps
      'latency': 40 // Minimum 40ms from request sent to response headers received
    });
    await commands.measure.start('https://google.com');
};

Save it to a file (e.g. test.js) and run it with:

docker run --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:8.12.0 --multi test.js

[soulgalore]

Thank you @MasonM for adding the example!

[soulgalore]

Lets close this since it's doable using scripting. No need to add it by default since not even the Chrome team likes it.