Performance of GitHub-hosted macOS runner

[BrightRan]

Associated community ticket: https://github.community/t/macos-performance/125281

The customer setup a matrix in his workflow to run the same job on different OSs (ubuntu-latest and macos-latest).
See the workflow here: https://github.com/gitoleg/bap/actions/runs/187608792/workflow

On some steps in the job, the macOS runner takes far more longer time than the Linux runner (even more than 2 times) to run the steps.

Earlier, I also noticed that some users have reported similar problems. Especially some users who need to pay for runners, they complained that the performance of GitHub-hosted macOS runner is slower and they have to pay more.
As we have known, the billing per-minute for macOS runner is 10 times that for Linux runner.

[maxim-lobanov]

Hello @gitoleg,
Could you please clarify: did you notice any macOS performance issues recently or they always existed?
We are working on some infrastructure changes related to macOS so we need to understand if it could be related

[gitoleg]

Hello @maxim-lobanov,
We use Github Actions only for 2 months, maybe even less. And I observe the issue in question during all this time.

[gitoleg]

@maxim-lobanov,
Maybe it will help somehow. This run was even worse. The whole workflow takes 1 hour on Ubuntu and more than 5 on macOS!
https://github.com/gitoleg/bap/actions/runs/188588845

[MaksimZhukov]

Hello @gitoleg ! Sorry for the late reply!
It is expected that MacOS performance is lower than Ubuntu (approximately 20 - 30%), but your build time is really longer than expected. Most likely, it is related to known issue with infrastructure but we can't provide ETA right now.
We will keep this issue open until it is resolved. Thank you for your patience! We will keep you informed!

[gitoleg]

@MaksimZhukov Thanks!

[sio]

In the past couple of weeks I've also experienced severe performance degradation on macOS runners.

In one project I've had to increase the timeout value for a function that gets called multiple times by the test suite. Previously timeout was set to sensible 60 seconds since the whole suite runs in 40-50 seconds on Ubuntu, but macOS runners weren't keeping up.

In another project I've moved macOS jobs to another CI provider.

[paresy]

I can confirm that MacOS performance is very flaky and not consistent. Sometimes it is quite fast and sometimes it degrades to 1/4 of the the normal performance. This is very unsatisfying especially on large/long builds due to the high costs.

[martinbonnin]

Sometimes our 30min build takes more than 6h. For an exemple: https://github.com/apollographql/apollo-android/runs/1325891021?check_suite_focus=true
This happens randomly but seems more frequent recently. Maybe it's a different thing but it's hard to tell. When this happens, I can't get any log to display. Is it possible to see the logs in real time or maybe even ssh into the vm to debug?

[bdruth]

Chiming in ... running xcodebuild locally: 266s, on macOS runner: 921s ... that's almost a 3.5x slowdown, with a 10x charge penalty ... that's a bit ridiculous, no? Local: MBP 16 (2.3 GHz 8-Core Intel Core i9).

[detrin]

I can confirm that MacOS performance is 2x-2.5x slower than ubuntu-latest, when testing Julia package https://github.com/detrin/OpenQuantumSystems.jl

[syoder]

Some anecdata from a few of my projects:

Step	time on macOS	time on Ubunutu
setting up the job	11s-27s	3s
checking out the repo	10s-12s	2s-5s

I can run my test suite locally in about 1.5m, but it takes ~6m on Github. When you throw in all the other setup and teardown steps, it's ~10m. If it really were only 20%-30% slower than Ubuntu instances, that would be great.

[locus313]

I am also seeing very slow performance on the macos runners. The recent runs take about an hour to run:
https://github.com/azerothcore/azerothcore-wotlk/actions/runs/449468989
But when testing the build locally on a mac it takes about 12-16 minutes.

I also noticed the queue time is quite long. sometimes the macos builds are queued for 8+ hours.

[kariharju]

Building for Win(self-hosted for signing), Mac and Linux we are seeing build times ~8min for Linux and over 20min for Mac.
On top of this Mac builds still quite often fail to some mysterious network errors, which causes the Mac builds to run even more.
Add to this the 10x cost aspect of Mac builds in GHA and the end result is that just about all the costs for us come from Mac builds and we need to start looking to self-hosted Mac machines because this is not sustainable.

[neil-119]

8 minutes to build on a 2018 MacBook Pro.

GitHub Runner:

Literally unusable with that $0.08/minute pricing.

[mfkrause]

+1. Larger (and probably more expensive) macOS runners won't resolve the underlying problem. There is absolutely no reason for these VMs being up 500% to 1000% slower when they have double the RAM and a full virtual core more than their Linux counterparts.
Adding together the pricing and the performance, macOS runners currently run at about 50-100x the pricing of a Linux runner. Definitely not paying this longterm and if you think we're going to pay even more for a larger macOS runner just so we can maybe get to the performance level of a base Ubuntu runner, you must be out of your mind. We'll buy three M2 Pro Minis at that rate, host the runner ourselves and will save money in the process.

[jozefizso]

We also have performance issues with macOS runners.

MacBook Pro Quad-Core: 60 seconds to build our project
MacBook Pro M2 Pro: 14 seconds
GitHub macOS 12 runner: 9 minutes

With the current pricing model for macOS runners this makes our builds super overpriced.

[ilia-shipitsin]

@jozefizso , can I ask to open an issue on https://support.github.com ?

this particular issue tracker is related to image generation (i.e. software included and so on), we cannot help much with operations

[sidekick-eimantas]

@jozefizso , can I ask to open an issue on https://support.github.com ?

this particular issue tracker is related to image generation (i.e. software included and so on), we cannot help much with operations

I've had an open issue since November of last year regarding this problem

It is clear GitHub are unable to fix this problem and are trying to sweep it under a rug, lest MacOS runners be labeled by the community as unusable

For those with access, and who can afford them - large macos runners do not seem to have this problem

[jozefizso]

@jozefizso , can I ask to open an issue on support.github.com ?

this particular issue tracker is related to image generation (i.e. software included and so on), we cannot help much with operations

Great, we are waiting a year for macOS 13 build machine.

[synth]

This should not be closed. We are also experiencing insanely slow performance of macOS runners. This is one issue among many that result in extremely degraded trust in Github, sadly.

[Ziao]

No official answer or any form of resolution, and the issue is closed? This is really uncool, Github..

[aviramha]

This affects us as well.

[Lessica]

Debugging my first iOS app CI script and fail, over 40 times. 3x slower than my M1 Pro MacBook Pro but $100 gone. Not affordable, at all.