[MacOS] psutil.cpu_freq() broken on Apple M1

[Lomanic]

Summary

• OS: MacOS
• Architecture: Apple M1
• Psutil version: N/A
• Python version: N/A
• Type: core

Description

Sorry that I couldn't provide some of the info in the template as I don't own an Apple M1 device, but it is highly suspected that calling psutil.cpu_freq() on Apple M1 is failing as the hw.cpufrequency sysctl call (used here) was removed on this arch (can be checked with sysctl hw.cpufrequency on any Apple M1 device, while it is working on amd64).

See shirou/gopsutil#999 and shirou/gopsutil#1000

In shirou/gopsutil#999 a fix was suggested but I couldn't confirm it.

psutil user-base is probably bigger than gopsutil's so I hope this will help both projects find a solution or confirm @shoenig fix in his PR.

[giampaolo]

Mmm... is "hw.cpufrequency" the only one that fails? Later on we also use "hw.cpufrequency_min" and "hw.cpufrequency_max". Do they fail as well?

Sorry that I couldn't provide some of the info in the template

It's OK, no worries.

[tmm1]

None are present.

tmm1@Mac-mini ~ % sysctl hw.cpufrequency
tmm1@Mac-mini ~ % sysctl hw.cpufrequency_min
tmm1@Mac-mini ~ % sysctl hw.cpufrequency_max
tmm1@Mac-mini ~ %

[giampaolo]

For a possible solution see #975 pointing to https://stackoverflow.com/a/14355710/376587 (use sysctl + HW_CPU_FREQ or sysctl + KERN_CLOCKRATE). It looks like both approaches report the advertised frequency (it never changes), same as the current code.

[tmm1]

tmm1@Mac-mini /tmp % cat freq.c 

#include <stdio.h>
#include <sys/sysctl.h>

int main() {
        int mib[2];
        unsigned int freq;
        size_t len;

        mib[0] = CTL_HW;
        mib[1] = HW_CPU_FREQ;
        len = sizeof(freq);
        sysctl(mib, 2, &freq, &len, NULL, 0);

        printf("%u\n", freq);

      mib[0] = CTL_KERN;
      mib[1] = KERN_CLOCKRATE;
      struct clockinfo clockinfo;
      len = sizeof(clockinfo);
      sysctl(mib, 2, &clockinfo, &len, NULL, 0);
      printf("clockinfo.hz: %d\n", clockinfo.hz);
      printf("clockinfo.tick: %d\n", clockinfo.tick);

        return 0;
}
tmm1@Mac-mini /tmp % gcc -o freq freq.c 
tmm1@Mac-mini /tmp % ./freq
0
clockinfo.hz: 100
clockinfo.tick: 10000

tmm1@Mac-mini /tmp % sysctl kern.clockrate
kern.clockrate: { hz = 100, tick = 10000, tickadj = 0, profhz = 100, stathz = 100 }

[giampaolo]

According to doc we should return 0 if min and max frequency cannot be determined, so we can ignore those on M1. This was tracked in #1456 (we return error instead of 0) which should also be fixed.

[giampaolo]

Fixed by #1895.

[Lomanic]

psutil still uses hw.cpufrequency from what I see (but using the mib mechanism), this is still returning 0 on Apple M1 as per #1892 (comment), so I'm quite confused by the fix. Is the fix that now it returns 0 instead of returning an error?

[giampaolo]

Oh you're right. I misread #1892 (comment).

[giampaolo]

The problem is that on my virtualized macOS I get this:

HW_CPU_FREQ: 2590000000
KERN_CLOCKRATE -> clockinfo.hz: 100

...so it appears they are 2 different things.

[ccrvlh]

Same happening in Ubuntu 20.04

[dehydratedpotato]

This is my first time posting in the psutil repo, but I have been trying to figure out how to pull the current CPU frequency on Apple Silicon for a few months now, and I've finally managed to create a script that can do that.

In case anyone wants to take a look at it, the entire project is here, but this is the important part.

Good day 👍

[piskvorky]

I see this wasn't really fixed in #1895 – I'm still getting:

>>> psutil.__version__
5.9.0
>>> psutil.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

on M1 (MBP 2021, OS 12.2.1). Let me know how I could help with testing!

[dbwiddis]

I just took a grand tour of almost every GitHub issue regarding this, and think I've cracked this nut.

This comment pointing to the clockrate.hz field is relevant, but the problem is what to multiply it by. The answer appears to be the time base frequency.

On M1s, sysctl hw.tbfrequency returns 24,000,000 which when multiplied by 100 Hz gives the expected 2.4 GHz. However, that sysctl is not reliable on Intel hardware: on my Intel chip I get 1,000,000,000.

This would obviously need testing by anyone with an M1 (or M2?) but I think the general logic should be:

• check if sysctl hw.cpufrequency returns. If so, use it.
• If not, multiply sysctl hw.tbfrequency by sysctl kern.clockrate (hz).

[shoenig]

@dbwiddis FYI I went down this rabbit hole in https://github.com/shirou/gopsutil/pull/999/files

[dbwiddis]

@dbwiddis FYI I went down this rabbit hole in https://github.com/shirou/gopsutil/pull/999/files

Indeed, and yours was one of the ones I visited in my grand tour, but I wasn't sure why you gave up on it; in the example given (a Hackintosh VM) the getFrequencyIntel() call (using hw.cpufrequency) succeeded, so getFrequencyArm() wouldn't. And there appears to be a comment here that implied the test using the HW_CPU_FREQ sysctl was the same thing, but it wasn't. The HW_CPU_FREQ MIB approach is essentially the same ashw.cpufrequency.

However, that said, I think I'm back to agreeing with you that it can't be done.

There are ample ioreg -l outputs on ARM hardware out there for various platforms (iPhones, iPads) and the timebase-frequency appears consistent. See this gist for an iPhone 4 with a known 800 MHz frequency. (Note the byte arrays values shown are little-endian.)

    | +-o cpu0@0  <class IOPlatformDevice, registered, matched, active, busy 0, retain 12>
    | |   {
    | |     "timebase-frequency" = <00366e01>
    | |     "clock-frequency" = <0008af2f>
    | |     "compatible" = <"ARM,cortex-a9","ARM,v7">
    | |   }

Time base frequency 0x016e3600 = 24,000,000 which disproves my proposal.... it's just the useless default.

However, clock-frequency DOES work:
Clock frequency 0x2faf0800 = 800,000,000 = the 800 MHz frequency of the Cortex a9.

But clock-frequency isn't reliable. Those same numbers on an M1:

    | +-o cpu0@0  <class IOPlatformDevice, id 0x10000010f, registered, matched, active, busy 0 (175 ms), retain 8>
    | | | {
    | | |   "clock-frequency" = <00366e01>
    | | |   "timebase-frequency" = <00366e01>
    | | |   "compatible" = <"apple,icestorm","ARM,v8">
    | | | }

So, yeah. Hardcoding 2.4 GHz seems to be the thing to do.

[johnny12150]

I see this wasn't really fixed in #1895 – I'm still getting:

>>> psutil.__version__
5.9.0
>>> psutil.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

on M1 (MBP 2021, OS 12.2.1). Let me know how I could help with testing!

Still facing this issue with M2 Max and psutil 5.9.4

>>> import psutil
>>> psutil.cpu_freq()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/john/miniconda3/envs/pytorch-m2/lib/python3.8/site-packages/psutil/__init__.py", line 1864, in cpu_freq
    ret = _psplatform.cpu_freq()
  File "/Users/john/miniconda3/envs/pytorch-m2/lib/python3.8/site-packages/psutil/_psosx.py", line 179, in cpu_freq
    curr, min_, max_ = cext.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

[k-nearest-neighbor]

Same on 5.9.4 on an M1 Pro with Ventura 13.2.1

>>> psutil.__version__
'5.9.4'
>>> psutil.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

[snOm3ad]

@giampaolo I have seen this issue fixed in other OSS projects. For example, osquery retrieves the voltage-states5-sram from the ARM IO device in the IO Kit registry then walks through the array, 4-bytes at a time and returns the maximum as the CPU frequency:

https://github.com/osquery/osquery/blob/042cb999f298e81da610bd630ae24bcf0740296c/osquery/tables/system/darwin/cpu_info.cpp#L124-L128

On my machine, this leads a max frequency of 3.5Ghz which matches what I see in powermetrics. I can prepare a PR with these changes, I just need some guidance in terms of where to put the IO Kit traversal code?

[giampaolo]

Sure, please go for it!

I just need some guidance in terms of where to put the IO Kit traversal code?

I would say psutil/arch/osx/cpu.c. We already use IOKit elsewhere but we didn't put it into a specific/separate file:

$ grep.py Kit
psutil/_psutil_osx.c                                                                                                                                                                          
 31   │ #include <IOKit/IOKitLib.h>
 32   │ #include <IOKit/storage/IOBlockStorageDriver.h>
 33   │ #include <IOKit/storage/IOMedia.h>
 34   │ #include <IOKit/IOBSD.h>
 35   │ #include <IOKit/ps/IOPowerSources.h>
 36   │ #include <IOKit/ps/IOPSKeys.h>

[shoenig]

For reference here's a simple C version we use for Go
https://github.com/shoenig/go-m1cpu/blob/main/cpu.go

[dbwiddis]

And here's a Java/JNA based solution based on this Issue/thread:
https://github.com/oshi/oshi/blob/e78a0ffb32d5bc2fe6943b903ed5eeb97f949fc5/oshi-core/src/main/java/oshi/hardware/platform/mac/MacCentralProcessor.java#L322-L357

[snOm3ad]

PR is up, could someone please review it? Thank you @shoenig and @dbwiddis both your examples where useful.

[SansGuidon]

I'm also interested by thix fix, as on M1, without that, no way it seems to use BabyAGI :-) (https://github.com/oliveirabruno01/babyagi-asi)

 ✘  py[babyagi-asi]  ~/c/babyagi-asi   main   python babyagi.py
Traceback (most recent call last):
  File "/Users/xxx/Code/babyagi-asi/babyagi.py", line 1, in <module>
    import openai, prompts, consts, pinecone, os, subprocess, tiktoken, json
  File "/Users/xxx/Code/babyagi-asi/prompts.py", line 9, in <module>
    I am running on a {platform.system()} {platform.architecture()[0]} system with {round(psutil.virtual_memory().total / (1024 ** 3), 2)} GB RAM and a {psutil.cpu_freq().current/1000 if psutil.cpu_freq() else "unknown"} GHz CPU. I am using OpenAI API. I must remember to use '|' instead of '&&' or '&' in my commands if using windows' cmd or pws.
  File "/Users/xxx/Library/Python/3.9/lib/python/site-packages/psutil/__init__.py", line 1864, in cpu_freq
    ret = _psplatform.cpu_freq()
  File "/Users/xxx/Library/Python/3.9/lib/python/site-packages/psutil/_psosx.py", line 179, in cpu_freq
    curr, min_, max_ = cext.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

However this works: in the requirements.txt file, replace the line with that dependency, e.g

psutil==5.9.4

with

git+https://github.com/snOm3ad/psutil.git@fix-cpu-freq-apple-silicon

then pip install ...