RPi hangs and crashes with lowered idle/min clock on Linux 5.4.51+

[maartenlangeveld]

Creating a bug report/issue

Required Information

• DietPi version
  G_DIETPI_VERSION_CORE=6
  G_DIETPI_VERSION_SUB=31
  G_DIETPI_VERSION_RC=2
  G_GITBRANCH='master'
  G_GITOWNER='MichaIng'
• Distro version
  buster
• Kernel version
  5.4.51+ DietPi-Software | PHP: Fix php-xsl installation #1326
• SBC model
  RPi Zero W (armv6l)
• Power supply used
  Samsung 5V 2A
• SDcard used
  SanDisk 400GB micro SD ULTRA memeory card UHS-I A1

Additional Information (if applicable)

Steps to reproduce

1. G_AGUP
2. G_AGUG
3. reboot
4. Wait couple of minutes and it hangs, time varies. It seems very unpredictable, but after a while it hangs.

Expected behaviour

Every working normal

Actual behaviour

RPi Zero W suddenly hangs after a while

Extra details

I have now reverted to firmware 4.19.118+ #1311, is there a way to block kernel update while have other updates pass using G_AGUP, G_AGUG ?

[MichaIng]

Many thanks for your report. Did you set a lower arm_freq_min in /boot/config.txt or ARM Idle Frequency via dietpi-config?
At least it sounds exactly like what I faced with my RPi2 yesterday: raspberrypi/firmware#1431

[maartenlangeveld]

 ─────────────────────────────────────────────────────
 DietPi CPU Info
 Use dietpi-config to change CPU / performance options
 ─────────────────────────────────────────────────────
 Architecture |     armv6l
 Temperature  |     41'C : 105'F (Optimal temperature)
 Governor     |     ondemand
 Throttle up  |     85% CPU usage

                 Current Freq    Min Freq   Max Freq
 CPU0         |      1000 MHz      300 MHz    1000 MHz

[MichaIng]

Can you paste: grep 'arm_freq' /boot/config.txt

[maartenlangeveld]

root@DietPi:/# grep 'arm_freq' /boot/config.txt
#arm_freq=900
arm_freq_min=300

[MichaIng]

Okay please try to comment arm_freq_min and redo the upgrade, at least if you are okay with the risk of some back and forth 😉.

sed -i 's/^arm_freq_min/#arm_freq_min/' /boot/config.txt

[maartenlangeveld]

Not much risk I have a backup, glad to help you

[MichaIng]

Just got it confirmed, as well on RPi 3+ (not on RPi4): raspberrypi/firmware#1431 (comment)

Hmm, shall we do an emergency hack via MOTD to remove this setting form all non-4 RPis proactively? It took me half a day yesterday to recover everything since the ongoing crashes finally corrupted my SD card (several perl modules files) which broke even dpkg and of course debugging back and forth to finally identify the culprit and assuring that it is not some other corrupted files. In worst case raising some users idle clocks/temps is much less of a problem compared to going through the risk of data loss (not everyone has a backup or SD card clone).

[maartenlangeveld]

Agree, or block kernel update until issues are resolved ?

[MichaIng]

Blocking kernel upgrade may already be too late, removing the setting at least fixes it on next reboot/crash automatically 😄. Also the kernel upgrade prevention has an effect on all systems while the settings removal only those which use a variable frequency governor and actually lowered the idle frequencies. So I'll stay with removing the setting and as well add a link to here to the MOTD to inform why it has been done.

[MichaIng]

MOTD hack in place:

# - https://github.com/MichaIng/DietPi/issues/3690
if [[ $G_HW_MODEL == [0-3] ]]; then

        [[ -w '/boot/dietpi/dietpi-config' ]] && grep -q "G_WHIP_MENU_ARRAY+=('ARM Idle Frequency'" /boot/dietpi/dietpi-config && sed -i "s/G_WHIP_MENU_ARRAY+=('ARM Idle Frequency'/#G_WHIP_MENU_ARRAY+=('ARM Idle Frequency'/" /boot/dietpi/dietpi-config
        if [[ -w '/boot/config.txt' ]] && grep -q '^[[:blank:]]*arm_freq_min' /boot/config.txt; then

                sed -i 's/^[[:blank:]]*arm_freq_min/#arm_freq_min/' /boot/config.txt
                motd='Your idle/min CPU frequency has been reset to system defaults
          to resolve an issue with hanging/crashing system on newest kernel.
          Further info: https://github.com/MichaIng/DietPi/issues/3690'
                return 0

        fi

fi

---

Applied for next DietPi release as well: 9e6152d
Better to do this now and not forget, since we don't know how quickly it's fixed. We can revert the commit once fix has been released.

[maartenlangeveld]

@MichaIng, thanks for your help!

[maartenlangeveld]

@MichaIng, will you let us know when it's safe to lower arm idle clock frequency again?

[MichaIng]

Jep I will. Also subscribe to raspberrypi/firmware#1431 to get new info about the topic ASAP.

[MichaIng]

Fix is available via rpi-update: raspberrypi/firmware#1431 (comment)
This means the next kernel package release will contain it most likely. I re-enabled the feature for DietPi v6.32 already but left commenting the setting during dietpi-update in place. Seems like the best balance between preventing hanging/crashing systems with the update and not disabling a feature that is close to be fixed for all. When user use the option before the fix has been released, at least they can likely associate issues with this option in particular.

[MichaIng]

The issue has not been 100% solved or has been reintroduced with a later commit. Still investigating...
Re-implemented settings removal: 2688194

[MichaIng]

My personal testing issues are solved when not lowering over_voltage anymore, while over_voltage_min can be lowered. Since this is a trail&error value anyway, this is fine.

[maartenlangeveld]

@MichaIng,

Does this mean I can lower the idle frequency to 300 again on my RPi Zero W (5.4.72+ #1356 Thu Oct 22 13:56:00 BST 2020 armv6l GNU/Linux) ? And how do I do this, since dietpi-config won't let me anymore...

[MichaIng]

You can set it in /boot/config.txt directly, uncomment and adjust the arm_freq_min option, or:

G_CONFIG_INJECT 'arm_freq_min=' 'arm_freq_min=300' /boot/config.txt

Reboot required to have it take effect.

dietpi-config will allow it again with next release.