Kubeadm init 1700MB limit bug #2365

abelbarrera15 · 2020-12-16T14:24:23Z

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:57:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/arm"}

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:51:19Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/arm"}

Cloud provider or hardware configuration:

Raspberry Pi 4 Model B 8GB

OS (e.g. from /etc/os-release):

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Kernel (e.g. uname -a):

Linux k8s-master 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux

Others:

N/A

What happened?

I tried to run sudo kubeadm init --pod-network-cidr=10.244.0.0/16 on the master node which had 6.9GB available of free memory as per the below:
pi@k8s-master:~ $ free -h
total used free shared buff/cache available
Mem: 7.7Gi 112Mi 6.9Gi 8.0Mi 679Mi 7.3Gi
Swap: 0B 0B 0B

And it threw the following error:

[ERROR Mem]: the system RAM (1 MB) is less than the minimum 1700 MB

Which is obviously incorrect. There is 6.9 GB of memory available not < 1MB

What you expected to happen?

kubeadm init runs with no issues and no errors minus small pre-flight checks that are warnings.

How to reproduce it (as minimally and precisely as possible)?

Run sudo kubeadm init --pod-network-cidr=10.244.0.0/16 on a Raspberry Pi 4 Model B with swap disabled/uninstalled and 8GBs of total memory 6.9 of which are available.

Anything else we need to know?

Installation succeeds if I add "--ignore-preflight-errors=Mem"

The text was updated successfully, but these errors were encountered:

neolit123 · 2020-12-16T17:25:17Z

Installation succeeds if I add "--ignore-preflight-errors=Mem"

at least you can skip the check, which is good.

Linux k8s-master 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux

i cannot reproduce this problem on:

$ lsb_release -a && uname -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal
Linux ubuntu 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 11:33:25 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

our code uses the Golang standard library to detect memory, which is calling the Linux sysinfo (syscall).
i suspect a bug in Golang here, alternatively if sysinfo is returning wrong values on this Linux distribution that can be quite bad for other applications too.

note this is the first report we get about this problem, and we do have a number of RPI / kubeadm users.
can you help us debug this?

if you install Go for your architecture locally you can test this small snippet. call the file main.go and run it with go run main.go.

package main

import (
	"fmt"
	"syscall"
)

func main() {
	info := syscall.Sysinfo_t{}
	err := syscall.Sysinfo(&info)
	if err != nil {
		fmt.Printf("failed to get system info: %v\n", err)
		return
	}
	fmt.Printf("actual MB: %v, bytes: %v\n", uint64(info.Totalram) / 1024 / 1024, info.Totalram)
}

A) what output do you get if you run this locally on that machine?

also if you install gcc (on Debian that is apt-get install gcc) and call this file main.c what output do you get from gcc main.c && ./a.out?

#include <sys/sysinfo.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

int main(void) {
        struct sysinfo info = {};
        if (sysinfo(&info) != 0) {
                printf("errno %d: %s\n", errno, strerror(errno));
                return 1;
        }
        printf("mem_unit (bytes): %d, totalram: %lu\n", info.mem_unit, info.totalram);
        return 0;
}

B) what output do you get if you run this locally on that machine?

please share the output of A) and B).

abelbarrera15 · 2020-12-16T18:44:23Z

Happy to help debug!

Output of A)

actual MB: 1, bytes: 2016237

Output of B)

totalram (bytes): 2016237

neolit123 · 2020-12-16T19:12:00Z

thanks for running the apps.

totalram (bytes): 2016237

this is not great as the Linux kernel is reporting that you only have 2 million bytes of RAM or around 2 MB.

i don't know what free -h is using but the sysinfo call (example in C above) should be returning the correct memory of the system. you can try contacting Raspbian showing them the C code above and see what they think about this problem:
https://www.raspbian.org/RaspbianBugs

you can also check with other Raspbian users for feedback.

i'm going to close this as this is not a kubeadm bug, but feel free to get back to us with a response (once you find more about it).
thank you

/close

k8s-ci-robot · 2020-12-16T19:12:14Z

@neolit123: Closing this issue.

In response to this:

thanks for running the apps.

totalram (bytes): 2016237

this is not great as the Linux kernel is reporting that you only have 2 million bytes of RAM or slightly more than 2 MB.

i don't know what free -h is using but the sysinfo call (example in C above) should be returning the correct memory of the system. you can try contacting Raspbian showing them the C code above and see what they think about this problem:
https://www.raspbian.org/RaspbianBugs

you can also check with other Rasbian users for feedback.

i'm going to close this as this is not a kubeadm bug, but feel free to get back to us with a response.
thank you

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

neolit123 · 2020-12-16T19:15:39Z

i've also shared this issue in the kubernetes slack channel #arm64 to see if someone has see this.

abelbarrera15 · 2020-12-17T14:12:06Z

@neolit123 , I'll go ahead and share w/ raspbian etc -- thanks for the help.

plugwash · 2020-12-19T14:23:38Z

According to https://man7.org/linux/man-pages/man2/sysinfo.2.html

"In the above structure, sizes of the memory and swap fields are given as multiples of mem_unit bytes."

So it seems to get the total memory in bytes, you need to multiply totalram by mem_unit.

neolit123 · 2020-12-19T14:43:21Z

What units do you get in your case? Please see the updated C code above, recompile it with gcc and show the output.

plugwash · 2020-12-19T14:47:16Z

Just done some testing to confirm this.

On a 4GB Pi4 I get

mem_unit (bytes): 1, totalram: 3982721024

But on an 8GB Pi4 I get

mem_unit (bytes): 4096, totalram: 2000948

So it looks to me like Linux switches memory units when it needs to do so to avoid overflow, i.e. on 32-bit systems with more than 4GB of ram.

neolit123 · 2020-12-19T14:59:10Z

Thanks for the details. I actually found about this field after my initial tests above, but it was 1 and i did not think about overflow. Looks like we need to multiply by mem units regardless and hopefully the golang field Unit is populated correctly. If anyone wishes to send a pr for that in kubeadm please go ahead. /reopen

k8s-ci-robot · 2020-12-19T14:59:22Z

@neolit123: Reopened this issue.

In response to this:

Thanks for the details. I actually found about this field after my initial
tests above, but it was 1 and i did not think about overflow.

Looks like we need to multiply by mem units regardless and hopefully the
golang field Unit is populated correctly.

If anyone wishes to send a pr for that in kubeadm please go ahead.

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

plugwash · 2020-12-19T15:15:56Z

It looks like the offending code is in https://github.com/kubernetes/kubernetes/blob/cea1d4e20b4a7886d8ff65f34c6d4f95efcb4742/cmd/kubeadm/app/preflight/checks_linux.go

"actual := uint64(info.Totalram) / 1024 / 1024"

I believe should be changed to

"actual := uint64(info.Totalram) * info.Unit / 1024 / 1024"

I'm not a kubernates user or developer though (I was just alerted to this issue through the filing of a raspbian bug), so I can't easilly test this myself.

abelbarrera15 · 2020-12-19T15:20:40Z

@plugwash , @neolit123 -- thanks. I follow. Let me try to test and put a PR in today :)

…ght 15/03/2021 complained for too old firmware - updated kubernetes version to latest stable (1.20.4) - updated flannel to latest stable (0.13.0) - fixed haproxy.cfg bug where haproxy complained about equal backend and frontend names - fixed kubeadm init error wrongly reporting insufficient memory on pi 4B 8GiB (known bug #kubernetes/kubeadm/issues/2365)

neolit123 added area/UX priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. area/external kind/bug Categorizes issue or PR as related to a bug. labels Dec 16, 2020

neolit123 added this to the v1.21 milestone Dec 16, 2020

k8s-ci-robot closed this as completed Dec 16, 2020

k8s-ci-robot reopened this Dec 19, 2020

abelbarrera15 mentioned this issue Dec 19, 2020

32bit > 4GB integer overflow edge case fix kubernetes/kubernetes#97403

Merged

neolit123 added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Dec 21, 2020

k8s-ci-robot closed this as completed in kubernetes/kubernetes#97403 Dec 21, 2020

aojea mentioned this issue Jan 3, 2021

kubelet not work in raspbian version 1.20.0. kubernetes/kubernetes#97646

Closed

msfukui mentioned this issue Feb 13, 2021

Switching the OS used from Raspberry Pi OS to Ubuntu msfukui/pitanetes#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubeadm init 1700MB limit bug #2365

Kubeadm init 1700MB limit bug #2365

abelbarrera15 commented Dec 16, 2020

neolit123 commented Dec 16, 2020 •

edited

Loading

abelbarrera15 commented Dec 16, 2020 •

edited

Loading

neolit123 commented Dec 16, 2020 •

edited

Loading

k8s-ci-robot commented Dec 16, 2020

neolit123 commented Dec 16, 2020 •

edited

Loading

abelbarrera15 commented Dec 17, 2020

plugwash commented Dec 19, 2020

neolit123 commented Dec 19, 2020 via email

plugwash commented Dec 19, 2020

neolit123 commented Dec 19, 2020 via email

k8s-ci-robot commented Dec 19, 2020

plugwash commented Dec 19, 2020

abelbarrera15 commented Dec 19, 2020

Kubeadm init 1700MB limit bug #2365

Kubeadm init 1700MB limit bug #2365

Comments

abelbarrera15 commented Dec 16, 2020

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

neolit123 commented Dec 16, 2020 • edited Loading

abelbarrera15 commented Dec 16, 2020 • edited Loading

neolit123 commented Dec 16, 2020 • edited Loading

k8s-ci-robot commented Dec 16, 2020

neolit123 commented Dec 16, 2020 • edited Loading

abelbarrera15 commented Dec 17, 2020

plugwash commented Dec 19, 2020

neolit123 commented Dec 19, 2020 via email

plugwash commented Dec 19, 2020

neolit123 commented Dec 19, 2020 via email

k8s-ci-robot commented Dec 19, 2020

plugwash commented Dec 19, 2020

abelbarrera15 commented Dec 19, 2020

neolit123 commented Dec 16, 2020 •

edited

Loading

abelbarrera15 commented Dec 16, 2020 •

edited

Loading

neolit123 commented Dec 16, 2020 •

edited

Loading

neolit123 commented Dec 16, 2020 •

edited

Loading