Debian 11 kernel version issue #2374

serhatcalik · 2023-01-25T07:40:01Z

Describe the bug

I was trying to use falco on my kubernetes cluster with drivers, on the init container it's trying to download the kernel module but it's trying to download 5.10.0-20-amd64.o file from download.falco.org domain, on debian it's showing 5.10.0-20 kernel version but it's actually 5.10.158-2 but script is getting uname -r's response so it's trying to download wrong file.

How to reproduce it

On any debian 11 machine with k8s cluster try to install falco.

Expected behaviour

It needs to download 5.10.0-20-amd64.o file from download server.

Screenshots

Environment

Falco version:
0.33.1
System info:
Wed Jan 25 10:37:56 2023: Falco version: 0.33.1 (x86_64) 
Wed Jan 25 10:37:56 2023: Falco initialized with configuration file: /etc/falco/falco.yaml 
Wed Jan 25 10:37:56 2023: Loading rules from file /etc/falco/falco_rules.yaml 
Wed Jan 25 10:37:56 2023: Loading rules from file /etc/falco/falco_rules.local.yaml 
null
Cloud provider or hardware configuration:
OS:

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" 
NAME="Debian GNU/Linux" 
VERSION_ID="11" 
VERSION="11 (bullseye)" 
VERSION_CODENAME=bullseye ID=debian
 HOME_URL="[https://www.debian.org/"](https://www.debian.org/%22) 
 SUPPORT_URL="[https://www.debian.org/support"](https://www.debian.org/support%22) 
 BUG_REPORT_URL="[https://bugs.debian.org/"](https://bugs.debian.org/%22)

Kernel:

Linux ranchm0 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux

Installation method:
Kubernetes

The text was updated successfully, but these errors were encountered:

Andreagit97 · 2023-01-25T09:40:29Z

Uhm this is more an issue with uname, it should report the running kernel version but in this case, it seems to report the wrong one :/ Have you rebooted the machine? Are you sure that you are running the new kernel version? maybe it is just installed but not running

serhatcalik · 2023-01-25T09:59:47Z

Uhm this is more an issue with uname, it should report the running kernel version but in this case, it seems to report the wrong one :/ Have you rebooted the machine? Are you sure that you are running the new kernel version? maybe it is just installed but not running

I tried but still same output from uname -r :(

Andreagit97 · 2023-01-25T13:55:51Z

Uhm really strange :/ Could you provide the output of the following commands?

uname -a
hostnamectl
cat /proc/version
sudo dmesg | grep Linux

FedeDP · 2023-01-25T14:21:53Z

I think i found something: https://lists.debian.org/debian-user/2017/03/msg00485.html

It seems like you are running kernel 5.10.158-2 that is compatible with the kernel ABI used in Debian kernel package 5.10.0-20-amd64.

We did never encountered such an uname -a output :/ it seems a bit crazy to me that uname -r is reporting the "kernel ABI package" and not the real running version, if i understood that correctly.

serhatcalik · 2023-01-25T14:39:26Z

Uhm really strange :/ Could you provide the output of the following commands?

uname -a

hostnamectl

cat /proc/version

sudo dmesg | grep Linux

Updated to the newer kernel for testing but still same

serhatcalik · 2023-01-25T14:42:34Z

I think i found something: https://lists.debian.org/debian-user/2017/03/msg00485.html

It seems like you are running kernel 5.10.158-2 that is compatible with the kernel ABI used in Debian kernel package 5.10.0-20-amd64.

We did never encountered such an uname -a output :/ it seems a bit crazy to me that uname -r is reporting the "kernel ABI package" and not the real running version, if i understood that correctly.

Maybe for debian 11 we can use symbolic link kernels on falco's download page. It's really weird :(

FedeDP · 2023-01-25T14:42:51Z

@serhatcalik yep, the issue is:

every piece of software uses uname -r
on your specific debian installation, uname -r does not return the real running kernel release, but instead returns the Debian kernel package providing the compatible kernel ABI

I am not sure how to tackle it :(

FedeDP · 2023-01-25T14:46:41Z

Maybe for debian 11 we can use symbolic link kernels on falco's download page. It's really weird :(

We might want to do that, but we haven't got any way to link a kernel release to its ABI compatible Debian kernel package version :)
I think we might want to instead parse the kernel version (uname -v) string, and if it contains a semver, we could try both uname -r (so called kernel release) and the semver contained inside the kernel version.

Given we are really near Falco 0.34.0, i will put this for 0.35.0 (perhaps it could get merged in a 0.34.1 patch release, if needed).
/milestone 0.35.0

I will try to fix it and perhaps i can publish a falco-driver-loader image on my own dockerhub repo, if you are willing to test it :)
NOTE: we still haven't got any 5.10.162-1 kernel build for debian, therefore you must wait for next week kernel-crawler sync before actually finding drivers for your new kernel!

serhatcalik · 2023-01-25T15:00:02Z

Maybe for debian 11 we can use symbolic link kernels on falco's download page. It's really weird :(

We might want to do that, but we haven't got any way to link a kernel release to its ABI compatible Debian kernel package version :) I think we might want to instead parse the kernel version (uname -v) string, and if it contains a semver, we could try both uname -r (so called kernel release) and the semver contained inside the kernel version.

Given we are really near Falco 0.34.0, i will put this for 0.35.0 (perhaps it could get merged in a 0.34.1 patch release, if needed). /milestone 0.35.0

I will try to fix it and perhaps i can publish a falco-driver-loader image on my own dockerhub repo, if you are willing to test it :) NOTE: we still haven't got any 5.10.162-1 kernel build for debian, therefore you must wait for next week kernel-crawler sync before actually finding drivers for your new kernel!

I would be glad if you can share your image :)

FedeDP · 2023-01-25T15:07:43Z

I just pushed the 0.33.1_debianfix tag on my own dockerhub: https://hub.docker.com/layers/fededp/falco-driver-loader/0.33.1_debianfix/images/sha256-06db31d8d31bcd80b451329348f63c65f37249b8c9d95032cccd700849919f57?context=repo

As i said, we just need to test that the kernel release it tries to find is the correct one now, but the prebuilt driver won't be available :)
The patch is the one in #2377

serhatcalik · 2023-01-25T15:08:54Z

I just pushed the 0.33.1_debianfix tag on my own dockerhub: https://hub.docker.com/layers/fededp/falco-driver-loader/0.33.1_debianfix/images/sha256-06db31d8d31bcd80b451329348f63c65f37249b8c9d95032cccd700849919f57?context=repo

As i said, we just need to test that the kernel release it tries to find is the correct one now, but the prebuilt driver won't be available :) The patch is the one in #2377

Thank you so much :)

serhatcalik · 2023-01-26T14:06:42Z

@FedeDP Hey I have bad news

Trying to download a prebuilt falco module from https://download.falco.org/driver/3.0.1%2Bdriver/x86_64/falco_debian_5.10.158-2_1.ko but file is;

https://download.falco.org/driver/3.0.1%2Bdriver/x86_64/falco_debian_5.10.158-2-amd64_1.ko

https://download.falco.org/driver/3.0.1%2Bdriver/x86_64/falco_debian_5.10.158-2-amd64_1.o

We just need to add arch :/

FedeDP · 2023-01-26T14:30:52Z

Uh you are right! I forgot about it! Will ping you back once i've got an updated image :)

FedeDP · 2023-01-26T15:06:03Z

I pushed a new version of my falco-driver-loader, care to try?
Thank you!

serhatcalik · 2023-01-29T18:28:17Z

Falco-driver-loader works fine now but the funny part is kernel is getting invalid module format error

I'm starting to hate Debian 11 so much

FedeDP · 2023-01-30T08:55:56Z

Wow, so in the end the 5.10.158 is not your running kernel :/ It seems so, at least.

Updated to the newer kernel for testing but still same

Didn't you upgrade to 5.10.162 though?

serhatcalik · 2023-01-30T09:02:15Z

Downgraded kernel to the 5.10.158 for testing faster but I can upgrade it.

FedeDP · 2023-01-30T09:27:56Z

Oh i see, thank you! It makes sense.
I don't get the error then...Can you share the dmesg output after trying to insmod the driver?

serhatcalik · 2023-01-31T06:47:36Z

disagrees about version of symbol module_layout
This is the output of falco from dmesg

FedeDP · 2023-01-31T08:06:30Z

Wow; it seems the prebuilt module is wrong :/
From SO:

This indicates you have compiled the module against a different version of the kernel than is running. Note that even if the running kernel and kernel source have the same numerical value (e.g. both are 2.6.31-20-server), if the two use different configuration options, you may see this error.

Can you test with eBPF by chance?

serhatcalik · 2023-01-31T08:47:59Z

I tested eBPF and it was not created for debian kernel :(

An error occurred in an event source, forcing termination... 
Error: BPF probe is compiled for 5.10.0-20-rt-amd64, but running version is 5.10.0-20-amd64

FedeDP · 2023-01-31T08:51:22Z

5.10.0-20-rt-amd64, but running version is 5.10.0-20-amd64

O.O
Can you share output of ls /lib/modules/ ?

serhatcalik · 2023-01-31T08:53:41Z

5.10.0-20-amd64 5.10.0-21-amd64
But I'm using 20 not 21 and it's not rt for sure :D

FedeDP · 2023-02-14T08:24:19Z

Hi! Sorry for the long time with no answer.
I am really not sure how to proceed from here :/ Your only chance is to try to let falco-driver-loader building a driver for you, instead of downloading a prebuilt module.
Can you try?
You should attach to a node and run falco-driver-loader --compile. Then, on the same node, just try to start falco and see if it works.
Thank you!

FedeDP · 2023-04-13T11:15:39Z

There have been more and more issues about this:

I think we need a fix.
@serhatcalik were you able to make it work by chance (ie: after upgrading the kernel perhaps)?

serhatcalik · 2023-04-13T11:17:46Z

I stopped using debian in my home cluster started using RHEL. But I can reproduce it for to try fixing it on debian we can work on it together :)

rao-hassan-1 · 2023-04-13T11:44:52Z

I am also affected by this bug. Please provide a resolution for this. I have opened a pull request for the issue I was seeing but that won't fix the problem

FedeDP · 2023-04-13T11:46:24Z

@serhatcalik that was the correct move indeed 🤣
I will test with the help of @rao-hassan-1 :)

FedeDP · 2023-04-13T11:52:03Z

I just pushed a new image: fededp/falco-driver-loader:0.34.1_fixed.
It is the same as falcosecurity/falco-driver-loader:0.34.1 image but with #2377 fix applied.

serhatcalik added the kind/bug label Jan 25, 2023

poiana added this to the 0.35.0 milestone Jan 25, 2023

FedeDP mentioned this issue Jan 25, 2023

fix(scripts): fix falco-driver-loader for some debian kernels. #2377

Merged

This was referenced Apr 13, 2023

Add new driver kit config falcosecurity/test-infra#1097

Closed

[Feature] compile driver for Digitalocean k8s and provide it to download.falco.org falcosecurity/kernel-crawler#114

Closed

poiana closed this as completed in #2377 Apr 27, 2023

This was referenced May 11, 2023

chore: add 4.0.0+driver support for kernel-5.10.0-0.bpo.15 x86_64 falcosecurity/test-infra#1131

Closed

feat: ✨ New drivers falcosecurity/test-infra#1157

Closed

glpfloss mentioned this issue Jun 7, 2023

Debian pre-built modules failing #2625

Closed

FedeDP mentioned this issue Jun 12, 2023

(Debian/Ubuntu) Attempt to download a version which doesn't exist falcosecurity/charts#496

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Debian 11 kernel version issue #2374

Debian 11 kernel version issue #2374

serhatcalik commented Jan 25, 2023 •

edited by FedeDP

Loading

Andreagit97 commented Jan 25, 2023

serhatcalik commented Jan 25, 2023

Andreagit97 commented Jan 25, 2023

FedeDP commented Jan 25, 2023

serhatcalik commented Jan 25, 2023 •

edited

Loading

serhatcalik commented Jan 25, 2023

FedeDP commented Jan 25, 2023

FedeDP commented Jan 25, 2023

serhatcalik commented Jan 25, 2023

FedeDP commented Jan 25, 2023

serhatcalik commented Jan 25, 2023

serhatcalik commented Jan 26, 2023

FedeDP commented Jan 26, 2023

FedeDP commented Jan 26, 2023

serhatcalik commented Jan 29, 2023 •

edited

Loading

FedeDP commented Jan 30, 2023

serhatcalik commented Jan 30, 2023

FedeDP commented Jan 30, 2023

serhatcalik commented Jan 31, 2023 •

edited

Loading

FedeDP commented Jan 31, 2023

serhatcalik commented Jan 31, 2023 •

edited

Loading

FedeDP commented Jan 31, 2023

serhatcalik commented Jan 31, 2023 •

edited

Loading

FedeDP commented Feb 14, 2023

FedeDP commented Apr 13, 2023

serhatcalik commented Apr 13, 2023

rao-hassan-1 commented Apr 13, 2023 •

edited

Loading

FedeDP commented Apr 13, 2023

FedeDP commented Apr 13, 2023

Debian 11 kernel version issue #2374

Debian 11 kernel version issue #2374

Comments

serhatcalik commented Jan 25, 2023 • edited by FedeDP Loading

Andreagit97 commented Jan 25, 2023

serhatcalik commented Jan 25, 2023

Andreagit97 commented Jan 25, 2023

FedeDP commented Jan 25, 2023

serhatcalik commented Jan 25, 2023 • edited Loading

serhatcalik commented Jan 25, 2023

FedeDP commented Jan 25, 2023

FedeDP commented Jan 25, 2023

serhatcalik commented Jan 25, 2023

FedeDP commented Jan 25, 2023

serhatcalik commented Jan 25, 2023

serhatcalik commented Jan 26, 2023

FedeDP commented Jan 26, 2023

FedeDP commented Jan 26, 2023

serhatcalik commented Jan 29, 2023 • edited Loading

FedeDP commented Jan 30, 2023

serhatcalik commented Jan 30, 2023

FedeDP commented Jan 30, 2023

serhatcalik commented Jan 31, 2023 • edited Loading

FedeDP commented Jan 31, 2023

serhatcalik commented Jan 31, 2023 • edited Loading

FedeDP commented Jan 31, 2023

serhatcalik commented Jan 31, 2023 • edited Loading

FedeDP commented Feb 14, 2023

FedeDP commented Apr 13, 2023

serhatcalik commented Apr 13, 2023

rao-hassan-1 commented Apr 13, 2023 • edited Loading

FedeDP commented Apr 13, 2023

FedeDP commented Apr 13, 2023

serhatcalik commented Jan 25, 2023 •

edited by FedeDP

Loading

serhatcalik commented Jan 25, 2023 •

edited

Loading

serhatcalik commented Jan 29, 2023 •

edited

Loading

serhatcalik commented Jan 31, 2023 •

edited

Loading

serhatcalik commented Jan 31, 2023 •

edited

Loading

serhatcalik commented Jan 31, 2023 •

edited

Loading

rao-hassan-1 commented Apr 13, 2023 •

edited

Loading