[Ubuntu Bionic 18.04 - AWS AMI] 404 Falco probe binary not found

[JPLachance]

What happened:
I am testing Falco minimal in Kubernetes 1.11 on AWS on top of a Ubuntu 18.04 AMI.

I see the following logs in the probeloader container:

2019-10-23T19:52:34Z [✿]  FALCO_VERSION: 0.17.1
2019-10-23T19:52:34Z [✿]  FALCO_PROBE_URL: 
2019-10-23T19:52:34Z [✿]  FALCO_PROBE_REPO: https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/
2019-10-23T19:52:34Z [✿]  KERNEL_VERSION: 4.15.0-1044-aws
2019-10-23T19:52:34Z [✿]  KERNEL_CONFIG_HASH: e801ad66ab7152e98bb6a89ff383a5f2
2019-10-23T19:52:34Z [✿]  Downloading kernel module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.17.1-x86_64-4.15.0-1044-aws-e801ad66ab7152e98bb6a89ff383a5f2.ko
2019-10-23T19:52:34Z [✿]  Recevied HTTP Status Code: 404
2019-10-23T19:52:34Z [✖]  Non-200 Status code received 404
2019-10-23T19:52:34Z [✖]  Error opening kernel module: /falco-probe.ko
2019-10-23T19:52:34Z [✖]  Error loading module: open /falco-probe.ko: no such file or directory

What you expected to happen:
I expected the probeloader container to download the required version of the Falco probe.

How to reproduce it (as minimally and precisely as possible):

1. Create an instance that uses the following base AMI:

       "source_ami_filter": {
        "filters": {
          "virtualization-type": "hvm",
          "name": "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*",
          "root-device-type": "ebs"
        },
        "owners": [
          "099720109477"
        ],
        "most_recent": true

2. Install things required for Docker and Docker
3. Launch the probeloader container

Anything else we need to know?:
I had a discussion in Slack with @mfdii and he seems pretty aware of the issue.

Environment:

• Falco version : 0.17.1 (minimal)
• Cloud provider or hardware configuration: AWS
• OS: Ubuntu 18.04.03
• Kernel: Linux 4.15.0-1044-aws Add a "falco safe" shell variant #46-Ubuntu SMP Thu Jul 4 13:38:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools : Kubernetes (the minimal image with the latest probeloader image (0.17.1)

[mfdii]

The probe builder wasn't taking into account Ubuntu aws kernels. We are working with the teams that maintain that system to start building probes for these kernels.

[fntlnz]

This process will become more clear once the Falco project owns the infrastructure to build those. For now we are relying on Sysdig (company) infrastructure to build those.

This is tracked down in this issue: falcosecurity/test-infra#53

[mfdii]

This needs to be fixed by adding -aws to this section of the kernel-crawler.py

[mfdii]

Created draios/sysdig#1552 to address this issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[JPLachance]

This issue was not solved. We should have a tag we can put on important issues like this one and the stale bot should stop closing important issues. 😄

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[fntlnz]

@JPLachance - we have the pre-built ubuntu driver now. Can you check by installing Falco from master?

The downloadable .ko files are here https://dl.bintray.com/falcosecurity/driver/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/

[JPLachance]

Hello @fntlnz,

Do we have an up-to-date version of the Falco DaemonSet that uses the probeloader init container somewhere?

I tried it, but I still get the following errors:

jplachance@WKS-001157 kops_daemonset % k logs falco-48h5f probeloader
2020-04-28T20:32:57Z [✿]  FALCO_VERSION: 0.22.1
2020-04-28T20:32:57Z [✿]  FALCO_PROBE_URL: 
2020-04-28T20:32:57Z [✿]  FALCO_PROBE_REPO: https://dl.bintray.com/falcosecurity/driver/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/
2020-04-28T20:32:57Z [✿]  KERNEL_VERSION: 4.15.0-1063-aws
2020-04-28T20:32:57Z [✿]  KERNEL_CONFIG_HASH: ca784ed50e06a5ba87f7b0d3e5b4b210
2020-04-28T20:32:57Z [✿]  Downloading kernel module from https://dl.bintray.com/falcosecurity/driver/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/falco-probe-0.22.1-x86_64-4.15.0-1063-aws-ca784ed50e06a5ba87f7b0d3e5b4b210.ko
2020-04-28T20:32:58Z [✿]  Recevied HTTP Status Code: 404
2020-04-28T20:32:58Z [✖]  Non-200 Status code received 404
2020-04-28T20:32:58Z [✖]  Error opening kernel module: /falco-probe.ko
2020-04-28T20:32:58Z [✖]  Error loading module: open /falco-probe.ko: no such file or directory

Also, by looking at tags in Docker hub for the probeloader image, the last update was two months ago.

Thanks in advance for the help!

[fntlnz]

I can find both the module and bpf probe for your kernel on the new bucket.

• https://dl.bintray.com/falcosecurity/driver/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/falco_ubuntu-aws_4.15.0-1063-aws_67.ko
• https://dl.bintray.com/falcosecurity/driver/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/falco_ubuntu-aws_4.15.0-1063-aws_67.o
  So it looks like the probe loader still needs to be updated to follow the new URL scheme.

I added a note to the today's community call mentioning your comment.

Thanks for checking all this with us before we release!

[leogr]

Note that the probeloader is not supported anymore 👉 falcosecurity/contrib#5

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[leodido]

Now we have a new infrastructure to provide prebuilt Falco kernel modules and Falco eBPF probes (see https://github.com/falcosecurity/test-infra, driverkit folder).

It turns out that we are providing both the Falco kernel module (link) and the Falco eBPF probe (link) for the kernel that @JPLachance reported (on ubuntu-aws).

/close

[poiana]

@leodido: Closing this issue.

In response to this:

Now we have a new infrastructure to provide prebuilt Falco kernel modules and Falco eBPF probes (see https://github.com/falcosecurity/test-infra, driverkit folder).

It turns out that we are providing both the Falco kernel module (link) and the Falco eBPF probe (link) for the kernel that @JPLachance reported.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.