Why are all k8s.. fields are null? #629

tomklino · 2019-05-29T12:33:05Z

I've added to the output of all rules the fields:

k8s.pod.name
k8s.pod.id
k8s.pod.labels

But every time an event is logged these values come out as null

for example:

{"output":"12:20:10.995144237: Notice A shell was spawned in a container with an attached terminal (user=root k8s_falco_falco-b7fr6_security_e59d29ac-8208-11e9-bd23-1229902c6cc4_0 (id=1231e98ac26a) shell=bash parent=<NA> cmdline=bash terminal=34816 podid=<NA> podlabels=<NA>)","priority":"Notice","rule":"Terminal shell in container","time":"2019-05-29T12:20:10.995144237Z", "output_fields": {"container.id":"1231e98ac26a","container.name":"k8s_falco_falco-b7fr6_security_e59d29ac-8208-11e9-bd23-1229902c6cc4_0","evt.time":1559132410995144237,"k8s.pod.id":null,"k8s.pod.labels":null,"proc.cmdline":"bash","proc.name":"bash","proc.pname":null,"proc.tty":34816,"user.name":"root"}}

I'm have looked through the docs to see if there is some configuration I needed to set to get this to work, but havn't found anything

The text was updated successfully, but these errors were encountered:

tomklino · 2019-05-30T06:40:32Z

so turns out I left those arguments out by mistake:

args: [ "/usr/bin/falco", "--cri", "/host/run/containerd/containerd.sock", "-K", "/var/run/secrets/kubernetes.io/serviceaccount/token", "-k", "https://$(KUBERNETES_SERVICE_HOST)", "-pk"]

tomklino · 2019-05-30T11:35:35Z

I have to re-open that issue.
Even now with the arguments for Kubernetes enabled, for some of the events, all k8s.* fields are null. They are critical in my case for filtering events like privileged containers, and I need to understand if this is a bug or if there is something additional in the configurations/setup I need to add.

Here is an example:

11:07:47.478740297: Warning Sensitive file opened for reading by non-trusted program (user=root program=executor file=/etc/shadow parent=executor gparent=sh ggparent=<NA> gggparent=<NA> podid=<NA> podlabels=<NA> containerimage=gcr.io/kaniko-project/executor:debug) k8s.ns=<NA> k8s.pod=<NA> container=055736f1f238 k8s.ns=<NA> k8s.pod=<NA> container=055736f1f238 k8s.ns=<NA> k8s.pod=<NA> container=055736f1f238

And sometimes, for the same event, k8s.* fields are as expected:

10:43:42.051510527: Warning Sensitive file opened for reading by non-trusted program (user=root program=executor file=/etc/pam.d/other parent=executor gparent=sh ggparent=<NA> gggparent=<NA> podid=c2b94f06-82c7-11e9-bd23-1229902c6cc4 podlabels= containerimage=gcr.io/kaniko-project/executor:debug) k8s.ns=default k8s.pod=runner-tgajjktl-project-277-concurrent-0g6g7m container=deb108bd49d2 k8s.ns=default k8s.pod=runner-tgajjktl-project-277-concurrent-0g6g7m container=deb108bd49d2 k8s.ns=default k8s.pod=runner-tgajjktl-project-277-concurrent-0g6g7m container=deb108bd49d2

fntlnz · 2019-06-11T22:08:10Z

@tomklino can you give us more info about your environment?

Environment:

Falco version (use falco --version):
System info
Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Install tools (e.g. in kubernetes, rpm, deb, from source):
Others:

fntlnz · 2019-06-11T22:08:31Z

/triage support

tomklino · 2019-06-12T07:47:09Z

I would love to:

falco version 0.15.0

# falco --support | jq .system_info
{
  "machine": "x86_64",
  "nodename": "falco-wvxqm",
  "release": "4.15.0-1039-aws",
  "sysname": "Linux",
  "version": "#41-Ubuntu SMP Wed May 8 10:43:54 UTC 2019"
}

We are running kubernetes on aws - falco is running as a daemonset in a privileged container and tolerations to allow it to run on master nodes
Using the falco image as provided (falcosecurity/falco:0.15.0) for the container image

# uname -a
Linux falco-wvxqm 4.15.0-1039-aws #41-Ubuntu SMP Wed May 8 10:43:54 UTC 2019 x86_64 GNU/Linux

# cat /host/etc/os-release
Defaulting container name to falco.
Use 'kubectl describe pod/falco-wvxqm -n security' to see all of the containers in this pod.
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

tcotav · 2019-06-14T22:25:06Z

FWIW -- I'm also seeing this on AWS kubernetes/EKS. Run with daemonset.

# falco --version
falco version 0.15.3

# falco --support | jq .system_info
Fri Jun 14 22:20:12 2019: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Jun 14 22:20:12 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Jun 14 22:20:13 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Fri Jun 14 22:20:13 2019: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
{
  "machine": "x86_64",
  "nodename": "falco-daemonset-wqtdh",
  "release": "4.14.106-97.85.amzn2.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Fri Mar 15 17:07:54 UTC 2019"
}

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

# uname -a
Linux falco-daemonset-wqtdh 4.14.106-97.85.amzn2.x86_64 #1 SMP Fri Mar 15 17:07:54 UTC 2019 x86_64 GNU/Linux

rastut · 2019-06-18T12:19:57Z

Here the same, running on GKE 1.12.6

falco version 0.15.3

{
  "machine": "x86_64",
  "nodename": "gke-standard-cluster-2-default-pool-3e4415e6-gpnc",
  "release": "4.14.119+",
  "sysname": "Linux",
  "version": "#1 SMP Tue May 14 21:04:23 PDT 2019"
}

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

uname -a
Linux gke-standard-cluster-2-default-pool-3e4415e6-gpnc 4.14.119+ #1 SMP Tue May 14 21:04:23 PDT 2019 x86_64 GNU/Linux

Some times the events come with the data populated but sometimes doesn't contain the k8s data:

{"output":"12:17:41.118240629: Notice A shell was spawned in a container with an attached terminal (user=root k8s.ns=default k8s.pod=nginx-falco container=ac0705e08932 shell=bash parent=docker-runc cmdline=bash terminal=34816)","priority":"Notice","rule":"Terminal shell in container","time":"2019-06-18T12:17:41.118240629Z", "output_fields": {"container.id":"ac0705e08932","evt.time":1560860261118240629,"k8s.ns.name":"default","k8s.pod.name":"nginx-falco","proc.cmdline":"bash","proc.name":"bash","proc.pname":"docker-runc","proc.tty":34816,"user.name":"root"}}

{"output":"12:17:49.061394964: Notice A shell was spawned in a container with an attached terminal (user=root k8s.ns=<NA> k8s.pod=<NA> container=db69b394c785 shell=bash parent=docker-runc cmdline=bash terminal=34816)","priority":"Notice","rule":"Terminal shell in container","time":"2019-06-18T12:17:49.061394964Z", "output_fields": {"container.id":"db69b394c785","evt.time":1560860269061394964,"k8s.ns.name":null,"k8s.pod.name":null,"proc.cmdline":"bash","proc.name":"bash","proc.pname":"docker-runc","proc.tty":34816,"user.name":"root"}}

tcotav · 2019-06-19T18:14:55Z

fwiw, switching my ds deploy yaml to use :dev and I got the info

 18:11:58.187352558: Notice A shell was spawned in a container with an attached terminal (user=root k8s.ns=default k8s.pod=falco-daemonset-ws254 container=6a5c0fa2afde shell=bash parent=docker-runc cmdline=bash terminal=34816) k8s.ns=default k8s.pod=falco-daemonset-ws254 container=6a5c0fa2afde

18:12:07.147066927: Error File below /etc opened for writing (user=root command=touch normal.txt parent=bash pcmdline=bash file=/etc/normal.txt program=touch gparent=<NA> ggparent=<NA> gggparent=<NA>) k8s.ns=default k8s.pod=falco-daemonset-ws254 container=6a5c0fa2afde k8s.ns=default k8s.pod=falco-daemonset-ws254 container=6a5c0fa2afde

fntlnz · 2019-07-30T20:27:01Z

@tomklino as @tcotav is reporting this was solved in #731 would you be able to confirm it fixes for your case too?

tomklino · 2019-08-07T07:14:25Z

I will update Falco on our environment to 0.17.0 later this week and will get back to you

And thank you for checking in with me :-)

fntlnz · 2019-08-14T15:35:09Z

/close

I'm closing this but @tomklino please continue the discussion if you find anything new.

poiana · 2019-08-14T15:35:10Z

@fntlnz: Closing this issue.

In response to this:

/close

I'm closing this but @tomklino please continue the discussion if you find anything new.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tomklino · 2019-08-15T15:46:38Z

Hi guys, so I updated to 0.17.0 a few days ago and let it run for a while.

I'm sorry to be the bearer of bad news, but that is still happening in 0.17.0:

14:42:35.118505676: Notice Privileged container started (user=<NA> k8s.ns=<NA> k8s.pod=<NA> container=745ebd332ed5 image=nuvo/orca:0.9.3 podid=<NA> podlabels=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=745ebd332ed5 k8s.ns=<NA> k8s.pod=<NA> container=745ebd332ed5

If there's any info you need from me that can help I'll be happy to get it for you

leodido · 2019-08-22T15:36:44Z

Ok, reopening it and adding to milestone 0.18.0.

/kind bug

stale · 2019-12-02T11:18:38Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tomklino · 2019-12-03T12:27:21Z

Hi, just updated to 0.18.0 - unfortunately still experiencing the error

Is there any progress on this? Can I help in some way?

jjo · 2019-12-11T20:03:16Z

FYI we're facing the same issue with 0.18.0 as daemonset, kubernetes-1.15 on kops AMIs
/cc @dbarranco @jbianquetti-nami

leodido · 2019-12-20T14:18:54Z

Linking to #925 since that contains more info.
Closing this. Please go into #925 for further info.

tomklino closed this as completed May 30, 2019

tomklino reopened this May 30, 2019

poiana added the triage/support label Jun 11, 2019

poiana closed this as completed Aug 14, 2019

leodido added this to the 0.18.0 milestone Aug 22, 2019

leodido reopened this Aug 22, 2019

poiana added the needs-kind label Aug 22, 2019

poiana added kind/bug and removed needs-kind labels Aug 22, 2019

leodido modified the milestones: 0.18.0, 0.19.0 Oct 3, 2019

seanmcrw mentioned this issue Nov 20, 2019

Multiple slack notifications #930

Closed

stale bot added the wontfix label Dec 2, 2019

stale bot removed the wontfix label Dec 3, 2019

leodido closed this as completed Dec 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why are all k8s.. fields are null? #629

Why are all k8s.. fields are null? #629

tomklino commented May 29, 2019 •

edited

Loading

tomklino commented May 30, 2019

tomklino commented May 30, 2019

fntlnz commented Jun 11, 2019

fntlnz commented Jun 11, 2019

tomklino commented Jun 12, 2019 •

edited

Loading

tcotav commented Jun 14, 2019

rastut commented Jun 18, 2019

tcotav commented Jun 19, 2019

fntlnz commented Jul 30, 2019

tomklino commented Aug 7, 2019

fntlnz commented Aug 14, 2019

poiana commented Aug 14, 2019

tomklino commented Aug 15, 2019 •

edited

Loading

leodido commented Aug 22, 2019

stale bot commented Dec 2, 2019

tomklino commented Dec 3, 2019

jjo commented Dec 11, 2019 •

edited

Loading

leodido commented Dec 20, 2019

Why are all k8s.*.* fields are null? #629

Why are all k8s.*.* fields are null? #629

Comments

tomklino commented May 29, 2019 • edited Loading

tomklino commented May 30, 2019

tomklino commented May 30, 2019

fntlnz commented Jun 11, 2019

fntlnz commented Jun 11, 2019

tomklino commented Jun 12, 2019 • edited Loading

tcotav commented Jun 14, 2019

rastut commented Jun 18, 2019

tcotav commented Jun 19, 2019

fntlnz commented Jul 30, 2019

tomklino commented Aug 7, 2019

fntlnz commented Aug 14, 2019

poiana commented Aug 14, 2019

tomklino commented Aug 15, 2019 • edited Loading

leodido commented Aug 22, 2019

stale bot commented Dec 2, 2019

tomklino commented Dec 3, 2019

jjo commented Dec 11, 2019 • edited Loading

leodido commented Dec 20, 2019

Why are all k8s.. fields are null? #629

Why are all k8s.. fields are null? #629

tomklino commented May 29, 2019 •

edited

Loading

tomklino commented Jun 12, 2019 •

edited

Loading

tomklino commented Aug 15, 2019 •

edited

Loading

jjo commented Dec 11, 2019 •

edited

Loading