Regression v0.11.0 -> v1.0.0: filename monitoring with prefix matchArgs selector does not work for long file path #1764

ashishkurmi · 2023-11-17T03:45:30Z

What happened?

When I deploy a filtered filename monitoring namespace scoped policy in Tetragon v1.0.0, it does not generate Tetragon events for long file paths. However, this bug does not exist in Tetragon v0.11.0.

Bug Repro steps for Tetragon v1.0.0

Deploy Tetragon v1.0.0 with enablePolicyFilter set to true.

cat tetragon_values.yaml
tetragon:
  enablePolicyFilter: "true"
  
helm install tetragon cilium/tetragon -n kube-system -f tetragon_values.yaml

Deploy the following filename_monitoring_filtered policy for default namespace

kubectl create -f tetragon_policy.yaml
cat tetragon_policy.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: "filepolicynamespaced"
  namespace: default
spec:
  kprobes:
    - call: "security_file_permission"
      syscall: false
      return: true
      args:
      - index: 0
        type: "file"
      - index: 1
        type: "int"
      returnArg:
        index: 0
        type: "int"
      returnArgAction: "Post"
      selectors:
      - matchArgs:      
        - index: 0
          operator: "Prefix"
          values:
          - "/rootdirtomonitor/" 
        - index: 1
          operator: "Equal"
          values:
          - "2" # MAY_WRITE
    - call: "security_mmap_file"
      syscall: false
      return: true
      args:
      - index: 0
        type: "file"
      - index: 1
        type: "uint32"
      - index: 2
        type: "nop"
      returnArg:
        index: 0
        type: "int"
      returnArgAction: "Post"
      selectors:
      - matchArgs:
        - index: 0
          operator: "Prefix"
          values:
          - "/rootdirtomonitor/" 
        - index: 1
          operator: "Mask"
          values:
          - "2"
    - call: "security_path_truncate"
      syscall: false
      return: true
      args:
      - index: 0
        type: "path"
      returnArg:
        index: 0
        type: "int"
      returnArgAction: "Post"
      selectors:
      - matchArgs:
          - index: 0
            operator: "Prefix"
            values:
            - "/rootdirtomonitor/"

Create a sample pod to generate test file events.

kubectl run debugpod --image=golang -i --tty -- /bin/bash
apt update
apt install vim
vim test_script.sh
# Paste the following content in test_script.sh
#!/bin/bash

# Function to create directory and file
create_dir_and_file() {
    local dir_path=$1
    local file_num=$2

    # Create directory
    mkdir -p "$dir_path"
    echo "Directory created: $dir_path"

    # Create file with content "test"
    echo "test" > "$dir_path/$file_num.txt"
    echo "File created: $dir_path/$file_num.txt"
}

# Function to delete created directories and files
delete_dir() {
    local dir_path=$1

    echo "Deleting: $dir_path"
    rm -rf "$dir_path"
}

# Main script starts here

# Check if a path is provided
if [ -z "$1" ]; then
    echo "No path provided. Exiting."
    exit 1
fi

input_path=$1
current_path=""
counter=1

# Split path and iterate over each part
IFS='/' read -ra ADDR <<< "$input_path"
for i in "${ADDR[@]}"; do
    # Skip empty elements
    if [ -z "$i" ]; then
        continue
    fi

    # Construct the current path
    current_path="$current_path/$i"
    create_dir_and_file "$current_path" $counter
    let counter+=1
done

# Deleting created directories and files in reverse order
for (( j=counter-1; j>=1; j-- )); do
    dir_to_delete=$(dirname "$current_path")
    delete_dir "$dir_to_delete"
    current_path="$dir_to_delete"
done

echo "All created directories and files have been deleted."

In another bash console, run tetra to observe Tetragon events

kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact -n default

In the test pod, run test_script.sh. This script progressively creates directories based on the path input and creates a file inside. For example, if you pass "/rootdirtomonitor/firstdir/seconddir/thirddir" to the script, it would create /rootdirtomonitor dir, /rootdirtomonitor/1.txt file, /rootdirtomonitor/firstdir dir, /rootdirtomonitor/firstdir/2.txt file, /rootdirtomonitor/firstdir/seconddir dir, /rootdirtomonitor/firstdir/seconddir/3.txt file, and so on.

bash test_script.sh /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp

In tetra console window, you would observed that Tetragon does not catch file write events after a particular length. For me, it stopped after .../homedr/_work/ directory as shown by 📝 write events below.

📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/9.txt 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr 0 
🚀 process default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work 0 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/10.txt 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/10.txt 
🚀 process default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var 0 
🚀 process default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib 0 
🚀 process default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp 0 
🚀 process default/debugpod /usr/bin/dirname /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp 
💥 exit    default/debugpod /usr/bin/dirname /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp 0

Please follow these steps to confirm that the bug does not exist in Tetragon v0.11.0
7. Uninstall Tetragon v1.0.0 and install Tetragon v0.11.0

helm uninstall tetragon -n kube-system
helm install tetragon cilium/tetragon  --version 0.11.0 -n kube-system -f tetragon_values.yaml

Repeat steps 2 to 6. You would see all Tetragon events this time in tetra console window (unlike Tetragon v1.0.0, you would see write events for 11.txt, 12.txt and 13.txt). For me, it produced the following output:

📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/9.txt 
🚀 process default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work 0 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/10.txt 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/10.txt 
🚀 process default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var 0 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/11.txt 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/11.txt 
🚀 process default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib 0 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/12.txt 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/12.txt 
🚀 process default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp 
💥 exit    default/debugpod /usr/bin/mkdir -p /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp 0 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp/13.txt 
📝 write   default/debugpod /usr/bin/bash /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp/13.txt 
🚀 process default/debugpod /usr/bin/dirname /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp 
💥 exit    default/debugpod /usr/bin/dirname /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib/tmp 0 
🚀 process default/debugpod /usr/bin/rm -rf /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib 
💥 exit    default/debugpod /usr/bin/rm -rf /rootdirtomonitor/var/lib/kubelet/pods/eda539f1-1fa9-4451-92dc-9c2fea310347/volumes/kubernetes.io~empty-dir/homedr/_work/var/lib 0

Tetragon Version

This bug exists only in v1.0.0. The events are generated as expected in v0.11.

Kernel Version

I reproed the issue on an AWS EKS cluster.

Linux ip-10-0-55-186.us-west-2.compute.internal 5.10.198-187.748.amzn2.x86_64 #1 SMP Tue Oct 24 19:49:54 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:38Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"28+", GitVersion:"v1.28.3-eks-4f4795d", GitCommit:"e77944ea667aee90c84c48a74a2a0da7a23e0508", GitTreeState:"clean", BuildDate:"2023-10-20T23:21:39Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}

Bugtool

No response

Relevant log output

No response

Anything else?

No response

The text was updated successfully, but these errors were encountered:

inliquid · 2023-11-17T18:07:41Z

Hi, I think this case might be the same issue as described here.

ashishkurmi · 2023-11-20T07:57:19Z

Thanks @inliquid! Do we know the root cause for this regression?

kevsecurity · 2023-11-22T13:41:58Z

Fixed by #1779

kkourt · 2023-11-27T06:59:37Z

Was there an error in the logs? Was the tracing policy accepted or not by Tetragon?

If not, it would be nice to have an error message for users that add prefixes larger than whatever the current limit is and have the policy rejected.

kevsecurity · 2023-11-27T12:04:02Z

Was there an error in the logs?

The prefix provided in the policy is shorter than the max prefix length, so the policy was accepted as usual. It also worked fine for paths shorter than the max prefix length. Where it failed, was when the actual path accessed was longer than the max prefix length – i.e. a runtime issue, not a policy loading issue. I'll make a PR that addresses this.

kevsecurity · 2023-11-27T15:20:59Z

#1806

ashishkurmi added the kind/bug Something isn't working label Nov 17, 2023

kevsecurity closed this as completed Nov 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression v0.11.0 -> v1.0.0: filename monitoring with prefix matchArgs selector does not work for long file path #1764

Regression v0.11.0 -> v1.0.0: filename monitoring with prefix matchArgs selector does not work for long file path #1764

ashishkurmi commented Nov 17, 2023 •

edited

Loading

inliquid commented Nov 17, 2023

ashishkurmi commented Nov 20, 2023

kevsecurity commented Nov 22, 2023

kkourt commented Nov 27, 2023 •

edited

Loading

kevsecurity commented Nov 27, 2023

kevsecurity commented Nov 27, 2023

Regression v0.11.0 -> v1.0.0: filename monitoring with prefix matchArgs selector does not work for long file path #1764

Regression v0.11.0 -> v1.0.0: filename monitoring with prefix matchArgs selector does not work for long file path #1764

Comments

ashishkurmi commented Nov 17, 2023 • edited Loading

What happened?

Tetragon Version

Kernel Version

Kubernetes Version

Bugtool

Relevant log output

Anything else?

inliquid commented Nov 17, 2023

ashishkurmi commented Nov 20, 2023

kevsecurity commented Nov 22, 2023

kkourt commented Nov 27, 2023 • edited Loading

kevsecurity commented Nov 27, 2023

kevsecurity commented Nov 27, 2023

ashishkurmi commented Nov 17, 2023 •

edited

Loading

kkourt commented Nov 27, 2023 •

edited

Loading