Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

network processor condition to support arrays #41917

Open
jguay opened this issue Dec 5, 2024 · 2 comments
Open

network processor condition to support arrays #41917

jguay opened this issue Dec 5, 2024 · 2 comments
Labels
enhancement needs_team Indicates that the issue/PR needs a Team:* label :Processors

Comments

@jguay
Copy link
Contributor

jguay commented Dec 5, 2024

Describe the enhancement:
Not sure if this should be considered bug or enhancement

At the moment network processor condition does not work on arrays and documentation is not clear on what type of data it supports

example using auditd integration with these processors (tested with latest 8.16.1) :

- add_host_metadata:
- add_fields:
    when:
      network:
        host.ip: ['10.154.0.0/24']
    fields:
      network_condition_on_host_ip: met
- add_fields:
    when:
      contains:
        host.ip: '10.154.0.'
    fields:
      contains_condition: met
contains works but network processor does not match in document
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 12,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-auditd.log-default-2024.11.20-000001",
        "_id": "WYgGc5MBK2hKpFNRV6Lt",
        "_score": 1,
        "_source": {
          "agent": {
            "name": "ubu22-julien",
            "id": "f8a08d03-ff81-4fd7-8d82-783b2d0a09a4",
            "ephemeral_id": "44b2b408-bbbb-4fd3-9b0a-229e055443ff",
            "type": "filebeat",
            "version": "8.16.1"
          },
          "process": {
            "pid": 5786,
            "executable": "/usr/bin/sudo"
          },
          "log": {
            "file": {
              "path": "/var/log/audit/audit.log"
            },
            "offset": 345018
          },
          "elastic_agent": {
            "id": "f8a08d03-ff81-4fd7-8d82-783b2d0a09a4",
            "version": "8.16.1",
            "snapshot": false
          },
          "auditd": {
            "log": {
              "ses": "1",
              "op": "PAM:accounting",
              "record_type": "USER_ACCT",
              "sequence": 1370,
              "uid": "1043",
              "subj": "unconfined",
              "grantors": "pam_permit"
            }
          },
          "tags": [
            "preserve_original_event",
            "auditd-log"
          ],
          "cloud": {
            "availability_zone": "europe-west2-c",
            "instance": {
              "name": "ubu22-julien",
              "id": "3214280448576685592"
            },
            "provider": "gcp",
            "service": {
              "name": "GCE"
            },
            "machine": {
              "type": "e2-standard-2"
            },
            "project": {
              "id": "elastic-support"
            },
            "region": "europe-west2",
            "account": {
              "id": "elastic-support"
            }
          },
          "input": {
            "type": "log"
          },
          "@timestamp": "2024-11-28T13:46:38.666Z",
          "ecs": {
            "version": "8.11.0"
          },
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "auditd.log"
          },
          "host": {
            "hostname": "ubu22-julien",
            "os": {
              "kernel": "5.10.0-33-cloud-amd64",
              "codename": "bullseye",
              "name": "Debian GNU/Linux",
              "type": "linux",
              "family": "debian",
              "version": "11 (bullseye)",
              "platform": "debian"
            },
            "ip": [
              "10.154.0.83",
              "fe80::4001:aff:fe9a:53"
            ],
            "containerized": false,
            "name": "ubu22-julien",
            "id": "1b1d98a60f0b92b83dcde27ccce63e70",
            "mac": [
              "42-01-0A-9A-00-53"
            ],
            "architecture": "x86_64"
          },
          "fields": {
            "contains_condition": "met"
          },
          "event": {
            "agent_id_status": "verified",
            "ingested": "2024-11-28T13:46:55Z",
            "original": "type=USER_ACCT msg=audit(1732801598.666:1370): pid=5786 uid=1043 auid=1043 ses=1 subj=unconfined msg='op=PAM:accounting grantors=pam_permit acct=\"julien\" exe=\"/usr/bin/sudo\" hostname=? addr=? terminal=/dev/pts/0 res=success'\u001dUID=\"julien\" AUID=\"julien\"",
            "kind": "event",
            "action": [
              "was-authorized"
            ],
            "category": [
              "authentication"
            ],
            "type": [
              "info"
            ],
            "dataset": "auditd.log",
            "outcome": "success"
          },
          "user": {
            "effective": {
              "name": "julien"
            },
            "audit": {
              "id": "1043"
            },
            "name": "julien",
            "id": "1043",
            "terminal": "/dev/pts/0"
          }
        }
      }
    ]
  }
}

Debug logs contain "message": "Invalid IP address in field=host.ip for network condition" :

{"log.level":"debug","@timestamp":"2024-11-28T13:08:29.778Z","message":"Publish event: {\n  \"@timestamp\": \"2024-11-28T13:08:28.589Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.16.1\",\n    \"raw_index\": \"logs-elastic_agent.filebeat-default\",\n    \"input_id\": \"filestream-monitoring-agent\",\n    \"stream_id\": \"filestream-monitoring-agent\"\n  },\n  \"log\": {\n    \"source\": \"log-default\",\n    \"offset\": 1510461,\n    \"file\": {\n      \"device_id\": \"2049\",\n      \"inode\": \"262591\",\n      \"path\": \"/opt/Elastic/Agent/data/elastic-agent-8.16.1-b6da7f/logs/elastic-agent-20241128.ndjson\"\n    }\n  },\n  \"log.level\": \"debug\",\n  \"event\": {\n    \"dataset\": \"elastic_agent.filebeat\"\n  },\n  \"agent\": {\n    \"type\": \"filebeat\",\n    \"version\": \"8.16.1\",\n    \"ephemeral_id\": \"ff0d59ac-4eda-40ef-ad4c-ee478442f163\",\n    \"name\": \"ubu22-julien\",\n    \"id\": \"f8a08d03-ff81-4fd7-8d82-783b2d0a09a4\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"cloud\": {\n    \"region\": \"europe-west2\",\n    \"project\": {\n      \"id\": \"elastic-support\"\n    },\n    \"account\": {\n      \"id\": \"elastic-support\"\n    },\n    \"provider\": \"gcp\",\n    \"service\": {\n      \"name\": \"GCE\"\n    },\n    \"instance\": {\n      \"name\": \"ubu22-julien\",\n      \"id\": \"3214280448576685592\"\n    },\n    \"machine\": {\n      \"type\": \"e2-standard-2\"\n    },\n    \"availability_zone\": \"europe-west2-c\"\n  },\n  \"message\": \"Invalid IP address in field=host.ip for network condition\",\n  \"component\": {\n    \"id\": \"log-default\",\n    \"type\": \"log\",\n    \"binary\": \"filebeat\",\n    \"dataset\": \"elastic_agent.filebeat\"\n  },\n  \"service.name\": \"filebeat\",\n  \"elastic_agent\": {\n    \"id\": \"f8a08d03-ff81-4fd7-8d82-783b2d0a09a4\",\n    \"snapshot\": false,\n    \"version\": \"8.16.1\"\n  },\n  \"log.origin\": {\n    \"file.name\": \"conditions/network.go\",\n    \"function\": \"github.com/elastic/beats/v7/libbeat/conditions.(*Network).Check\",\n    \"file.line\": 162\n  },\n  \"host\": {\n    \"name\": \"ubu22-julien\",\n    \"ip\": [\n      \"10.154.0.83\",\n      \"fe80::4001:aff:fe9a:53\"\n    ],\n    \"mac\": [\n      \"42-01-0A-9A-00-53\"\n    ],\n    \"hostname\": \"ubu22-julien\",\n    \"architecture\": \"x86_64\",\n    \"os\": {\n      \"type\": \"linux\",\n      \"platform\": \"debian\",\n      \"version\": \"11 (bullseye)\",\n      \"family\": \"debian\",\n      \"name\": \"Debian GNU/Linux\",\n      \"kernel\": \"5.10.0-33-cloud-amd64\",\n      \"codename\": \"bullseye\"\n    },\n    \"id\": \"1b1d98a60f0b92b83dcde27ccce63e70\",\n    \"containerized\": false\n  },\n  \"log.logger\": \"conditions\",\n  \"input\": {\n    \"type\": \"filestream\"\n  },\n  \"data_stream\": {\n    \"type\": \"logs\",\n    \"dataset\": \"elastic_agent.filebeat\",\n    \"namespace\": \"default\"\n  }\n}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"processors","log.origin":{"file.line":215,"file.name":"processing/processors.go","function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1"},"service.name":"filebeat","log.type":"event","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Describe a specific use case for the enhancement or feature:

Basically allow network processor condition to support arrays as host.ip tend to return an array

Workaround if filter for network is /24 ie network mask255.255.255.0 is simple as this mean matching first 3 digits of IP address - however this is much more complicated if network filter were/25 for example meaning last digit contains 2 subnets

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 5, 2024
@botelastic
Copy link

botelastic bot commented Dec 5, 2024

This issue doesn't have a Team:<team> label.

@jguay jguay changed the title network processor to support arrays network processor condition to support arrays Dec 5, 2024
@fearful-symmetry
Copy link
Contributor

fix here: #41918

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement needs_team Indicates that the issue/PR needs a Team:* label :Processors
Projects
None yet
Development

No branches or pull requests

2 participants