Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tuning] Windows Discovery Rule Tuning for UEBA #3097

Merged
merged 47 commits into from
Oct 11, 2023
Merged

[Tuning] Windows Discovery Rule Tuning for UEBA #3097

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
b127b42
[Tuning] Win DR Tuning for UEBA
Aegrah Sep 14, 2023
92ebe05
Need to get used to Windows formatting
Aegrah Sep 14, 2023
5df753f
Added additional content
Aegrah Sep 14, 2023
1cb38b2
Updated min stack
Aegrah Sep 14, 2023
7750c9f
Added additional tuning
Aegrah Sep 14, 2023
7cd5750
Fixed unit testing for KQL optimization
Aegrah Sep 14, 2023
4778cb7
Update rules_building_block/discovery_internet_capabilities.toml
Aegrah Sep 14, 2023
802a06b
Additional tuning
Aegrah Sep 14, 2023
d988c5c
Merge branch 'win-rule-tuning-and-conversion-for-UEBA' of github.com:…
Aegrah Sep 14, 2023
2c18c5d
Kuery optimization
Aegrah Sep 14, 2023
8e682a3
Additional tuning
Aegrah Sep 14, 2023
f6e9315
Additional tuning
Aegrah Sep 14, 2023
ef1779a
Additional tuning
Aegrah Sep 14, 2023
83f63b5
Additional tuning
Aegrah Sep 14, 2023
16d20fd
Unit testing optimization fix
Aegrah Sep 14, 2023
8fcae0e
optimization
Aegrah Sep 14, 2023
120024a
tuning
Aegrah Sep 14, 2023
2276eba
Optimization
Aegrah Sep 14, 2023
c50714a
Update rules/windows/discovery_privileged_localgroup_membership.toml
Aegrah Sep 19, 2023
59a7b5f
Merge branch 'main' into win-rule-tuning-and-conversion-for-UEBA
Aegrah Sep 19, 2023
fec72d4
Added feedback
Aegrah Sep 20, 2023
0b733e3
Update rules/windows/discovery_privileged_localgroup_membership.toml
Aegrah Sep 21, 2023
27d5f9b
Update rules/windows/discovery_remote_system_discovery_commands_windo…
Aegrah Sep 21, 2023
f5367ae
Update rules/windows/discovery_system_service_discovery.toml
Aegrah Sep 21, 2023
4647b5d
added host.id as additional new_terms field
Aegrah Sep 21, 2023
c0dd576
Reworked a lot.
Aegrah Sep 21, 2023
299ce9e
kibana.alert.rule.rule_id to non-ecs-schema.json
Aegrah Sep 21, 2023
da33c09
Fixed index by adding a dot
Aegrah Sep 21, 2023
aee101f
fixed typo
Aegrah Sep 21, 2023
2909d31
Added host.os.type:windows for signals
Aegrah Sep 21, 2023
079c2c3
Added additional tag
Aegrah Sep 21, 2023
438f592
Added Higher-Order Rule tag
Aegrah Sep 21, 2023
347716b
Stripped down signal rules down to two
Aegrah Sep 22, 2023
9ec4b71
revert
Aegrah Oct 10, 2023
26451b6
Merge branch 'main' into win-rule-tuning-and-conversion-for-UEBA
Aegrah Oct 10, 2023
cc95dda
Update rules/windows/discovery_admin_recon.toml
Aegrah Oct 10, 2023
0428a43
Update rules/windows/discovery_enumerating_domain_trusts_via_nltest.toml
Aegrah Oct 10, 2023
ae154e3
Update rules_building_block/discovery_generic_registry_query.toml
Aegrah Oct 10, 2023
522cc3e
Update rules_building_block/discovery_system_time_discovery.toml
Aegrah Oct 10, 2023
85d11ed
Update rules/windows/discovery_privileged_localgroup_membership.toml
Aegrah Oct 10, 2023
2507943
Update discovery_generic_registry_query.toml
Aegrah Oct 10, 2023
97b1536
Readded exclusions
Aegrah Oct 10, 2023
d6a8c3a
Added trailing wildcards for KQL
Aegrah Oct 10, 2023
5f39660
Update discovery_privileged_localgroup_membership.toml
Aegrah Oct 10, 2023
f1585f7
Update rules_building_block/discovery_signal_unusual_user_host.toml
Aegrah Oct 10, 2023
eed00f9
Update rules/windows/discovery_signal_unusual_discovery_signal_proc_c…
Aegrah Oct 11, 2023
1fbec65
Formatting fix
Aegrah Oct 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions rules/windows/discovery_admin_recon.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ integration = ["endpoint", "windows"]
maturity = "production"
min_stack_comments = "New fields added: required_fields, related_integrations, setup"
min_stack_version = "8.3.0"
updated_date = "2023/06/22"
updated_date = "2023/09/14"

[rule]
author = ["Elastic"]
Expand Down Expand Up @@ -56,7 +56,14 @@ If enabling an EQL rule on a non-elastic-agent index (such as beats) for version
risk_score = 21
rule_id = "871ea072-1b71-4def-b016-6278b505138d"
severity = "low"
tags = ["Domain: Endpoint", "OS: Windows", "Use Case: Threat Detection", "Tactic: Discovery", "Resources: Investigation Guide", "Data Source: Elastic Endgame", "Data Source: Elastic Defend"]
tags = ["Domain: Endpoint",
"OS: Windows",
"Use Case: Threat Detection",
"Tactic: Discovery",
"Resources: Investigation Guide",
"Data Source: Elastic Endgame",
"Data Source: Elastic Defend"
]
timestamp_override = "event.ingested"
type = "eql"

Expand All @@ -74,42 +81,43 @@ process where host.os.type == "windows" and event.type == "start" and

((process.name : "wmic.exe" or process.pe.original_file_name == "wmic.exe") and
process.args : ("group", "useraccount"))
)
) and not user.id : "S-1-5-18"
Aegrah marked this conversation as resolved.
Show resolved Hide resolved
'''


[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1069"
name = "Permission Groups Discovery"
reference = "https://attack.mitre.org/techniques/T1069/"

[[rule.threat.technique.subtechnique]]
id = "T1069.001"
name = "Local Groups"
reference = "https://attack.mitre.org/techniques/T1069/001/"

[[rule.threat.technique.subtechnique]]
id = "T1069.002"
name = "Domain Groups"
reference = "https://attack.mitre.org/techniques/T1069/002/"


[[rule.threat.technique]]
id = "T1087"
name = "Account Discovery"
reference = "https://attack.mitre.org/techniques/T1087/"

[[rule.threat.technique.subtechnique]]
id = "T1087.001"
name = "Local Account"
reference = "https://attack.mitre.org/techniques/T1087/001/"

[[rule.threat.technique.subtechnique]]
id = "T1087.002"
name = "Domain Account"
reference = "https://attack.mitre.org/techniques/T1087/002/"


[rule.threat.tactic]
id = "TA0007"
name = "Discovery"
reference = "https://attack.mitre.org/tactics/TA0007/"

19 changes: 14 additions & 5 deletions rules/windows/discovery_enumerating_domain_trusts_via_nltest.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ integration = ["endpoint", "windows"]
maturity = "production"
min_stack_comments = "New fields added: required_fields, related_integrations, setup"
min_stack_version = "8.3.0"
updated_date = "2023/06/27"
updated_date = "2023/09/14"

[rule]
author = ["Elastic"]
Expand Down Expand Up @@ -62,7 +62,14 @@ references = [
risk_score = 21
rule_id = "84da2554-e12a-11ec-b896-f661ea17fbcd"
severity = "low"
tags = ["Domain: Endpoint", "OS: Windows", "Use Case: Threat Detection", "Tactic: Discovery", "Data Source: Elastic Endgame", "Resources: Investigation Guide", "Data Source: Elastic Defend"]
tags = ["Domain: Endpoint",
"OS: Windows",
"Use Case: Threat Detection",
"Tactic: Discovery",
"Data Source: Elastic Endgame",
"Resources: Investigation Guide",
"Data Source: Elastic Defend"
]
timestamp_override = "event.ingested"
type = "eql"

Expand All @@ -71,16 +78,19 @@ process where host.os.type == "windows" and event.type == "start" and
process.name : "nltest.exe" and process.args : (
"/DCLIST:*", "/DCNAME:*", "/DSGET*",
"/LSAQUERYFTI:*", "/PARENTDOMAIN",
"/DOMAIN_TRUSTS", "/BDC_QUERY:*")
"/DOMAIN_TRUSTS", "/BDC_QUERY:*"
) and
not process.parent.name : "PDQInventoryScanner.exe" and not user.id : "S-1-5-18"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this intended to exclude PDQInventoryScanner.exe by S-1-5-18 user or are these intended to be 2 separate exclusions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are two separate exclusions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to make it more obvious

Suggested change
not process.parent.name : "PDQInventoryScanner.exe" and not user.id : "S-1-5-18"
not process.parent.name : "PDQInventoryScanner.exe" and
not user.id : "S-1-5-18"

Aegrah marked this conversation as resolved.
Show resolved Hide resolved
'''


[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1482"
name = "Domain Trust Discovery"
reference = "https://attack.mitre.org/techniques/T1482/"

[[rule.threat.technique]]
id = "T1018"
name = "Remote System Discovery"
Expand All @@ -90,4 +100,3 @@ reference = "https://attack.mitre.org/techniques/T1018/"
id = "TA0007"
name = "Discovery"
reference = "https://attack.mitre.org/tactics/TA0007/"

42 changes: 28 additions & 14 deletions rules/windows/discovery_files_dir_systeminfo_via_cmd.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,19 @@
creation_date = "2022/11/01"
integration = ["endpoint", "windows"]
maturity = "production"
min_stack_comments = "New fields added: required_fields, related_integrations, setup"
min_stack_version = "8.3.0"
updated_date = "2023/06/22"
min_stack_comments = "Multiple field support in the New Terms rule type was added in Elastic 8.6"
min_stack_version = "8.6.0"
updated_date = "2023/09/20"

[rule]
author = ["Elastic"]
description = """
Identifies the execution of discovery commands to enumerate system information, files, and folders using the Windows Command Shell.
Identifies the execution of discovery commands to enumerate system information, files, and folders using the Windows
Command Shell.
"""
from = "now-9m"
index = ["winlogbeat-*", "logs-windows.*", "logs-endpoint.events.*"]
language = "eql"
language = "kuery"
brokensound77 marked this conversation as resolved.
Show resolved Hide resolved
license = "Elastic License v2"
name = "System Information Discovery via Windows Command Shell"
note = """## Triage and analysis
Expand Down Expand Up @@ -50,19 +51,25 @@ If enabling an EQL rule on a non-elastic-agent index (such as beats) for version
risk_score = 21
rule_id = "d68e95ad-1c82-4074-a12a-125fe10ac8ba"
severity = "low"
tags = ["Domain: Endpoint", "OS: Windows", "Use Case: Threat Detection", "Tactic: Discovery", "Tactic: Execution", "Resources: Investigation Guide", "Data Source: Elastic Defend"]
tags = ["Domain: Endpoint",
"OS: Windows",
"Use Case: Threat Detection",
"Tactic: Discovery",
"Tactic: Execution",
"Resources: Investigation Guide",
"Data Source: Elastic Defend"
]
timestamp_override = "event.ingested"
type = "eql"
type = "new_terms"

query = '''
process where host.os.type == "windows" and event.type == "start" and
process.name : "cmd.exe" and process.args : "/c" and process.args : ("set", "dir") and
not process.parent.executable : ("?:\\Program Files\\*", "?:\\Program Files (x86)\\*", "?:\\PROGRA~1\\*")
host.os.type:windows and event.category:process and event.type:start and process.name.caseless:"cmd.exe" and
process.args:("/c" and ("dir" or "set"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

process.args.caseless?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, so it is not ingested as a multi-field.

What you can do instead then to retain the resiliency here is

Suggested change
process.args:("/c" and ("dir" or "set"))
process.command_line.caseless:("cmd.exe * /c *" and ("cmd.exe * dir *" or "cmd.exe * set *"))

I would test it too

image

Copy link
Contributor Author

@Aegrah Aegrah Sep 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

outdated due to the BBR converions

Yea unfortunately we cannot use fields such as process.parent.name.caseless, process.parent.executable.caseless, process.args.caseless etc. Apparently, we can use process.command_line.caseless (I did not know, so thanks!).

This one would for example not execute on cmd.exe /C ... due to capitalization. However, the issue with it is, in KQL building queries with spaces and wildcards are a mess, as we can't use quotes like in your example above. What we can do is something like:

host.os.type:windows and event.category:process and event.type:start and process.name.caseless:"cmd.exe" and process.command_line.caseless:(
  (cmd.exe?/c* and (cmd.exe*dir* or cmd.exe*set*)) or (C\:\\\\Windows\\\\system32\\\\cmd.exe?/c* and (C\:\\\\Windows\\\\system32\\\\cmd.exe*dir* or C\:\\\\Windows\\\\system32\\\\cmd.exe*set*))
)

Or building it into one query if we expect the set and dir commands to be always used as first argument within the command:

host.os.type:windows and event.category:process and event.type:start and process.name.caseless:"cmd.exe" and process.command_line.caseless:(
  (cmd.exe?/c?dir* or cmd.exe?/c?set*) or (C\:\\\\Windows\\\\system32\\\\cmd.exe?/c?dir* or C\:\\\\Windows\\\\system32\\\\cmd.exe?/c?set*)
)

If we would change it to something like this:

host.os.type:windows and event.category:process and event.type:start and process.name.caseless:"cmd.exe" and process.command_line.caseless:(
  (cmd.exe?/c*dir* or cmd.exe?/c*set*) or (C\:\\\\Windows\\\\system32\\\\cmd.exe?/c*dir* or C\:\\\\Windows\\\\system32\\\\cmd.exe?/c*set*)
)

We would capture way more FPs as it can trigger on chunks of commands. What do you think?

'''


[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1082"
name = "System Information Discovery"
Expand All @@ -73,26 +80,33 @@ id = "T1083"
name = "File and Directory Discovery"
reference = "https://attack.mitre.org/techniques/T1083/"


[rule.threat.tactic]
id = "TA0007"
name = "Discovery"
reference = "https://attack.mitre.org/tactics/TA0007/"

[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1059"
name = "Command and Scripting Interpreter"
reference = "https://attack.mitre.org/techniques/T1059/"

[[rule.threat.technique.subtechnique]]
id = "T1059.003"
name = "Windows Command Shell"
reference = "https://attack.mitre.org/techniques/T1059/003/"



[rule.threat.tactic]
id = "TA0002"
name = "Execution"
reference = "https://attack.mitre.org/tactics/TA0002/"

[rule.new_terms]
field = "new_terms_fields"
value = ["user.id", "process.command_line"]
Aegrah marked this conversation as resolved.
Show resolved Hide resolved

[[rule.new_terms.history_window_start]]
field = "history_window_start"
value = "now-14d"
47 changes: 25 additions & 22 deletions rules/windows/discovery_net_view.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@
creation_date = "2020/12/04"
integration = ["endpoint", "windows"]
maturity = "production"
min_stack_comments = "New fields added: required_fields, related_integrations, setup"
min_stack_version = "8.3.0"
updated_date = "2023/06/22"
min_stack_comments = "Multiple field support in the New Terms rule type was added in Elastic 8.6"
min_stack_version = "8.6.0"
updated_date = "2023/09/20"

[rule]
author = ["Elastic"]
description = "Identifies attempts to enumerate hosts in a network using the built-in Windows net.exe tool."
from = "now-9m"
index = ["logs-endpoint.events.*", "winlogbeat-*", "logs-windows.*", "endgame-*"]
language = "eql"
language = "kuery"
license = "Elastic License v2"
name = "Windows Network Enumeration"
note = """## Triage and analysis
Expand Down Expand Up @@ -49,30 +49,27 @@ If enabling an EQL rule on a non-elastic-agent index (such as beats) for version
risk_score = 47
rule_id = "7b8bfc26-81d2-435e-965c-d722ee397ef1"
severity = "medium"
tags = ["Domain: Endpoint", "OS: Windows", "Use Case: Threat Detection", "Tactic: Discovery", "Resources: Investigation Guide", "Data Source: Elastic Endgame", "Data Source: Elastic Defend"]
tags = ["Domain: Endpoint",
"OS: Windows",
"Use Case: Threat Detection",
"Tactic: Discovery",
"Resources: Investigation Guide",
"Data Source: Elastic Endgame",
"Data Source: Elastic Defend"
]
timestamp_override = "event.ingested"
type = "eql"
type = "new_terms"

query = '''
process where host.os.type == "windows" and event.type == "start" and
((process.name : "net.exe" or process.pe.original_file_name == "net.exe") or
((process.name : "net1.exe" or process.pe.original_file_name == "net1.exe") and
not process.parent.name : "net.exe")) and
(process.args : "view" or (process.args : "time" and process.args : "\\\\*"))


/* expand when ancestry is available
and not descendant of [process where event.type == "start" and process.name : "cmd.exe" and
((process.parent.name : "userinit.exe") or
(process.parent.name : "gpscript.exe") or
(process.parent.name : "explorer.exe" and
process.args : "C:\\*\\Start Menu\\Programs\\Startup\\*.bat*"))]
*/
host.os.type:windows and event.category:process and event.type:start and
(process.name.caseless:"net.exe" or process.pe.original_file_name:"net.exe" or
(process.name.caseless:"net1.exe" or process.pe.original_file_name:"net1.exe") and not process.parent.name:"net.exe") and
process.args:("view" or "time" and \\\\\\\\*)
brokensound77 marked this conversation as resolved.
Show resolved Hide resolved
'''


[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1018"
name = "Remote System Discovery"
Expand All @@ -83,9 +80,15 @@ id = "T1135"
name = "Network Share Discovery"
reference = "https://attack.mitre.org/techniques/T1135/"


[rule.threat.tactic]
id = "TA0007"
name = "Discovery"
reference = "https://attack.mitre.org/tactics/TA0007/"

[rule.new_terms]
field = "new_terms_fields"
value = ["user.id", "process.command_line"]
Aegrah marked this conversation as resolved.
Show resolved Hide resolved

[[rule.new_terms.history_window_start]]
field = "history_window_start"
value = "now-14d"
Loading