Skip to content

Commit

Permalink
[ML] Adds a 30 day model prune window to non-rare Security jobs (elas…
Browse files Browse the repository at this point in the history
…tic#107752)

Adds the model_prune_window setting added in elastic/elasticsearch#75741
to all Security jobs that use functions that support model pruning.
This means that the models for split field values that are not seen for
30 days will be dropped. If those split field values are subsequently seen
again then new models will be created like for completely new entities.
The "rare" function does not support model pruning, so jobs that use
the "rare" function are not modified.
  • Loading branch information
droberts195 authored Aug 10, 2021
1 parent 283349a commit f479259
Show file tree
Hide file tree
Showing 10 changed files with 20 additions and 10 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@
"detector_index": 0
}
],
"influencers": []
"influencers": [],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
"source.ip",
"winlog.event_data.LogonType",
"user.name"
]
],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@
"detector_index": 0
}
],
"influencers": []
"influencers": [],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
"destination.as.organization.name",
"source.ip",
"destination.ip"
]
],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
"destination.as.organization.name",
"source.ip",
"destination.port"
]
],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
"destination.as.organization.name",
"source.ip",
"destination.ip"
]
],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
"host.name",
"user.name",
"source.ip"
]
],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@
"aws.cloudtrail.user_identity.arn",
"source.ip",
"source.geo.city_name"
]
],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@
"destination.ip",
"host.name",
"dns.question.etld_plus_one"
]
],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
"host.name",
"user.name",
"winlog.event_data.Path"
]
],
"model_prune_window": "30d"
},
"allow_lazy_open": true,
"analysis_limits": {
Expand Down

0 comments on commit f479259

Please sign in to comment.