[Windows] Simplifying Windows Events packages

[jamiehynds]

Windows Events are currently split across numerous packages:

- System: Application, System, Security Events
The rational behind including Windows events in the System integration was to ensure users automatically start ingesting Application, System and Security events as soon as Agent is installed. While this is valuable, it leads to confusion as users don't realise this is the case, and end up looking in the Windows integration for these sources.

- Windows: Forwarded Events, PowerShell and Sysmon
Should we consider splitting these data streams into standalone integrations, inline with other integrations?

- Custom Windows Events - anything outside the supported events in System and Windows integrations.
This integration currently limits users to just one event channel. This results in users having to maintain several integrations for Windows events, assuming they are collecting numerous event channels.

Feedback provided directly by @nicpenning:

1. Each channel has it's own mappings and pipelines so if you need to alter any of them you have to do so in many areas
2. The custom windows event logs is causing two field conflicts for sure (event.creation, winlog.event_id)
3. The fact that there are 3 integrations to do similar things is uneasy as I don't know the parity between the system integration for System, Security, Application logs vs doing that in the Custom Windows Event Logs, or the Windows Integration that only has Powershell, Sysmon, etc..
4. Data stream inconsistencies between each of the integrations which needs to be taken into account and changed per channel (if possible)
5. I shouldn't need 3 unique integrations which is a total of 13 integrations to read the Windows event logs. They all need the same pipeline and mappings, so it should be possible to assign the 1 custom pipeline and 1 custom mapping to all Windows events. Since the custom mappings for the integrations are components I can't add to the mapping other component templates and I don't want to alter and managed mappings and pipelines as those could change over upgrades.

This issue is intended to track discussions as to how we can streamline the discoverability and usability of Windows packages for both Elastic supported events/channels and custom Windows events.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[nicpenning]

I will add to my response above.

The goal: Replace Winlogbeat to leverage the power of the Elastic Agent while maintaining custom enrichments Windows event data.

The situation:

The problem:
To take full advantage of the Windows log parsing for System, Security, and Application events, we must use the System Integration as the pipelines are built into the integration and can change with upgrades.
Security example:

This same logic applies for the Windows Integration and processing Sysmon and PowerShell logs.
Sysmon example:

And lastly the Custom Windows integration:

Note: As far as I know, when you change the data set name the pipelines and mappings no longer work since it won't match on the template, which is why you need to create your own template and make sure you have all the Windows mappings set appropriately (which is not so bad since they are component templates, but still not ideal).

If an engineer decided to go this route then they have some considerations to make this work.

1. Inconsistent Data Streams - logs-system.application*, logs-windows*, logs-winlog* (this can be changed but then won't abide by built in pipelines and mappings so a special index template needs to be created and used.
2. Enriching data that applies to all three types of integrations requires careful planning to make sure they all have the appropriate pipelines and mappings. Since the mappings do apply to all of that integration type, that only works if you don't change the data set name which goes back to consideration 1.

Right now I am stuck because I want to be consistent for all Windows events but want the advantage of the managed pipelines that Elastic is maintaining, therefor I cannot simply just use only Custom Windows Event integrations without having to call the special pipeline versions that will change over time. Not to mention addressing all of the mappings as well. Too many things can go wrong.

Ideally we should be able to use the Windows Integration and it should include all current processed Windows events (Application, System, Security, PowerShell, Sysmon, and Forwarded) plus any new event channels, providers in those channels, and of course event IDs to fully align with the Winlogbeat capability.

That would at least centralize all of the logging to a base data stream (windows) that will have it's appropriate data sets (PowerShell, security, etc..). From there, it would be very useful to apply an integration ingest pipeline and mapping that could apply to all channels. The use case, again, is to easily add our enrich pipelines to all Windows events and make sure the mappings for those new enrichments take hold. I know managing each integration via the API is an option but due to the inconsistencies above and for the good of the users, this should be done in the UI.

Other considerations, we are moving away from the Logstash parsing for enrichments which is the need for ingest pipelines. The output for Logstash is useless for us in this use case since Elastic made the decision to stop processing on the endpoint and made it server side with ingest pipelines.

Lastly, I evaluated this in 8.5.0 and there was no changes that would impact the comments above.

Please let me know if any further clarification is needed.

Thanks!

[nicpenning]

Any movement on this? 👀

[jamiehynds]

Sorry @nicpenning, we haven't prioritised the Windows pacakge(s) improvements yet but may have an enhancement coming to address the inconsistent data streams problem. @P1llus do you think the dynamic ECS templates may address this problem to some degree?

[P1llus]

I think maybe the issue is a bit different than a usecase for the dynamic ECS template that is coming.

I am unsure if I understood this right, on why you had to create custom indices for this.
From my understanding, you want to add custom pipelines to the existing integrations, that uses the enrich processor to enrich the incoming events right @nicpenning ?

In the newer versions of integrations, there is a @custom ingest pipeline added to all the packages, so that users can add their own ingest pipelines to integrations without having to re-add them each time an integration is updated, would that not solve the issue with enrichment for now?

For the other comments around naming convention I think that is a separate issue that this issue is already tracking.

[nicpenning]

I have started down the path of using the @Custom for mappings and pipelines, and they work. The enrichments require custom mappings. Which requires either adding the same mappings to the following:

logs-system.system@custom
logs-system.application@custom
logs-system.security@custom

logs-windows.powershell@custom
logs-windows.sysmon_operational@custom
logs-windows.powershell_operational@custom

logs-winlog.winlog@custom

Or adding a component template that has our enriched mappings to the managed index templates which I am unclear if this component template will get wiped out after upgrades or if it is a good practice at all:

logs-system.system
logs-system.application
logs-system.security

logs-windows.powershell
logs-windows.sysmon_operational
logs-windows.powershell_operational

logs

It just seems very complicated to have to manage all of these pipelines and templates instead of doing it in 1 place.

Right now, all of the @Custom pipelines point to other pipelines as I don't want to get stuck putting raw pipelines in the @Custom since those could change and I don't want to have to make 7 changes, so that is okay at the moment.

We are running 8.6.0 and trying to replace winlogbeat still but it is way more effort than we had imagined just because we need custom pipelines and mappings that will persist after upgrades and are manageable.

Is it recommended to modify the managed index templates to add component templates to them? The @Custom mappings is not ideal because it is a component template we can't add component templates too, thus, that is still 7 mappings that will need to change whenever we need new mappings.

[P1llus]

Unfortunately right now the only way to do this is indeed to manage it through @custom, which means you would have multiple copies of your mappings and ingest pipelines.

I would not advice to modify the index templates at this point, that will most likely be overwritten on updates indeed.

There are some efforts being done here to be able to set this per integration rather than per datastream only: elastic/kibana#146792
Which would make the issue smaller but still there.

I have forwarded your comments and I think someone else might also give another comment here later.

[nicpenning]

I am glad I am not the only one who is experiencing this. In the meanwhile, we really need to migrate so we will have to copy pasta our mappings across the 7 @Custom mappings to make due. I just hope the changes that come later make it easy to consolidate these integrations without much breakage.

At least knowing that this needs to be done through the @Custom is helpful. But a standardized and streamlined way to handle Windows events is a big need in my opinion.

[nicpenning]

Upon further review, the Custom Windows Events poses another disadvantage that is quite unfortunate. The integration has a constant_keyword stuck in the logs-winlog.winlog@package managed index template:

Which prevents me from changing the event.dataset.

I can't index any events after I change the event.dataset to something that aligns with the Windows Integration and System Integration. Thinking this through further, it doesn't make sense to change the dataset anyways because the managed index template requires a mapping to logs-winlog.winlog-* so we a forced to use that dataset anyways.

{"type":"mapper_parsing_exception","reason":"failed to parse field [event.dataset] of type [constant_keyword] in document with id '1Bmex4UBSOTXuI8cu1SU'. Preview of field's value: 'winlog.winlog-applocker.exe_dll'","caused_by":{"type":"illegal_argument_exception","reason":"[constant_keyword] field [event.dataset] only accepts values that are equal to the value defined in the mappings [winlog.winlog], but got [winlog.winlog-applocker.exe_dll]"}}, dropping event!

Furthermore, ILM Policies will be a challenge because every custom channel is bound to the same managed index template.

Here is a small section snapshot of documentation trying to get my head around the complexities of implementing the Elastic Agent and various Windows Integrations.

[ruflin]

@nicpenning You are definitively not the only one. This issue might also be of interest for you: elastic/kibana#146792

[nicpenning]

Thanks @ruflin! That appears to be the same issue that Marius posted.

How would you like me to proceed to help with this conversation/issue?

[ruflin]

@nicpenning One question I have for you is: Would management per integration already be helfpul for you? In your example above, it would mean you do it 3 times instead of X times?

[nicpenning]

3 is better than 7, however, here are some things to consider:

1. I would rather all Windows Logs go to logs-windows-*
2. I would like to have each integration to have its own dataset or the ability to have its own ILM. The custom integration of winlog.winlog forces all custom windows events to use the same index template, ilm, pipelines and mappings.
3. I would like to be able to use any component template, not forced to use the single @Custom for all since that means updating all 7 or down to 3 when making changes to the required enrichments.

Hope this helps!

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[nicpenning]

I believe this is still relevant and at some point, I would like to see all Windows events in the Windows integration rather than the core Windows events in the System integration. Perhaps it can live in both. But when I find the time I can elaborate on the streamlined / simplification of all Windows events in the same integration instead of spread out across the 3 integrations (System, Windows, Custom Windows). With the move of AppLocker here this was a helpful. But Windows Defender and many other custom Windows Event logs should eventually be supported when they are built in. Such as WDAC, Terminal Services, etc.. I think Custom Windows Event logs integration is perfect for actual custom Windows event logs where system admins are creating their own channel and special event logs that only apply to their environment. But if any Windows Event log is natively in Windows, then ideally it should be in the Windows Integration. If I didn't know about Elastic Agent in the early days, it would be very confusing to know that the Windows Event Log ingestion capabilities exists across these 3 integrations.

More to come on this, but I think there is great opportunity on this topic and am willing to collaborate and enhance Windows integrations.

[jamiehynds]

Totally agree Nic. Thanks for the additional feedback. @andrewkroh and I were just discussing a streamlined workflow for Windows events this week. Will let you know as soon as we have an update.

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[mbudge]

Any new index patterns will need adding to the security rules if winlogbeat is moved to windows.

The system.security dataset name is a bit confusing as it doesn’t make it obvious this is the windows security event log data.

Windows.security would be a better dataset.

[nicpenning]

Any thoughts to moving/copying System, Application, and Security to the Windows Integration yet? If there was a way to centrally manage the same Integration code in both places I wouldn't mind having it in both spots and either/or could be enabled and have similar results.

[mbudge]

I’m pretty sure I enabled system security and Linux system/syslog on a Linux host recently and agent went unhealthy. Maybe the integrations in the system package aren’t OS aware. I create a policy for Linux and policy for windows. Both have metrics collection enabled, but we change the default metrics settings as those can cause serious performance issues on production servers running complex tasks.

Moving system application and security to another integration and dataset will require changing the data_stream.dataset. All security rules, dashboards, saved search and any custom processing customers have built using this data would need updating when that happens. It’s going to be a lot of work…. Customers will need to be notified before this happens.

[nicpenning]

Yes it be a lot of work, but likely manageable and worth it in the long run. Again they could be duplicated until everything is near perfect with lots of communication and migration docs, then deprecate it.

[mbudge]

Agree it would make it easier for users. I’d do it.

[nicpenning]

Regarding rules, this site (https://elastic.github.io/detection-rules-explorer/) might be a little misleading since it only shows 1 rule with the "System" integration.

Manually checking, it's much more than that. Using Windows as the source there are only 3, which is also not quite right.