[Filebeat][Fortinet Module] Adding support for forticlient

[P1llus]

This is an issue to track progress and information related to adding a forticlient fieldset.

Currently the fortinet module only contains a fileset for Fortigate Firewall logs

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[P1llus]

@fredtj If you are able to include some forticlient logs (anonymized), I can see if we can make a test fileset for it

[fredtj]

@P1llus forticlient.anon.log.txt

Let me know if you need more, and also let me know if you have some better method to anonymize them!

[P1llus]

@fredtj could you give me some quick info on which version you are using of forticlient and fortianalyzer, I am guessing it is being forwarded from there?

[P1llus]

My method when sanitizing logs is to try to find as many different type of logs as possible. Then either use bash/python or a editors search + replace function to quickly modify all parameters over all logs.

[fredtj]

Fortianalyser v6.2.5
Forticlients are different versions, check the fctver string in the logs.
The clients send their logs to a Forticlient EMS server, which then sends them to Fortianalyzer.

I will do some more robust capturing to try and capture all types of logs, but it will take me a little time. Thanks for your efforts, very much appreciated!

[0x00-null]

@P1llus, should we have other issues like this in order to track the progress of Fortiweb and/or fortimail ?

[P1llus]

@thesilvereagle That would always be great, especially if people are able to contribute any sort of test data etc.

[0x00-null]

@P1llus I would love to contribute. Are there any guidelines or things to consider before starting? Also, I would love to know more about how to test these parsers?

[P1llus]

Always good to hear people would want to contribute! @thesilvereagle . In terms of general knowledge around filebeat, I would recommend the "setting up dev environment part" and the filebeat part of:
https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html
and
https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html

If you want to contribute adding new products related to fortinet, then in theory you would find the module under /beats/x-pack/filebeat/module/fortinet, and can copy the firewall folder and call it for example "fortisandbox", then I would recommend getting to know each file and what they are, it's also referenced in the docs above.

I won't be able to give a step by step guide, but I would recommend joining our community slack and/or discuss forums as many of us hang out there as well :)

If you don't want to go through the hassle of creating a fully fledged module then we always accept test data, and if you just want to work on the actual parsing, then copying a log entry from the product you want to create, and using the simulate pipeline API is how I build up the parser before adding it to a module:

https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html

Any example parsers is in the "ingest" folder of any module, so plenty there to get you going in terms of examples, good luck and maybe I'l see you around in slack!

[fredtj]

been scratching my head for a while as was receiving the following error attached to my documents: "Cannot invoke "Object.getClass()" because "receiver" is null"

I traced this back to:

- gsub:
    field: fortinet.firewall.eventtime
    pattern: "\\d{6}$"
    replacement: ""
    if: "(ctx.fortinet?.firewall?.eventtime).length() > 18"

seems in all of the forticlient logs, there is no eventtime field

i fixed my pipeline with:

  {
    "set": {
      "field": "fortinet.firewall.eventtime",
      "value": "{{fortinet.firewall.timestamp}}",
      "if": "(ctx.fortinet?.firewall?.eventtime) == null"
    }

[P1llus]

The thing is that this pipeline was only made for Fortigate @fredtj . Any other sort of Fortinet product will current not work. It would need a brand new fileset and a new pipeline. I wouldn't be surprised there is many similarities, but it will still need to be made again from scratch.

Any error calling Object.getClass() null is because of if conditions or script processors.
The above could also have been resolved by changing:
if: "(ctx.fortinet?.firewall?.eventtime).length() > 18"
to
if: "(ctx?.fortinet?.firewall?.eventtime).length() > 18"

The reason I don't have that in there is simply because on fortigate logs, ctx.fortinet does always exist.

[fredtj]

thanks @P1llus

[fredtj]

some support for Forticlient has been added to the Fortinet module by @adriansr - however from the test logs they seem to be targeting really old (unsupported) versions - is this an oversight? some limited research shows those logs to be from version 3 of Forticlient, which was around at least in 2008. there are some more up to date forticlient logs attached above, but if more are required just let me know.

[jamiehynds]

@fredtj the Forticlient module will be experimental to begin with to ensure we can iterate on the parser to cover a broader set of events before we officially support the module. We have a limited dataset to base this module on, so thank you for providing the sample logs - they are really helpful. @adriansr and I will take a look at the logs you've attached and adjust the module accordingly. If there are additional events you'd like to cover, you can attach them to this issue. Thanks!

[jlind23]

Backlog grooming: Closing it for now until further activity.