Firewall Integrations | Supported Formats

[jamiehynds]

We currently support several firewall integrations including Cisco, Palo Alto, Check Point and more. However, there are inconsistencies across which formats are supported across each integration (e.g. RFC3164 and RFC5424). We need to ensure each integration is consistent across syslog format supported.

Cisco ASA/FTD/IOS
RFC5424 timestamp format seems to be an option for ASA logs.

• RFC3164
• RFC5424

Check Point
RFC3164 and RFC5424 are supported. Docs [here](https://sc1.checkpoint.com/documents/R80.20_GA/WebAdminGuides/EN/CP_R80.20_LoggingAndMonitoring_AdminGuide/html_frameset.htm?

• [ ] RFC3164 (Log exporter only supports RFC 5424)
• RFC5424

Fortinet
Syslog configuration now includes the option to enable RFC5424 format. Docs here.

• RFC3164

Juniper SRX

• RFC5424

Palo Alto

• RFC3164
• RFC5424 - [Palo Alto/panw.panos] Support Syslog RFC5424 beats#26430

Next round of supported formats tracked in #4077

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[taylor-swanson]

Looks like we already support RFC 5424 timestamps (technically, it's RFC 3339 for the timestamp) for Cisco ASA and FTD. I'll add some pipeline tests for regression tests.

I'm not sure about Cisco IOS, though. I'm not finding any concise answers about what formats Cisco IOS supports. Our sample logs seem to suggest it's derived from RFC 3164. I'm not sure if there are other variations (RFC 5424) that are supported.

@jamiehynds, do you know if we have access to a Cisco IOS device? Or if we can get sample logs of different formats (if they exist).

[jamiehynds]

Thanks for looking into it @taylor-swanson. We can probably get a license from Cisco, but I can try to get some samples to avoid spinning up an appliance and manually generating events. I'm not overly worried about IOS as firewalls are the priority for now.

[taylor-swanson]

@jamiehynds, I'm circling back to this issue again. No worries on the IOS side of things.

If we can get example logs for the missing items above, that would be great. I can crank them out quickly once I have samples (assuming no other issues come up). That list appears to be:

• Check Point: RFC 3164
• Fortinet: RFC 5424 (like Sophos, it has a strange log format, so I really need to see examples of this one)
• Juniper SRX: RFC 3164

Regarding Sophos XG, we are already supporting the Central Reporting Format, so no further actions required there. We do have a user complaining our integration doesn't handle TCP delimiters correctly. According them, we should be using null terminators instead of newlines. A PCAP can confirm this. Do we have any way of getting something like that? (This is being handled in #3482)

[jamiehynds]

@taylor-swanson I've reached out to our partnership contacts at CheckPoint, Fortinet and Sophos - so we can hopefully get access to samples/appliances pretty quickly.

Regarding Juniper, we have some sample data from our pipeline tests that I think are in RF3164 format - could we use these as sample data? https://github.com/elastic/integrations/tree/main/packages/juniper_srx/data_stream/log/_dev/test/pipeline

[taylor-swanson]

Thanks, @jamiehynds!

Those Juniper logs all appear to be RFC 5424 (that'd be anything that starts with <number>number, i.e. <165>1). They also leverage specific fields/features that don't exist in RFC3164. The sample logs for system tests are also in the same format.

[taylor-swanson]

@jamiehynds, I looked at Juniper SRX a bit closer for RFC 3164 logs. This looks to be a significant undertaking. Here would be an example log using that format:

Nov 4 16:23:09 cixi RT_FLOW: RT_FLOW_SESSION_CREATE: session created 50.0.0.100/24065->30.0.0.100/768 icmp 50.0.0.100/24065->30.0.0.100/768 None None 1 alg-policy untrust trust 100000165 N/A(N/A) reth2.0 UNKNOWN UNKNOWN UNKNOWN

The problem is that none of the fields are labeled, so we'd have to have full documentation on every log emitted to know what these fields mean. It might end up being a grok processor for every possible message type if they don't follow a consistent pattern.

According to our docs, we only support syslog messages in the format "structured-data + brief". The structured-data part is unique to RFC 5424, not something RFC 3164 can do.

Do we have a lot of requests for the RFC 3164 format? If not, I suggest we don't implement it. It's a much worse format that lacks precision in a lot of areas (timestamp, lack of field names, structure, etc).

[jamiehynds]

@taylor-swanson ++ on not implementing it and just sticking with 5424 support. Not enough demand to warrant such a heavy lift to support 3164. Thanks for looking into it nonetheless.

[jamiehynds]

We've now reached sufficient progress to close this issue. There are a few outstanding vendors where we need to expand support syslog formats, but will take some time to gather samples from those vendors.

Please see this new issue for updates: #4077