Firewall Integration Input Consistency

[jamiehynds]

We currently support several firewall integrations including Cisco, Palo Alto, Check Point and more. However, there are inconsistencies across the ingest options, such as supported protocols and syslog format (UDP/TCP/TCP+TLS) and syslog format (RFC3164 vs RFC5424). We need to ensure each integration is consistent across protocols and syslog format supported. Given how popular the firewall Beats modules are, we should consider enhancement to the modules as part of this effort too.

Cisco ASA/FTD/IOS

• UDP
• TCP
• TCP + TLS
• RFC3164
• RFC5424

Check Point

• UDP
• TCP
• TCP + TLS
• RFC3164
• RFC5424

Fortinet

• UDP
• TCP
• TCP + TLS
• RFC3164
• RFC5424

Juniper SRX

• UDP
• TCP
• TCP + TLS
• RFC3164
• RFC5424

Palo Alto

• UDP
• TCP
• TCP + TLS (IETF format recommended by PANW)
• RFC3164
• RFC5424 - [Palo Alto/panw.panos] Support Syslog RFC5424 beats#26430

Sophos XG

• UDP
• TCP
• TCP + TLS
• RFC3164
• RFC5424

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[poochwashere]

Excellent. Is there any estimated timeline for this? I am specifically interested in seeing the Palo Alto TCP + TLS support. Thanks!

[jamiehynds]

@poochwashere unfortunately I can't share an exact timeframe, but we're about to begin development, so you shouldn't have too long to wait. Will certainly keep you posted as we progress through it.

For Palo Alto TCP + TLS, do you know if you can use the BSD message format, or are you limited to IETF when using TCP + TLS? (cc: @taylor-swanson)

[jamiehynds]

@taylor-swanson as you work through this one, Palo Alto is the priority as we only support UDP currently, and get a large volume of requests for TCP, TCP+TLS, etc. We have a virtual PANW license if you need to avail of it for testing.

[taylor-swanson]

@poochwashere unfortunately I can't share an exact timeframe, but we're about to begin development, so you shouldn't have too long to wait. Will certainly keep you posted as we progress through it.

For Palo Alto TCP + TLS, do you know if you can use the BSD message format, or are you limited to IETF when using TCP + TLS? (cc: @taylor-swanson)

We should be able to use either message format for the Palo Alto integration as long as there's no limitation on the firewall side. It currently uses a loose version of the BSD format (RFC 3164, but it lacks the syslog priority field, at least from what I can see based on our system tests), but the grok pattern should be able to handle either format as long as the message itself doesn't change.

[taylor-swanson]

@jamiehynds, do we have any sample logs in the RFC 5424 (IETF) format for Palo Alto? I mainly want to know what the format of the actual message is, if it's CSV like RFC 3164 or if it uses the structured data fields from RFC 5424. If it's CSV, then we can support IETF with basically zero added effort.

EDIT: I think I answered my question based on an SDH linked to the Support RFC 5424 issue mentioned above, and CSV was the format used.

[taylor-swanson]

@jamiehynds, the syslog formats we currently support widely vary depending on the integration. I'm able to add TCP and TLS to every integration with relative ease, but I don't think the same can be said about the syslog formats. We are largely limited on what the device sends us, some may not even support both formats or even just one of them (some use a custom format).

For reference, BSD format is RFC 3164, IETF format is RFC 5424.
Here's the current state of things, along with some questions:

• Check Point uses IETF format, with their own format for structured data. Do we know if they support BSD format?

• The Cisco Products seem to use their own format loosely based on BSD (close enough that I'd call it that for the sake of this discussion). Do we know if they support IETF format?

• Fortinet uses their own style. The only thing I see that is based on syslog is that they use a priority value (for instance, <30>) at the start of the message. The rest of it is just key=value pairs. It'd be a stretch, but you could call it BSD format. Do we know if they support IETF format?

• Juniper SRX adheres to IETF. Do we know if they support BSD format?

• PANW supports both formats and the new syslog processor can handle either format.

• Sophos XG uses a format that is identical to Fortinet. The official Sophos documentation mentions two formats, "Device Standard Format" and "Central Reporting Format". I believe we are currently using "Central Reporting Format", since this is the default and the integration documentation says to use the default format. Do we have any examples of what the "Device Standard Format" looks like, and do we want to support it?

Depending on the answers, we probably want to split supporting the other formats to separate issues (since there's a larger amount of work and testing that will need to be done on the pipelines).

[poochwashere]

I am here if you need a beta tester for the PANW module.

[jamiehynds]

Thanks for investigating @taylor-swanson. I'll create a new issue for supporting the other formats and remove those items from this issue.

Based on some quick research:

• Check Point - both RFC3164 and RFC5424 are supported. Docs here.
• Cisco: RFC5424 timestamp format seems to be an option for ASA logs.
• Fortinet: Their syslog configuration now includes the option to enable RFC5424 format. Docs here.
• Juniper SRX: Can't determine if BSD is supported.
• Sophos XG: Device Standard Format isn't a standard syslog format according to their docs. Fine to limit our support to Central Reporting Format.

[jamiehynds]

@poochwashere thanks for the offer. In addition to supporting secure syslog and IETF for Palo Alto, we're also in the process of updating our integration to cover more of their event types, see here: #2988

Will definitely be seeking some beta testers, and will be in touch as soon as we have something to share.

[rhyguy]

Hello everyone. I've been closely watching this issue because I've had an open support ticket for proper Palo Alto setup over TLS for quite a while. I'm excited to see the progress!

Do you know what level of backward compatibility to expect? I'm currently running a 7.11.1 cluster. If I need to update the cluster, or FileBeat in order to take advantage of your efforts, I'd like to get ahead of the game if possible.

[taylor-swanson]

@rhyguy, we're currently targeting 8.2.1 and 8.3.0 for the Palo Alto integration (it'll support TCP, TCP with TLS, and both syslog formats). The Palo Alto integration package was recently updated and support for 7.x was unfortunately dropped.

If you happen to be using the filebeat module, I believe you should be able to edit the module config file using a method similar to the one mentioned in this issue: elastic/beats#26430. To enable TCP, you'd change the protocol.udp: line to say protocol.tcp:. If you need to support the RFC 5424 format as well, then add the format: auto line as well to support both formats.

[taylor-swanson]

The work on inputs has been completed and the issue for format consistency can be found here: #3377