Packetbeat flows enhancements

[urso]

Flows support in 5.0 is quite rudimentary, not really taking any network layers (arbitrary timeout) into account and only counting packets and bytes. Plus, there is a parent-child relation between flows and transactions, not yet represented in packetbeat events being published.

List of flow proposed enhancements:

• pass active flow to packet analyzer:
  • add protocol specific stats to flows (e.g. number of transactions with success/failure/dropped due to packet loss/timed out)
  • add flow id to protocol transaction events to establish some relationship between flows and transaction
• add support to tie flow lifetime to connection status:
  • Flows for TCP/SCTP streams do not timeout while TCP connection is active
  • Protocols on top of UDP managing connection state should be able to disable flow timeout
  • Stop flow if connection is closed or connection attempt failed:
    • for TCP stop flow on RST or normal TCP shutdown
    • consider ICMP messages if TCP/UDP port is not reachable
    • timeout flow in case of handshake being incomplete and no data being send in either direction
  • add indicator (string?) to indicate the reason a flow has ended
  • add indicator if connection has been good (established TCP connection), in case it ended forcefully
  • add indicator for flow starting to capture an older TCP connection (not having seen the connection attempt itself, as packetbeat was started after)
• report TCP level stats:
  • count flags usage (SYN, RST, FIN, PUSH)
  • report bytes lost due to packet-loss
  • report packets fully/partially resend (number of packets and bytes)

[tbragin]

@urso @adriansr Does this pull also partially address this issue? #5476

[urso]

@tbragin No. Flows support and integrating flows into application protocols is about collecting metrics only.

[ghost]

If using Packetbeat and not interested in any of the supported protocols can we have a generic "TCP" option where we can specify a list of ports to be able to filter on specific traffic flows before the analysis stage.

[urso]

@london2016 this can already be done by configuring your custom packet filter in the device configs. Please checkout the forums if you need any help.

[q2dg]

Well...version 7.0 is here and nothing has been resolved about this issue yet...

[Umarhayat3]

PB version 7.3.1 has been released but this issue is still there and many are facing problems due to lack of information in debugging mode. This flow enhancement is necessary for troubleshooting.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[faec]

Adding this to triage because we're still seeing requests for ways to measure tcp connection drops in particular and it looks like this fell off the radar for a while.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zez3]

would be a nice addition

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zez3]

Part of the proposed enhancements would still be useful to have.
@jamiehynds
Would it be possible to take some enhancements on the roadmap?

[wangxin688]

It would be a very nice feature to support translation layer for TCP/UDP, which will help measure network performance, especially for forwording devices(use linux as a gateway, running routing, NAT, and firewall function)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[willemdh]

.

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

[gaby]

This issue still work in progress after 7 years?