Incorporating IPFIX/Netflow fields into existing ECS field sets #732
Conversation
|
I have completed the CLA. Is there a way to rerun that check? |
|
Thanks for the PR! I haven't had a chance to go thru the whole thing, but wondering if you had seen the INTERFACE and VLAN fields proposed earlier - we didn't go with the rfc .1q names in the original PR, which I'm thinking are worth putting into the description if not considering renaming the field names. We had a couple of people internally who had run into more than 2 .1q tags in provider networks, etc, which led us towards discrete inner & outer vlan id & names, as well as a related.vlan.id/name for an array of any other additional encapsulations. Interface Fields: #689 We are looking at allowing the interface and vlan fields to be nested to allow for the egress / ingress interface and vlan info for firewalls, routers/switches, load balancers, while maintaining the core set of fields under the same header. t looked like there were a number of fields allowing for hex representations OR text descriptions, I think separating those out into 2 fields is likely a good thing. Thanks |
|
@dainperkins Thanks for the comments and feedback! I glanced over the discussion on VLAN identifiers, but had not noticed the INTERFACE PR, thanks for pointing out those. It looks like the VLAN field set and its fields as proposed in PR #688 could handle the IPFIX/Netflow use case. Similarly, the Interface field set and its fields as proposed in PR #689 might be able to handle most of the IPFIX/Netflow use cases. In network flow data, the interface information usually maps back to an SNMP ifIndex field, so it would be good to have the option to list both an ID and Name if someone wanted to map more comprehensible names/descriptions to their less comprehensible index values. What are your thoughts on how to annotate interface information where it is related to an observed network event as entering or exiting a particular interface at that observation point? Do you think "source.interface.id = network.egress_interface" and "destination.interface.id = network.ingress_interface"? I can see where this might confound other source/destination information, at least in the IPFIX/Netflow use case. I am interested in hearing more on your thoughts of how to approach this. Regarding breaking out fields to allow for numeric and text representations, I agree that would be great. I initially overloaded those in an effort to keep the number of fields I was proposing down to a minimum 😃. Thanks again for taking the time to review this PR. |
|
after talking with @webmat I've put in a new PR with the interface / vlan fields as well as updates to observer to include ingress/egress zones. #752 I'd very much like to get the flow specific stuff here worked thru and in as well, and will try to get some more thoughts down on things like tcp control bits vs syn/fin/rtxmt, naming conventions, etc. so we can push some good updates for ipfix/flow data |
|
ok, back to it!
TCP flags (I skipped echo & ecn)
For COS, DSCP, & IP Precedence we need to cover ingress & egress in case of a change, potentially making egress required only when theres a change
And then if we get into flexible netflow theres all sorts of really useful N/APM goodies (as well as the metrics I was previously questioning - I'm still not sure as a general "heres a netflow config to use with Elastic" standpoint I would be inclined to include observer aggregates, as opposed to just using visualizations to aggregate, but I'm unsure how relevant that is)
theres a whole lot there, but its pretty much standard as well, so it should be relatively easy to work thru. |
|
probably need to add vrfs as well... |
| @@ -574,6 +574,12 @@ | |||
| description: 'IP address of the destination. | |||
|
|
|||
| Can be one or multiple IPv4 or IPv6 addresses.' | |||
| - name: ip.network_prefix | |||
There was a problem hiding this comment.
just to know if this type of definition is possible:
ip is already a field of type IP not a json object
There was a problem hiding this comment.
Perhaps the IP address should become ip.address?
There was a problem hiding this comment.
honestly, "ip" fields exist all around since the first incarnation of the ECS. I would rather prefer to have a simple "network_prefix" field at the same level as the corresponding ip one
Splitting out the ICMP type and code sounds like a good idea, would make querying a bit easier. For the TCP flags, an array of flags sounds like a good approach, again to make querying easier. |
There was a problem hiding this comment.
Do you by any chance have e.g. a fnf / ipfix output you could share? I remember being able to track retransmits, resets, etc. from netflow/ipfix in other products, but i can't for the life of me recall how that was provided (duplicate syns are pretty straightforward, but if they both come in the same netflow interval is the an indicator in the flow record?).
| - name: port | ||
| format: string | ||
| level: core | ||
| type: long | ||
| description: > | ||
| Port of the source. | ||
|
|
||
| - name: port_name |
There was a problem hiding this comment.
Does this make sense thinking about a typical network connection? Source ports will generally be random high ports (specific protocols not withstanding - e.g. dhcp, kerberos)
| @@ -31,13 +31,31 @@ | |||
|
|
|||
| Can be one or multiple IPv4 or IPv6 addresses. | |||
|
|
|||
| - name: ip.network_prefix | |||
There was a problem hiding this comment.
Definitely seems like a potentially useful representation, but curious how the implementation/ingest would work from a typical network observer (sensor, fw, netflow, etc).
|
A stage 0 ECS RFC proposal has been submitted to expand the network headers currently defined in ECS. It appears there's some overlap between that RFC and the proposed changes here? This PR been inactive for awhile now, and it's also expanded from the original description quite a bit. I'd like to centralize the discussion in the network headers RFC and will close this out for now. We can always re-open this PR if needed. |
Added 2 additional fields to both the destination and source field sets to capture IP network prefixes/masks and text descriptions of common network ports. Added 11 additional fields to the network field set to capture interface, VLAN (Virtual Local Area Network), ICMP, IP, and TCP information as well as bit rates and packet rates.