Support version skew between Antrea Agent and Flow Aggregator #6777
Assignees
Labels
area/flow-visibility/aggregator
Issues or PRs related to Flow Aggregator
area/flow-visibility
Issues or PRs related to flow visibility support in Antrea
kind/feature
Categorizes issue or PR as related to a new feature.
Milestone
Comments
antoninbas
added
kind/feature
|
@tnqn @heanlan @yuntanghsu for visibility |
|
Example reproduction (updating Agent first): The FlowAggregator will panic once: The panic is when decoding a data set: the same template ID is used by the (new) Agent, the new template is rejected because of an unknown registry element ( After the FlowAggregator restarts, the template is never accepted and all messages end up being dropped. |
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Jan 8, 2025
When a new IPFIX Information Element (IE) is introduced, a version mismatch between the Agent and the Flow Aggregator can be problematic. A "new" Agent can send an IE which is unknown to the "old" Flow Aggregator, or the "new" Flow Aggregator may expect an IE which is not sent by an "old" Agent. Prior to this change, we required the list of IEs sent by the Agent to be the same as the list of IEs expected by the Flow Aggregator. This is impossible to ensure during upgrade, as it may take a long time for all Agents in the cluster to be upgraded. After this change, Agents and Flow Aggregator can be upgraded in any order (although we would recommend the Flow Aggregator to be upgraded last). To achieve this, we introduce a new "process" between IPFIX collection and aggregation in the Flow Aggregator: the "preprocessor". The preprocessor is in charge of processing messages received from the IPFIX collector, prior to handling records over to the aggregation process. At the moment, its only task is to ensure that all records have the expected fields. If a record has extra fields, they will be discarded. If some fields are missing, they will be "appended" to the record with a "zero" value. For example, we will use 0 for integral types, "" for strings, 0.0.0.0 for IPv4 address, etc. Note that we are able to keep the implementation simple by assuming that a record either has missing fields or extra fields (not a combination of both), and that such fields are always at the tail of the field list. This assumption is based on implementation knowledge of the FlowExporter and the FlowAggregator. When we introduce a new IE, it always comes after all existing IEs, and we never deprecate / remove an existing IE across versions. Note that when the preprocessor adds a missing field, it is no longer possible to determine whether the field was originally missing, or was sent by the Agent with a zero value. This is why we recommend upgrading the Flow Aggregator last (to avoid this situation altogether). However, we do not believe that it is a significant drawback based on current usage. Fixes antrea-io#6777 Signed-off-by: Antonin Bas <[email protected]>
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Jan 9, 2025
When a new IPFIX Information Element (IE) is introduced, a version mismatch between the Agent and the Flow Aggregator can be problematic. A "new" Agent can send an IE which is unknown to the "old" Flow Aggregator, or the "new" Flow Aggregator may expect an IE which is not sent by an "old" Agent. Prior to this change, we required the list of IEs sent by the Agent to be the same as the list of IEs expected by the Flow Aggregator. This is impossible to ensure during upgrade, as it may take a long time for all Agents in the cluster to be upgraded. After this change, Agents and Flow Aggregator can be upgraded in any order (although we would recommend the Flow Aggregator to be upgraded last). To achieve this, we introduce a new "process" between IPFIX collection and aggregation in the Flow Aggregator: the "preprocessor". The preprocessor is in charge of processing messages received from the IPFIX collector, prior to handling records over to the aggregation process. At the moment, its only task is to ensure that all records have the expected fields. If a record has extra fields, they will be discarded. If some fields are missing, they will be "appended" to the record with a "zero" value. For example, we will use 0 for integral types, "" for strings, 0.0.0.0 for IPv4 address, etc. Note that we are able to keep the implementation simple by assuming that a record either has missing fields or extra fields (not a combination of both), and that such fields are always at the tail of the field list. This assumption is based on implementation knowledge of the FlowExporter and the FlowAggregator. When we introduce a new IE, it always comes after all existing IEs, and we never deprecate / remove an existing IE across versions. Note that when the preprocessor adds a missing field, it is no longer possible to determine whether the field was originally missing, or was sent by the Agent with a zero value. This is why we recommend upgrading the Flow Aggregator last (to avoid this situation altogether). However, we do not believe that it is a significant drawback based on current usage. Fixes antrea-io#6777 Signed-off-by: Antonin Bas <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the problem/challenge you have
At the moment, we do not support version skew between Antrea Agent and Flow Aggregator. Let's take a concrete example:
In Antrea v2.0, we introduced a new Information Element (IE),
egressNodeName(#6012). It so happened that this was introduced in a major version release, but we also routinely introduce new IEs in minor version releases. If one tries to update the Antrea Agent (from v1.15 to v2.0) before the Flow Aggregator, the existing Flow Aggregator (v1.15) will reject the new IPFIX templates sent by the new Antrea Agent (v2.0) - see https://github.com/vmware/go-ipfix/blob/main/pkg/collector/process.go. If one tries to update the Flow Aggregator first it will also not work, as the aggregation process reuses the IPFIX data "record" received from the Agent, which will not match the template sent by the Flow Aggregator to the external IPFIX collector.For large clusters, a rolling update of the antrea-agent DaemonSet can take a while, so a version mismatch between some Agents and the Flow Aggregator is expected and that situation will remain until the update completes.
Describe the solution you'd like
We should tolerate some version skew between the Antrea Agent and the Flow Aggregator (N-2/N+2), for graceful updates and to ensure that connection data can still be exported during the update window.
For example:
Because item 2) is more problematic than 1) (what's an appropriate default value?), we could specify than in order to achieve graceful update, the Flow Aggregator should be updated last. In that case, we would have
version(FlowAggregator) <= version(Agent), and 1) would be sufficient. Once the FlowAggregator is itself updated, it will be able to "forward" the newly introduced IEs.The text was updated successfully, but these errors were encountered: