Network Performance tests: negative UDP latency values for some test runs.

[VivekThrivikraman-est]

What happened:
UDP latency metrics collected while running network performance test cases (clusterloader2/testing/network) intermittently contains negative values.

What you expected to happen:
UDP latency should be always positive.

How to reproduce it (as minimally and precisely as possible):
Issue is not seen when running locally in Kind or local vm cluster. Negative values are seen when the testcases are run in k8s CI/CD(GCE).

Anything else we need to know?:
It is confirmed from the logs that Iperf2 tool which is used for collecting the metrics return these -ve values. Raised this in iperf2 forum :
https://sourceforge.net/p/iperf2/discussion/general/thread/a1bcdb67d8/

They suspect this to be clock synchronization issue(they mentioned NTP synchronized nodes also can have latency).

Aggregated metrics output for UDP attached:
NetworkPerformanceMetrics_N_M_UDP_P2P_network_performance_udp-50_50_2021-05-21T08_39_04Z.json.txt

Environment:
Testcases are run as part of prow job in k8s CI/CD, in GCE.

[aojea]

We've been talking about this in slack, and since the VM synchronisation seems complex problem to fix , we though that the best solution moving forward will be to gather more data to get better insights, maybe we only need the trend and not the absolute values.

The doubt that we have is if it would be good to operate only with positive numbers, i.e. truncate negative values to 0

@wojtek-t what do you think?

[wojtek-t]

We do clock-synchronization in GCE - I personally doubt it's clock difference. I bet it's a real bug...

[VivekThrivikraman-est]

@wojtek-t is it NTP ?

[hanamantagoudvk]

@wojtek-t @aojea : What do you think of considering negative values as zero. ? Shall we proceed with that idea ?

[mm4tt]

It's NTP - https://cloud.google.com/compute/docs/instances/managing-instances#configure-ntp

[mm4tt]

If we're seeing negative values in order of microseconds then, IMHO, it's 99% clock synchronization / precision issue.

Have we seen a negative value strictly greater than microseconds? We can wait for perf-dash to contain the network-performance dashboards to confirm.

If the answer is not then the question is whether we care about this level of precision. My intuition is that we don't and it's not worth fixing. Changing the measurement to treat negative values as zero is probably the preferred course of action.

[VivekThrivikraman-est]

Thanks @mm4tt , as you mentioned we will first fix perfdash and monitor to make sure latency is always in microseconds and then make a decision.

[aojea]

Thanks @mm4tt , as you mentioned we will first fix perfdash and monitor to make sure latency is always in microseconds and then make a decision.

http://perf-dash.k8s.io/#/?jobname=network-performance&metriccategoryname=Network_Baseline&metricname=UDP_P2P&Metric=Latency

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[wojtek-t]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[wojtek-t]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.