//knative/serving/test/e2e:TestAutoscaleUpDownUp is super flaky

[adrcunha]

Expected Behavior

//knative/serving/test/e2e:TestAutoscaleUpDownUp flakiness is close to 0.

Actual Behavior

For the last 16 CI runs, TestAutoscaleUpDownUp failed 10 times, or ~60% of the time. From these 10 failures, 9 were because ("got/wanted" numbers change for each failure)

autoscale_test.go:212: Error during initial scale up: Error making requests for scale up. Got 160 successful requests. Wanted 162.

https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-serving-continuous/1057301584175697921

Steps to Reproduce the Problem

1. Run the test a few times.

Additional Info

[mattmoor]

/assign @dgerd

@josephburnett Dan volunteered to help a bit here.

[tcnghia]

The autoscaling issue seems to be caused by requests hitting Terminating pods, in which istio-proxy usually dies first causing the Terminating pods to reject all message. that results in 503 upstream reset/disconnect before header. we used to have a prestop sleep like this tcnghia@20ca8e5#diff-d4961cee95b4d08627915faf1c62c4d3L1114, but later we could avoid doing that by adding retries. however, we are seeing the same issue again now.

I think we should add back the prestop sleep to unblock PRs, while working on a better fix.

[tcnghia]

/assign @tcnghia

[josephburnett]

One other action item might be to create another E2E test that just serves variable traffic over time. That would help narrow in on the cause more quickly I think. Up Down and Up is a lot to debug all at once.

[tcnghia]

@dgerd has an excellent way to repro this consistently by killing the revision's Pod. I think we should add that test.

[lvjing2]

Thanks for your help, this also fixed part of problems in #2311 and #2344

[lvjing2]

Hi, is this problem still there? if yes, then I'd like try to dig it also.

[adrcunha]

Actually it's way better now: in the last 24 runs, it only failed once. I'm closing this issue, thanks everyone.