Bazel tests are flaky

[johanbrandhorst]

The bazel tests fail with connection refused errors every now and then. Seems we might need to wait a little longer for the test server to start.

[johanbrandhorst]

Happened again today, but seems to be bazel consuming too many resources and the host killing the process? Not quite sure.

https://circleci.com/gh/grpc-ecosystem/grpc-gateway/3484

[achew22]

My vote is that we activate the Bazel team's CI and deactivate the CircleCI based one. Step 1 in that process is here #983

[srenatus]

Cool. So, in buildkite.com/bazel, test fail consistently. Like this one: https://buildkite.com/bazel/grpc-ecosystem-grpc-gateway/builds/104#8f57d1c1-3082-4a3d-8ad3-f1c35f0bf987

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //examples/integration:go_default_test
-----------------------------------------------------------------------------
E0911 09:09:51.099595      14 main.go:73] Failed to listen and serve: listen tcp :8080: bind: address already in use
cannot run gateway service: listen tcp :8080: bind: address already in use

(from the uploaded test logs in buildkite artifacts)

👀 I guess it's either running test in parallel that shouldn't, or... something else is bound to 8080. Either way, making the tests use a random available port (i.e. binding to :0, then retrieving the port that was used) could be an alternative 🤔

[johanbrandhorst]

TIL. Sounds like unintentional parallelism to me. Unfortunately I understand nothing of how this works, so I defer to @achew22.

[srenatus]

Well, I'm sure our tests would be more robust if they could run in parallel. 😃 But yeah, that's the harder route for sure.

[drigz]

FYI, if you add tags = ["exclusive"] for a Bazel test it will not be run in parallel with other tests, so you can use this for tests that bind to a fixed port.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[johanbrandhorst]

Flipping stale bot giving me lip

[johanbrandhorst]

Still flaky: https://circleci.com/gh/grpc-ecosystem/grpc-gateway/5222

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[johanbrandhorst]

Sigh

[adasari]

@johanbrandhorst still there tests which is uses ports (secondary) in test. listener.Close() is not synchronous and never guarantee that port is available for immediate use in other test. Using unique ports fixes the problem.

[johanbrandhorst]

Thanks for pointing that out, it could be worth doing something about, but this issue is specifically around the bazel tests, which are failing independently of the other tests (which run the same tests) due to what looks like resource constraints a lot of the time.

[johanbrandhorst]

Might be fixed by caching the bazel artifacts and using explicit limits