Sometimes postgres write broken pipe 500

[vlad-tokarev]

Describe the bug

Error: Got error from server: write tcp 10.44.0.100:38068->10.44.138.33:5432: write: broken pipe (status: 500)
Makefile:214: recipe for target 'setup-e2e-robot' failed

https://jenkins.cicd.odahu.org/job/GCP_Create_cluster/1881/console
To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Actual behavior
A clear and concise description of what really happened.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS, Win10]
• Browser (if applicable) [e.g. chrome, mozilla]
• cli, sdk versions [e.g. 1.2]

Additional context
Add any other context about the problem here.

[easokol]

Additional info
lib/pq#870
lib/pq#871
Not merged for now yet.

[karbyshevds]

TCP connection got terminated by envoy sidecar with default timeout 60m. We can increase it but it will only prorogue the issue. Meanwhile postgresql deal with it correctly and closes broken connections but go driver is not.
We need to use driver that can deal with such situations or create a workaround in application logic and do not return broken pipe.
Run separate query to check connection status before starting actual query is also not a good option - connection in the pool should be checked with keepalive of sample requests separately.