postgres check, Broken pipe, doesn't reestablish connection

[miketheman]

 postgres
    --------
      - instance #0 [ERROR]: error(32, 'Broken pipe')
      - Collected 0 metrics, 0 events & 0 service checks

This occurs after a pg-related blip, like a pg service restart. The connection was broken, but it has recovered. It will remain in this state until the Agent is restarted.

I would expect a connection retry at some point.

[remh]

2014-08-07 14:52:50 UTC | ERROR | dd.collector | checks.postgres(__init__.py:552) | Check 'postgres' instance #0 failed
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/checks/__init__.py", line 543, in run
    self.check(copy.deepcopy(instance))
  File "/opt/datadog-agent/agent/checks.d/postgres.py", line 325, in check
    self._collect_stats(key, db, tags, relations)
  File "/opt/datadog-agent/agent/checks.d/postgres.py", line 176, in _collect_stats
    cursor.execute(query.replace(r'%', r'%%'))
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/pg8000/core.py", line 486, in execute
    self._conn, operation, args, statement_name="")
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/pg8000/core.py", line 1907, in __init__
    self.c.parse(self, self.statement)
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/pg8000/core.py", line 1390, in parse
    (PARSE, val), (DESCRIBE, desc_data), SYNC, FLUSH)
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/pg8000/core.py", line 1469, in _send_messages
    self._write(msg[0] + i_pack(len(msg_data) + 4))
  File "/opt/datadog-agent/embedded/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/opt/datadog-agent/embedded/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_of

[alq666]

Since I'm doing a bit of work with postgres, I can take a look at this one too.

[remh]

An easy way is to catch the broken pipe error and to raise a ShouldRestartException

[remh]

https://github.com/DataDog/dd-agent/blob/master/checks.d/postgres.py#L8

[LeoCavaille]

Reopening this issue as it was reported to still occur with agent 5.3.

[yannmh]

@LeoCavaille do we have more details on this new case ? I am pushing back the issue to 5.5.0 for lack of time.

[talwai]

Bumping to 5.6.0 in the absence of more definition on this

[remh]

Closing for lack of details.