PostgreSQL module (#2253)

A start for the Metricbeat Postgresql module. The core functionality for connecting and an `activity` metricset is done.

CHANGELOG.asciidoc

@@ -68,6 +68,7 @@ https://github.com/elastic/beats/compare/v5.0.0-alpha5...master[Check the HEAD d
 
 - Use the new scaled_float Elasticsearch type for the percentage values. {pull}2156[2156]
 - Add cgroup metrics to the system/process MetricSet. {pull}2184[2184]
+- Added a PostgreSQL module. {pull}2253[2253]
 
 *Packetbeat*
 

metricbeat/docker-compose.yml

@@ -5,6 +5,7 @@ beat:
     - mongodb
     - mysql
     - nginx
+    - postgresql
     - redis
     - zookeeper
   environment:
@@ -19,6 +20,7 @@ beat:
     - MYSQL_DSN=root:test@tcp(mysql:3306)/
     - MYSQL_HOST=mysql
     - MYSQL_PORT=3306
+    - POSTGRESQL_DSN=postgres://postgres@postgresql:5432?sslmode=disable
     - ZOOKEEPER_HOST=zookeeper
     - ZOOKEEPER_PORT=2181
     - TEST_ENVIRONMENT=false
@@ -48,6 +50,9 @@ mysql:
 nginx:
   build: ${PWD}/module/nginx/_meta
 
+postgresql:
+  image: postgres:9.5.3
+
 redis:
   image: redis:3.2.3
 

metricbeat/docs/fields.asciidoc

@@ -18,6 +18,7 @@ grouped in the following categories:
 * <<exported-fields-mongodb>>
 * <<exported-fields-mysql>>
 * <<exported-fields-nginx>>
+* <<exported-fields-postgresql>>
 * <<exported-fields-redis>>
 * <<exported-fields-system>>
 * <<exported-fields-zookeeper>>
@@ -1258,6 +1259,153 @@ type: integer
 The current number of idle client connections waiting for a request.
 
 
+[[exported-fields-postgresql]]
+== PostgreSQL Fields
+
+Metrics collected from PostgreSQL servers.
+
+
+
+[float]
+== postgresql Fields
+
+PostgreSQL metrics.
+
+
+
+[float]
+== activity Fields
+
+One document per server process, showing information related to the current activity of that process, such as state and current query.
+
+
+
+[float]
+=== postgresql.activity.database.oid
+
+type: long
+
+OID of the database this backend is connected to.
+
+
+[float]
+=== postgresql.activity.database.name
+
+type: keyword
+
+Name of the database this backend is connected to.
+
+
+[float]
+=== postgresql.activity.pid
+
+type: integer
+
+Process ID of this backend.
+
+
+[float]
+=== postgresql.activity.user.id
+
+type: long
+
+OID of the user logged into this backend.
+
+
+[float]
+=== postgresql.activity.user.name
+
+Name of the user logged into this backend.
+
+
+[float]
+=== postgresql.activity.application_name
+
+Name of the application that is connected to this backend.
+
+
+[float]
+=== postgresql.activity.client.address
+
+IP address of the client connected to this backend.
+
+
+[float]
+=== postgresql.activity.client.hostname
+
+Host name of the connected client, as reported by a reverse DNS lookup of client_addr.
+
+
+[float]
+=== postgresql.activity.client.port
+
+type: integer
+
+TCP port number that the client is using for communication with this backend, or -1 if a Unix socket is used.
+
+
+[float]
+=== postgresql.activity.backend_start
+
+type: date
+
+Time when this process was started, i.e., when the client connected to the server.
+
+
+[float]
+=== postgresql.activity.transaction_start
+
+type: date
+
+Time when this process' current transaction was started.
+
+
+[float]
+=== postgresql.activity.query_start
+
+type: date
+
+Time when the currently active query was started, or if state is not active, when the last query was started.
+
+
+[float]
+=== postgresql.activity.state_change
+
+type: date
+
+Time when the state was last changed.
+
+
+[float]
+=== postgresql.activity.waiting
+
+type: boolean
+
+True if this backend is currently waiting on a lock.
+
+
+[float]
+=== postgresql.activity.state
+
+Current overall state of this backend. Possible values are:
+
+  * active: The backend is executing a query.
+  * idle: The backend is waiting for a new client command.
+  * idle in transaction: The backend is in a transaction, but is not
+    currently executing a query.
+  * idle in transaction (aborted): This state is similar to idle in
+    transaction, except one of the statements in the transaction caused
+    an error.
+  * fastpath function call: The backend is executing a fast-path function.
+  * disabled: This state is reported if track_activities is disabled in this backend.
+
+
+[float]
+=== postgresql.activity.query
+
+Text of this backend's most recent query. If state is active this field shows the currently executing query. In all other states, it shows the last query that was executed.
+
+
 [[exported-fields-redis]]
 == Redis Fields
 

metricbeat/docs/modules.asciidoc

@@ -12,6 +12,7 @@ about each module can be found under the links below.
   * <<metricbeat-module-mysql,MySQL>>
   * <<metricbeat-module-nginx,Nginx>>
   * <<metricbeat-module-redis,Redis>>
+  * <<metricbeat-module-postgresql,Postgresql>>
   * <<metricbeat-module-system,System>>
   * <<metricbeat-module-zookeeper,ZooKeeper>>
 
@@ -21,6 +22,7 @@ include::modules/apache.asciidoc[]
 include::modules/mongodb.asciidoc[]
 include::modules/mysql.asciidoc[]
 include::modules/nginx.asciidoc[]
+include::modules/postgresql.asciidoc[]
 include::modules/redis.asciidoc[]
 include::modules/system.asciidoc[]
 include::modules/zookeeper.asciidoc[]

metricbeat/docs/modules/postgresql.asciidoc

@@ -0,0 +1,42 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-module-postgresql]]
+== postgresql Module
+
+This is the postgresql Module.
+
+
+
+[float]
+=== Example Configuration
+
+The PostgreSQL module supports the standard configuration options that are described
+in <<configuration-metricbeat>>. Here is an example configuration:
+
+[source,yaml]
+----
+metricbeat.modules:
+#- module: postgresql
+  #metricsets: ["activity"]
+  #enabled: true
+  #period: 10s
+
+  # The host must be passed as PostgreSQL DSN. Example:
+  # postgres://pqgotest:password@localhost:5432?sslmode=disable
+  # The available parameters are documented here:
+  # https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters
+  #hosts: ["postgres://postgres@localhost:5432"]
+
+----
+
+[float]
+=== Metricsets
+
+The following metricsets are available:
+
+* <<metricbeat-metricset-postgresql-activity,activity>>
+
+include::postgresql/activity.asciidoc[]
+

metricbeat/docs/modules/postgresql/activity.asciidoc

@@ -0,0 +1,19 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-metricset-postgresql-activity]]
+include::../../../module/postgresql/activity/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-postgresql,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../module/postgresql/activity/_meta/data.json[]
+----

metricbeat/etc/beat.full.yml

@@ -103,6 +103,19 @@ metricbeat.modules:
   # Path to server status. Default server-status
   #server_status_path: "server-status"
 
+#----------------------------- PostgreSQL Module -----------------------------
+#- module: postgresql
+  #metricsets: ["activity"]
+  #enabled: true
+  #period: 10s
+
+  # The host must be passed as PostgreSQL DSN. Example:
+  # postgres://pqgotest:password@localhost:5432?sslmode=disable
+  # The available parameters are documented here:
+  # https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters
+  #hosts: ["postgres://postgres@localhost:5432"]
+
+
 #-------------------------------- Redis Module -------------------------------
 #- module: redis
   #metricsets: ["info", "keyspace"]

metricbeat/etc/fields.yml

@@ -727,6 +727,97 @@
               type: integer
               description: >
                 The current number of idle client connections waiting for a request.
+- key: postgresql
+  title: "PostgreSQL"
+  description: >
+    Metrics collected from PostgreSQL servers.
+  short_config: false
+  fields:
+    - name: postgresql
+      type: group
+      description: >
+        PostgreSQL metrics.
+      fields:
+        - name: activity
+          type: group
+          description: >
+            One document per server process, showing information related to the current
+            activity of that process, such as state and current query.
+          fields:
+            - name: database.oid
+              type: long
+              description: >
+                OID of the database this backend is connected to.
+            - name: database.name
+              type: keyword
+              description: >
+                Name of the database this backend is connected to.
+            - name: pid
+              type: integer
+              description: >
+                Process ID of this backend.
+            - name: user.id
+              type: long
+              description: >
+                OID of the user logged into this backend.
+            - name: user.name
+              description: >
+                Name of the user logged into this backend.
+            - name: application_name
+              description: >
+                Name of the application that is connected to this backend.
+            - name: client.address
+              description: >
+                IP address of the client connected to this backend.
+            - name: client.hostname
+              description: >
+                Host name of the connected client, as reported by a reverse DNS lookup of client_addr.
+            - name: client.port
+              type: integer
+              description: >
+                TCP port number that the client is using for communication with this
+                backend, or -1 if a Unix socket is used.
+            - name: backend_start
+              type: date
+              description: >
+                Time when this process was started, i.e., when the client connected to
+                the server.
+            - name: transaction_start
+              type: date
+              description: >
+                Time when this process' current transaction was started.
+            - name: query_start
+              type: date
+              description: >
+                Time when the currently active query was started, or if state is not
+                active, when the last query was started.
+            - name: state_change
+              type: date
+              description: >
+                Time when the state was last changed.
+            - name: waiting
+              type: boolean
+              description: >
+                True if this backend is currently waiting on a lock.
+            - name: state
+              description: >
+                Current overall state of this backend. Possible values are:
+        
+                  * active: The backend is executing a query.
+                  * idle: The backend is waiting for a new client command.
+                  * idle in transaction: The backend is in a transaction, but is not
+                    currently executing a query.
+                  * idle in transaction (aborted): This state is similar to idle in
+                    transaction, except one of the statements in the transaction caused
+                    an error.
+                  * fastpath function call: The backend is executing a fast-path function.
+                  * disabled: This state is reported if track_activities is disabled in this backend.
+            - name: query
+              description: >
+                Text of this backend's most recent query. If state is active this field
+                shows the currently executing query. In all other states, it shows the
+                last query that was executed.
+
 - key: redis
   title: "Redis"
   description: >