Add new static_single_node config option

In essence, this configuration option will ensure that a static
single-node Patroni cluster does not demote the master (the one member
of the cluster) unnecessarily.

This changeset modifies the behavior of is_failover_possible. If the
cluster is configured with static_single_node=True, then no failover
will take place.

Transient failures to update the leader lock in the DCS will not cause a
demotion when running with static_single_node=True.

When running as leader under normal circumstances, DCS exceptions will
not cause a demotion when running with static_single_node=True.

docs/ENVIRONMENT.rst

@@ -11,6 +11,7 @@ Global/Universal
 -  **PATRONI\_NAME**: name of the node where the current instance of Patroni is running. Must be unique for the cluster.
 -  **PATRONI\_NAMESPACE**: path within the configuration store where Patroni will keep information about the cluster. Default value: "/service"
 -  **PATRONI\_SCOPE**: cluster name
+-  **PATRONI\_STATIC\_SINGLE\_NODE**: instructs Patroni to operate under the guarantee that the cluster will only be operated as a single-node cluster while this config value is True. In effect, this ensures that the master of the cluster does not unnecessarily demote itself under some circumstances.
 
 Log
 ---

docs/SETTINGS.rst

@@ -18,7 +18,8 @@ Dynamic configuration is stored in the DCS (Distributed Configuration Store) and
 -  **maximum\_lag\_on\_syncnode**: the maximum bytes a synchronous follower may lag before it is considered as an unhealthy candidate and swapped by healthy asynchronous follower. Patroni utilize the max replica lsn if there is more than one follower, otherwise it will use leader's current wal lsn. Default is -1, Patroni will not take action to swap synchronous unhealthy follower when the value is set to 0 or below. Please set the value high enough so Patroni won't swap synchrounous follower fequently during high transaction volume.
 -  **max\_timelines\_history**: maximum number of timeline history items kept in DCS.  Default value: 0. When set to 0, it keeps the full history in DCS.
 -  **master\_start\_timeout**: the amount of time a master is allowed to recover from failures before failover is triggered (in seconds). Default is 300 seconds. When set to 0 failover is done immediately after a crash is detected if possible. When using asynchronous replication a failover can cause lost transactions. Worst case failover time for master failure is: loop\_wait + master\_start\_timeout + loop\_wait, unless master\_start\_timeout is zero, in which case it's just loop\_wait. Set the value according to your durability/availability tradeoff.
-- **master\_stop\_timeout**: The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
+-  **master\_stop\_timeout**: The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
+-  **static\_single\_node**: instructs Patroni to operate under the guarantee that the cluster will only be operated as a single-node cluster while this config value is True. In effect, this ensures that the master of the cluster does not unnecessarily demote itself under some circumstances.
 -  **synchronous\_mode**: turns on synchronous replication mode. In this mode a replica will be chosen as synchronous and only the latest leader and synchronous replica are able to participate in leader election. Synchronous mode makes sure that successfully committed transactions will not be lost at failover, at the cost of losing availability for writes when Patroni cannot ensure transaction durability. See :ref:`replication modes documentation <replication_modes>` for details.
 -  **synchronous\_mode\_strict**: prevents disabling synchronous replication if no synchronous replicas are available, blocking all client writes to the master. See :ref:`replication modes documentation <replication_modes>` for details.
 -  **postgresql**:
@@ -182,7 +183,7 @@ ZooKeeper
 -  **key**: (optional) File with the client key.
 -  **key_password**: (optional) The client key password.
 -  **verify**: (optional) Whether to verify certificate or not. Defaults to ``true``.
--  **set_acls**: (optional) If set, configure Kazoo to apply a default ACL to each ZNode that it creates. ACLs will assume 'x509' schema and should be specified as a dictionary with the principal as the key and one or more permissions as a list in the value.  Permissions may be one of ``CREATE``, ``READ``, ``WRITE``, ``DELETE`` or ``ADMIN``.  For example, ``set_acls: {CN=principal1: [CREATE, READ], CN=principal2: [ALL]}``. 
+-  **set_acls**: (optional) If set, configure Kazoo to apply a default ACL to each ZNode that it creates. ACLs will assume 'x509' schema and should be specified as a dictionary with the principal as the key and one or more permissions as a list in the value.  Permissions may be one of ``CREATE``, ``READ``, ``WRITE``, ``DELETE`` or ``ADMIN``.  For example, ``set_acls: {CN=principal1: [CREATE, READ], CN=principal2: [ALL]}``.
 
 .. note::
     It is required to install ``kazoo>=2.6.0`` to support SSL.

docs/releases.rst

@@ -3,6 +3,15 @@
 Release notes
 =============
 
+Version 2.2.0
+-------------
+
+**New features**
+
+- Added support for static_single_node configuration ``patronictl`` (Anthony Dodd)
+
+  This can be configured using the ``static_single_node=True`` config value, which enables a few optimizations to ensure a single-node master does not demote.
+
 Version 2.1.3
 -------------
 
@@ -1036,7 +1045,7 @@ Version 1.6.1
 
 - Kill all children along with the callback process before starting the new one (Alexander Kukushkin)
 
-  Not doing so makes it hard to implement callbacks in bash and eventually can lead to the situation when two callbacks are running at the same time. 
+  Not doing so makes it hard to implement callbacks in bash and eventually can lead to the situation when two callbacks are running at the same time.
 
 - Fix 'start failed' issue (Alexander Kukushkin)
 

patroni/config.py

@@ -65,6 +65,7 @@ class Config(object):
         'check_timeline': False,
         'master_start_timeout': 300,
         'master_stop_timeout': 0,
+        'static_single_node': False,
         'synchronous_mode': False,
         'synchronous_mode_strict': False,
         'synchronous_node_count': 1,

patroni/ha.py

@@ -128,6 +128,13 @@ def is_leader(self):
         with self._is_leader_lock:
             return self._is_leader > time.time()
 
+    def is_static_single_node(self):
+        if self.patroni.config is None:
+            return False
+        # QUESTION to reviewers: should we check the last held DCS config as well just to ensure there is no conflict?
+        # Perhaps we should just force this value to only be considered as part of non-dynamic config, thoughts?
+        return self.patroni.config.get('static_single_node', False)
+
     def set_is_leader(self, value):
         with self._is_leader_lock:
             self._is_leader = time.time() + self.dcs.ttl if value else 0
@@ -689,7 +696,8 @@ def _is_healthiest_node(self, members, check_replication_lag=True):
     def is_failover_possible(self, members, check_synchronous=True, cluster_lsn=None):
         ret = False
         cluster_timeline = self.cluster.timeline
-        members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url]
+        is_static_single_node = self.is_static_single_node()
+        members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url and not is_static_single_node]
         if check_synchronous and self.is_synchronous_mode():
             members = [m for m in members if self.cluster.sync.matches(m.name)]
         if members:
@@ -1043,6 +1051,8 @@ def process_healthy_cluster(self):
                 if self.state_handler.is_leader():
                     if self.is_paused():
                         return 'continue to run as master after failing to update leader lock in DCS'
+                    if self.is_static_single_node():
+                        return 'continue to run as master after failing to update leader lock in DCS due to static_single_node config'
                     self.demote('immediate-nolock')
                     return 'demoted self because failed to update leader lock in DCS'
                 else:
@@ -1487,7 +1497,7 @@ def _run_cycle(self):
         except DCSError:
             dcs_failed = True
             logger.error('Error communicating with DCS')
-            if not self.is_paused() and self.state_handler.is_running() and self.state_handler.is_leader():
+            if not self.is_paused() and self.state_handler.is_running() and self.state_handler.is_leader() and not self.is_static_single_node():
                 self.demote('offline')
                 return 'demoted self because DCS is not accessible and i was a leader'
             return 'DCS is not accessible'

patroni/version.py

@@ -1 +1 @@
-__version__ = '2.1.3'
+__version__ = '2.2.0'