Consul KV consistency checks

[shlomi-noach]

Work in progress to be able to audit and fix KV pair inconsistency, especially external KV pair store inconsistencies.

e.g. if upon failover Consul is unavailable, orchestrator could run a consistency check against its local KV store to detect and resolve such inconsistency.

Initial commit: writing Nano timestamp in all KV stores.

[shlomi-noach]

What this PR turned to be doing:

Assuming ConsulCrossDataCenterDistribution is set, when the orchestrator leader distributes master KV values, we run a thorough check on all known Consul datacenters to verify all KVs exist on all these clusters and are correct. This only happens on the orchestrator leader node.

There's caching to avoid abusing consul. On first iteration as leader, orchestrator will query all KV on all consul clusters. From there on, it will rely on caching of last-known value.

What this effectively does: if a consul cluster was down for any reason, and if some failover took place at that time, and as result orcehstartor could not update that cluster with the new KV data -- the change is not lost. Every minute, orchestrator will retry updating the cluster with the most up-to-date KV data.