diff --git a/website/content/api-docs/variables/index.mdx b/website/content/api-docs/variables/index.mdx index 92ba5e78359..8f1e8bcb217 100644 --- a/website/content/api-docs/variables/index.mdx +++ b/website/content/api-docs/variables/index.mdx @@ -11,8 +11,11 @@ description: |- The `/var` and `/vars` endpoints are used to query for and interact with variables, and set up locks and leases over them. -See the [Variables] documentation for information how these capabilities are +See the [Variables][] documentation for information how these capabilities are used. For a CLI to perform these operations manually, please see the -documentation for the [`nomad var`] commands. +documentation for the [`nomad var`][] commands. Please choose a sub-section in the navigation for more information + +[`nomad var`]: /nomad/docs/commands/var +[Variables]: /nomad/docs/concepts/variables \ No newline at end of file diff --git a/website/content/api-docs/variables/locks.mdx b/website/content/api-docs/variables/locks.mdx index f575cdf9244..bbb79d75835 100644 --- a/website/content/api-docs/variables/locks.mdx +++ b/website/content/api-docs/variables/locks.mdx @@ -204,6 +204,7 @@ regular variables, locked variables can be created with or without any items. The lock TTL and Delay must be values between 10 seconds and 24 hours. [Variables]: /nomad/docs/concepts/variables +[restrictions]: /nomad/api-docs/variables/variables#restrictions [`nomad var`]: /nomad/docs/commands/var [blocking queries]: /nomad/api-docs#blocking-queries [required ACLs]: /nomad/api-docs#acls diff --git a/website/content/api-docs/variables/variables.mdx b/website/content/api-docs/variables/variables.mdx index bd071f36403..f53b0c7be4f 100644 --- a/website/content/api-docs/variables/variables.mdx +++ b/website/content/api-docs/variables/variables.mdx @@ -9,9 +9,8 @@ description: The /var endpoints are used to query for and interact with variable The `/var` and `/vars` endpoints are used to query for and interact with variables. -See the [Variables] documentation for information how these capabilities are -used. For a CLI to perform these operations manually, please see the -documentation for the [`nomad var`] commands. +See the [Variables][] documentation for information how these capabilities are +used. ## List Variables @@ -294,10 +293,8 @@ the response will include only metadata and not the `Items` field: } ``` - [Variables]: /nomad/docs/concepts/variables -[locks section]: /nomad/docs/commands/var/lock -[`nomad var`]: /nomad/docs/commands/var +[locks section]:/nomad/api-docs/variables/locks [blocking queries]: /nomad/api-docs#blocking-queries [required ACLs]: /nomad/api-docs#acls [RFC3986]: https://www.rfc-editor.org/rfc/rfc3986#section-2 diff --git a/website/content/docs/commands/var/index.mdx b/website/content/docs/commands/var/index.mdx index 16f7bd53347..695bf78dd96 100644 --- a/website/content/docs/commands/var/index.mdx +++ b/website/content/docs/commands/var/index.mdx @@ -21,6 +21,7 @@ subcommands are available: - [`var get`][get] - Retrieve a variable - [`var put`][put] - Insert or update a variable - [`var purge`][purge] - Permanently delete a variable +- [`var lock`][lock] - Acquire a lock over a variable ## Examples @@ -62,3 +63,4 @@ user = dba [list]: /nomad/docs/commands/var/list [put]: /nomad/docs/commands/var/put [purge]: /nomad/docs/commands/var/purge +[lock]: /nomad/docs/commands/var/lock diff --git a/website/content/docs/concepts/variables.mdx b/website/content/docs/concepts/variables.mdx index 85016a45e7b..6390de96f14 100644 --- a/website/content/docs/concepts/variables.mdx +++ b/website/content/docs/concepts/variables.mdx @@ -186,6 +186,55 @@ apply. See [Workload Associated ACL Policies] for more details. +## Locks + +Nomad provides the ability to block a variable from being updated for a period +of time by setting a lock on it. Once a variable is locked, it can be read by +everyone, but it can only be updated by the lock holder. + +The locks are designed to provide granular locking and are heavily inspired by +[The Chubby Lock Service for Loosely-Coupled Distributed Systems][]. + +A lock is composed of an ID, a TTL, and a lock delay. The ID is generated by +the Nomad Server and must be provided on every request to modify the variable's +items or the lock itself. The TTL defines the time the lock will be held; if the +lock needs to be in place for longer, it can be renewed for as many new periods +as wanted. + +Once it is no longer needed, it must be released. If by the time the TTL expires, +and no renew or release calls were made, the variable will remain locked for at +least the lock delay duration, to avoid a possible split-brain situation, where +there are two holders at the same time. + +### Leader election backed by Nomad Variable Locks + +For some applications, like HDFS or the Nomad Autoscaler, it is necessary to +have multiple instances running to ensure redundancy in case of a failure, but +only one of them may be active at a time as a leader. + +As part of the [Go Package][], Nomad offers a helper that takes one variable and +uses a lock over it as a sync mechanism to run multiple instances but always +keeping just one running at any given time, using the following algorithm: + +[![Leader Election Helper](/img/nomad-vars-leader-election-sdk.png)](/img/nomad-vars-leader-election-sdk.png) + +As soon as any instance starts, it tries to lock the sync variable. If it succeeds, +it continues to execute while a secondary thread is in charge of keeping track of +the lock and renewing it when necessary. If by any chance the renewal fails, +the main process is forced to return, and the instance goes into standby until it +attempts to acquire the lock over the sync variable. + +Only threads 1 and 3 or thread 2 are running at any given time, because every +instance is either executing as normal while renewing the lock or waiting for a +chance to acquire it and run. + +When the main process, or protected function, returns, the helper releases the +lock, allowing a second instance to start running. + +To see it implemented live, look for the [`nomad var lock`][] command +implementation or the [Nomad Autoscaler][] High Availability implementation. + + [HashiCorp Consul]: https://www.consul.io/ [HashiCorp Vault]: https://www.vaultproject.io/ [Key Management]: /nomad/docs/operations/key-management @@ -194,3 +243,8 @@ See [Workload Associated ACL Policies] for more details. [workload identity]: /nomad/docs/concepts/workload-identity [Workload Associated ACL Policies]: /nomad/docs/concepts/workload-identity#workload-associated-acl-policies [ACL policy namespace rules]: /nomad/docs/other-specifications/acl-policy#namespace-rules +[The Chubby Lock Service for Loosely-Coupled Distributed Systems]: https://research.google/pubs/pub27897/ +[`nomad var lock`]: /nomad/docs/commands/var +[Go Package]: https://pkg.go.dev/github.com/hashicorp/nomad/api +[implementation]: https://github.com/hashicorp/nomad/blob/release/1.7.0/command/var_lock.go#L240 +[Nomad Autoscaler]: https://github.com/hashicorp/nomad-autoscaler/release/0.3.7command/agent.go#L368 \ No newline at end of file diff --git a/website/public/img/nomad-vars-leader-election-sdk.png b/website/public/img/nomad-vars-leader-election-sdk.png new file mode 100644 index 00000000000..8f57c160981 Binary files /dev/null and b/website/public/img/nomad-vars-leader-election-sdk.png differ