Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[f-gh-17449] Add lock sdk to documentation and update missing links #18520

Merged
merged 11 commits into from
Sep 19, 2023
7 changes: 5 additions & 2 deletions website/content/api-docs/variables/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,11 @@ description: |-
The `/var` and `/vars` endpoints are used to query for and interact with
variables, and set up locks and leases over them.

See the [Variables] documentation for information how these capabilities are
See the [Variables][] documentation for information how these capabilities are
used. For a CLI to perform these operations manually, please see the
documentation for the [`nomad var`] commands.
documentation for the [`nomad var`][] commands.

Please choose a sub-section in the navigation for more information

[`nomad var`]: /nomad/docs/commands/var
[Variables]: /nomad/docs/concepts/variables
1 change: 1 addition & 0 deletions website/content/api-docs/variables/locks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,7 @@ regular variables, locked variables can be created with or without any items.
The lock TTL and Delay must be values between 10 seconds and 24 hours.

[Variables]: /nomad/docs/concepts/variables
[restrictions]: /nomad/api-docs/variables/variables#restrictions
[`nomad var`]: /nomad/docs/commands/var
[blocking queries]: /nomad/api-docs#blocking-queries
[required ACLs]: /nomad/api-docs#acls
Expand Down
9 changes: 3 additions & 6 deletions website/content/api-docs/variables/variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,8 @@ description: The /var endpoints are used to query for and interact with variable
The `/var` and `/vars` endpoints are used to query for and interact with
variables.

See the [Variables] documentation for information how these capabilities are
used. For a CLI to perform these operations manually, please see the
documentation for the [`nomad var`] commands.
See the [Variables][] documentation for information how these capabilities are
used.

## List Variables

Expand Down Expand Up @@ -294,10 +293,8 @@ the response will include only metadata and not the `Items` field:
}
```


[Variables]: /nomad/docs/concepts/variables
gulducat marked this conversation as resolved.
Show resolved Hide resolved
[locks section]: /nomad/docs/commands/var/lock
[`nomad var`]: /nomad/docs/commands/var
[locks section]:/nomad/api-docs/variables/locks
[blocking queries]: /nomad/api-docs#blocking-queries
[required ACLs]: /nomad/api-docs#acls
[RFC3986]: https://www.rfc-editor.org/rfc/rfc3986#section-2
2 changes: 2 additions & 0 deletions website/content/docs/commands/var/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ subcommands are available:
- [`var get`][get] - Retrieve a variable
- [`var put`][put] - Insert or update a variable
- [`var purge`][purge] - Permanently delete a variable
- [`var lock`][lock] - Acquire a lock over a variable

## Examples

Expand Down Expand Up @@ -62,3 +63,4 @@ user = dba
[list]: /nomad/docs/commands/var/list
[put]: /nomad/docs/commands/var/put
[purge]: /nomad/docs/commands/var/purge
[lock]: /nomad/docs/commands/var/lock
54 changes: 54 additions & 0 deletions website/content/docs/concepts/variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,55 @@ apply.

See [Workload Associated ACL Policies] for more details.

## Locks

Nomad provides the ability to block a variable from being updated for a period
of time by setting a lock on it. Once a variable is locked, it can be read by
everyone, but it can only be updated by the lock holder.

The locks are designed to provide granular locking and are heavily inspired by
[The Chubby Lock Service for Loosely-Coupled Distributed Systems][].

A lock is composed of an ID, a TTL, and a lock delay. The ID is generated by
the Nomad Server and must be provided on every request to modify the variable's
items or the lock itself. The TTL defines the time the lock will be held; if the
lock needs to be in place for longer, it can be renewed for as many new periods
as wanted.

Once it is no longer needed, it must be released. If by the time the TTL expires,
and no renew or release calls were made, the variable will remain locked for at
least the lock delay duration, to avoid a possible split-brain situation, where
there are two holders at the same time.

### Leader election backed by Nomad Variable Locks

For some applications, like HDFS or the Nomad Autoscaler, it is necessary to
have multiple instances running to ensure redundancy in case of a failure, but
only one of them may be active at a time as a leader.

As part of the [Go Package][], Nomad offers a helper that takes one variable and
uses a lock over it as a sync mechanism to run multiple instances but always
keeping just one running at any given time, using the following algorithm:

[![Leader Election Helper](/img/nomad-vars-leader-election-sdk.png)](/img/nomad-vars-leader-election-sdk.png)

As soon as any instance starts, it tries to lock the sync variable. If it succeeds,
it continues to execute while a secondary thread is in charge of keeping track of
the lock and renewing it when necessary. If by any chance the renewal fails,
the main process is forced to return, and the instance goes into standby until it
attempts to acquire the lock over the sync variable.

Only threads 1 and 3 or thread 2 are running at any given time, because every
instance is either executing as normal while renewing the lock or waiting for a
chance to acquire it and run.

When the main process, or protected function, returns, the helper releases the
lock, allowing a second instance to start running.

To see it implemented live, look for the [`nomad var lock`][] command
implementation or the [Nomad Autoscaler][] High Availability implementation.


[HashiCorp Consul]: https://www.consul.io/
[HashiCorp Vault]: https://www.vaultproject.io/
[Key Management]: /nomad/docs/operations/key-management
Expand All @@ -194,3 +243,8 @@ See [Workload Associated ACL Policies] for more details.
[workload identity]: /nomad/docs/concepts/workload-identity
[Workload Associated ACL Policies]: /nomad/docs/concepts/workload-identity#workload-associated-acl-policies
[ACL policy namespace rules]: /nomad/docs/other-specifications/acl-policy#namespace-rules
[The Chubby Lock Service for Loosely-Coupled Distributed Systems]: https://research.google/pubs/pub27897/
[`nomad var lock`]: /nomad/docs/commands/var
[Go Package]: https://pkg.go.dev/github.com/hashicorp/nomad/api
[implementation]: https://github.com/hashicorp/nomad/blob/release/1.7.0/command/var_lock.go#L240
[Nomad Autoscaler]: https://github.com/hashicorp/nomad-autoscaler/release/0.3.7command/agent.go#L368
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.