[docs] Add server cluster docs #1407

agdb_web/e2e/allLinks.spec.ts

@@ -6,6 +6,7 @@ const validateLinks = async (page: Page) => {
     const links = await page
         .locator("a")
         .evaluateAll((els) => els.map((el) => el.getAttribute("href")));
+    const sourcePageTitle = await page.title();
 
     for (const href of links) {
         if (
@@ -16,11 +17,12 @@ const validateLinks = async (page: Page) => {
             !href.startsWith("javascript") &&
             !href.startsWith("#")
         ) {
+            const error = `${sourcePageTitle} -> ${href}`;
             await page.goto(href);
 
             const pageTitle = await page.title();
-            expect(pageTitle.length).toBeGreaterThan(0);
-            expect(pageTitle).not.toContain("404");
+            expect(pageTitle.length, error).toBeGreaterThan(0);
+            expect(pageTitle, error).not.toContain("404");
 
             validatedLinks.push(href);
 

agdb_web/pages/en-US/docs/examples/_meta.ts

@@ -14,6 +14,11 @@ const config = {
         href: "https://github.com/agnesoft/agdb/tree/main/examples/joins",
         newWindow: true,
     },
+    k8s: {
+        title: "kubernetes (k8s)",
+        href: "https://github.com/agnesoft/agdb/tree/main/examples/k8s",
+        newWindow: true,
+    },
     "schema-migration": {
         title: "schema migration",
         href: "https://github.com/agnesoft/agdb/tree/main/examples/schema_migration",

agdb_web/pages/en-US/docs/guides/concepts.md

@@ -67,7 +67,7 @@ The database durability is provided by the write-ahead-log (WAL) file which reco
 
 Just like the memory the main database file will get fragmented over time. Sectors of the file used for the data that was later reallocated will remain unused (fragmented) until the database file is defragmented. That operation is performed automatically on database object instance drop.
 
-The storage taken by individual elements are properties is generally as follows:
+The storage taken by individual elements and properties is generally as follows:
 
 -   node: 32 bytes
 -   edge: 32 bytes

agdb_web/pages/en-US/docs/guides/how-to-run-server.mdx

@@ -7,120 +7,9 @@ import { Callout, Steps } from "nextra/components";
 
 # How to run the server?
 
-The following is a guide how to run a local instance of the `agdb_server` on any platform/OS supported by Rust building from source.
+The `agdb_server` can be run in multiple ways. The following table directs you to the correct documentation based on your use case. It differentiates between running the server as a single server or as a cluster of multiple replicated nodes (using Raft consensus protocol). The documentation is provided for running the server on bare metal, using docker or Kubernetes (K8s). Please refer to the [server](/docs/references/server) documentation for general information.
 
-<Steps>
-
-### Install git
-
-From the [official source](https://git-scm.com/) (skip if you already have it).
-
-### Install Rust toolchain
-
-From the [official source](https://www.rust-lang.org/tools/install) (minimum required version is `1.75.0`).
-
-### Install `agdb_server`
-
-For non-production use:
-
-```
-cargo install agdb_server
-```
-
-For production use there are several options:
-
-1. Manual build & install. You would build `agdb_server` from source in release mode with a custom `pepper` file. The `pepper` file located in sources as `agdb_server/pepper` contains a random 16 character value that is used internally to additionally "season" the encrypted passwords. When building for production you should change this value to a different one and keep the pepper file as secret in case you needed to rebuild the server or build a new version.
-
-The steps for a production/manual build (use `bash` on Unix or `git bash` on Windows):
-
-```bash
-git clone https://github.com/agnesoft/agdb.git
-cd agdb/
-git checkout $(git describe --tags) # checkout the latest released version
-echo "1234567891234567" > agdb_server/pepper #use a different value, this value will be a secret
-cargo build --release -p agdb_server
-mv target/release/agdb_server "<location available on your PATH>"
-# Windows: target/release/agdb_server.exe
-```
-
-<Callout type="warning">
-    Server with a different pepper value (e.g. default non-prod version) won't
-    be able to decode passwords in the internal database. If you lose the pepper
-    value of your server and need to rebuild it you should generate a new pepper
-    and then you will need to create a new admin account (by changing the config
-    value to a non-existent user) and using that account you can reset passwords
-    of all your users via `/api/v1/admin/user/{username}/change_password` API
-    (including the old admin account).
-</Callout>
-
-2. Use default pepper value (it is still recommended to build manually and use a different value) but specify in configuration the "pepper_path" from which the pepper would be loaded in runtime. This file and location should be treated as secret and all of the caveats from step 1 still apply. On the other hand you would not need to rebuild the server in the future.
-
-### Run the server
-
-```bash
-agdb_server
-```
-
-The server upon starting will create few things in its working directory:
-
--   `agdb_server.yaml`: Configuration file. You can alter it as you wish. You would need to restart the server for the changes to take effect.
--   `agdb_server.agdb` (`.agdb_server.agdb`): Internal database of the server (uses `agdb` itself) + its write ahead file (the dotfile).
--   `agdb_data_dir/`: Folder for storing the user data. It can be changed in the configuration file (requires restart of the server).
-
-and report where it listens at:
-
-```
-2024-01-26T17:47:30.956260Z  INFO agdb_server: Listening at localhost:3000
-```
-
-<Callout>
-    You can prepare the configuration file before starting the server.
-</Callout>
-
-The config supports following values:
-
-```yaml
-# agdb_server.yaml
-bind: :::3000 # host address to listen on
-address: localhost:3000 # address of incoming connections
-basepath: "" # optional prefix to allow running behind a reverse proxy
-admin: admin # the admin user that will be created automatically for the server, the password will be the same as name (admin by default, it is recommended to change it after startup)
-data_dir: agdb_server_data # directory to store user data
-```
-
-The server will be available on `host:port` as per configuration (i.e. `localhost:3000` by default). The server logs every request-response as a single entry each time to `STDOUT`. You can redirect the output to a file, e.g. `agdb_server > server.log`. It is recommended to **change the admin password from the default** (same as admin username by default).
-
-### Test that the server is up with `curl`
-
-```bash
-curl -v localhost:3000/api/v1/status # should return 200 OK
-```
-
-### Create a database user
-
-It is recommended (but optional) to create a regular user rather than using the `admin` user (which is however still possible):
-
-```bash
- # produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5"
-token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}')
-# using admin token to create a user
-curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/user/my_db_user/add -d '{"password":"password123"}'
-# login as the new user and producing their token
-token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"my_db_user","password":"password123"}')
-```
-
-### Interact with the database server
-
-You can either continue using `curl`, interactive OpenAPI GUI from any browser `localhost:3000/api/v1` (provided by `rapidoc`) or choose one of the [available API clients](/api-docs/openapi). The raw OpenAPI specification can be downloaded from the server at `localhost:3000/api/v1/openapi.json` as well.
-
-### Shutdown the server
-
-The server can be shutdown with `CTRL+C` or programmatically posting to the shutdown endpoint as logged in server admin:
-
-```bash
-# this will produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5"
-token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}')
-curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/shutdown
-```
-
-</Steps>
+| Type / Target | Bare Metal                                                | Docker                                                | K8s                                                |
+| ------------- | --------------------------------------------------------- | ----------------------------------------------------- | -------------------------------------------------- |
+| Server        | [LINK](/docs/guides/how-to-run-server/server-bare-metal)  | [LINK](/docs/guides/how-to-run-server/server-docker)  | [LINK](/docs/guides/how-to-run-server/server-k8s)  |
+| Cluster       | [LINK](/docs/guides/how-to-run-server/cluster-bare-metal) | [LINK](/docs/guides/how-to-run-server/cluster-docker) | [LINK](/docs/guides/how-to-run-server/cluster-k8s) |

agdb_web/pages/en-US/docs/guides/how-to-run-server/_meta.ts

@@ -0,0 +1,31 @@
+const config = {
+    "*": {
+        display: "hidden",
+    },
+    "server-bare-metal": {
+        title: "Server (Bare Metal)",
+        display: "normal",
+    },
+    "server-docker": {
+        title: "Server (Docker)",
+        display: "normal",
+    },
+    "server-k8s": {
+        title: "Server (K8s)",
+        display: "normal",
+    },
+    "cluster-bare-metal": {
+        title: "Cluster (Bare Metal)",
+        display: "normal",
+    },
+    "cluster-docker": {
+        title: "Cluster (Docker)",
+        display: "normal",
+    },
+    "cluster-k8s": {
+        title: "Cluster (K8s)",
+        display: "normal",
+    },
+};
+
+export default config;

agdb_web/pages/en-US/docs/guides/how-to-run-server/cluster-bare-metal.mdx

@@ -0,0 +1,96 @@
+---
+title: "How to run the cluster on bare metal?"
+description: "How to run the cluster on bare metal, Agnesoft Graph Database"
+---
+
+import { Callout, Steps } from "nextra/components";
+
+# How to run the cluster on bare metal?
+
+The `agdb_server` can be run as a cluster on bare metal. First you should build the server as described in the [server - bare metal](/docs/guides/how-to-run-server/server-bare-metal) guide. In the following steps we will run a cluster of 3 nodes on a local machine however it should work essentially the same even when each server was run on its own machine:
+
+<Steps>
+
+### Prepare cluster configuration
+
+First we create the configuration files for our nodes. Notice that the only difference is the port in the address option. As described in the main [server](/docs/references/server) documentation the address field determines the index of the local node (position in the cluster list). The list of nodes is then the same and having the same order for each instance as it provides the cluster "hash" for establishing trust in the cluster (in addition to the `cluster_token`):
+
+```yaml
+#node0: ~/agdb_cluster/node0/agdb_server.yaml
+bind: :::3000
+address: http://localhost:3000
+basepath: ""
+admin: admin
+log_level: INFO
+data_dir: agdb_server_data
+pepper_path: ""
+cluster_token: cluster
+cluster_heartbeat_timeout_ms: 1000
+cluster_term_timeout_ms: 3000
+cluster: [http://localhost:3000, http://localhost:3001, http://localhost:3002]
+
+#node1: ~/agdb_cluster/node1/agdb_server.yaml
+bind: :::3000
+address: http://localhost:3001
+basepath: ""
+admin: admin
+log_level: INFO
+data_dir: agdb_server_data
+pepper_path: ""
+cluster_token: cluster
+cluster_heartbeat_timeout_ms: 1000
+cluster_term_timeout_ms: 3000
+cluster: [http://localhost:3000, http://localhost:3001, http://localhost:3002]
+
+#node2: ~/agdb_cluster/node2/agdb_server.yaml
+bind: :::3000
+address: http://localhost:3002
+basepath: ""
+admin: admin
+log_level: INFO
+data_dir: agdb_server_data
+pepper_path: ""
+cluster_token: cluster
+cluster_heartbeat_timeout_ms: 1000
+cluster_term_timeout_ms: 3000
+cluster: [http://localhost:3000, http://localhost:3001, http://localhost:3002]
+```
+
+### Run the server
+
+Next we run all 3 nodes as background processes in their respective directories with the prepared config files. It is recommended to run each in its own shell so you can observe the logs otherwise they would be all writing to the same shell if run as background processes (i.e. `agdb_server &`). If you decide to run all of them in the same shell each log messages clearly indicates to which node it belongs using the node's index (e.g. `[0]`, `[1]` etc.)
+
+```bash
+cd ~/agdb_cluster/node0/ #run each node in its respective directory
+agdb_server
+```
+
+### Test that the cluster is up with `curl`
+
+The following commands will hit each node and return the list of nodes, their status and which one is the leader. If the servers are connected and operating normally the returned list should be the same from each node.
+
+```bash
+curl -v localhost:3000/api/v1/cluster/status
+curl -v localhost:3001/api/v1/cluster/status
+curl -v localhost:3002/api/v1/cluster/status
+```
+
+### Shutdown the servers
+
+The cluster must be shutdown one by one using the same mechanism as with single server including the `CTRL+C`. Using curl:
+
+```bash
+# this will produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5"
+token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}')
+curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/shutdown
+
+token=$(curl -X POST -H 'Content-Type: application/json' localhost:3001/api/v1/user/login -d '{"username":"admin","password":"admin"}')
+curl -X POST -H "Authorization: Bearer ${token}" localhost:3001/api/v1/admin/shutdown
+
+token=$(curl -X POST -H 'Content-Type: application/json' localhost:3002/api/v1/user/login -d '{"username":"admin","password":"admin"}')
+curl -X POST -H "Authorization: Bearer ${token}" localhost:3002/api/v1/admin/shutdown
+```
+
+While it is technically possible to use cluster login and avoid logins to each node separately it might be fragile and not work well in situations where the cluster is in the bad shapes with nodes not being available etc. Local login and shutdown are guaranteed to work regardless of the overall cluster status.
+
+</Steps>

agdb_web/pages/en-US/docs/guides/how-to-run-server/cluster-docker.mdx

@@ -0,0 +1,77 @@
+---
+title: "How to run the cluster in docker?"
+description: "How to run the cluster in docker, Agnesoft Graph Database"
+---
+
+import { Callout, Steps } from "nextra/components";
+
+# How to run the cluster in docker?
+
+The `agdb_server` can be run as a cluster in docker . Optionally you can [build the image](https://github.com/agnesoft/agdb/blob/main/agdb_server/containerfile) yourself.
+
+<Steps>
+
+### Install docker
+
+-   Windows: https://www.docker.com/products/docker-desktop/
+-   Linux: https://docs.docker.com/desktop/setup/install/linux/
+
+### Pull or build the agdb_server image
+
+The image is based on [Alpine Linux](https://alpinelinux.org/) using musl libc. The image is made available on Docker Hub or GitHub packages:
+
+| Vendor     | Tag    | Command                                  | Description                                                                               |
+| ---------- | ------ | ---------------------------------------- | ----------------------------------------------------------------------------------------- |
+| Docker Hub | latest | docker pull agnesoft/agdb:latest         | Equals latest released version                                                            |
+| Docker Hub | 0.x.x  | docker pull agnesoft/agdb:0.x.x          | Released version, e.g. 0.10.0                                                             |
+| Docker Hub | dev    | docker pull agnesoft/agdb:dev            | Equals latest development version on the main branch, refreshed with every commit to main |
+| GitHub     | latest | docker pull ghcr.io/agnesoft/agdb:latest | Equals latest released version                                                            |
+| GitHub     | 0.x.x  | docker pull ghcr.io/agnesoft/agdb:0.x.x  | Released version, e.g. 0.10.0                                                             |
+| GitHub     | dev    | docker pull ghcr.io/agnesoft/agdb:dev    | Equals latest development version on the main branch, refreshed with every commit to main |
+
+If you want to build the image yourself run the following in the root of the checked out `agdb` repository:
+
+```bash
+docker build --pull -t agnesoft/agdb:dev -f agdb_server/containerfile .
+```
+
+### Run the cluster
+
+You will need the `compose.yaml` file from the sources at: https://github.com/agnesoft/agdb/blob/main/agdb_server/compose.yaml
+
+```bash
+# the -f path is where the file resides in the sources, you can change it to the actual location of the compose.yaml file
+docker compose -f agdb_server/compose.yaml up --wait
+```
+
+This command runs the 3 nodes as a docker cluster using docker compose that contains valid cluster configuration. The volumes are provided for each node so that the data is persisted. It exposes the nodes at the ports `3000`, `3001` and `3002`.
+
+### Test that the cluster is up with `curl`
+
+The following commands will hit each node and return the list of nodes, their status and which one is the leader. If the servers are connected and operating normally the returned list should be the same from each node.
+
+```bash
+curl -v localhost:3000/api/v1/cluster/status
+curl -v localhost:3001/api/v1/cluster/status
+curl -v localhost:3002/api/v1/cluster/status
+```
+
+### Shutdown the cluster
+
+The cluster can be shutdown either by stopping the containers or programmatically posting to the shutdown endpoints as logged in server admin:
+
+```bash
+# this will produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5"
+token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}')
+curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/shutdown
+
+token=$(curl -X POST -H 'Content-Type: application/json' localhost:3001/api/v1/user/login -d '{"username":"admin","password":"admin"}')
+curl -X POST -H "Authorization: Bearer ${token}" localhost:3001/api/v1/admin/shutdown
+
+token=$(curl -X POST -H 'Content-Type: application/json' localhost:3002/api/v1/user/login -d '{"username":"admin","password":"admin"}')
+curl -X POST -H "Authorization: Bearer ${token}" localhost:3002/api/v1/admin/shutdown
+```
+
+While it is technically possible to use cluster login and avoid logins to each node separately it might be fragile and not work well in situations where the cluster is in the bad shapes with nodes not being available etc. Local login and shutdown are guaranteed to work regardless of the overall cluster status.
+
+</Steps>