Add a clustering guide

src/core/1/guide/guides/essentials/plugins/index.md

@@ -11,40 +11,37 @@ Our prepackaged multi-feature backend solution will meet basic project requireme
 
 For example, imagine you are developing a mobile application that accesses a **third-party payment platform**, such as Braintree, through its third-party's API. For **security** reasons, you will want to avoid accessing the third-party's API directly from the mobile device. Also, you will not want users to purchase more items than are currently in stock, so your backend will need to **monitor** what has been purchased. To achieve all this, you will want to develop a custom Plugin that lets Kuzzle communicate directly with the third-party payment platform.
 
-Kuzzle's **[Plugin Engine](/plugins/1)** is a powerful feature that ensures that Kuzzle meets any project requirement:
+Kuzzle's **[Plugin Engine](/core/1/guide/guides/essentials/introduction/)** is a powerful feature that ensures that Kuzzle meets any project requirement:
 
-- select from a set of prebuilt plugins (such as the [OAuth2 Authentication Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth) or the [MQTT Protocol](https://github.com/kuzzleio/protocol-mqtt)).
-- [create your own plugin](/core/1/plugins/essentials) to meet your specific requirements.
+- select from a set of prebuilt plugins (such as the [OAuth2 Authentication Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth) or the [Cluster Plugin](https://github.com/kuzzleio/kuzzle-plugin-cluster)).
+- [create your own plugin](/core/1/guide/guides/essentials/introduction/) to meet your specific requirements.
 
 ---
 
 ## Plugins
 
 Plugins are used to extend Kuzzle's functionalities. They are loaded into Kuzzle during startup and share its execution thread. A plugin can implement one or multiple of the following interfaces:
 
-[Hooks](/core/1/plugins/essentials/hooks): adds asynchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is sent to the listeners and Kuzzle continues its process without waiting for the listener to complete.
+[Hooks](/core/1/plugins/plugins/hooks/): adds asynchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is sent to the listeners and Kuzzle continues its process without waiting for the listener to complete.
 
 _Example - "Write a log to a third-party logging service every time a document is deleted"_. The [Logger Plugin](https://github.com/kuzzleio/kuzzle-plugin-logger) (shipped with Kuzzle) uses this feature to log all the data-related events.
 
-[Pipes](/core/1/plugins/essentials/pipes): adds synchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is passed synchronously to listeners, each modifying the input data and returning the result to the next listener. Kuzzle waits until the last listener completes and returns its data. If any listener returns an error, it will interrupt the Kuzzle lifecycle, and the thrown error will be used as a response by Kuzzle.
+[Pipes](/core/1/plugins/plugins/pipes/): adds synchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is passed synchronously to listeners, each modifying the input data and returning the result to the next listener. Kuzzle waits until the last listener completes and returns its data. If any listener returns an error, it will interrupt the Kuzzle lifecycle, and the thrown error will be used as a response by Kuzzle.
 
 _Example - "Compare the ordered quantity with the available stock and return an error if the amount of ordered items exceeds the amount in stock"_.
 
-[Controllers](/core/1/plugins/essentials/controllers): extends Kuzzle API.
+[Controllers](/core/1/plugins/plugins/controllers): extends Kuzzle API.
 
 _Example - "Expose a `checkout` API endpoint that handles a third-party payment process"_.
 
-[Strategies](/core/1/plugins/essentials/strategies): add an authentication strategy to identify and authenticate users.
+[Strategies](/core/1/plugins/plugins/strategies): add an authentication strategy to identify and authenticate users.
 
 _Example - "Enable OAuth based authentication in Kuzzle"_
 Kuzzle ships with the [Local Strategy Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-local) and thanks to PassportJS, more than 300 authentication strategies are readily available.
 
 ## Protocols
 
-[Protocols](/core/1/protocols) add extended networking capabilities to your Kuzzle installation. These are useful if you need to handle other, even proprietary transport protocols.
-
-_Example - "Allow Kuzzle to interact with XMPP-oriented services"_
-Kuzzle ships with the [MQTT Protocol](https://github.com/kuzzleio/protocol-mqtt).
+[Protocols](/core/1/protocols/protocols/essentials/getting-started) add extended networking capabilities to your Kuzzle installation. These are useful if you need to handle other, even proprietary transport protocols.
 
 ---
 
@@ -66,7 +63,7 @@ Go to the Kuzzle installation folder and type:
 
 ```bash
 # Open plugins/available folder
-cd plugins/available
+cd <kuzzle directory>/plugins/available
 
 # Download Plugin to plugins/available folder
 git clone https://github.com/kuzzleio/kuzzle-core-plugin-boilerplate.git
@@ -75,13 +72,11 @@ git clone https://github.com/kuzzleio/kuzzle-core-plugin-boilerplate.git
 cd kuzzle-core-plugin-boilerplate
 npm install # add --unsafe-perm if installing from inside a docker container
 
-# Open plugins/enabled folder
+# Enable the installed plugin. Delete this link to disable it
 cd ../../enabled
+ln -s ../available/kuzzle-core-plugin-boilerplate
 
-# Creata the symbolic link from the enabled folder to the available folder
-ln -s ../available/kuzzle-core-plugin-boilerplate .
-
-# Restart Kuzzle to reload Plugins
+# Restart Kuzzle to reload plugins
 ```
 
 ---
@@ -143,36 +138,22 @@ Note that the plugin description above contains a property for each plugin compo
 
 The steps to install a new protocol are exactly the same than for plugins, except that you have to use the `protocols/` directory, instead of the `plugins/` one.
 
-For instance, to install the MQTT protocol:
+To install a protocol:
 
 ```bash
 # In Kuzzle's directory:
 cd protocols/available
 
-git clone https://github.com/kuzzleio/protocol-mqtt.git
+# copy the protocol folder into the current directory
+cp -r <protocol_dir> .
 
 # Install the protocol's dependencies
-cd protocol-mqtt
+cd <protocol_dir>
 npm install # add --unsafe-perm if installing from inside a docker container
 
-# Open plugins/enabled folder
+# Enable the installed plugin. Delete this link to disable it
 cd ../../enabled
-
-# Creata the symbolic link from the enabled folder to the available folder
-ln -s ../available/protocol-mqtt .
+ln -s ../available/<protocol_dir>
 
 # Restart Kuzzle to reload protocols
 ```
-
----
-
-## Going Further
-
-To get more insight into how plugins work, please refer to the [Plugin Reference](/plugins/1).
-
-Here is a list of official plugins:
-
-- [**kuzzle-plugin-auth-passport-local**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-local): authentication Plugin shipped with Kuzzle
-- [**kuzzle-plugin-logger**](https://github.com/kuzzleio/kuzzle-plugin-logger): plugin shipped with Kuzzle
-- [**kuzzle-plugin-auth-passport-oauth**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth): authentication plugin
-- [**protocol-mqtt**](https://github.com/kuzzleio/protocol-mqtt): MQTT network protocol

src/core/1/guide/guides/kuzzle-depth/scalability/index.md

@@ -0,0 +1,233 @@
+---
+code: false
+type: page
+title: Scalability
+---
+
+# Scalability
+
+Kuzzle can scale horizontally, provided our [official Cluster Plugin](https://github.com/kuzzleio/kuzzle-plugin-cluster) is installed.
+
+This guide covers how clustering capabilities can be added to Kuzzle.
+
+---
+
+## Quick start
+
+This chapter shows how to quickly create a Kuzzle cluster stack for development purposes. If you already have an existing Kuzzle server running, you may want to read the manual install chapter instead.
+
+::: info
+This development stack is for demonstration and test purposes only and should not be used as-is on production.  
+
+Notably, this stack only starts Kuzzle in cluster mode: Elasticsearch and Redis are not clustered.
+:::
+
+Install and run: 
+
+```bash
+git clone https://github.com/kuzzleio/kuzzle-plugin-cluster.git
+cd kuzzle-plugin-cluster
+docker-compose -p cluster up --scale kuzzle=3
+```
+
+You should now have a Kuzzle cluster stack running with 3 Kuzzle nodes.
+
+### ENOSPC error
+
+On some Linux environments, you may get `ENOSPC` errors from the filesystem watcher, because of limits set too low.
+
+If that happens, simply raise the limits on the number of files that can be watched:
+
+`sudo sysctl -w fs.inotify.max_user_watches=524288`
+
+That configuration change will last until the next reboot. 
+
+To make it permanent, add the following line to your `/etc/sysctl.conf` file:
+
+```
+fs.inotify.max_user_watches=524288
+```
+
+---
+
+## Manual install on an existing Kuzzle installation
+
+To add cluster capabilities to an existing Kuzzle installation, the cluster plugin must be installed by following the [Plugin Install Guide](/core/1/guide/guides/essentials/plugins/#installing-a-plugin).
+
+::: info
+If you are running Kuzzle in a Docker container, you will need to access the running container's shell and then the Kuzzle installation folder inside the container.
+:::
+
+To install the cluster plugin, follow these steps:
+
+```bash
+cd <kuzzle directory>/plugins/available
+
+git clone https://github.com/kuzzleio/kuzzle-plugin-cluster.git
+
+cd kuzzle-plugin-cluster
+npm install # add --unsafe-perm if installing from inside a docker container
+
+# Enable the installed plugin. Delete this link to disable it
+cd ../../enabled
+ln -s ../available/kuzzle-plugin-cluster
+```
+
+### Cluster plugin configuration
+
+* The cluster plugin requires a privileged context from Kuzzle. This context is granted by Kuzzle via the global configuration.
+* The cluster plugin registers a few [pipes](/core/1/plugins/plugins/pipes/), and some of them may exceed the default pipe timeouts. 
+
+Add the following to your kuzzlerc configuration file (see our [Kuzzle configuration guide](/core/1/guide/guides/essentials/configuration/)):
+
+```js
+"plugins": {
+  "common": {
+    "pipeWarnTime": 5000,
+    "pipeTimeout": 10000
+  },
+  "cluster": {
+    "privileged": true
+  }
+}
+```
+
+Once the plugin installed and configured, you can start as many Kuzzle instances as you need, and they will automatically synchronize and work together.
+
+---
+
+## Extended API
+
+The cluster plugin adds an [API controller](/core/1/plugins/plugins/controllers) named `cluster`, with the following actions defined:
+
+### health
+
+The `cluster:health` API action returns the cluster health status.
+
+#### HTTP
+
+```
+GET http://<host>:<port>/_plugin/cluster/health
+```
+
+#### Other Protocols
+
+```js
+{
+  "controller": "cluster/cluster",
+  "action": "health"
+}
+```
+
+#### Result
+
+```js
+{
+  "status": 200,
+  "error": null,
+  "controller": "cluster/cluster",
+  "action": "health",
+  "result": "ok"
+}
+````
+
+### reset
+
+The `cluster:reset` API action resets the cluster state and forces a resync.
+
+#### HTTP
+
+```
+POST http://<host>:<port>/_plugin/cluster/reset
+```
+
+#### Other Protocols
+
+```js
+{
+  "controller": "cluster/cluster",
+  "action": "reset"
+}
+```
+
+#### Result
+
+```js
+{
+  "status": 200,
+  "error": null,
+  "controller": "cluster/cluster",
+  "action": "reset",
+  "result": "ok"
+}
+````
+
+### status
+
+The `cluster:status` API action returns the current cluster status.
+
+#### HTTP
+
+```
+GET http://<host>:<port>/_plugin/cluster/status
+```
+
+#### Other Protocols
+
+```js
+{
+  "controller": "cluster/cluster",
+  "action": "status"
+}
+```
+
+#### Result
+
+```js
+{
+  "status": 200,
+  "error": null,
+  "controller": "cluster/cluster",
+  "action": "status",
+  "result": {
+    "count": 3,
+    "current": {
+      "pub": "tcp://<kuzzle node IP>:7511",
+      "router": "tcp://<kuzzle node IP>:7510",
+      "ready": true
+    },
+    "pool": [
+      {
+        "pub": "tcp://<kuzzle node IP>:7511",
+        "router": "tcp://<kuzzle node IP>:7510",
+        "ready": true
+      },
+      {
+        "pub": "tcp://<kuzzle node IP>:7511",
+        "router": "tcp://<kuzzle node IP>:7510",
+        "ready": true
+      }
+    ]
+  }
+}
+```
+
+---
+
+## How a Kuzzle cluster works
+
+### Auto-discovery and Synchronization
+
+Kuzzle nodes are synchronized by maintaining their state in a [Redis](https://redis.io/) server instance, and they constantly exchange information using the [0mq](http://zeromq.org/) messaging library.
+
+What this means is that, to scale horizontally, all a Kuzzle node needs is a reachable Redis instance, and to be able to connect to other nodes.  
+When these conditions are met, a Kuzzle node with the cluster plugin installed only needs to be started to automatically synchronize its state and to work together with the other nodes.
+
+Check our [Kuzzle configuration guide](/core/1/guide/guides/essentials/configuration/) to know how to make Kuzzle connect to specific Redis instances.
+
+### Load Balancing
+
+A load balancer in front of a Kuzzle cluster is hugely advised, to dispatch user connections to different Kuzzle nodes.  
+Once assigned to a Kuzzle node, a client stays attached to it until their connection drop; when needed, a Kuzzle node automatically dispatches valuable information to other nodes.
+
+Any load balancer will do. For instance, our development stack uses nginx for the sake of example.