From 512526323b667c9602ad677d74e9a003ff582925 Mon Sep 17 00:00:00 2001 From: scottinet Date: Wed, 29 May 2019 16:57:36 +0200 Subject: [PATCH 1/6] [guide] add clustering guides --- .../guide/guides/essentials/plugins/index.md | 59 +++-- .../guides/kuzzle-depth/scalability/index.md | 230 ++++++++++++++++++ 2 files changed, 258 insertions(+), 31 deletions(-) create mode 100644 src/core/1/guide/guides/kuzzle-depth/scalability/index.md diff --git a/src/core/1/guide/guides/essentials/plugins/index.md b/src/core/1/guide/guides/essentials/plugins/index.md index 2b8a45cb8..deed533b1 100644 --- a/src/core/1/guide/guides/essentials/plugins/index.md +++ b/src/core/1/guide/guides/essentials/plugins/index.md @@ -11,10 +11,10 @@ Our prepackaged multi-feature backend solution will meet basic project requireme For example, imagine you are developing a mobile application that accesses a **third-party payment platform**, such as Braintree, through its third-party's API. For **security** reasons, you will want to avoid accessing the third-party's API directly from the mobile device. Also, you will not want users to purchase more items than are currently in stock, so your backend will need to **monitor** what has been purchased. To achieve all this, you will want to develop a custom Plugin that lets Kuzzle communicate directly with the third-party payment platform. -Kuzzle's **[Plugin Engine](/plugins/1)** is a powerful feature that ensures that Kuzzle meets any project requirement: +Kuzzle's **[Plugin Engine]({{ site_base_path }}plugins/1)** is a powerful feature that ensures that Kuzzle meets any project requirement: -- select from a set of prebuilt plugins (such as the [OAuth2 Authentication Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth) or the [MQTT Protocol](https://github.com/kuzzleio/protocol-mqtt)). -- [create your own plugin](/core/1/plugins/essentials) to meet your specific requirements. +- select from a set of prebuilt plugins (such as the [OAuth2 Authentication Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth) or the [Cluster Plugin](https://github.com/kuzzleio/kuzzle-plugin-cluster)). +- [create your own plugin]({{ site_base_path }}plugins/1/essentials) to meet your specific requirements. --- @@ -22,29 +22,26 @@ Kuzzle's **[Plugin Engine](/plugins/1)** is a powerful feature that ensures that Plugins are used to extend Kuzzle's functionalities. They are loaded into Kuzzle during startup and share its execution thread. A plugin can implement one or multiple of the following interfaces: -[Hooks](/core/1/plugins/essentials/hooks): adds asynchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is sent to the listeners and Kuzzle continues its process without waiting for the listener to complete. +[Hooks]({{ site_base_path }}plugins/1/hooks): adds asynchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is sent to the listeners and Kuzzle continues its process without waiting for the listener to complete. _Example - "Write a log to a third-party logging service every time a document is deleted"_. The [Logger Plugin](https://github.com/kuzzleio/kuzzle-plugin-logger) (shipped with Kuzzle) uses this feature to log all the data-related events. -[Pipes](/core/1/plugins/essentials/pipes): adds synchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is passed synchronously to listeners, each modifying the input data and returning the result to the next listener. Kuzzle waits until the last listener completes and returns its data. If any listener returns an error, it will interrupt the Kuzzle lifecycle, and the thrown error will be used as a response by Kuzzle. +[Pipes]({{ site_base_path }}plugins/1/pipes): adds synchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is passed synchronously to listeners, each modifying the input data and returning the result to the next listener. Kuzzle waits until the last listener completes and returns its data. If any listener returns an error, it will interrupt the Kuzzle lifecycle, and the thrown error will be used as a response by Kuzzle. _Example - "Compare the ordered quantity with the available stock and return an error if the amount of ordered items exceeds the amount in stock"_. -[Controllers](/core/1/plugins/essentials/controllers): extends Kuzzle API. +[Controllers]({{ site_base_path }}plugins/1/controllers): extends Kuzzle API. _Example - "Expose a `checkout` API endpoint that handles a third-party payment process"_. -[Strategies](/core/1/plugins/essentials/strategies): add an authentication strategy to identify and authenticate users. +[Strategies]({{ site_base_path }}plugins/1/essentials/strategies): add an authentication strategy to identify and authenticate users. _Example - "Enable OAuth based authentication in Kuzzle"_ Kuzzle ships with the [Local Strategy Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-local) and thanks to PassportJS, more than 300 authentication strategies are readily available. ## Protocols -[Protocols](/core/1/protocols) add extended networking capabilities to your Kuzzle installation. These are useful if you need to handle other, even proprietary transport protocols. - -_Example - "Allow Kuzzle to interact with XMPP-oriented services"_ -Kuzzle ships with the [MQTT Protocol](https://github.com/kuzzleio/protocol-mqtt). +[Protocols]({{ site_base_path }}protocols/1) add extended networking capabilities to your Kuzzle installation. These are useful if you need to handle other, even proprietary transport protocols. --- @@ -66,22 +63,20 @@ Go to the Kuzzle installation folder and type: ```bash # Open plugins/available folder -cd plugins/available +cd /plugins/available # Download Plugin to plugins/available folder -git clone https://github.com/kuzzleio/kuzzle-core-plugin-boilerplate.git +git clone git@github.com:kuzzleio/kuzzle-core-plugin-boilerplate # Install the Plugin dependencies cd kuzzle-core-plugin-boilerplate npm install # add --unsafe-perm if installing from inside a docker container -# Open plugins/enabled folder +# Enable the installed plugin. Delete this link to disable it cd ../../enabled +ln -s ../available/kuzzle-core-plugin-boilerplate -# Creata the symbolic link from the enabled folder to the available folder -ln -s ../available/kuzzle-core-plugin-boilerplate . - -# Restart Kuzzle to reload Plugins +# Restart Kuzzle to reload plugins ``` --- @@ -143,23 +138,22 @@ Note that the plugin description above contains a property for each plugin compo The steps to install a new protocol are exactly the same than for plugins, except that you have to use the `protocols/` directory, instead of the `plugins/` one. -For instance, to install the MQTT protocol: +To install a protocol: ```bash # In Kuzzle's directory: cd protocols/available -git clone https://github.com/kuzzleio/protocol-mqtt.git +# copy the protocol folder into the current directory +cp -r . # Install the protocol's dependencies -cd protocol-mqtt +cd npm install # add --unsafe-perm if installing from inside a docker container -# Open plugins/enabled folder +# Enable the installed plugin. Delete this link to disable it cd ../../enabled - -# Creata the symbolic link from the enabled folder to the available folder -ln -s ../available/protocol-mqtt . +ln -s ../available/ # Restart Kuzzle to reload protocols ``` @@ -168,11 +162,14 @@ ln -s ../available/protocol-mqtt . ## Going Further -To get more insight into how plugins work, please refer to the [Plugin Reference](/plugins/1). +To get more insight into how plugins work, please refer to the [Plugin Reference]({{ site_base_path }}plugins/1). -Here is a list of official plugins: +Official plugins: -- [**kuzzle-plugin-auth-passport-local**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-local): authentication Plugin shipped with Kuzzle -- [**kuzzle-plugin-logger**](https://github.com/kuzzleio/kuzzle-plugin-logger): plugin shipped with Kuzzle -- [**kuzzle-plugin-auth-passport-oauth**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth): authentication plugin -- [**protocol-mqtt**](https://github.com/kuzzleio/protocol-mqtt): MQTT network protocol +- [**boilerplate**](https://github.com/kuzzleio/kuzzle-core-plugin-boilerplate): boilerplate code to easily get started with plugin development +- [**logger**](https://github.com/kuzzleio/kuzzle-plugin-logger): log plugin, shipped with Kuzzle +- [**auth-local**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-local): add support for login/password authentication , shipped with Kuzzle +- [**auth-oauth**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth): add support for OAuth authentications +- [**cluster**](https://github.com/kuzzleio/kuzzle-plugin-cluster): add clustering capabilities (more information in our [scalability]({{ site_base_path }}guide/1/kuzzle-depth/scalability/) guide) +- [**s3**](https://github.com/kuzzleio/kuzzle-plugin-s3): Amazon S3 plugin +- [**probe**](https://github.com/kuzzleio/kuzzle-plugin-probe): add data quality measurement capabilities diff --git a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md new file mode 100644 index 000000000..cc94e9f50 --- /dev/null +++ b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md @@ -0,0 +1,230 @@ +--- +code: false +type: page +title: Scalability +--- + +# Scalability + +Kuzzle can scale horizontally, provided our [official Cluster Plugin](https://github.com/kuzzleio/kuzzle-plugin-cluster) is installed. + +This guide covers how clustering capabilities can be added to Kuzzle. + +--- + +## Quick start + +This chapter shows how to quickly create a Kuzzle cluster stack for development purposes. If you already have an existing Kuzzle server running, you may want to read the manual install chapter instead. + +
+This development stack is for demonstration and test purposes only and should not be used as-is on production.
+Notably, this stack only starts Kuzzle in cluster mode: Elasticsearch and Redis are not clustered. +
+ +Install and run: + +```bash +git clone git@github.com:kuzzleio/kuzzle-plugin-cluster +cd kuzzle-plugin-cluster +docker-compose -p cluster up --scale kuzzle=3 +``` + +You should now have a Kuzzle cluster stack running with 3 Kuzzle nodes. + +### ENOSPC error + +On some Linux environments, you may get `ENOSPC` errors from the filesystem watcher, because of limits set too low. + +If that happens, simply raise the limits on the number of files that can be watched: + +`sudo sysctl -w fs.inotify.max_user_watches=524288` + +That configuration change will last until the next reboot. + +To make it permanent, add the following line to your `/etc/sysctl.conf` file: + +``` +fs.inotify.max_user_watches=524288 +``` + +--- + +## Manual install on an existing Kuzzle installation + +To add cluster capabilities to an existing Kuzzle installation, the cluster plugin must be installed by following the [Plugin Install Guide]({{ site_base_path }}/guide/1/essentials/plugins/#installing-a-plugin). + +
+If you are running Kuzzle in a Docker container, you will need to access the running container's shell and then the Kuzzle installation folder inside the container. +
+ +To install the cluster plugin, follow these steps: + +```bash +cd /plugins/available + +git clone git@github.com:kuzzleio/kuzzle-plugin-cluster + +cd kuzzle-plugin-cluster +npm install # add --unsafe-perm if installing from inside a docker container + +# Enable the installed plugin. Delete this link to disable it +cd ../../enabled +ln -s ../available/kuzzle-plugin-cluster +``` + +### Cluster plugin configuration + +* The cluster plugin requires a privileged context from Kuzzle. This context is granted by Kuzzle via the global configuration. +* The cluster plugin registers a few [pipes]({{ site_base_path }}plugins/1/pipes), and some of them may exceed the default pipe timeouts. + +Add the following to your kuzzlerc configuration file (see our [Kuzzle configuration guide]({{ site_base_path }}guide/1/essentials/configuration/)): + +```js +"plugins": { + "common": { + "pipeWarnTime": 5000, + "pipeTimeout": 10000 + }, + "cluster": { + "privileged": true + } +} +``` + +Once the plugin installed and configured, you can start as many Kuzzle instances as you need, and they will automatically synchronize and work together. + +--- + +## Extended API + +The cluster plugin adds an [API controller]({{ site_base_path }}plugins/1/controllers) named `cluster`, with the following actions defined: + +### health + +The `cluster:health` API action returns the cluster health status. + +#### HTTP + +``` +GET http://:/_plugin/cluster/health +``` + +#### Other Protocols + +```js +{ + "controller": "cluster/cluster", + "action": "health" +} +``` + +#### Result + +```js +{ + "status": 200, + "error": null, + "controller": "cluster/cluster", + "action": "health", + "result": "ok" +} +```` + +### reset + +The `cluster:reset` API action resets the cluster state and forces a resync. + +#### HTTP + +``` +POST http://:/_plugin/cluster/reset +``` + +#### Other Protocols + +```js +{ + "controller": "cluster/cluster", + "action": "reset" +} +``` + +#### Result + +```js +{ + "status": 200, + "error": null, + "controller": "cluster/cluster", + "action": "reset", + "result": "ok" +} +```` + +### status + +The `cluster:status` API action returns the current cluster status. + +#### HTTP + +``` +GET http://:/_plugin/cluster/status +``` + +#### Other Protocols + +```js +{ + "controller": "cluster/cluster", + "action": "status" +} +``` + +#### Result + +```js +{ + "status": 200, + "error": null, + "controller": "cluster/cluster", + "action": "status", + "result": { + "count": 3, + "current": { + "pub": "tcp://:7511", + "router": "tcp://:7510", + "ready": true + }, + "pool": [ + { + "pub": "tcp://:7511", + "router": "tcp://:7510", + "ready": true + }, + { + "pub": "tcp://:7511", + "router": "tcp://:7510", + "ready": true + } + ] + } +} +``` + +--- + +## How a Kuzzle cluster works + +### Auto-discovery and Synchronization + +Kuzzle nodes are synchronized by maintaining their state in the Redis server instance. +What this means is that, as long as Kuzzle nodes connect to the same Redis instance, they see each others and they work together. + +Check our [Kuzzle configuration guide]({{ site_base_path }}guide/1/essentials/configuration/) to know how to make Kuzzle connect to specific Redis instances. + +### Load Balancing + +A load balancer in front of a Kuzzle cluster is hugely advised, to dispatch user connections to different Kuzzle nodes. +Once assigned to a Kuzzle node, a client stays attached to it until their connection drop; when needed, a Kuzzle node automatically dispatches valuable information to other nodes. + +Any load balancer will do. For instance, our development stack uses nginx for the sake of example. From 9d0c370611c3475d4d1ba373e46fd5c3210d6fbc Mon Sep 17 00:00:00 2001 From: scottinet Date: Wed, 29 May 2019 17:07:17 +0200 Subject: [PATCH 2/6] [v3] update urls --- .../guide/guides/essentials/plugins/index.md | 30 +++++-------------- .../guides/kuzzle-depth/scalability/index.md | 10 +++---- 2 files changed, 12 insertions(+), 28 deletions(-) diff --git a/src/core/1/guide/guides/essentials/plugins/index.md b/src/core/1/guide/guides/essentials/plugins/index.md index deed533b1..47e60cf40 100644 --- a/src/core/1/guide/guides/essentials/plugins/index.md +++ b/src/core/1/guide/guides/essentials/plugins/index.md @@ -11,10 +11,10 @@ Our prepackaged multi-feature backend solution will meet basic project requireme For example, imagine you are developing a mobile application that accesses a **third-party payment platform**, such as Braintree, through its third-party's API. For **security** reasons, you will want to avoid accessing the third-party's API directly from the mobile device. Also, you will not want users to purchase more items than are currently in stock, so your backend will need to **monitor** what has been purchased. To achieve all this, you will want to develop a custom Plugin that lets Kuzzle communicate directly with the third-party payment platform. -Kuzzle's **[Plugin Engine]({{ site_base_path }}plugins/1)** is a powerful feature that ensures that Kuzzle meets any project requirement: +Kuzzle's **[Plugin Engine](/core/1/guide/guides/essentials/introduction/)** is a powerful feature that ensures that Kuzzle meets any project requirement: - select from a set of prebuilt plugins (such as the [OAuth2 Authentication Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth) or the [Cluster Plugin](https://github.com/kuzzleio/kuzzle-plugin-cluster)). -- [create your own plugin]({{ site_base_path }}plugins/1/essentials) to meet your specific requirements. +- [create your own plugin](/core/1/guide/guides/essentials/introduction/) to meet your specific requirements. --- @@ -22,26 +22,26 @@ Kuzzle's **[Plugin Engine]({{ site_base_path }}plugins/1)** is a powerful featur Plugins are used to extend Kuzzle's functionalities. They are loaded into Kuzzle during startup and share its execution thread. A plugin can implement one or multiple of the following interfaces: -[Hooks]({{ site_base_path }}plugins/1/hooks): adds asynchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is sent to the listeners and Kuzzle continues its process without waiting for the listener to complete. +[Hooks](/core/1/plugins/plugins/hooks/): adds asynchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is sent to the listeners and Kuzzle continues its process without waiting for the listener to complete. _Example - "Write a log to a third-party logging service every time a document is deleted"_. The [Logger Plugin](https://github.com/kuzzleio/kuzzle-plugin-logger) (shipped with Kuzzle) uses this feature to log all the data-related events. -[Pipes]({{ site_base_path }}plugins/1/pipes): adds synchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is passed synchronously to listeners, each modifying the input data and returning the result to the next listener. Kuzzle waits until the last listener completes and returns its data. If any listener returns an error, it will interrupt the Kuzzle lifecycle, and the thrown error will be used as a response by Kuzzle. +[Pipes](/core/1/plugins/plugins/pipes/): adds synchronous listeners that perform operations triggered by data events. When a listened event occurs, the data is passed synchronously to listeners, each modifying the input data and returning the result to the next listener. Kuzzle waits until the last listener completes and returns its data. If any listener returns an error, it will interrupt the Kuzzle lifecycle, and the thrown error will be used as a response by Kuzzle. _Example - "Compare the ordered quantity with the available stock and return an error if the amount of ordered items exceeds the amount in stock"_. -[Controllers]({{ site_base_path }}plugins/1/controllers): extends Kuzzle API. +[Controllers](/core/1/plugins/plugins/controllers): extends Kuzzle API. _Example - "Expose a `checkout` API endpoint that handles a third-party payment process"_. -[Strategies]({{ site_base_path }}plugins/1/essentials/strategies): add an authentication strategy to identify and authenticate users. +[Strategies](/core/1/plugins/plugins/strategies): add an authentication strategy to identify and authenticate users. _Example - "Enable OAuth based authentication in Kuzzle"_ Kuzzle ships with the [Local Strategy Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-local) and thanks to PassportJS, more than 300 authentication strategies are readily available. ## Protocols -[Protocols]({{ site_base_path }}protocols/1) add extended networking capabilities to your Kuzzle installation. These are useful if you need to handle other, even proprietary transport protocols. +[Protocols](/core/1/protocols/protocols/essentials/getting-started) add extended networking capabilities to your Kuzzle installation. These are useful if you need to handle other, even proprietary transport protocols. --- @@ -157,19 +157,3 @@ ln -s ../available/ # Restart Kuzzle to reload protocols ``` - ---- - -## Going Further - -To get more insight into how plugins work, please refer to the [Plugin Reference]({{ site_base_path }}plugins/1). - -Official plugins: - -- [**boilerplate**](https://github.com/kuzzleio/kuzzle-core-plugin-boilerplate): boilerplate code to easily get started with plugin development -- [**logger**](https://github.com/kuzzleio/kuzzle-plugin-logger): log plugin, shipped with Kuzzle -- [**auth-local**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-local): add support for login/password authentication , shipped with Kuzzle -- [**auth-oauth**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth): add support for OAuth authentications -- [**cluster**](https://github.com/kuzzleio/kuzzle-plugin-cluster): add clustering capabilities (more information in our [scalability]({{ site_base_path }}guide/1/kuzzle-depth/scalability/) guide) -- [**s3**](https://github.com/kuzzleio/kuzzle-plugin-s3): Amazon S3 plugin -- [**probe**](https://github.com/kuzzleio/kuzzle-plugin-probe): add data quality measurement capabilities diff --git a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md index cc94e9f50..3d164f9f7 100644 --- a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md +++ b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md @@ -51,7 +51,7 @@ fs.inotify.max_user_watches=524288 ## Manual install on an existing Kuzzle installation -To add cluster capabilities to an existing Kuzzle installation, the cluster plugin must be installed by following the [Plugin Install Guide]({{ site_base_path }}/guide/1/essentials/plugins/#installing-a-plugin). +To add cluster capabilities to an existing Kuzzle installation, the cluster plugin must be installed by following the [Plugin Install Guide](/core/1/guide/guides/essentials/plugins/#installing-a-plugin).
If you are running Kuzzle in a Docker container, you will need to access the running container's shell and then the Kuzzle installation folder inside the container. @@ -75,9 +75,9 @@ ln -s ../available/kuzzle-plugin-cluster ### Cluster plugin configuration * The cluster plugin requires a privileged context from Kuzzle. This context is granted by Kuzzle via the global configuration. -* The cluster plugin registers a few [pipes]({{ site_base_path }}plugins/1/pipes), and some of them may exceed the default pipe timeouts. +* The cluster plugin registers a few [pipes](/core/1/plugins/plugins/pipes/), and some of them may exceed the default pipe timeouts. -Add the following to your kuzzlerc configuration file (see our [Kuzzle configuration guide]({{ site_base_path }}guide/1/essentials/configuration/)): +Add the following to your kuzzlerc configuration file (see our [Kuzzle configuration guide](/core/1/guide/guides/essentials/configuration/)): ```js "plugins": { @@ -97,7 +97,7 @@ Once the plugin installed and configured, you can start as many Kuzzle instances ## Extended API -The cluster plugin adds an [API controller]({{ site_base_path }}plugins/1/controllers) named `cluster`, with the following actions defined: +The cluster plugin adds an [API controller](/core/1/plugins/plugins/controllers) named `cluster`, with the following actions defined: ### health @@ -220,7 +220,7 @@ GET http://:/_plugin/cluster/status Kuzzle nodes are synchronized by maintaining their state in the Redis server instance. What this means is that, as long as Kuzzle nodes connect to the same Redis instance, they see each others and they work together. -Check our [Kuzzle configuration guide]({{ site_base_path }}guide/1/essentials/configuration/) to know how to make Kuzzle connect to specific Redis instances. +Check our [Kuzzle configuration guide](/core/1/guide/guides/essentials/configuration/) to know how to make Kuzzle connect to specific Redis instances. ### Load Balancing From c29f8d9a146dd73ca7cff4049c103d2f06fd5e46 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Cottinet?= Date: Mon, 3 Jun 2019 09:28:22 +0200 Subject: [PATCH 3/6] Apply fixes from @Aschen Co-Authored-By: Adrien Maret --- src/core/1/guide/guides/essentials/plugins/index.md | 2 +- .../1/guide/guides/kuzzle-depth/scalability/index.md | 11 ++++++----- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/src/core/1/guide/guides/essentials/plugins/index.md b/src/core/1/guide/guides/essentials/plugins/index.md index 47e60cf40..791cea5e9 100644 --- a/src/core/1/guide/guides/essentials/plugins/index.md +++ b/src/core/1/guide/guides/essentials/plugins/index.md @@ -66,7 +66,7 @@ Go to the Kuzzle installation folder and type: cd /plugins/available # Download Plugin to plugins/available folder -git clone git@github.com:kuzzleio/kuzzle-core-plugin-boilerplate +git clone https://github.com/kuzzleio/kuzzle-core-plugin-boilerplate.git # Install the Plugin dependencies cd kuzzle-core-plugin-boilerplate diff --git a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md index 3d164f9f7..5b483b076 100644 --- a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md +++ b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md @@ -16,10 +16,11 @@ This guide covers how clustering capabilities can be added to Kuzzle. This chapter shows how to quickly create a Kuzzle cluster stack for development purposes. If you already have an existing Kuzzle server running, you may want to read the manual install chapter instead. -
-This development stack is for demonstration and test purposes only and should not be used as-is on production.
+::: info +This development stack is for demonstration and test purposes only and should not be used as-is on production. + Notably, this stack only starts Kuzzle in cluster mode: Elasticsearch and Redis are not clustered. -
+::: Install and run: @@ -53,9 +54,9 @@ fs.inotify.max_user_watches=524288 To add cluster capabilities to an existing Kuzzle installation, the cluster plugin must be installed by following the [Plugin Install Guide](/core/1/guide/guides/essentials/plugins/#installing-a-plugin). -
+::: info If you are running Kuzzle in a Docker container, you will need to access the running container's shell and then the Kuzzle installation folder inside the container. -
+::: To install the cluster plugin, follow these steps: From c63487d631fbc37410ac69b017a13631c21574d1 Mon Sep 17 00:00:00 2001 From: scottinet Date: Mon, 3 Jun 2019 09:31:32 +0200 Subject: [PATCH 4/6] [guide] switch to https url for git clone examples --- src/core/1/guide/guides/kuzzle-depth/scalability/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md index 5b483b076..c7aa6cc88 100644 --- a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md +++ b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md @@ -25,7 +25,7 @@ Notably, this stack only starts Kuzzle in cluster mode: Elasticsearch and Redis Install and run: ```bash -git clone git@github.com:kuzzleio/kuzzle-plugin-cluster +git clone https://github.com/kuzzleio/kuzzle-plugin-cluster.git cd kuzzle-plugin-cluster docker-compose -p cluster up --scale kuzzle=3 ``` @@ -63,7 +63,7 @@ To install the cluster plugin, follow these steps: ```bash cd /plugins/available -git clone git@github.com:kuzzleio/kuzzle-plugin-cluster +git clone https://github.com/kuzzleio/kuzzle-plugin-cluster.git cd kuzzle-plugin-cluster npm install # add --unsafe-perm if installing from inside a docker container From 85d9d3483e5fec2711c0cd80098dab1be7c97b31 Mon Sep 17 00:00:00 2001 From: scottinet Date: Mon, 3 Jun 2019 09:53:07 +0200 Subject: [PATCH 5/6] [cluster guide] more detailed technical stuff --- src/core/1/guide/guides/kuzzle-depth/scalability/index.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md index c7aa6cc88..76a4a822c 100644 --- a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md +++ b/src/core/1/guide/guides/kuzzle-depth/scalability/index.md @@ -218,8 +218,10 @@ GET http://:/_plugin/cluster/status ### Auto-discovery and Synchronization -Kuzzle nodes are synchronized by maintaining their state in the Redis server instance. -What this means is that, as long as Kuzzle nodes connect to the same Redis instance, they see each others and they work together. +Kuzzle nodes are synchronized by maintaining their state in a [Redis](https://redis.io/) server instance, and they constantly exchange information using the [0mq](http://zeromq.org/) messaging library. + +What this means is that, to scale horizontally, all a Kuzzle node needs is a reachable Redis instance, and to be able to connect to other nodes. +When these conditions are met, a Kuzzle node with the cluster plugin installed only needs to be started to automatically synchronize its state and to work together with the other nodes. Check our [Kuzzle configuration guide](/core/1/guide/guides/essentials/configuration/) to know how to make Kuzzle connect to specific Redis instances. From b349fdf283c9bb15df2250c12f1c6bacab5e56e4 Mon Sep 17 00:00:00 2001 From: scottinet Date: Tue, 4 Jun 2019 11:51:21 +0200 Subject: [PATCH 6/6] [guide] move scalability guide to the new arborescence --- .../guides/kuzzle-depth/scalability/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) rename src/core/1/{guide => }/guides/kuzzle-depth/scalability/index.md (90%) diff --git a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md b/src/core/1/guides/kuzzle-depth/scalability/index.md similarity index 90% rename from src/core/1/guide/guides/kuzzle-depth/scalability/index.md rename to src/core/1/guides/kuzzle-depth/scalability/index.md index 76a4a822c..6f01c40fa 100644 --- a/src/core/1/guide/guides/kuzzle-depth/scalability/index.md +++ b/src/core/1/guides/kuzzle-depth/scalability/index.md @@ -52,7 +52,7 @@ fs.inotify.max_user_watches=524288 ## Manual install on an existing Kuzzle installation -To add cluster capabilities to an existing Kuzzle installation, the cluster plugin must be installed by following the [Plugin Install Guide](/core/1/guide/guides/essentials/plugins/#installing-a-plugin). +To add cluster capabilities to an existing Kuzzle installation, the cluster plugin must be installed by following the [Plugin Install Guide](/core/1/guides/essentials/plugins/#installing-a-plugin). ::: info If you are running Kuzzle in a Docker container, you will need to access the running container's shell and then the Kuzzle installation folder inside the container. @@ -76,9 +76,9 @@ ln -s ../available/kuzzle-plugin-cluster ### Cluster plugin configuration * The cluster plugin requires a privileged context from Kuzzle. This context is granted by Kuzzle via the global configuration. -* The cluster plugin registers a few [pipes](/core/1/plugins/plugins/pipes/), and some of them may exceed the default pipe timeouts. +* The cluster plugin registers a few [pipes](/core/1/plugins/guides/pipes/), and some of them may exceed the default pipe timeouts. -Add the following to your kuzzlerc configuration file (see our [Kuzzle configuration guide](/core/1/guide/guides/essentials/configuration/)): +Add the following to your kuzzlerc configuration file (see our [Kuzzle configuration guide](/core/1/guides/essentials/configuration/)): ```js "plugins": { @@ -98,7 +98,7 @@ Once the plugin installed and configured, you can start as many Kuzzle instances ## Extended API -The cluster plugin adds an [API controller](/core/1/plugins/plugins/controllers) named `cluster`, with the following actions defined: +The cluster plugin adds an [API controller](/core/1/plugins/guides/controllers) named `cluster`, with the following actions defined: ### health @@ -223,7 +223,7 @@ Kuzzle nodes are synchronized by maintaining their state in a [Redis](https://re What this means is that, to scale horizontally, all a Kuzzle node needs is a reachable Redis instance, and to be able to connect to other nodes. When these conditions are met, a Kuzzle node with the cluster plugin installed only needs to be started to automatically synchronize its state and to work together with the other nodes. -Check our [Kuzzle configuration guide](/core/1/guide/guides/essentials/configuration/) to know how to make Kuzzle connect to specific Redis instances. +Check our [Kuzzle configuration guide](/core/1/guides/essentials/configuration/) to know how to make Kuzzle connect to specific Redis instances. ### Load Balancing