diff --git a/src/core/1/guides/essentials/plugins/index.md b/src/core/1/guides/essentials/plugins/index.md index ac1832152..c267e8ced 100644 --- a/src/core/1/guides/essentials/plugins/index.md +++ b/src/core/1/guides/essentials/plugins/index.md @@ -13,8 +13,8 @@ For example, imagine you are developing a mobile application that accesses a **t Kuzzle's **[Plugin Engine](/core/1/plugins)** is a powerful feature that ensures that Kuzzle meets any project requirement: -- select from a set of prebuilt plugins (such as the [OAuth2 Authentication Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth) or the [MQTT Protocol](https://github.com/kuzzleio/protocol-mqtt)). -- [create your own plugin](/core/1/plugins/essentials) to meet your specific requirements. +- select from a set of prebuilt plugins (such as the [OAuth2 Authentication Plugin](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth) or the [Cluster Plugin](https://github.com/kuzzleio/kuzzle-plugin-cluster)). +- [create your own plugin](/core/1/guides/essentials/introduction/) to meet your specific requirements. --- @@ -41,10 +41,7 @@ Kuzzle ships with the [Local Strategy Plugin](https://github.com/kuzzleio/kuzzle ## Protocols -[Protocols](/core/1/protocols) add extended networking capabilities to your Kuzzle installation. These are useful if you need to handle other, even proprietary transport protocols. - -_Example - "Allow Kuzzle to interact with XMPP-oriented services"_ -Kuzzle ships with the [MQTT Protocol](https://github.com/kuzzleio/protocol-mqtt). +[Protocols](/core/1/protocols/essentials/getting-started) add extended networking capabilities to your Kuzzle installation. These are useful if you need to handle other, even proprietary transport protocols. --- @@ -66,7 +63,7 @@ Go to the Kuzzle installation folder and type: ```bash # Open plugins/available folder -cd plugins/available +cd /plugins/available # Download Plugin to plugins/available folder git clone https://github.com/kuzzleio/kuzzle-core-plugin-boilerplate.git @@ -75,13 +72,11 @@ git clone https://github.com/kuzzleio/kuzzle-core-plugin-boilerplate.git cd kuzzle-core-plugin-boilerplate npm install # add --unsafe-perm if installing from inside a docker container -# Open plugins/enabled folder +# Enable the installed plugin. Delete this link to disable it cd ../../enabled +ln -s ../available/kuzzle-core-plugin-boilerplate -# Creata the symbolic link from the enabled folder to the available folder -ln -s ../available/kuzzle-core-plugin-boilerplate . - -# Restart Kuzzle to reload Plugins +# Restart Kuzzle to reload plugins ``` --- @@ -143,36 +138,23 @@ Note that the plugin description above contains a property for each plugin compo The steps to install a new protocol are exactly the same than for plugins, except that you have to use the `protocols/` directory, instead of the `plugins/` one. -For instance, to install the MQTT protocol: +To install a protocol: ```bash # In Kuzzle's directory: cd protocols/available -git clone https://github.com/kuzzleio/protocol-mqtt.git +# copy the protocol folder into the current directory +cp -r . # Install the protocol's dependencies -cd protocol-mqtt +cd npm install # add --unsafe-perm if installing from inside a docker container -# Open plugins/enabled folder +# Enable the installed plugin. Delete this link to disable it cd ../../enabled - -# Creata the symbolic link from the enabled folder to the available folder -ln -s ../available/protocol-mqtt . +ln -s ../available/ # Restart Kuzzle to reload protocols ``` ---- - -## Going Further - -To get more insight into how plugins work, please refer to the [Plugin Reference](/core/1/plugins). - -Here is a list of official plugins: - -- [**kuzzle-plugin-auth-passport-local**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-local): authentication Plugin shipped with Kuzzle -- [**kuzzle-plugin-logger**](https://github.com/kuzzleio/kuzzle-plugin-logger): plugin shipped with Kuzzle -- [**kuzzle-plugin-auth-passport-oauth**](https://github.com/kuzzleio/kuzzle-plugin-auth-passport-oauth): authentication plugin -- [**protocol-mqtt**](https://github.com/kuzzleio/protocol-mqtt): MQTT network protocol diff --git a/src/core/1/guides/kuzzle-depth/scalability/index.md b/src/core/1/guides/kuzzle-depth/scalability/index.md new file mode 100644 index 000000000..6f01c40fa --- /dev/null +++ b/src/core/1/guides/kuzzle-depth/scalability/index.md @@ -0,0 +1,233 @@ +--- +code: false +type: page +title: Scalability +--- + +# Scalability + +Kuzzle can scale horizontally, provided our [official Cluster Plugin](https://github.com/kuzzleio/kuzzle-plugin-cluster) is installed. + +This guide covers how clustering capabilities can be added to Kuzzle. + +--- + +## Quick start + +This chapter shows how to quickly create a Kuzzle cluster stack for development purposes. If you already have an existing Kuzzle server running, you may want to read the manual install chapter instead. + +::: info +This development stack is for demonstration and test purposes only and should not be used as-is on production. + +Notably, this stack only starts Kuzzle in cluster mode: Elasticsearch and Redis are not clustered. +::: + +Install and run: + +```bash +git clone https://github.com/kuzzleio/kuzzle-plugin-cluster.git +cd kuzzle-plugin-cluster +docker-compose -p cluster up --scale kuzzle=3 +``` + +You should now have a Kuzzle cluster stack running with 3 Kuzzle nodes. + +### ENOSPC error + +On some Linux environments, you may get `ENOSPC` errors from the filesystem watcher, because of limits set too low. + +If that happens, simply raise the limits on the number of files that can be watched: + +`sudo sysctl -w fs.inotify.max_user_watches=524288` + +That configuration change will last until the next reboot. + +To make it permanent, add the following line to your `/etc/sysctl.conf` file: + +``` +fs.inotify.max_user_watches=524288 +``` + +--- + +## Manual install on an existing Kuzzle installation + +To add cluster capabilities to an existing Kuzzle installation, the cluster plugin must be installed by following the [Plugin Install Guide](/core/1/guides/essentials/plugins/#installing-a-plugin). + +::: info +If you are running Kuzzle in a Docker container, you will need to access the running container's shell and then the Kuzzle installation folder inside the container. +::: + +To install the cluster plugin, follow these steps: + +```bash +cd /plugins/available + +git clone https://github.com/kuzzleio/kuzzle-plugin-cluster.git + +cd kuzzle-plugin-cluster +npm install # add --unsafe-perm if installing from inside a docker container + +# Enable the installed plugin. Delete this link to disable it +cd ../../enabled +ln -s ../available/kuzzle-plugin-cluster +``` + +### Cluster plugin configuration + +* The cluster plugin requires a privileged context from Kuzzle. This context is granted by Kuzzle via the global configuration. +* The cluster plugin registers a few [pipes](/core/1/plugins/guides/pipes/), and some of them may exceed the default pipe timeouts. + +Add the following to your kuzzlerc configuration file (see our [Kuzzle configuration guide](/core/1/guides/essentials/configuration/)): + +```js +"plugins": { + "common": { + "pipeWarnTime": 5000, + "pipeTimeout": 10000 + }, + "cluster": { + "privileged": true + } +} +``` + +Once the plugin installed and configured, you can start as many Kuzzle instances as you need, and they will automatically synchronize and work together. + +--- + +## Extended API + +The cluster plugin adds an [API controller](/core/1/plugins/guides/controllers) named `cluster`, with the following actions defined: + +### health + +The `cluster:health` API action returns the cluster health status. + +#### HTTP + +``` +GET http://:/_plugin/cluster/health +``` + +#### Other Protocols + +```js +{ + "controller": "cluster/cluster", + "action": "health" +} +``` + +#### Result + +```js +{ + "status": 200, + "error": null, + "controller": "cluster/cluster", + "action": "health", + "result": "ok" +} +```` + +### reset + +The `cluster:reset` API action resets the cluster state and forces a resync. + +#### HTTP + +``` +POST http://:/_plugin/cluster/reset +``` + +#### Other Protocols + +```js +{ + "controller": "cluster/cluster", + "action": "reset" +} +``` + +#### Result + +```js +{ + "status": 200, + "error": null, + "controller": "cluster/cluster", + "action": "reset", + "result": "ok" +} +```` + +### status + +The `cluster:status` API action returns the current cluster status. + +#### HTTP + +``` +GET http://:/_plugin/cluster/status +``` + +#### Other Protocols + +```js +{ + "controller": "cluster/cluster", + "action": "status" +} +``` + +#### Result + +```js +{ + "status": 200, + "error": null, + "controller": "cluster/cluster", + "action": "status", + "result": { + "count": 3, + "current": { + "pub": "tcp://:7511", + "router": "tcp://:7510", + "ready": true + }, + "pool": [ + { + "pub": "tcp://:7511", + "router": "tcp://:7510", + "ready": true + }, + { + "pub": "tcp://:7511", + "router": "tcp://:7510", + "ready": true + } + ] + } +} +``` + +--- + +## How a Kuzzle cluster works + +### Auto-discovery and Synchronization + +Kuzzle nodes are synchronized by maintaining their state in a [Redis](https://redis.io/) server instance, and they constantly exchange information using the [0mq](http://zeromq.org/) messaging library. + +What this means is that, to scale horizontally, all a Kuzzle node needs is a reachable Redis instance, and to be able to connect to other nodes. +When these conditions are met, a Kuzzle node with the cluster plugin installed only needs to be started to automatically synchronize its state and to work together with the other nodes. + +Check our [Kuzzle configuration guide](/core/1/guides/essentials/configuration/) to know how to make Kuzzle connect to specific Redis instances. + +### Load Balancing + +A load balancer in front of a Kuzzle cluster is hugely advised, to dispatch user connections to different Kuzzle nodes. +Once assigned to a Kuzzle node, a client stays attached to it until their connection drop; when needed, a Kuzzle node automatically dispatches valuable information to other nodes. + +Any load balancer will do. For instance, our development stack uses nginx for the sake of example.