Does mosca support clustered Redis?

[vpeng23]

Before v2.0.x, I knew that mosca didn't support clustered Redis. I'm hoping this can be solved by changing the redis driver to ioredis. However, I am still having problems using clustered Redis.

Error messages:
{"pid":2765,"hostname":"vincent-devbox","name":"mosca","level":30,"msg":"server started","time":1466722028912,"mqtt":1883,"v":1}
/home/vincent.peng/git/infra/mosca/node_modules/bluebird/js/main/async.js:43
fn = function () { throw arg; };
^

SyntaxError: Unexpected token u in JSON at position 0
at Object.parse (native)
at /home/vincent.peng/git/infra/mosca/lib/persistence/redis.js:301:17
at tryCatcher (/home/vincent.peng/git/infra/mosca/node_modules/bluebird/js/main/util.js:26:23)
at Promise.errorAdapter (/home/vincent.peng/git/infra/mosca/node_modules/bluebird/js/main/nodeify.js:36:34)
at Promise._settlePromiseAt (/home/vincent.peng/git/infra/mosca/node_modules/bluebird/js/main/promise.js:579:21)
at Promise._settlePromises (/home/vincent.peng/git/infra/mosca/node_modules/bluebird/js/main/promise.js:697:14)
at Async._drainQueue (/home/vincent.peng/git/infra/mosca/node_modules/bluebird/js/main/async.js:123:16)
at Async._drainQueues (/home/vincent.peng/git/infra/mosca/node_modules/bluebird/js/main/async.js:133:10)
at Immediate.Async.drainQueues as _onImmediate
at tryOnImmediate (timers.js:543:15)
at processImmediate as _immediateCallback

It seems that in lib/persistence/redis.js:301, the subs is undefined.
Any suggestion?

[vpeng23]

BTW, mosca can initialize and connect to the clustered Redis correctly. Ths exception thrown when a client tried to connect to it.

[mcollina]

Between v1 and v2 the format of data on Redis changed from JSON to msgpack.
flush the db.
Il giorno mar 28 giu 2016 alle 19:02 Vincent Peng [email protected]
ha scritto:

BTW, mosca can initialize and connect to the clustered Redis correctly.
Ths exception thrown when a client tried to connect to it.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#509 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AADL48jnLG54NQVYgikFVek5T-jtt6OWks5qQVPBgaJpZM4JAURr
.

[vpeng23]

The DB is clean. I got this issue on a brand new Redis cluster.

[mcollina]

that error means there is some data in there that Mosca cannot parse. You might want to put a logline in there to see what's going on, and which key it is, and what content it has.

[vpeng23]

I added two lines in lib/persistence/redis.js in RedisPersistence.prototype._cleanClient function:

this._client.get(key, function(err, subs) {
/* Log key and content here /
console.log("Key: " + key);
console.log("Content: " + subs);
/***********************************/
subs = JSON.parse(subs) || {};

Not sure if I log this correctly.
I'm pretty sure that the redis db is clean.

Below please find the detail log:

{"pid":22323,"hostname":"vincent-devbox","name":"mosca","level":20,"msg":"setting keepalive timeout","time":1467135884667,"timeout":90000,"client":"mosqsub/22329-vincent-d","v":1}
Key: client:sub:mosqsub/22329-vincent-d
Content: undefined
/home/vincent.peng/git/vincent.peng/mosca/node_modules/bluebird/js/main/async.js:43
fn = function () { throw arg; };
^

SyntaxError: Unexpected token u in JSON at position 0
at Object.parse (native)
at /home/vincent.peng/git/vincent.peng/mosca/lib/persistence/redis.js:304:17
at tryCatcher (/home/vincent.peng/git/vincent.peng/mosca/node_modules/bluebird/js/main/util.js:26:23)
at Promise.errorAdapter (/home/vincent.peng/git/vincent.peng/mosca/node_modules/bluebird/js/main/nodeify.js:36:34)
at Promise._settlePromiseAt (/home/vincent.peng/git/vincent.peng/mosca/node_modules/bluebird/js/main/promise.js:579:21)
at Promise._settlePromises (/home/vincent.peng/git/vincent.peng/mosca/node_modules/bluebird/js/main/promise.js:697:14)
at Async._drainQueue (/home/vincent.peng/git/vincent.peng/mosca/node_modules/bluebird/js/main/async.js:123:16)
at Async._drainQueues (/home/vincent.peng/git/vincent.peng/mosca/node_modules/bluebird/js/main/async.js:133:10)
at Immediate.Async.drainQueues as _onImmediate
at tryOnImmediate (timers.js:543:15)
at processImmediate as _immediateCallback

Thanks a lot for your prompt responses!

[behrad]

Are you trying to clean a clean=true client? What is your client code?

[vpeng23]

I'm using mosquitto_sub/mosquitto_pub for testing.
I think the default behavior is to have clean session for connection.

$ mosquitto_sub -h localhost -p 1883 -t "mqtt/test" -d
Client mosqsub/22329-vincent-d sending CONNECT

BTW, I got the same error for mosquitto_pub
$ mosquitto_pub -h localhost -p 1883 -t "mqtt/test" -m "test1234567888999" -d
Client mosqpub/22357-vincent-d sending CONNECT
Error: The connection was lost.

[jeanadrien]

Any update on this one ? I have the exact same issue using AWS elasticache with cluster mode enabled.

Also note that I override mosca.persistence.Redis function _buildClient in order to construct ioredis new Redis.Cluster instead of new Redis. It's probably useless, but I don't get how it works without this.

update : Actually constructing a Redis.Cluster solves the above problem, but then I have a type error that._client.scanStream is not a function mosca/lib/persistence/redis.js:136:35`

According to redis/ioredis#175 it's not something to be used in cluster mode.

[mcollina]

@jeanadrien I have little time to add the support for this. But I would be very happy to review and merge a PR.

[jeanadrien]

@mcollina sure, if I manage to have a working solution, I'll send you a patch.

[rasalasantosh]

@mcollina ,any updates on this. I am facing the same issue as above while connecting to clustered redis with AWS elastic cache .

While starting Mosca server getting below error ,but server able to run.

Unhandled rejection ReplyError: ERR SELECT is not allowed in cluster mode
at JavascriptReplyParser.returnError (/apps/mosca/server/node_modules/ascoltatori/node_modules/ioredis/lib/redis/parser.js:25:25)
at JavascriptReplyParser.run (/apps/mosca/server/node_modules/redis-parser/lib/javascript.js:135:18)
at JavascriptReplyParser.execute (/apps/mosca/server/node_modules/redis-parser/lib/javascript.js:112:10)
at Socket. (/apps/mosca/server/node_modules/ascoltatori/node_modules/ioredis/lib/redis/event_handler.js:107:22)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at readableAddChunk (_stream_readable.js:176:18)
at Socket.Readable.push (_stream_readable.js:134:10)
at TCP.onread (net.js:548:20)

But while connecting to server from client ,getting below error.

SyntaxError: Unexpected token u in JSON at position 0
at Object.parse (native)
at /apps/mosca/server/node_modules/mosca/lib/persistence/redis.js:322:17
at tryCatcher (/apps/mosca/server/node_modules/ioredis/node_modules/bluebird/js/main/util.js:26:23)
at Promise.errorAdapter (/apps/mosca/server/node_modules/ioredis/node_modules/bluebird/js/main/nodeify.js:36:34)
at Promise._settlePromiseAt (/apps/mosca/server/node_modules/ioredis/node_modules/bluebird/js/main/promise.js:582:21)
at Promise._settlePromises (/apps/mosca/server/node_modules/ioredis/node_modules/bluebird/js/main/promise.js:700:14)
at Async._drainQueue (/apps/mosca/server/node_modules/ioredis/node_modules/bluebird/js/main/async.js:123:16)
at Async._drainQueues (/apps/mosca/server/node_modules/ioredis/node_modules/bluebird/js/main/async.js:133:10)
at Immediate.Async.drainQueues (/apps/mosca/server/node_modules/ioredis/node_modules/bluebird/js/main/async.js:15:14)
at runCallback (timers.js:649:20)
at tryOnImmediate (timers.js:622:5)
at processImmediate [as _immediateCallback] (timers.js:594:5)

,The same issue is happening with Aedes also.

[mcollina]

Code does not write itself, and currently Mosca does not support Redis cluster. Feel free to step in. For Aedes, there is moscajs/aedes-persistence-redis#6