Redis cluster connections issue

[ackerL]

In our use cases, the client fleet is super big, in extreme use case, it could exceed 1000 host capacity. And the Redis server has 100 nodes(20 shards, 1 primary node, 4 replica nodes).

When we initialize the connection, the implementation is:

      try {
            List<RedisURI> redisURIs = getURIs(hostPortPairs);
            if (redisURIs != null) {
                this.hostPortPairs = hostPortPairs;
                createReadConnection(redisURIs);
                createWriteConnection(redisURIs);
            }
        } catch (Exception e) {
            close();
        }

 void createReadConnection(List<RedisURI> redisURIs) {
        closeReadConnectionAndClient();
        this.readClusterClient = RedisClusterClient.create(CLIENT_RESOURCES, redisURIs);
        readClusterClient.setOptions(createClusterClientOptions());

        this.readClusterConnection = readClusterClient.connect(new ByteArrayCodec());

        Duration timeout = Duration.ofMillis(readTimeoutMs);
        readClusterConnection.setTimeout(timeout);

        readClusterConnection.setReadFrom(readFromStrategy);

void createWriteConnection(List<RedisURI> redisURIs) {
        closeWriteConnectionAndClient();
        this.writeClusterClient = RedisClusterClient.create(CLIENT_RESOURCES, redisURIs);
        writeClusterClient.setOptions(createClusterClientOptions());
        this.writeClusterConnection = writeClusterClient.connect(new ByteArrayCodec());

        Duration timeout = Duration.ofMillis(writeTimeoutMs);
        writeClusterConnection.setTimeout(timeout);

        writeClusterConnection.setReadFrom(ReadFrom.MASTER);
    }

It means for one host in the client fleet, when it started, it will initialize the Read and Write connection to the Redis cluster. As Redis cluster has 100 nodes(hosts), i believe, the client host will setup connection to the 100 host separately, and for one host, it will initialize two connections.

From the redis cluster side, given the same client host, for any Redis node, Ideally it should hold two connections from the client client.

After we verify the behavior, we found that some redis hosts hold more than 2 connections from the client host.

10.117.154.244 is the client host.

id=118437258 addr=10.117.154.244:36328 fd=2093 name= age=166473 idle=453 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=124013924 addr=10.117.154.244:38930 fd=524 name= age=111 idle=111 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly
id=123908775 addr=10.117.154.244:33272 fd=925 name= age=3389 idle=3389 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=123912325 addr=10.117.154.244:50334 fd=44 name= age=3284 idle=3284 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=123991268 addr=10.117.154.244:38712 fd=1365 name= age=825 idle=825 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly

We want to know:

1. How does the 5 connections occurred instead of max 2 connections in some Redis nodes?
2. More connections to the Redis cluster will have impact on redis server performance, considering current client fleet capacity, is it feasible to use Lettuce connection pool to reduce the connection size? Currently given 1000 client hosts, in average, for each redis node, it will hold more than 2000 connections.
3. Follow up question 2, if there any other way we can configure to mitigate the large connection issues?

Similar issue: #3031 (comment)

[tishun]

Hey @ackerL ,

There are some things missing in your sample, like the CLIENT_RESOURCES and more importantly the createClusterClientOptions() that you are using. There are a multitude of reasons why there might be a connection existing from a client to a given server. Since you are using a RedisClusterClient this means that you are leaving to the driver how many connections and to which of the nodes it should make (since it would load the topology initially and route commands according to the slot allocation).

If you need a 1:1 relationship perhaps you can just use the RedisClient instead? But then you must be sure you are handling the slot resolution for each key yourself. Do you require the driver to handle MOVED responses? Do you use the driver to monitor for topology updates? Depending on your use case this could be a solution or not.