Sentinels are not working in failover case #3472

mahanam · 2020-08-19T19:52:58Z

NAME: redis-ha
REVISION: 1
RELEASED: Wed Aug 19 19:04:31 2020
CHART: redis-10.7.16
USER-SUPPLIED VALUES:
{}

Actually i triggered my redis-go client towards my setup => [single master {redis+sentinel} and 3 slaves{redis+sentinel}]

The first time it's working fine it's connected to the right Redis master.
Now after some time I have deleted the current master pod {redis+sentinel}
After failover, the client able to pick the right master at this time.
Now after some time I have deleted the master pod {redis+sentinel}
But here, the client stuck into the loop with the old IP address which is a stale entry.
=========================================================
Magic comes here, It will connect to the right Redis master when you restart your client.
While in this process, it reconnects the appropriate master after a client restart
======================================================
After (6) point have killed the running and working client just to check whether it will work for next time.
=========================================================
When i start the client again it stucks with stale ip address as mentioned in (5) point

Logs:
2020/08/19 19:13:07 Initializing the redis client...
2020/08/19 19:13:07 Getting the sentinel redis master...
redis: 2020/08/19 19:13:07 sentinel.go:470: sentinel: discovered new sentinel="7a732b2572c10a61e22e6e79885d1286b943669d" for master="mymaster"
redis: 2020/08/19 19:13:07 sentinel.go:470: sentinel: discovered new sentinel="852f11a9d3634574e8786838ce5fb26c43f6bec1" for master="mymaster"
redis: 2020/08/19 19:13:07 sentinel.go:470: sentinel: discovered new sentinel="3301ae7a855f25d2a6b0085a879c3609be823e72" for master="mymaster"
redis: 2020/08/19 19:13:07 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.7:6379"
2020/08/19 19:13:07 Successfully connected to the redis master server:[10.233.13.207:26379]
2020/08/19 19:13:07 Retrieved key 0:0
2020/08/19 19:13:08 Retrieved key 1:1
2020/08/19 19:13:09 Retrieved key 2:2
2020/08/19 19:13:10 Retrieved key 3:3
2020/08/19 19:13:11 Retrieved key 4:4
2020/08/19 19:13:12 Retrieved key 5:5
2020/08/19 19:13:13 Retrieved key 6:6
2020/08/19 19:13:14 Retrieved key 7:7
2020/08/19 19:13:15 Retrieved key 8:8
2020/08/19 19:13:16 Retrieved key 9:9
2020/08/19 19:13:17 Retrieved key 10:10
2020/08/19 19:13:18 Retrieved key 11:11
2020/08/19 19:13:19 Retrieved key 12:12
2020/08/19 19:13:20 Retrieved key 13:13
2020/08/19 19:13:21 Retrieved key 14:14
2020/08/19 19:13:22 Retrieved key 15:15
2020/08/19 19:13:25 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:27 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:27 Retrieved key 16:
2020/08/19 19:13:29 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:30 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:30 Retrieved key 17:
2020/08/19 19:13:33 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:35 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:35 Retrieved key 18:
2020/08/19 19:13:38 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:40 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:40 Retrieved key 19:
2020/08/19 19:13:43 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:44 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:44 Retrieved key 20:
2020/08/19 19:13:47 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:49 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:49 Retrieved key 21:
2020/08/19 19:13:51 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:52 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:52 Retrieved key 22:
redis: 2020/08/19 19:13:53 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.11:6379"
2020/08/19 19:13:53 Retrieved key 23:23
2020/08/19 19:13:54 Retrieved key 24:24
2020/08/19 19:13:55 Retrieved key 25:25
2020/08/19 19:13:56 Retrieved key 26:26
2020/08/19 19:13:57 Retrieved key 27:27
2020/08/19 19:13:58 Retrieved key 28:28
...
2020/08/19 19:15:11 Retrieved key 100:100
2020/08/19 19:15:12 Retrieved key 101:101
redis: 2020/08/19 19:15:13 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.12:6379"
2020/08/19 19:15:13 Retrieved key 102:102
2020/08/19 19:15:14 Retrieved key 103:103
2020/08/19 19:15:15 Retrieved key 104:104
2020/08/19 19:15:16 Retrieved key 105:105
...
2020/08/19 19:16:47 Retrieved key 195:195
redis: 2020/08/19 19:16:48 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.11:6379"
2020/08/19 19:16:50 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:16:52 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:16:52 Retrieved key 196:
2020/08/19 19:16:54 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:16:56 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:16:56 Retrieved key 197:
2020/08/19 19:16:58 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:00 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:00 Retrieved key 198:
2020/08/19 19:17:02 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:03 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:03 Retrieved key 199:

2020/08/19 19:17:05 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:07 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:07 Retrieved key 200:
2020/08/19 19:17:09 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:10 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:10 Retrieved key 201:
2020/08/19 19:17:13 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:14 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:14 Retrieved key 202:
2020/08/19 19:17:17 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:18 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:18 Retrieved key 203:
2020/08/19 19:17:21 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:23 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:23 Retrieved key 204:
2020/08/19 19:17:26 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:27 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:27 Retrieved key 205:
2020/08/19 19:17:30 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:31 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:31 Retrieved key 206:
2020/08/19 19:17:34 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:35 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:35 Retrieved key 207:
2020/08/19 19:17:37 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:38 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:38 Retrieved key 208:
2020/08/19 19:17:41 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:43 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:43 Retrieved key 209:
2020/08/19 19:17:46 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:48 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:48 Retrieved key 210:
2020/08/19 19:17:50 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:51 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:51 Retrieved key 211:
2020/08/19 19:17:54 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:56 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:56 Retrieved key 212:
2020/08/19 19:17:58 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:59 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:59 Retrieved key 213:
2020/08/19 19:18:02 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:04 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:04 Retrieved key 214:
2020/08/19 19:18:06 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:08 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:08 Retrieved key 215:
2020/08/19 19:18:10 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:12 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:12 Retrieved key 216:
2020/08/19 19:18:14 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:17 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:17 Retrieved key 217:
2020/08/19 19:18:19 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:20 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:20 Retrieved key 218:
2020/08/19 19:18:22 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:24 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:24 Retrieved key 219:

===============================================================================
2020/08/19 19:18:58 Initializing the redis client...
2020/08/19 19:18:58 Getting the sentinel redis master...
redis: 2020/08/19 19:18:58 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.13:6379"
2020/08/19 19:18:58 Successfully connected to the redis master server:[10.233.13.207:26379]
2020/08/19 19:18:58 Retrieved key 0:0
2020/08/19 19:18:59 Retrieved key 1:1
2020/08/19 19:19:00 Retrieved key 2:2
2020/08/19 19:19:01 Retrieved key 3:3
2020/08/19 19:19:02 Retrieved key 4:4
2020/08/19 19:19:03 Retrieved key 5:5
2020/08/19 19:19:04 Retrieved key 6:6
2020/08/19 19:19:05 Retrieved key 7:7
2020/08/19 19:19:06 Retrieved key 8:8
2020/08/19 19:19:07 Retrieved key 9:9

=============================================================================================
2020/08/19 19:20:20 Initializing the redis client...
2020/08/19 19:20:20 Getting the sentinel redis master...
redis: 2020/08/19 19:20:20 sentinel.go:470: sentinel: discovered new sentinel="852f11a9d3634574e8786838ce5fb26c43f6bec1" for master="mymaster"
redis: 2020/08/19 19:20:20 sentinel.go:470: sentinel: discovered new sentinel="3301ae7a855f25d2a6b0085a879c3609be823e72" for master="mymaster"
redis: 2020/08/19 19:20:20 sentinel.go:470: sentinel: discovered new sentinel="57bff9e9e2cca87c135ec4eefc146a08f4654b9d" for master="mymaster"
redis: 2020/08/19 19:20:20 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.11:6379"
2020/08/19 19:20:22 Connection failed with redis master serverdial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:22 Successfully connected to the redis master server:[10.233.13.207:26379]
2020/08/19 19:20:24 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:25 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:25 Retrieved key 0:
2020/08/19 19:20:27 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:29 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:29 Retrieved key 1:
2020/08/19 19:20:32 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:34 dial tcp 10.233.77.11:6379: connect: invalid argument
2

Expected behavior
If you delete the current master pod the client should be able to connect the right redis master after failover window

Version of Helm and Kubernetes:
demo@demo1:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:25Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

demo@demo1:~$ helm version
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

Output of kubectl version:
demo@demo1:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Configuration of debug manifest files:

[debug] Created tunnel using local port: '35084'

[debug] SERVER: "127.0.0.1:35084"

Release "redis-ha" does not exist. Installing it now.
[debug] CHART PATH: /home/demo/git/sdpon-manifest/templates/sdponcharts/charts/voltha/charts/redis-ha

NAME: redis-ha
REVISION: 1
RELEASED: Wed Aug 19 19:04:31 2020
CHART: redis-10.7.16
USER-SUPPLIED VALUES:
{}

COMPUTED VALUES:
cluster:
enabled: true
slaveCount: 3
clusterDomain: cluster.local
configmap: |-

Enable AOF https://redis.io/topics/persistence#append-only-file

appendonly yes
appendfsync everysec
#no-appendfsync-on-rewrite no
#save 900 1
#save 300 10
#save 60 10000

Disable RDB persistence, AOF persistence already enabled.

save ""
global:
redis: {}
image:
pullPolicy: IfNotPresent
registry: docker-registry.com:5000
repository: redis
tag: 6.0.6
master:
affinity: {}
command: redis-server
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
- ReadWriteOnce
enabled: true
matchExpressions: {}
matchLabels: {}
path: /data
size: 8Gi
subPath: ""
podAnnotations: {}
podLabels: {}
priorityClassName: {}
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
loadBalancerIP: null
port: 6379
type: ClusterIP
statefulset:
updateStrategy: RollingUpdate
metrics:
enabled: false
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-exporter
tag: 1.9.0-debian-10-r20
podAnnotations:
prometheus.io/port: "9121"
prometheus.io/scrape: "true"
prometheusRule:
additionalLabels: {}
enabled: false
namespace: ""
rules: []
service:
annotations: {}
labels: {}
type: ClusterIP
serviceMonitor:
enabled: false
selector:
prometheus: kube-prometheus
networkPolicy:
enabled: true
ingressNSMatchLabels: {}
ingressNSPodMatchLabels: {}
password: null
persistence:
existingClaim: null
podDisruptionBudget:
enabled: false
minAvailable: 1
podSecurityPolicy:
create: false
rbac:
create: false
role:
rules: []
redisPort: 6379
securityContext:
enabled: true
fsGroup: 1001
runAsUser: 0
sentinel:
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
downAfterMilliseconds: 10000
enabled: true
failoverTimeout: 9000
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-sentinel
tag: 6.0.6-debian-10-r11
initialCheckTimeout: 5
livenessProbe:
enabled: false
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
masterSet: mymaster
parallelSyncs: 1
port: 26379
quorum: 2
readinessProbe:
enabled: false
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
loadBalancerIP: null
redisPort: 6379
sentinelPort: 26379
type: ClusterIP
staticID: false
usePassword: true
serviceAccount:
create: false
name: null
slave:
affinity: {}
command: redis-server
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
- ReadWriteOnce
enabled: true
matchExpressions: {}
matchLabels: {}
path: /data
size: 8Gi
subPath: ""
podAnnotations: {}
podLabels: {}
port: 6379
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
service:
annotations: {}
labels: {}
loadBalancerIP: null
port: 6379
type: ClusterIP
spreadConstraints: {}
statefulset:
updateStrategy: RollingUpdate
sysctlImage:
command: []
enabled: false
mountHostSys: false
pullPolicy: Always
registry: docker.io
repository: bitnami/minideb
resources: {}
tag: buster
tls:
authClients: true
certCAFilename: null
certFilename: null
certKeyFilename: null
certificatesSecret: null
enabled: false
usePassword: true
usePasswordFile: false
volumePermissions:
enabled: false
image:
pullPolicy: Always
registry: docker.io
repository: bitnami/minideb
tag: buster
resources: {}

HOOKS:
MANIFEST:

Source: redis/templates/networkpolicy.yaml

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: redis-ha
namespace: default
labels:
app: redis
chart: redis-10.7.16
release: redis-ha
heritage: Tiller
spec:
podSelector:
matchLabels:
app: redis
release: redis-ha
policyTypes:
- Ingress
- Egress
egress:
# Allow dns resolution
- ports:
- port: 53
protocol: UDP
# Allow outbound connections to other cluster pods
- ports:
- port: 6379
- port: 26379
to:
- podSelector:
matchLabels:
app: redis
release: redis-ha
ingress:
# Allow inbound connections
- ports:
- port: 6379
- port: 26379
from:
- podSelector:
matchLabels:
redis-ha-client: "true"
- podSelector:
matchLabels:
app: redis
release: redis-ha

Source: redis/templates/secret.yaml

apiVersion: v1
kind: Secret
metadata:
name: redis-ha
namespace: default
labels:
app: redis
chart: redis-10.7.16
release: "redis-ha"
heritage: "Tiller"
type: Opaque
data:
redis-password: "aUZGUkFoc1hxMA=="

Source: redis/templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: redis-ha
namespace: default
labels:
app: redis
chart: redis-10.7.16
heritage: Tiller
release: redis-ha
data:
redis.conf: |-
# User-supplied configuration:
# Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
appendfsync everysec
#no-appendfsync-on-rewrite no
#save 900 1
#save 300 10
#save 60 10000
# Disable RDB persistence, AOF persistence already enabled.
save ""
master.conf: |-
dir /data
replica.conf: |-
dir /data
slave-read-only yes
sentinel.conf: |-
dir "/tmp"
bind 0.0.0.0
port 26379
sentinel monitor mymaster redis-ha-master-0.redis-ha-headless.default.svc.cluster.local 6379 2
sentinel down-after-milliseconds mymaster 10000
sentinel failover-timeout mymaster 9000
sentinel parallel-syncs mymaster 1

Source: redis/templates/health-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: redis-ha-health
namespace: default
labels:
app: redis
chart: redis-10.7.16
heritage: Tiller
release: redis-ha
data:
ping_readiness_local.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_PASSWORD $no_auth_warning
-h localhost
-p $REDIS_PORT
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
ping_liveness_local.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_PASSWORD $no_auth_warning
-h localhost
-p $REDIS_PORT
ping
)
if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then
echo "$response"
exit 1
fi
ping_sentinel.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_PASSWORD $no_auth_warning
-h localhost
-p $REDIS_SENTINEL_PORT
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
parse_sentinels.awk: |-
/ip/ {FOUND_IP=1}
/port/ {FOUND_PORT=1}
/runid/ {FOUND_RUNID=1}
!/ip|port|runid/ {
if (FOUND_IP==1) {
IP=$1; FOUND_IP=0;
}
else if (FOUND_PORT==1) {
PORT=$1;
FOUND_PORT=0;
} else if (FOUND_RUNID==1) {
printf "\nsentinel known-sentinel mymaster %s %s %s", IP, PORT, $0; FOUND_RUNID=0;
}
}
ping_readiness_master.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_MASTER_PASSWORD $no_auth_warning
-h $REDIS_MASTER_HOST
-p $REDIS_MASTER_PORT_NUMBER
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
ping_liveness_master.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.*) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_MASTER_PASSWORD $no_auth_warning
-h $REDIS_MASTER_HOST
-p $REDIS_MASTER_PORT_NUMBER
ping
)
if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then
echo "$response"
exit 1
fi
ping_readiness_local_and_master.sh: |-
script_dir="$(dirname "$0")"
exit_status=0
"$script_dir/ping_readiness_local.sh" $1 || exit_status=$?
"$script_dir/ping_readiness_master.sh" $1 || exit_status=$?
exit $exit_status
ping_liveness_local_and_master.sh: |-
script_dir="$(dirname "$0")"
exit_status=0
"$script_dir/ping_liveness_local.sh" $1 || exit_status=$?
"$script_dir/ping_liveness_master.sh" $1 || exit_status=$?
exit $exit_status

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-ha-slave-2
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-ha-slave-1
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-ha-master-0
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-ha-slave-0
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-ha-slave-2
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-ha-slave-1
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-ha-master-0
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-ha-slave-0
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

Source: redis/templates/headless-svc.yaml

apiVersion: v1
kind: Service
metadata:
name: redis-ha-headless
namespace: default
labels:
app: redis
chart: redis-10.7.16
release: redis-ha
heritage: Tiller
spec:
type: ClusterIP
clusterIP: None
ports:
- name: redis
port: 6379
targetPort: redis
- name: redis-sentinel
port: 26379
targetPort: redis-sentinel
selector:
app: redis
release: redis-ha

Source: redis/templates/redis-with-sentinel-svc.yaml

apiVersion: v1
kind: Service
metadata:
name: redis-ha
namespace: default
labels:
app: redis
chart: redis-10.7.16
release: redis-ha
heritage: Tiller
spec:
type: ClusterIP
ports:
- name: redis
port: 6379
targetPort: redis
- name: redis-sentinel
port: 26379
targetPort: redis-sentinel
selector:
app: redis
release: redis-ha

Source: redis/templates/redis-master-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-ha-master
namespace: default
labels:
app: redis
chart: redis-10.7.16
release: redis-ha
heritage: Tiller
spec:
selector:
matchLabels:
app: redis
release: redis-ha
role: master
serviceName: redis-ha-headless
template:
metadata:
labels:
app: redis
chart: redis-10.7.16
release: redis-ha
role: master
annotations:
checksum/health: 72098cadec6e6d8bf78527a9626bc96fe2f2dae070bb60d27a7c046c37eea287
checksum/configmap: 7ebd09346d8dfcacb96ee1ddb30e75d7e7e5344a985f5eacd4c2fdeeedfa7fa7
checksum/secret: 3c1003cf7a0ca691891f8285ee98aeb4465fa121f821b0d07e77579849a19cff
spec:

  securityContext:
    fsGroup: 1001
  serviceAccountName: default
  containers:
    - name: redis
      image: docker-registry.com:5000/redis:6.0.6
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsUser: 0
      command:
        - /bin/bash
        - -c
        - |
          useradd redis
          chown -R redis /data
          if [[ -n $REDIS_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_PASSWORD_FILE}`
            export REDIS_PASSWORD=$password_aux
          fi
          if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
            cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
          fi
          if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
            cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
          fi
          ARGS=("--port" "${REDIS_PORT}")
          ARGS+=("--requirepass" "${REDIS_PASSWORD}")
          ARGS+=("--masterauth" "${REDIS_PASSWORD}")
          ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
          ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
          redis-server ${ARGS[@]}
      env:
        - name: REDIS_REPLICATION_MODE
          value: master
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-ha
              key: redis-password
        - name: REDIS_TLS_ENABLED
          value: "no"
        - name: REDIS_PORT
          value: "6379"
      ports:
        - name: redis
          containerPort: 6379
      livenessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_liveness_local.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_readiness_local.sh 5
      resources:
        null
        
      volumeMounts:
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
          subPath: 
        - name: config
          mountPath: /opt/bitnami/redis/mounted-etc
        - name: redis-tmp-conf
          mountPath: /opt/bitnami/redis/etc/
    - name: sentinel
      image: "docker.io/bitnami/redis-sentinel:6.0.6-debian-10-r11"
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsUser: 0
      command:
        - /bin/bash
        - -c
        - |
          if [[ -n $REDIS_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_PASSWORD_FILE}`
            export REDIS_PASSWORD=$password_aux
          fi
          if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]];then
            cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
            printf "\nsentinel auth-pass mymaster $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
            printf "\nrequirepass $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
          fi
          echo "Getting information about current running sentinels"
          # Get information from existing sentinels
          existing_sentinels=$(timeout -s 10 5 redis-cli --raw -h redis-ha -a "$REDIS_PASSWORD" -p 26379 SENTINEL sentinels mymaster)
          echo $existing_sentinels | awk -f /health/parse_sentinels.awk | tee -a  /opt/bitnami/redis-sentinel/etc/sentinel.conf

          redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel 
      env:
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-ha
              key: redis-password
        - name: REDIS_SENTINEL_TLS_ENABLED
          value: "no"
        - name: REDIS_SENTINEL_PORT
          value: "26379"
      ports:
        - name: redis-sentinel
          containerPort: 26379
      resources:
        null
        
      volumeMounts:
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
          subPath: 
        - name: config
          mountPath: /opt/bitnami/redis-sentinel/mounted-etc
        - name: sentinel-tmp-conf
          mountPath: /opt/bitnami/redis-sentinel/etc/
  volumes:
    - name: health
      configMap:
        name: redis-ha-health
        defaultMode: 0755
    - name: config
      configMap:
        name: redis-ha
    - name: redis-tmp-conf
      emptyDir: {}
    - name: sentinel-tmp-conf
      emptyDir: {}

volumeClaimTemplates:
- metadata:
name: redis-data
labels:
app: redis
release: redis-ha
heritage: Tiller
component: master
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"

    selector:

updateStrategy:
type: RollingUpdate

Source: redis/templates/redis-slave-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-ha-slave
namespace: default
labels:
app: redis
chart: redis-10.7.16
release: redis-ha
heritage: Tiller
spec:
replicas: 3
serviceName: redis-ha-headless
selector:
matchLabels:
app: redis
release: redis-ha
role: slave
template:
metadata:
labels:
app: redis
release: redis-ha
chart: redis-10.7.16
role: slave
annotations:
checksum/health: 72098cadec6e6d8bf78527a9626bc96fe2f2dae070bb60d27a7c046c37eea287
checksum/configmap: 7ebd09346d8dfcacb96ee1ddb30e75d7e7e5344a985f5eacd4c2fdeeedfa7fa7
checksum/secret: e8633c629f9222be7f704afcdf994d39c2d3297da0bdbab784b8c0d689308566
spec:

  securityContext:
    fsGroup: 1001
  serviceAccountName: default
  containers:
    - name: redis
      image: docker-registry.com:5000/redis:6.0.6
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsUser: 0
      command:
        - /bin/bash
        - -c
        - |
          useradd redis
          chown -R redis /data
          if [[ -n $REDIS_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_PASSWORD_FILE}`
            export REDIS_PASSWORD=$password_aux
          fi
          if [[ -n $REDIS_MASTER_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_MASTER_PASSWORD_FILE}`
            export REDIS_MASTER_PASSWORD=$password_aux
          fi
          if [[ ! -f /opt/bitnami/redis/etc/replica.conf ]];then
            cp /opt/bitnami/redis/mounted-etc/replica.conf /opt/bitnami/redis/etc/replica.conf
          fi
          if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
            cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
          fi
          ARGS=("--port" "${REDIS_PORT}")
          ARGS+=("--slaveof" "${REDIS_MASTER_HOST}" "${REDIS_MASTER_PORT_NUMBER}")
          ARGS+=("--requirepass" "${REDIS_PASSWORD}")
          ARGS+=("--masterauth" "${REDIS_MASTER_PASSWORD}")
          ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
          ARGS+=("--include" "/opt/bitnami/redis/etc/replica.conf")
          redis-server "${ARGS[@]}"
      env:
        - name: REDIS_REPLICATION_MODE
          value: slave
        - name: REDIS_MASTER_HOST
          value: redis-ha-master-0.redis-ha-headless.default.svc.cluster.local
        - name: REDIS_MASTER_PORT_NUMBER
          value: "6379"
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-ha
              key: redis-password
        - name: REDIS_MASTER_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-ha
              key: redis-password
        - name: REDIS_TLS_ENABLED
          value: "no"
        - name: REDIS_PORT
          value: "6379"
      ports:
        - name: redis
          containerPort: 6379
      livenessProbe:
        initialDelaySeconds: 30
        periodSeconds: 10
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_liveness_local.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 10
        timeoutSeconds: 10
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_readiness_local.sh 5
      resources:
        null
        
      volumeMounts:
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
        - name: config
          mountPath: /opt/bitnami/redis/mounted-etc
        - name: redis-tmp-conf
          mountPath: /opt/bitnami/redis/etc
    - name: sentinel
      image: docker.io/bitnami/redis-sentinel:6.0.6-debian-10-r11
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsUser: 0
      command:
        - /bin/bash
        - -c
        - |
          if [[ -n $REDIS_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_PASSWORD_FILE}`
            export REDIS_PASSWORD=$password_aux
          fi
          if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]];then
            cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
            printf "\nsentinel auth-pass mymaster $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
            printf "\nrequirepass $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
          fi

          redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel 
      env:
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-ha
              key: redis-password
        - name: REDIS_SENTINEL_TLS_ENABLED
          value: "no"
        - name: REDIS_SENTINEL_PORT
          value: "26379"
      ports:
        - name: redis-sentinel
          containerPort: 26379
      resources:
        null
        
      volumeMounts:
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
          subPath: 
        - name: config
          mountPath: /opt/bitnami/redis-sentinel/mounted-etc
        - name: sentinel-tmp-conf
          mountPath: /opt/bitnami/redis-sentinel/etc
  volumes:
    - name: health
      configMap:
        name: redis-ha-health
        defaultMode: 0755
    - name: config
      configMap:
        name: redis-ha
    - name: sentinel-tmp-conf
      emptyDir: {}
    - name: redis-tmp-conf
      emptyDir: {}

volumeClaimTemplates:
- metadata:
name: redis-data
labels:
app: redis
release: redis-ha
heritage: Tiller
component: slave
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"

    selector:

updateStrategy:
type: RollingUpdate
LAST DEPLOYED: Wed Aug 19 19:04:31 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME AGE
redis-ha 0s
redis-ha-health 0s

==> v1/NetworkPolicy
NAME AGE
redis-ha 0s

==> v1/PersistentVolume
NAME AGE
redis-data-redis-ha-master-0 0s
redis-data-redis-ha-slave-0 0s
redis-data-redis-ha-slave-1 0s
redis-data-redis-ha-slave-2 0s

==> v1/PersistentVolumeClaim
NAME AGE
redis-data-redis-ha-master-0 0s
redis-data-redis-ha-slave-0 0s
redis-data-redis-ha-slave-1 0s
redis-data-redis-ha-slave-2 0s

==> v1/Pod(related)
NAME AGE
redis-ha-master-0 0s
redis-ha-slave-0 0s

==> v1/Secret
NAME AGE
redis-ha 0s

==> v1/Service
NAME AGE
redis-ha 0s
redis-ha-headless 0s

==> v1/StatefulSet
NAME AGE
redis-ha-master 0s
redis-ha-slave 0s

NOTES:
** Please be patient while the chart is being deployed **
Redis can be accessed via port 6379 on the following DNS name from within your cluster:

redis-ha.default.svc.cluster.local for read only operations

For read/write operations, first access the Redis Sentinel cluster, which is available in port 26379 using the same domain name above.

Note: Since NetworkPolicy is enabled, only pods with label
redis-ha-client=true"
will be able to connect to redis.

The text was updated successfully, but these errors were encountered:

javsalgar · 2020-08-20T10:13:26Z

Hi,

It seems that you are using the redis-ha chart. Note that this one is not maintained by Bitnami. We are maintaining the redis and redis-cluster charts. I advise you to contact the redis-ha maintainers with the issue.

mahanam · 2020-08-24T05:40:12Z

Hi,

Its the Redis charts from bitnami charts, I have used just the naming redis-ha.

KIndly check out the full configuration by which the redis sentinel based ha is deployed.

Thanks,
Mahadev

javsalgar · 2020-08-24T11:23:36Z

Hi,

Sorry I misread the yaml. Could you add proper code blocks so it's more readable?

Regarding the client, do you have the go code you are using for testing? I would like to reproduce the exact issue.

mahanam · 2020-08-24T15:52:36Z

Hi,

Please find the snippet of my current client.

if config.RedisSentinel {
log.Print("Getting the sentinel redis master...")
RClient = redis.NewFailoverClient(&redis.FailoverOptions{
MasterName: "mymaster",
SentinelAddrs: []string{"10.233.13.207:26379"},
SentinelPassword: "iFFRAhsXq0",
Password: "iFFRAhsXq0",
MaxRetries: 10,
DB: 0,
})
} else {
log.Print("Getting the redis client...")
RClient = redis.NewClient(&redis.Options{
Addr: config.RedisAddr,
Password: config.RedisPassword,
DB: config.RedisDB,
})
}

err := RClient.Ping(ctx).Err()
if err != nil {
log.Print("Connection failed with redis master server", err)
}

log.Print("Successfully connected to the redis master server:", config.RedisSentinelAddrs)

Thanks,
Mahadev

javsalgar · 2020-08-25T09:32:55Z

Hi,

I imagine this will require importing some libraries and executing it with a set of commands. Could you provide a link to a github repo that I can clone, execute make and do the testing? Sorry for the inconvenience but it would be very helpful for the engineering team.

mahanam · 2020-08-25T09:38:43Z

Hi,

We are not using any kind of git repo and its same code, which we are using as a client.
Could you help me with the required libraries as you suggested?

Thanks,
Mahadev

javsalgar · 2020-08-25T09:44:17Z

Hi,

I imagine that, in your code, the use of redis. and other parts is thank to some includes in your code. Am i right? That's what I would like to know

mahanam · 2020-08-25T09:46:25Z

Hi,

package main

import (
    "context"
    "time"
    "log"
    "strconv"
    "github.com/go-redis/redis/v8"
)

var (
        ctx = context.Background()
    RClient *redis.Client
)

type Config struct {
    RedisSentinelAddrs []string
    RedisAddr            string
    RedisMasterName      string
    RedisPassword        string
    RedisDB              int
    RedisSentinel        bool
}

func main() {
    config := &Config{
        RedisSentinelAddrs: []string{"redis-ha:26379"},
        RedisAddr: "",
        RedisMasterName: "mymaster",
        RedisPassword: "OwIyBdHKx0",
        RedisDB: 0,
        RedisSentinel: true,
    }
    RClient := NewRedisClient(config)

    var i int = 0
    for true {
        s1 := strconv.Itoa(i)
        err := RClient.Set(ctx, s1 , i , 0).Err()
        if err != nil {
            log.Print("Write Error", err)
        }
        val, err := RClient.Get(ctx, s1).Result()
        if err != nil {
            log.Print(err)
        }
        log.Print("Retrieved key ", s1, ":", val)
        time.Sleep(time.Second)
        i++
    }

}

func NewRedisClient(config *Config) *redis.Client {
    var client *redis.Client
    if config.RedisSentinel {
        client = redis.NewFailoverClient(&redis.FailoverOptions{
            MasterName:    "mymaster",
            SentinelAddrs: []string{"redis-ha:26379"},
            SentinelPassword: "OwIyBdHKx0",
            Password: "OwIyBdHKx0",
            MaxRetries:       10,
            DB: 0,
        })
    } else {
        client = redis.NewClient(&redis.Options{
            Addr:     config.RedisAddr,
            Password: config.RedisPassword,
            DB:       config.RedisDB,
        })
    }

    _, err := client.Ping(ctx).Result()
    if err != nil {
        log.Fatal("COnnection failed with redis master server", err)
    }

    log.Print("Successfully connected to the redis master server:", config.RedisSentinelAddrs)

    return client

}

javsalgar · 2020-08-26T10:40:52Z

Hi,

Just a quick note that I was able to reproduce a failover error issue. I will forward this to the engineering team so we can work on a fix. As soon as I have more news, we will update this ticket.

mahanam · 2020-08-26T10:46:58Z

Hi,

Thanks for the confirmation on this issue.

Thanks,
Mahadev

mahanam · 2020-08-31T10:29:49Z

Can I hear some noise on this? :)

javsalgar · 2020-08-31T11:05:10Z

Hi,

We have planned working on this during the following weeks, as soon as there are more news, we will update the ticket.

mahanam · 2020-08-31T11:05:40Z

Great! Thanks

rafariossaa · 2020-09-02T11:22:21Z

Hi @mahanam ,
I am working on this, could you indicate in which kubernetes cluster did this happen to you ?

mahanam · 2020-09-03T05:44:54Z

Hi @rafariossaa

It is basically a 3 node setup on which am deploying my redis-ha but
the same was reproduced on a single node as well.

Let me know if you need more details regarding the system.
Thanks,
Mahadev

mahanam · 2020-09-07T05:49:44Z

Hi,

I hope your queries got answered in the previous comment, Any possibility to update on the status as it's still in the hold state.

Thanks
Mahadev

rafariossaa · 2020-09-07T07:15:44Z

Hi,
We are currently working on this. The hold state is mainly to avoid this issue to be marked as stale and auto-closed.
There is not ETA, but I hope this to be solved soon.

mahanam · 2020-09-17T07:56:38Z

Hi,

I have tried this fix from this branch "rafariossaa:working_redis_fix" and its the same as previous issue.

Please find the simple steps below which I have followed for your reference.
Is this bug fix tested or am I misconfigured something?

I have deployed a fresh redis-ha
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-master-node-0 2/2 Running 0 2m30s 10.233.121.246 localhost
redis-master-node-1 2/2 Running 0 116s 10.233.121.249 localhost
redis-master-node-2 2/2 Running 0 104s 10.233.121.250 localhost
deleted the master pod 10.233.121.246
Now at this point, other slave [10.233.121.250] became a new master as expected
but the problem here is the deleted master pod came up with new ip [10.233.121.251] and when i see the logs inside that redis container, its trying for the connect the old master.
20:S 17 Sep 2020 07:10:36.234 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:10:36.234 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:11:37.651 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:11:37.651 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:11:37.651 * MASTER <-> REPLICA sync started

=======================================================

#""For more logs,""

sentinel logs of the new pod [10.233.121.251]
admin@localhost:/mnt/onl/sdpon/templates/sdponcharts/charts/redis-ha$ docker logs 605cf63fab95
11:X 17 Sep 2020 07:03:29.380 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
11:X 17 Sep 2020 07:03:29.380 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=11, just started
11:X 17 Sep 2020 07:03:29.380 # Configuration loaded
11:X 17 Sep 2020 07:03:29.382 * Running mode=sentinel, port=26379.
11:X 17 Sep 2020 07:03:29.382 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
11:X 17 Sep 2020 07:03:29.386 # Sentinel ID is 7df5adeca0eec70d960e5f930e2cb45c3d021529
11:X 17 Sep 2020 07:03:29.386 # +monitor master mymaster 10.233.121.246 6379 quorum 2
11:X 17 Sep 2020 07:04:29.407 # +sdown master mymaster 10.233.121.246 6379

redis logs of the new pod [10.233.121.251]
20:C 17 Sep 2020 07:03:29.073 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
20:C 17 Sep 2020 07:03:29.073 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=20, just started
20:C 17 Sep 2020 07:03:29.073 # Configuration loaded
20:S 17 Sep 2020 07:03:29.075 * Running mode=standalone, port=6379.
20:S 17 Sep 2020 07:03:29.076 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
20:S 17 Sep 2020 07:03:29.076 # Server initialized
20:S 17 Sep 2020 07:03:29.076 * Reading RDB preamble from AOF file...
20:S 17 Sep 2020 07:03:29.077 * Loading RDB produced by version 6.0.8
20:S 17 Sep 2020 07:03:29.077 * RDB age 209 seconds
20:S 17 Sep 2020 07:03:29.077 * RDB memory usage when created 94.77 Mb
20:S 17 Sep 2020 07:03:29.077 * RDB has an AOF tail
20:S 17 Sep 2020 07:03:29.413 * Reading the remaining AOF tail...
20:S 17 Sep 2020 07:03:29.413 * DB loaded from append only file: 0.336 seconds
20:S 17 Sep 2020 07:03:29.413 * Ready to accept connections
20:S 17 Sep 2020 07:03:29.413 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:03:29.413 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:04:30.803 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:04:30.803 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:04:30.803 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:05:31.172 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:05:31.172 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:05:31.172 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:06:32.561 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:06:32.561 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:06:32.561 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:07:33.969 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:07:33.969 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:07:33.969 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:08:34.385 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:08:34.385 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:08:34.385 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:09:35.797 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:09:35.797 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:09:35.797 * MASTER <-> REPLICA sync started

=================================================
#Configuration made in values-production.yaml these are only Highlighted changes made:
command: helm upgrade --install redis-master -f values-production.yaml . --debug

Sentinel:
enabled: true
usePassword: false

networkPolicy:
enabled: false

usePassword: false

securityContext:
runAsUser: 0

command: "redis-server"

disableCommands

- FLUSHDB

- FLUSHALL

command: "redis-server" #for slave

metrics:
enabled: false

configmap: |-
# Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
appendfsync everysec

===========================================================
#Complete configuration:
[debug] Created tunnel using local port: '43627'

[debug] SERVER: "127.0.0.1:43627"

Release "redis-master" does not exist. Installing it now.
[debug] CHART PATH: /home/admin/redis-work/redis-ha

NAME: redis-master
REVISION: 1
RELEASED: Thu Sep 17 07:37:59 2020
CHART: redis-11.0.0
USER-SUPPLIED VALUES:
cluster:
enabled: true
slaveCount: 3
clusterDomain: cluster.local
configmap: |-

Enable AOF https://redis.io/topics/persistence#append-only-file

appendonly yes
appendfsync everysec
#no-appendfsync-on-rewrite no
#save 900 1
#save 300 10
#save 60 10000

Disable RDB persistence, AOF persistence already enabled.

save ""
global:
redis: {}
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis
tag: 6.0.8-debian-10-r0
master:
affinity: {}
command: redis-server
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
disableCommands: null
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
- ReadWriteOnce
enabled: true
matchExpressions: {}
matchLabels: {}
path: /data
size: 8Gi
subPath: ""
podAnnotations: {}
podLabels: {}
priorityClassName: {}
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
loadBalancerIP: null
port: 6379
type: ClusterIP
statefulset:
updateStrategy: RollingUpdate
metrics:
enabled: false
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-exporter
tag: 1.11.1-debian-10-r12
podAnnotations:
prometheus.io/port: "9121"
prometheus.io/scrape: "true"
prometheusRule:
additionalLabels: {}
enabled: false
namespace: ""
rules: []
service:
annotations: {}
labels: {}
type: ClusterIP
serviceMonitor:
enabled: false
selector:
prometheus: kube-prometheus
networkPolicy:
enabled: false
ingressNSMatchLabels: {}
ingressNSPodMatchLabels: {}
password: null
persistence:
existingClaim: null
podDisruptionBudget:
enabled: false
minAvailable: 1
podSecurityPolicy:
create: false
rbac:
create: false
role:
rules: []
redisPort: 6379
securityContext:
enabled: true
fsGroup: 1001
runAsUser: 0
sentinel:
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
downAfterMilliseconds: 60000
enabled: true
failoverTimeout: 18000
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-sentinel
tag: 6.0.8-debian-10-r1
initialCheckTimeout: 5
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
masterSet: mymaster
parallelSyncs: 1
port: 26379
quorum: 2
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
loadBalancerIP: null
redisPort: 6379
sentinelPort: 26379
type: ClusterIP
staticID: false
usePassword: false
serviceAccount:
create: false
name: null
slave:
affinity: {}
command: redis-server
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
disableCommands:

FLUSHDB
FLUSHALL
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
- ReadWriteOnce
  enabled: true
  matchExpressions: {}
  matchLabels: {}
  path: /data
  size: 8Gi
  subPath: ""
  podAnnotations: {}
  podLabels: {}
  port: 6379
  readinessProbe:
  enabled: true
  failureThreshold: 5
  initialDelaySeconds: 5
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 10
  service:
  annotations: {}
  labels: {}
  loadBalancerIP: null
  port: 6379
  type: ClusterIP
  spreadConstraints: {}
  statefulset:
  updateStrategy: RollingUpdate
  sysctlImage:
  command: []
  enabled: false
  mountHostSys: false
  pullPolicy: Always
  registry: docker.io
  repository: bitnami/minideb
  resources: {}
  tag: buster
  tls:
  authClients: true
  certCAFilename: null
  certFilename: null
  certKeyFilename: null
  certificatesSecret: null
  enabled: false
  usePassword: false
  usePasswordFile: false
  volumePermissions:
  enabled: false
  image:
  pullPolicy: Always
  registry: docker.io
  repository: bitnami/minideb
  tag: buster
  resources: {}

COMPUTED VALUES:
cluster:
enabled: true
slaveCount: 3
clusterDomain: cluster.local
configmap: |-

Enable AOF https://redis.io/topics/persistence#append-only-file

appendonly yes
appendfsync everysec
#no-appendfsync-on-rewrite no
#save 900 1
#save 300 10
#save 60 10000

Disable RDB persistence, AOF persistence already enabled.

save ""
global:
redis: {}
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis
tag: 6.0.8-debian-10-r0
master:
affinity: {}
command: redis-server
customLivenessProbe: {}
customReadinessProbe: {}
extraEnvVars: []
extraEnvVarsCM: []
extraEnvVarsSecret: []
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
- ReadWriteOnce
enabled: true
matchExpressions: {}
matchLabels: {}
path: /data
size: 8Gi
subPath: ""
podAnnotations: {}
podLabels: {}
priorityClassName: {}
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
port: 6379
type: ClusterIP
shareProcessNamespace: false
statefulset:
updateStrategy: RollingUpdate
metrics:
enabled: false
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-exporter
tag: 1.11.1-debian-10-r12
podAnnotations:
prometheus.io/port: "9121"
prometheus.io/scrape: "true"
prometheusRule:
additionalLabels: {}
enabled: false
namespace: ""
rules: []
service:
annotations: {}
labels: {}
type: ClusterIP
serviceMonitor:
enabled: false
selector:
prometheus: kube-prometheus
networkPolicy:
enabled: false
ingressNSMatchLabels: {}
ingressNSPodMatchLabels: {}
persistence: {}
podDisruptionBudget:
enabled: false
minAvailable: 1
podSecurityPolicy:
create: false
rbac:
create: false
role:
rules: []
redisPort: 6379
securityContext:
enabled: true
fsGroup: 1001
runAsUser: 0
sentinel:
customLivenessProbe: {}
customReadinessProbe: {}
downAfterMilliseconds: 60000
enabled: true
failoverTimeout: 18000
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-sentinel
tag: 6.0.8-debian-10-r1
initialCheckTimeout: 5
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
masterSet: mymaster
parallelSyncs: 1
port: 26379
quorum: 2
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
redisPort: 6379
sentinelPort: 26379
type: ClusterIP
staticID: false
usePassword: false
serviceAccount:
create: false
slave:
affinity: {}
command: redis-server
customLivenessProbe: {}
customReadinessProbe: {}
disableCommands:

FLUSHDB
FLUSHALL
extraEnvVars: []
extraEnvVarsCM: []
extraEnvVarsSecret: []
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
- ReadWriteOnce
  enabled: true
  matchExpressions: {}
  matchLabels: {}
  path: /data
  size: 8Gi
  subPath: ""
  podAnnotations: {}
  podLabels: {}
  port: 6379
  readinessProbe:
  enabled: true
  failureThreshold: 5
  initialDelaySeconds: 5
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 10
  service:
  annotations: {}
  labels: {}
  port: 6379
  type: ClusterIP
  shareProcessNamespace: false
  spreadConstraints: {}
  statefulset:
  updateStrategy: RollingUpdate
  sysctlImage:
  command: []
  enabled: false
  mountHostSys: false
  pullPolicy: Always
  registry: docker.io
  repository: bitnami/minideb
  resources: {}
  tag: buster
  tls:
  authClients: true
  enabled: false
  usePassword: false
  usePasswordFile: false
  volumePermissions:
  enabled: false
  image:
  pullPolicy: Always
  registry: docker.io
  repository: bitnami/minideb
  tag: buster
  resources: {}

HOOKS:
MANIFEST:

Source: redis/templates/configmap-scripts.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: redis-master-scripts
namespace: default
labels:
app: redis
chart: redis-11.0.0
heritage: Tiller
release: redis-master
data:
start-node.sh: |
#!/bin/bash
is_boolean_yes() {
local -r bool="${1:-}"
# comparison is performed without regard to the case of alphabetic characters
shopt -s nocasematch
if [[ "$bool" = 1 || "$bool" =~ ^(yes|true)$ ]]; then
true
else
false
fi
}

export REDIS_REPLICATION_MODE="slave"
if [[ $HOSTNAME =~ (.*)-([0-9]+)$ ]]; then
  if [[ ${BASH_REMATCH[2]} == "0" ]]; then
    if [[ ! -f /data/redisboot.lock ]]; then
      export REDIS_REPLICATION_MODE="master"
    else
      if is_boolean_yes "$REDIS_TLS_ENABLED"; then
        sentinel_info_command="redis-cli -a $REDIS_PASSWORD -h redis-master-headless.default.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} info"
      else
        sentinel_info_command="redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 info"
      fi
      if [[ ! ($($sentinel_info_command)) ]]; then
         export REDIS_REPLICATION_MODE="master"
         rm /data/redisboot.lock
      fi
    fi
  fi
fi
useradd redis
chown -R redis /data

if [[ -n $REDIS_PASSWORD_FILE ]]; then
  password_aux=`cat ${REDIS_PASSWORD_FILE}`
  export REDIS_PASSWORD=$password_aux
fi

if [[ -n $REDIS_MASTER_PASSWORD_FILE ]]; then
  password_aux=`cat ${REDIS_MASTER_PASSWORD_FILE}`
  export REDIS_MASTER_PASSWORD=$password_aux
fi

if [[ "$REDIS_REPLICATION_MODE" == "master" ]]; then
  echo "I am master"
  if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
    cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
  fi
else
  if [[ ! -f /opt/bitnami/redis/etc/replica.conf ]];then
    cp /opt/bitnami/redis/mounted-etc/replica.conf /opt/bitnami/redis/etc/replica.conf
  fi

  if is_boolean_yes "$REDIS_TLS_ENABLED"; then
    sentinel_info_command="timeout -s 5 10 redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} sentinel get-master-addr-by-name mymaster"
  else
    sentinel_info_command="timeout -s 5 10 redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster"
  fi
  REDIS_SENTINEL_INFO=($($sentinel_info_command))
  REDIS_MASTER_HOST=${REDIS_SENTINEL_INFO[0]}
  REDIS_MASTER_PORT_NUMBER=${REDIS_SENTINEL_INFO[1]}
fi

if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
  cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
fi
ARGS=("--port" "${REDIS_PORT}")

if [[ "$REDIS_REPLICATION_MODE" == "slave" ]]; then
  ARGS+=("--slaveof" "${REDIS_MASTER_HOST}" "${REDIS_MASTER_PORT_NUMBER}")
fi
ARGS+=("--protected-mode" "no")

if [[ "$REDIS_REPLICATION_MODE" == "master" ]]; then
  ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
else
  ARGS+=("--include" "/opt/bitnami/redis/etc/replica.conf")
fi

ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")

touch /data/redisboot.lock
redis-server "${ARGS[@]}"

start-sentinel.sh: |
#!/bin/bash
replace_in_file() {
local filename="${1:?filename is required}"
local match_regex="${2:?match regex is required}"
local substitute_regex="${3:?substitute regex is required}"
local posix_regex=${4:-true}

    local result

    # We should avoid using 'sed in-place' substitutions
    # 1) They are not compatible with files mounted from ConfigMap(s)
    # 2) We found incompatibility issues with Debian10 and "in-place" substitutions
    del=$'\001' # Use a non-printable character as a 'sed' delimiter to avoid issues
    if [[ $posix_regex = true ]]; then
        result="$(sed -E "s${del}${match_regex}${del}${substitute_regex}${del}g" "$filename")"
    else
        result="$(sed "s${del}${match_regex}${del}${substitute_regex}${del}g" "$filename")"
    fi
    echo "$result" > "$filename"
}
sentinel_conf_set() {
    local -r key="${1:?missing key}"
    local value="${2:-}"

    # Sanitize inputs
    value="${value//\\/\\\\}"
    value="${value//&/\\&}"
    value="${value//\?/\\?}"
    [[ "$value" = "" ]] && value="\"$value\""

    replace_in_file "/opt/bitnami/redis-sentinel/etc/sentinel.conf" "^#*\s*${key} .*" "${key} ${value}" false
}
is_boolean_yes() {
    local -r bool="${1:-}"
    # comparison is performed without regard to the case of alphabetic characters
    shopt -s nocasematch
    if [[ "$bool" = 1 || "$bool" =~ ^(yes|true)$ ]]; then
        true
    else
        false
    fi
}

if [[ -n $REDIS_PASSWORD_FILE ]]; then
  password_aux=`cat ${REDIS_PASSWORD_FILE}`
  export REDIS_PASSWORD=$password_aux
fi

if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]]; then
  cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
fi

export REDIS_REPLICATION_MODE="slave"
if [[ $HOSTNAME =~ (.*)-([0-9]+)$ ]]; then
  if [[ ${BASH_REMATCH[2]} == "0" ]]; then
    if [[ ! -f /data/sentinelboot.lock ]]; then
      export REDIS_REPLICATION_MODE="master"
    else
      if is_boolean_yes "$REDIS_TLS_ENABLED"; then
        sentinel_info_command="redis-cli -a $REDIS_PASSWORD -h redis-master-headless.default.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} info"
      else
        sentinel_info_command="redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 info"
      fi
      if [[ ! ($($sentinel_info_command)) ]]; then
         export REDIS_REPLICATION_MODE="master"
         rm /data/sentinelboot.lock
      fi
    fi
  fi
fi

if [[ "$REDIS_REPLICATION_MODE" == "master" ]]; then
  sentinel_conf_set "sentinel monitor" "mymaster redis-master-node-0.redis-master-headless.default.svc.cluster.local 6379 2"
else
  if is_boolean_yes "$REDIS_TLS_ENABLED"; then
    sentinel_info_command="redis-cli -a $REDIS_PASSWORD -h redis-master-headless.default.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} sentinel get-master-addr-by-name mymaster"
  else
    sentinel_info_command="redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster"
  fi
  REDIS_SENTINEL_INFO=($($sentinel_info_command))
  REDIS_MASTER_HOST=${REDIS_SENTINEL_INFO[0]}
  REDIS_MASTER_PORT_NUMBER=${REDIS_SENTINEL_INFO[1]}

  sentinel_conf_set "sentinel monitor" "mymaster "$REDIS_MASTER_HOST" "$REDIS_MASTER_PORT_NUMBER" 2"
fi
touch /data/sentinelboot.lock
redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel

Source: redis/templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: redis-master
namespace: default
labels:
app: redis
chart: redis-11.0.0
heritage: Tiller
release: redis-master
data:
redis.conf: |-
# User-supplied configuration:
# Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
appendfsync everysec
#no-appendfsync-on-rewrite no
#save 900 1
#save 300 10
#save 60 10000
# Disable RDB persistence, AOF persistence already enabled.
save ""
master.conf: |-
dir /data
replica.conf: |-
dir /data
slave-read-only yes
rename-command FLUSHDB ""
rename-command FLUSHALL ""
sentinel.conf: |-
dir "/tmp"
bind 0.0.0.0
port 26379
sentinel monitor mymaster redis-master-node-0.redis-master-headless.default.svc.cluster.local 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 18000
sentinel parallel-syncs mymaster 1

Source: redis/templates/health-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: redis-master-health
namespace: default
labels:
app: redis
chart: redis-11.0.0
heritage: Tiller
release: redis-master
data:
ping_readiness_local.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h localhost
-p $REDIS_PORT
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
ping_liveness_local.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h localhost
-p $REDIS_PORT
ping
)
if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then
echo "$response"
exit 1
fi
ping_sentinel.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h localhost
-p $REDIS_SENTINEL_PORT
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
parse_sentinels.awk: |-
/ip/ {FOUND_IP=1}
/port/ {FOUND_PORT=1}
/runid/ {FOUND_RUNID=1}
!/ip|port|runid/ {
if (FOUND_IP==1) {
IP=$1; FOUND_IP=0;
}
else if (FOUND_PORT==1) {
PORT=$1;
FOUND_PORT=0;
} else if (FOUND_RUNID==1) {
printf "\nsentinel known-sentinel mymaster %s %s %s", IP, PORT, $0; FOUND_RUNID=0;
}
}
ping_readiness_master.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h $REDIS_MASTER_HOST
-p $REDIS_MASTER_PORT_NUMBER
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
ping_liveness_master.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h $REDIS_MASTER_HOST
-p $REDIS_MASTER_PORT_NUMBER
ping
)
if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then
echo "$response"
exit 1
fi
ping_readiness_local_and_master.sh: |-
script_dir="$(dirname "$0")"
exit_status=0
"$script_dir/ping_readiness_local.sh" $1 || exit_status=$?
"$script_dir/ping_readiness_master.sh" $1 || exit_status=$?
exit $exit_status
ping_liveness_local_and_master.sh: |-
script_dir="$(dirname "$0")"
exit_status=0
"$script_dir/ping_liveness_local.sh" $1 || exit_status=$?
"$script_dir/ping_liveness_master.sh" $1 || exit_status=$?
exit $exit_status

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-3
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-2
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-1
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
#redis-data-redis-master-node-0
name: redis-data-redis-master-node-0
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-2
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-1
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-3
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-0
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

Source: redis/templates/headless-svc.yaml

apiVersion: v1
kind: Service
metadata:
name: redis-master-headless
namespace: default
labels:
app: redis
chart: redis-11.0.0
release: redis-master
heritage: Tiller
spec:
type: ClusterIP
clusterIP: None
ports:
- name: redis
port: 6379
targetPort: redis
- name: redis-sentinel
port: 26379
targetPort: redis-sentinel
selector:
app: redis
release: redis-master

Source: redis/templates/redis-with-sentinel-svc.yaml

apiVersion: v1
kind: Service
metadata:
name: redis-master
namespace: default
labels:
app: redis
chart: redis-11.0.0
release: redis-master
heritage: Tiller
spec:
type: ClusterIP
ports:
- name: redis
port: 6379
targetPort: redis
- name: redis-sentinel
port: 26379
targetPort: redis-sentinel
selector:
app: redis
release: redis-master

Source: redis/templates/redis-node-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-master-node
namespace: default
labels:
app: redis
chart: redis-11.0.0
release: redis-master
heritage: Tiller
spec:
replicas: 3
serviceName: redis-master-headless
selector:
matchLabels:
app: redis
release: redis-master
role: node
template:
metadata:
labels:
app: redis
release: redis-master
chart: redis-11.0.0
role: node
annotations:
checksum/health: 08ddae951303a7ae9b56253bb4a68c17c05fee1074a472a53302403f9f7bd160
checksum/configmap: 2d8d1cfc2398f046811b068339a9ac33db8b20767cff63bb2a2d09095eab8750
checksum/secret: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
spec:

  securityContext:
    fsGroup: 1001
  serviceAccountName: default
  containers:
    - name: redis
      image: docker.io/bitnami/redis:6.0.8-debian-10-r0
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsUser: 0
      command:
        - /bin/bash
        - -c
        - /opt/bitnami/scripts/start-scripts/start-node.sh
      env:
        - name: REDIS_MASTER_PORT_NUMBER
          value: "6379"
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        - name: REDIS_TLS_ENABLED
          value: "no"
        - name: REDIS_PORT
          value: "6379"
        - name: REDIS_DATA_DIR
          value: /data
      ports:
        - name: redis
          containerPort: 6379
      livenessProbe:
        initialDelaySeconds: 30
        periodSeconds: 10
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_liveness_local.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 10
        timeoutSeconds: 10
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_readiness_local.sh 5
      resources:
        null

      volumeMounts:
        - name: start-scripts
          mountPath: /opt/bitnami/scripts/start-scripts
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
        - name: config
          mountPath: /opt/bitnami/redis/mounted-etc
        - name: redis-tmp-conf
          mountPath: /opt/bitnami/redis/etc
    - name: sentinel
      image: docker.io/bitnami/redis-sentinel:6.0.8-debian-10-r1
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsUser: 0
      command:
        - /bin/bash
        - -c
        - /opt/bitnami/scripts/start-scripts/start-sentinel.sh
      env:
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        - name: REDIS_SENTINEL_TLS_ENABLED
          value: "no"
        - name: REDIS_SENTINEL_PORT
          value: "26379"
      ports:
        - name: redis-sentinel
          containerPort: 26379
      livenessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_sentinel.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_sentinel.sh 5
      resources:
        null

      volumeMounts:
        - name: start-scripts
          mountPath: /opt/bitnami/scripts/start-scripts
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
          subPath:
        - name: config
          mountPath: /opt/bitnami/redis-sentinel/mounted-etc
        - name: sentinel-tmp-conf
          mountPath: /opt/bitnami/redis-sentinel/etc
  volumes:
    - name: start-scripts
      configMap:
        name: redis-master-scripts
        defaultMode: 0755
    - name: health
      configMap:
        name: redis-master-health
        defaultMode: 0755
    - name: config
      configMap:
        name: redis-master
    - name: sentinel-tmp-conf
      emptyDir: {}
    - name: redis-tmp-conf
      emptyDir: {}

volumeClaimTemplates:
- metadata:
name: redis-data
labels:
app: redis
release: redis-master
heritage: Tiller
component: slave
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"

    selector:

updateStrategy:
type: RollingUpdate
LAST DEPLOYED: Thu Sep 17 07:37:59 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME AGE
redis-master 0s
redis-master-health 0s
redis-master-scripts 0s

==> v1/PersistentVolume
NAME AGE
redis-data-redis-master-node-0 0s
redis-data-redis-master-node-1 0s
redis-data-redis-master-node-2 0s
redis-data-redis-master-node-3 0s

==> v1/PersistentVolumeClaim
NAME AGE
redis-data-redis-master-node-0 0s
redis-data-redis-master-node-1 0s
redis-data-redis-master-node-2 0s
redis-data-redis-master-node-3 0s

==> v1/Pod(related)
NAME AGE
redis-master-node-0 0s

==> v1/Service
NAME AGE
redis-master 0s
redis-master-headless 0s

==> v1/StatefulSet
NAME AGE
redis-master-node 0s

NOTES:
** Please be patient while the chart is being deployed **
Redis can be accessed via port 6379 on the following DNS name from within your cluster:

redis-master.default.svc.cluster.local for read only operations

For read/write operations, first access the Redis Sentinel cluster, which is available in port 26379 using the same domain name above.

To connect to your Redis server:

Run a Redis pod that you can use as a client:
kubectl run --namespace default redis-master-client --rm --tty -i --restart='Never' \

--image docker.io/bitnami/redis:6.0.8-debian-10-r0 -- bash
Connect using the Redis CLI:
redis-cli -h redis-master -p 6379 # Read only operations
redis-cli -h redis-master -p 26379 # Sentinel access

To connect to your database from outside the cluster execute the following commands:

kubectl port-forward --namespace default svc/redis-master-master 6379:6379 &
redis-cli -h 127.0.0.1 -p 6379

etavene · 2020-11-29T14:44:41Z

@javsalgar

we created chart using image built from redis 6.0.8 code base, and ran couple of performance comparision tests
and observed that bitnami charts gave less performance.

./redis-benchmark -h 10.233.127.60 -t hset -r 100000 -n 1000000

The above command provided 64968.81 requests per second on a stand-alone redis server with the helm chart built using redis-6.0.8 code base, where as, bitnami charts you provided, gave us around 62262.62 requests per second.
Is there any reason for this latency differences? As both DB config's are same, what could be the reason for this behaviour ?

snippet of our dockerfile

###############################################################
FROM debian:buster-slim
ENV REDIS_VERSION 6.0.8
ENV REDIS_DOWNLOAD_URL http://download.redis.io/releases/redis-6.0.8.tar.gz
###################################################################

Bitnami Templates Results:
====== HSET ======
1000000 requests completed in 16.06 seconds (62262.62 requests per second)
50 parallel clients
3 bytes payload
keep alive: 1
host configuration "save":
host configuration "appendonly": yes
multi-thread: no

**: Our template using redis code base **

====== HSET ======
1000000 requests completed in 15.39 seconds (64968.81 requests per second)
50 parallel clients
3 bytes payload
keep alive: 1
host configuration "save":
host configuration "appendonly": yes
multi-thread: no

Bitnami Templates DB config ( defaults are not changed)

"rdbchecksum"
"yes"
"daemonize"
"no"
"io-threads-do-reads"
"no"
"lua-replicate-commands"
"yes"
"always-show-logo"
"no"
"protected-mode"
"no"
"rdbcompression"
"yes"
"rdb-del-sync-files"
"no"
"activerehashing"
"yes"
"stop-writes-on-bgsave-error"
"yes"
"dynamic-hz"
"yes"
"lazyfree-lazy-eviction"
"no"
"lazyfree-lazy-expire"
"no"
"lazyfree-lazy-server-del"
"no"
"lazyfree-lazy-user-del"
"no"
"repl-disable-tcp-nodelay"
"no"
"repl-diskless-sync"
"no"
"gopher-enabled"
"no"
"aof-rewrite-incremental-fsync"
"yes"
"no-appendfsync-on-rewrite"
"no"
"cluster-require-full-coverage"
"yes"
"rdb-save-incremental-fsync"
"yes"
"aof-load-truncated"
"yes"
"aof-use-rdb-preamble"
"yes"
"cluster-replica-no-failover"
"no"
"cluster-slave-no-failover"
"no"
"replica-lazy-flush"
"no"
"slave-lazy-flush"
"no"
"replica-serve-stale-data"
"yes"
"slave-serve-stale-data"
"yes"
"replica-read-only"
"yes"
"slave-read-only"
"yes"
"replica-ignore-maxmemory"
"yes"
"slave-ignore-maxmemory"
"yes"
"jemalloc-bg-thread"
"yes"
"activedefrag"
"no"
"syslog-enabled"
"no"
"cluster-enabled"
"no"
"appendonly"
"yes"
"cluster-allow-reads-when-down"
"no"
"oom-score-adj"
"no"
"aclfile"
""
"unixsocket"
""
"pidfile"
""
"replica-announce-ip"
""
"slave-announce-ip"
""
"masteruser"
""
"masterauth"
""
"cluster-announce-ip"
""
"syslog-ident"
"redis"
"dbfilename"
"dump.rdb"
"appendfilename"
"appendonly.aof"
"server_cpulist"
""
"bio_cpulist"
""
"aof_rewrite_cpulist"
""
"bgsave_cpulist"
""
"supervised"
"no"
"syslog-facility"
"local0"
"repl-diskless-load"
"disabled"
"loglevel"
"notice"
"maxmemory-policy"
"noeviction"
"appendfsync"
"everysec"
"databases"
"16"
"port"
"6379"
"io-threads"
"1"
"auto-aof-rewrite-percentage"
"100"
"cluster-replica-validity-factor"
"10"
"cluster-slave-validity-factor"
"10"
"list-max-ziplist-size"
"-2"
"tcp-keepalive"
"300"
"cluster-migration-barrier"
"1"
"active-defrag-cycle-min"
"1"
"active-defrag-cycle-max"
"25"
"active-defrag-threshold-lower"
"10"
"active-defrag-threshold-upper"
"100"
"lfu-log-factor"
"10"
"lfu-decay-time"
"1"
"replica-priority"
"100"
"slave-priority"
"100"
"repl-diskless-sync-delay"
"5"
"maxmemory-samples"
"5"
"timeout"
"0"
"replica-announce-port"
"0"
"slave-announce-port"
"0"
"tcp-backlog"
"511"
"cluster-announce-bus-port"
"0"
"cluster-announce-port"
"0"
"repl-timeout"
"60"
"repl-ping-replica-period"
"10"
"repl-ping-slave-period"
"10"
"list-compress-depth"
"0"
"rdb-key-save-delay"
"0"
"key-load-delay"
"0"
"active-expire-effort"
"1"
"hz"
"10"
"min-replicas-to-write"
"0"
"min-slaves-to-write"
"0"
"min-replicas-max-lag"
"10"
"min-slaves-max-lag"
"10"
"maxclients"
"10000"
"active-defrag-max-scan-fields"
"1000"
"slowlog-max-len"
"128"
"acllog-max-len"
"128"
"lua-time-limit"
"5000"
"cluster-node-timeout"
"15000"
"slowlog-log-slower-than"
"10000"
"latency-monitor-threshold"
"0"
"proto-max-bulk-len"
"536870912"
"stream-node-max-entries"
"100"
"repl-backlog-size"
"1048576"
"maxmemory"
"0"
"hash-max-ziplist-entries"
"512"
"set-max-intset-entries"
"512"
"zset-max-ziplist-entries"
"128"
"active-defrag-ignore-bytes"
"104857600"
"hash-max-ziplist-value"
"64"
"stream-node-max-bytes"
"4096"
"zset-max-ziplist-value"
"64"
"hll-sparse-max-bytes"
"3000"
"tracking-table-max-keys"
"1000000"
"repl-backlog-ttl"
"3600"
"auto-aof-rewrite-min-size"
"67108864"
"tls-port"
"0"
"tls-session-cache-size"
"20480"
"tls-session-cache-timeout"
"300"
"tls-cluster"
"no"
"tls-replication"
"no"
"tls-auth-clients"
"yes"
"tls-prefer-server-ciphers"
"no"
"tls-session-caching"
"yes"
"tls-cert-file"
""
"tls-key-file"
""
"tls-dh-params-file"
""
"tls-ca-cert-file"
""
"tls-ca-cert-dir"
""
"tls-protocols"
""
"tls-ciphers"
""
"tls-ciphersuites"
""
"logfile"
""
"client-query-buffer-limit"
"1073741824"
"watchdog-period"
"0"
"dir"
"/data"
"save"
""
"client-output-buffer-limit"
"normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
"unixsocketperm"
"0"
"slaveof"
""
"notify-keyspace-events"
""
"bind"
""
"requirepass"
""
"oom-score-adj-values"
"0 200 800"

Our Templates DB config:

DB config:

"rdbchecksum"
"yes"
"daemonize"
"no"
"io-threads-do-reads"
"no"
"lua-replicate-commands"
"yes"
"always-show-logo"
"yes"
"protected-mode"
"no"
"rdbcompression"
"yes"
"rdb-del-sync-files"
"no"
"activerehashing"
"yes"
"stop-writes-on-bgsave-error"
"yes"
"dynamic-hz"
"yes"
"lazyfree-lazy-eviction"
"no"
"lazyfree-lazy-expire"
"no"
"lazyfree-lazy-server-del"
"no"
"lazyfree-lazy-user-del"
"no"
"repl-disable-tcp-nodelay"
"no"
"repl-diskless-sync"
"no"
"gopher-enabled"
"no"
"aof-rewrite-incremental-fsync"
"yes"
"no-appendfsync-on-rewrite"
"no"
"cluster-require-full-coverage"
"yes"
"rdb-save-incremental-fsync"
"yes"
"aof-load-truncated"
"yes"
"aof-use-rdb-preamble"
"yes"
"cluster-replica-no-failover"
"no"
"cluster-slave-no-failover"
"no"
"replica-lazy-flush"
"no"
"slave-lazy-flush"
"no"
"replica-serve-stale-data"
"yes"
"slave-serve-stale-data"
"yes"
"replica-read-only"
"yes"
"slave-read-only"
"yes"
"replica-ignore-maxmemory"
"yes"
"slave-ignore-maxmemory"
"yes"
"jemalloc-bg-thread"
"yes"
"activedefrag"
"no"
"syslog-enabled"
"no"
"cluster-enabled"
"no"
"appendonly"
"yes"
"cluster-allow-reads-when-down"
"no"
"oom-score-adj"
"no"
"aclfile"
""
"unixsocket"
""
"pidfile"
"/var/run/redis_6379.pid"
"replica-announce-ip"
""
"slave-announce-ip"
""
"masteruser"
""
"masterauth"
""
"cluster-announce-ip"
""
"syslog-ident"
"redis"
"dbfilename"
"dump.rdb"
"appendfilename"
"appendonly.aof"
"server_cpulist"
""
"bio_cpulist"
""
"aof_rewrite_cpulist"
""
"bgsave_cpulist"
""
"supervised"
"no"
"syslog-facility"
"local0"
"repl-diskless-load"
"disabled"
"loglevel"
"notice"
"maxmemory-policy"
"noeviction"
"appendfsync"
"everysec"
"databases"
"16"
"port"
"6379"
"io-threads"
"1"
"auto-aof-rewrite-percentage"
"100"
"cluster-replica-validity-factor"
"10"
"cluster-slave-validity-factor"
"10"
"list-max-ziplist-size"
"-2"
"tcp-keepalive"
"300"
"cluster-migration-barrier"
"1"
"active-defrag-cycle-min"
"1"
"active-defrag-cycle-max"
"25"
"active-defrag-threshold-lower"
"10"
"active-defrag-threshold-upper"
"100"
"lfu-log-factor"
"10"
"lfu-decay-time"
"1"
"replica-priority"
"100"
"slave-priority"
"100"
"repl-diskless-sync-delay"
"5"
"maxmemory-samples"
"5"
"timeout"
"0"
"replica-announce-port"
"0"
"slave-announce-port"
"0"
"tcp-backlog"
"511"
"cluster-announce-bus-port"
"0"
"cluster-announce-port"
"0"
"repl-timeout"
"60"
"repl-ping-replica-period"
"10"
"repl-ping-slave-period"
"10"
"list-compress-depth"
"0"
"rdb-key-save-delay"
"0"
"key-load-delay"
"0"
"active-expire-effort"
"1"
"hz"
"10"
"min-replicas-to-write"
"0"
"min-slaves-to-write"
"0"
"min-replicas-max-lag"
"10"
"min-slaves-max-lag"
"10"
"maxclients"
"10000"
"active-defrag-max-scan-fields"
"1000"
"slowlog-max-len"
"128"
"acllog-max-len"
"128"
"lua-time-limit"
"5000"
"cluster-node-timeout"
"15000"
"slowlog-log-slower-than"
"10000"
"latency-monitor-threshold"
"0"
"proto-max-bulk-len"
"536870912"
"stream-node-max-entries"
"100"
"repl-backlog-size"
"1048576"
"maxmemory"
"0"
"hash-max-ziplist-entries"
"512"
"set-max-intset-entries"
"512"
"zset-max-ziplist-entries"
"128"
"active-defrag-ignore-bytes"
"104857600"
"hash-max-ziplist-value"
"64"
"stream-node-max-bytes"
"4096"
"zset-max-ziplist-value"
"64"
"hll-sparse-max-bytes"
"3000"
"tracking-table-max-keys"
"1000000"
"repl-backlog-ttl"
"3600"
"auto-aof-rewrite-min-size"
"67108864"
"tls-port"
"0"
"tls-session-cache-size"
"20480"
"tls-session-cache-timeout"
"300"
"tls-cluster"
"no"
"tls-replication"
"no"
"tls-auth-clients"
"yes"
"tls-prefer-server-ciphers"
"no"
"tls-session-caching"
"yes"
"tls-cert-file"
""
"tls-key-file"
""
"tls-dh-params-file"
""
"tls-ca-cert-file"
""
"tls-ca-cert-dir"
""
"tls-protocols"
""
"tls-ciphers"
""
"tls-ciphersuites"
""
"logfile"
""
"client-query-buffer-limit"
"1073741824"
"watchdog-period"
"0"
"dir"
"/data"
"save"
""
"client-output-buffer-limit"
"normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
"unixsocketperm"
"0"
"slaveof"
""
"notify-keyspace-events"
""
"bind"
""
"requirepass"
""
"oom-score-adj-values"
"0 200 800"

Bitnami charts redis server info
redis_version:6.0.8
redis_build_id:e46fe36ddd420afe
redis_mode:standalone
os:Linux 4.4.0-131-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:8.3.0
executable:/redis-server

our template redis server info:
redis_version:6.0.8
redis_build_id:bea1c0ecc8184f2f
redis_mode:standalone
os:Linux 4.4.0-131-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:8.3.0
config_file:/etc/redis/redis.conf

rafariossaa · 2020-11-30T11:26:50Z

Hi @etavene ,
This thread is closed and your question is not related to this issue.
Do you mind opening a new one so it can be properly handled ?

Thanks forehand.

javsalgar added the on-hold Issues or Pull Requests with this label will never be considered stale label Aug 31, 2020

rafariossaa mentioned this issue Sep 11, 2020

[bitnami/redis] Removes master/slave when using sentinel #3658

Merged

4 tasks

javsalgar closed this as completed in #3658 Sep 17, 2020

carrodher removed the on-hold Issues or Pull Requests with this label will never be considered stale label Dec 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sentinels are not working in failover case #3472

Sentinels are not working in failover case #3472

mahanam commented Aug 19, 2020

javsalgar commented Aug 20, 2020

mahanam commented Aug 24, 2020

javsalgar commented Aug 24, 2020

mahanam commented Aug 24, 2020

javsalgar commented Aug 25, 2020

mahanam commented Aug 25, 2020

javsalgar commented Aug 25, 2020

mahanam commented Aug 25, 2020 •

edited by javsalgar

Loading

javsalgar commented Aug 26, 2020

mahanam commented Aug 26, 2020 •

edited

Loading

mahanam commented Aug 31, 2020

javsalgar commented Aug 31, 2020

mahanam commented Aug 31, 2020

rafariossaa commented Sep 2, 2020

mahanam commented Sep 3, 2020

mahanam commented Sep 7, 2020

rafariossaa commented Sep 7, 2020

mahanam commented Sep 17, 2020 •

edited

Loading

etavene commented Nov 29, 2020

rafariossaa commented Nov 30, 2020

Sentinels are not working in failover case #3472

Sentinels are not working in failover case #3472

Comments

mahanam commented Aug 19, 2020

Configuration of debug manifest files:

Enable AOF https://redis.io/topics/persistence#append-only-file

Disable RDB persistence, AOF persistence already enabled.

Source: redis/templates/networkpolicy.yaml

Source: redis/templates/secret.yaml

apiVersion: v1 kind: Secret metadata: name: redis-ha namespace: default labels: app: redis chart: redis-10.7.16 release: "redis-ha" heritage: "Tiller" type: Opaque data: redis-password: "aUZGUkFoc1hxMA=="

Source: redis/templates/configmap.yaml

Source: redis/templates/health-configmap.yaml

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: redis-data-redis-ha-slave-2 labels: type: local spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: redis-data-redis-ha-slave-1 labels: type: local spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: #path: "/mnt/data" path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: redis-data-redis-ha-master-0 labels: type: local spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: #path: "/mnt/data" path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: redis-data-redis-ha-slave-0 labels: type: local spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: #path: "/mnt/data" path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data-redis-ha-slave-2 spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data-redis-ha-slave-1 spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data-redis-ha-master-0 spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data-redis-ha-slave-0 spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "2Gi"

Source: redis/templates/headless-svc.yaml

Source: redis/templates/redis-with-sentinel-svc.yaml

Source: redis/templates/redis-master-statefulset.yaml

updateStrategy: type: RollingUpdate

Source: redis/templates/redis-slave-statefulset.yaml

javsalgar commented Aug 20, 2020

mahanam commented Aug 24, 2020

javsalgar commented Aug 24, 2020

mahanam commented Aug 24, 2020

javsalgar commented Aug 25, 2020

mahanam commented Aug 25, 2020

javsalgar commented Aug 25, 2020

mahanam commented Aug 25, 2020 • edited by javsalgar Loading

javsalgar commented Aug 26, 2020

mahanam commented Aug 26, 2020 • edited Loading

mahanam commented Aug 31, 2020

javsalgar commented Aug 31, 2020

mahanam commented Aug 31, 2020

rafariossaa commented Sep 2, 2020

mahanam commented Sep 3, 2020

mahanam commented Sep 7, 2020

rafariossaa commented Sep 7, 2020

mahanam commented Sep 17, 2020 • edited Loading

- FLUSHDB

- FLUSHALL

Enable AOF https://redis.io/topics/persistence#append-only-file

Disable RDB persistence, AOF persistence already enabled.

Enable AOF https://redis.io/topics/persistence#append-only-file

Disable RDB persistence, AOF persistence already enabled.

Source: redis/templates/configmap-scripts.yaml

Source: redis/templates/configmap.yaml

Source: redis/templates/health-configmap.yaml

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: redis-data-redis-master-node-3 labels: type: local spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: redis-data-redis-master-node-2 labels: type: local spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: #path: "/mnt/data" path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: redis-data-redis-master-node-1 labels: type: local spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: #path: "/mnt/data" path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolume metadata: #redis-data-redis-master-node-0 name: redis-data-redis-master-node-0 labels: type: local spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce hostPath: #path: "/mnt/data" path: "/data"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data-redis-master-node-2 spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data-redis-master-node-1 spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data-redis-master-node-3 spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "2Gi"

Source: redis/templates/pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data-redis-master-node-0 spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "2Gi"

Source: redis/templates/headless-svc.yaml

Source: redis/templates/redis-with-sentinel-svc.yaml

Source: redis/templates/redis-node-statefulset.yaml

etavene commented Nov 29, 2020

apiVersion: v1
kind: Secret
metadata:
name: redis-ha
namespace: default
labels:
app: redis
chart: redis-10.7.16
release: "redis-ha"
heritage: "Tiller"
type: Opaque
data:
redis-password: "aUZGUkFoc1hxMA=="

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-ha-slave-2
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data"

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-ha-slave-1
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-ha-master-0
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-ha-slave-0
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-ha-slave-2
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-ha-slave-1
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-ha-master-0
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-ha-slave-0
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

updateStrategy:
type: RollingUpdate

mahanam commented Aug 25, 2020 •

edited by javsalgar

Loading

mahanam commented Aug 26, 2020 •

edited

Loading

mahanam commented Sep 17, 2020 •

edited

Loading

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-3
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data"

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-2
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-1
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

apiVersion: v1
kind: PersistentVolume
metadata:
#redis-data-redis-master-node-0
name: redis-data-redis-master-node-0
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-2
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-1
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-3
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-0
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"