Patroni list all timescaleDB servers but I have no leaders

[throrin19]

Hello,

I try to launch the single-node chart in my kubernetes cluster. But I have a big problem : I have a timeout on try to connecting to the server using AWS NLB for the leader service.

I try to connect to servers and launch patronictl list. I have the list of all my timescaleDB servers, but none of them is leader and the cluster is uninitialized :

+ Cluster: timescaledb-single (uninitialized) +---------+----+-----------+
|        Member        |      Host     | Role |  State  | TL | Lag in MB |
+----------------------+---------------+------+---------+----+-----------+
| timescaledb-single-0 | 172.31.13.213 |      | running |  1 |         0 |
| timescaledb-single-1 | 172.31.32.227 |      | running |  1 |         0 |
| timescaledb-single-2 |  172.31.25.84 |      | running |  1 |         0 |
+----------------------+---------------+------+---------+----+-----------+

Is it normal ?

This is my value.yml passed to the chart :

replicaCount: 3
loadBalancer:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
persistentVolumes:
  data:
    size: 10Gi
secretNames:
  credentials: credentials
  certificate: certificate
  # pgbackrest: pgbackrest
patroni:
  postgresql:
    pg_hba:
    - local     all             all                                   peer
    - hostssl   all             all                127.0.0.1/32       md5
    - hostssl   all             all                ::1/128            md5
    - hostssl   replication     standby            all                md5
    - hostssl   all             all                all                md5
    - host      all             all                all                md5

[throrin19]

After research, If I see patroni logs with service patroni status in pods, I have this error :

[FAIL] Patroni configuration file /etc/patroni/postgres.yml does not exist. ... failed!

It's weird because, the patroni config is mount in /etc/timescaledb/patroni.yaml. This is an error in the chart declaration ?

[feikesteenbergen]

What do the logs of the containers say?

kubectl logs pod/timescaledb-single-0 -c timescaledb
kubectl logs pod/timescaledb-single-1 -c timescaledb
kubectl logs pod/timescaledb-single-2 -c timescaledb

service patroni status

That will not work within a pod, Patroni is running as pid 1, there is no notion of services inside the container.
The command you use would be useful on a VM deployment, but not inside a container.

[throrin19]

After clean recreate deployment, I can connect to the master server.

Ok, I understand for Patroni part.

This is the log for my pods :

• pod/timescaledb-single-0 :

2020-06-24 07:28:14 UTC [32202]: [5ef3008e.7dca-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:28:14 UTC [32202]: [5ef3008e.7dca-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:28:14 UTC [32202]: [5ef3008e.7dca-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
2020-06-24 07:28:14 UTC [32203]: [5ef3008e.7dcb-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=172.31.22.118 port=8051
2020-06-24 07:28:14 UTC [32203]: [5ef3008e.7dcb-2] [unknown]@[unknown],app=[unknown] [08P01] LOG:  incomplete startup packet
2020-06-24 07:28:15 UTC [32204]: [5ef3008f.7dcc-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=172.31.1.35 port=33543
2020-06-24 07:28:15 UTC [32204]: [5ef3008f.7dcc-2] [unknown]@[unknown],app=[unknown] [08P01] LOG:  incomplete startup packet
2020-06-24 07:28:15 UTC [32205]: [5ef3008f.7dcd-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=172.31.33.61 port=59116
2020-06-24 07:28:15 UTC [32205]: [5ef3008f.7dcd-2] [unknown]@[unknown],app=[unknown] [08P01] LOG:  incomplete startup packet
2020-06-24 07:28:15 UTC [32206]: [5ef3008f.7dce-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=172.31.22.118 port=17906
2020-06-24 07:28:15 UTC [32206]: [5ef3008f.7dce-2] [unknown]@[unknown],app=[unknown] [08P01] LOG:  incomplete startup packet
2020-06-24 07:28:15 UTC [32207]: [5ef3008f.7dcf-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=172.31.40.184 port=27819
2020-06-24 07:28:15 UTC [32207]: [5ef3008f.7dcf-2] [unknown]@[unknown],app=[unknown] [08P01] LOG:  incomplete startup packet
2020-06-24 07:28:15 UTC [32208]: [5ef3008f.7dd0-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=172.31.1.35 port=32921
2020-06-24 07:28:15 UTC [32208]: [5ef3008f.7dd0-2] [unknown]@[unknown],app=[unknown] [08P01] LOG:  incomplete startup packet
2020-06-24 07:28:15 UTC [32209]: [5ef3008f.7dd1-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=172.31.33.61 port=49244
2020-06-24 07:28:15 UTC [32209]: [5ef3008f.7dd1-2] [unknown]@[unknown],app=[unknown] [08P01] LOG:  incomplete startup packet

• pod/timescaledb-single-1 :

2020-06-24 07:25:45 UTC [16644]: [5ef2fff9.4104-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:25:45 UTC [16644]: [5ef2fff9.4104-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.004 user=postgres database=postgres host=[local]
2020-06-24 07:26:15 UTC [16652]: [5ef30017.410c-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:26:15 UTC [16652]: [5ef30017.410c-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:26:15 UTC [16652]: [5ef30017.410c-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.001 user=postgres database=postgres host=[local]
2020-06-24 07:26:45 UTC [16661]: [5ef30035.4115-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:26:45 UTC [16661]: [5ef30035.4115-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:26:45 UTC [16661]: [5ef30035.4115-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.003 user=postgres database=postgres host=[local]
2020-06-24 07:27:15 UTC [16668]: [5ef30053.411c-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:27:15 UTC [16668]: [5ef30053.411c-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:27:15 UTC [16668]: [5ef30053.411c-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
2020-06-24 07:27:45 UTC [16675]: [5ef30071.4123-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:27:45 UTC [16675]: [5ef30071.4123-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:27:45 UTC [16675]: [5ef30071.4123-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.003 user=postgres database=postgres host=[local]

• pod/timescaledb-single-2 :

2020-06-24 07:28:30 UTC [16815]: [5ef3009e.41af-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:28:30 UTC [16815]: [5ef3009e.41af-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:28:30 UTC [16815]: [5ef3009e.41af-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
2020-06-24 07:29:00 UTC [16822]: [5ef300bc.41b6-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:29:00 UTC [16822]: [5ef300bc.41b6-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:29:00 UTC [16822]: [5ef300bc.41b6-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.005 user=postgres database=postgres host=[local]
2020-06-24 07:29:30 UTC [16829]: [5ef300da.41bd-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:29:30 UTC [16829]: [5ef300da.41bd-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:29:30 UTC [16829]: [5ef300da.41bd-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.003 user=postgres database=postgres host=[local]
2020-06-24 07:30:00 UTC [16836]: [5ef300f8.41c4-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
2020-06-24 07:30:00 UTC [16836]: [5ef300f8.41c4-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres
2020-06-24 07:30:00 UTC [16836]: [5ef300f8.41c4-3] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.005 user=postgres database=postgres host=[local]

[feikesteenbergen]

That looks good, those connection attempts are pg_isready and are most likely the readinessProbe of kubernetes

[throrin19]

Thanks 😄 I continue test this chart.

I close the issue