Operator can't handle pods in Error state after initialisation

[FlavioF]

When one of the pods of a AS cluster goes from Running state to Error state, AS operator is not able to recover the cluster.

Observed Behaviour
For cluster cluster-1 with 3 nodes Running:

• When pod cluster-1-2 goes to error state (because aerospike server container stoped working due different reasons)
• AS Operator will kill cluster-1-0 in a infinite loop
• AS Operator will never handle the pod cluster-1-2

How to replicate

1. SSH into one vm with one AS Cluster pod and kill the container with the aerospike cluster running
2. Check that Pod is now in Error state
3. Check that operator is not able to handle with this error

Temporary workaround
The problem is solved by killing the pod in error state manually

[pires]

Kubernetes allows one to configure the behavior to be taken or not taken at all when a pod container(s) fail, the PodSpec.restartPolicy. By default, the behavior is to always restart a pod in case of a failure.

However, this operator explicitly disables said default behavior and instead tries and reconcile in the face of an error. Aren't logs telling you the pod is failed?

[FlavioF]

logs for what is described above.

as-cluster-0-0                        2/2       Running   0          2m
as-cluster-0-1                        2/2       Running   0          2m
as-cluster-0-2                        2/2       Running   0          2m


as-cluster-0-2   1/2       Error     0         3m # Container killed manually inside the vm

time="2019-09-06T11:13:00Z" level=debug msg="processing object: data-postgresql-ex-vote-postgresql-0" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:13:00Z" level=debug msg="processing object: as-cluster-0-1-as-namespace-0-w7djc" controller=aerospikegarbagecollectorcontroller=aerospikegarbagecollector
time="2019-09-06T11:13:00Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-nr8tw" controller=aerospikegarbagecollector
time="2019-09-06T11:13:00Z" level=debug msg="processing object: as-cluster-0-2-as-namespace-0-gljzv" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:13:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-1-as-namespace-0-w7djc
time="2019-09-06T11:13:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-nr8tw
time="2019-09-06T11:13:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-1-as-namespace-0-w7djc'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-nr8tw'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:00Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-bg8jj
time="2019-09-06T11:13:00Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-cn57t
time="2019-09-06T11:13:00Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-cn57t'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:00Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-bg8jj'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-2-as-namespace-0-gljzv
time="2019-09-06T11:13:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-2-as-namespace-0-gljzv'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:13:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:00Z" level=info msg="processing cluster" aerospikecluster=fferreira/as-cluster-0
time="2019-09-06T11:13:00Z" level=debug msg="service already exists" aerospikecluster=fferreira/as-cluster-0 service=as-cluster-0
time="2019-09-06T11:13:00Z" level=debug msg="configmap exists and is up to date" aerospikecluster=fferreira/as-cluster-0 configmap=as-cluster-0
time="2019-09-06T11:13:00Z" level=debug msg="networkpolicy already exists" aerospikecluster=fferreira/as-cluster-0
time="2019-09-06T11:13:00Z" level=debug msg="checking if pods need to be updated" aerospikecluster=fferreira/as-cluster-0 currentSize=3 desiredSize=3

time="2019-09-06T11:13:00Z" level=debug msg="processing object: as-cluster-0-2" controller=aerospikecluster
...processing object..
time="2019-09-06T11:13:30Z" level=debug msg="processing object: as-cluster-0-1-as-namespace-0-w7djc" controller=aerospikegarbagecollector
time="2019-09-06T11:13:30Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-nr8tw" controller=aerospikegarbagecollector
time="2019-09-06T11:13:30Z" level=debug msg="processing object: as-cluster-0-2-as-namespace-0-gljzv" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:13:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-1-as-namespace-0-w7djc
time="2019-09-06T11:13:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-1-as-namespace-0-w7djc'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-nr8tw
time="2019-09-06T11:13:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-nr8tw'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:30Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-cn57t
time="2019-09-06T11:13:30Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-cn57t'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:30Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-bg8jj
time="2019-09-06T11:13:30Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-bg8jj'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-2-as-namespace-0-gljzv
time="2019-09-06T11:13:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-2-as-namespace-0-gljzv'" controller=aerospikegarbagecollector
time="2019-09-06T11:13:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:13:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:14:00Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-nr8tw" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="processing object: as-cluster-0-2-as-namespace-0-gljzv" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-2w4gk" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="processing object: as-cluster-0-1-as-namespace-0-w7djc" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:14:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-nr8tw
time="2019-09-06T11:14:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-nr8tw'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-cn57t
time="2019-09-06T11:14:00Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-cn57t'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-bg8jj
time="2019-09-06T11:14:00Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-bg8jj'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-2-as-namespace-0-gljzv
time="2019-09-06T11:14:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-2-as-namespace-0-gljzv'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:14:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-1-as-namespace-0-w7djc
time="2019-09-06T11:14:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-1-as-namespace-0-w7djc'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-2w4gk" controller=aerospikegarbagecollector
time="2019-09-06T11:14:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:14:00Z" level=debug msg="no expiration set for pvc" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:14:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:14:00Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
time="2019-09-06T11:14:01Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster


as-cluster-0-0   2/2       Terminating   0         4m

time="2019-09-06T11:14:08Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
time="2019-09-06T11:14:08Z" level=debug msg="pod has been deleted" aerospikecluster=fferreira/as-cluster-0 pod=fferreira/as-cluster-0-0
time="2019-09-06T11:14:08Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
time="2019-09-06T11:14:08Z" level=error msg="failed tip-clear ip on pod \"fferreira/as-cluster-0-2\"" aerospikecluster=as-cluster-0 pod=fferreira/as-cluster-0-0
time="2019-09-06T11:14:08Z" level=error msg="failed alumni-reset on pod \"fferreira/as-cluster-0-2\"" aerospikecluster=as-cluster-0 pod=fferreira/as-cluster-0-0
time="2019-09-06T11:14:08Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-2w4gk" controller=aerospikegarbagecollector
time="2019-09-06T11:14:08Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:14:08Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:14:09Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
time="2019-09-06T11:14:09Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
time="2019-09-06T11:14:09Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
...processing object..

as-cluster-0-0   0/2       Pending   0         1s
as-cluster-0-0   0/2       Init:0/1   0         1s
as-cluster-0-0   0/2       PodInitializing   0         12s
as-cluster-0-0   1/2       Running   0         14s
as-cluster-0-0   2/2       Running   0         18s

time="2019-09-06T11:14:18Z" level=error msg="failed tip-clear ip on pod \"fferreira/as-cluster-0-0\"" aerospikecluster=as-cluster-0 pod=fferreira/as-cluster-0-0
time="2019-09-06T11:14:20Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
time="2019-09-06T11:14:22Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
...processing object..
time="2019-09-06T11:14:28Z" level=error msg="failed alumni-reset on pod \"fferreira/as-cluster-0-0\"" aerospikecluster=as-cluster-0 pod=fferreira/as-cluster-0-0
E0906 11:14:28.902207       1 generic.go:165] error syncing 'fferreira/as-cluster-0': detected incorrect cluster size for pod "fferreira/as-cluster-0-0"
time="2019-09-06T11:14:28Z" level=info msg="processing cluster" aerospikecluster=fferreira/as-cluster-0
time="2019-09-06T11:14:28Z" level=debug msg="service already exists" aerospikecluster=fferreira/as-cluster-0 service=as-cluster-0
time="2019-09-06T11:14:28Z" level=debug msg="configmap exists and is up to date" aerospikecluster=fferreira/as-cluster-0 configmap=as-cluster-0
time="2019-09-06T11:14:28Z" level=debug msg="networkpolicy already exists" aerospikecluster=fferreira/as-cluster-0
time="2019-09-06T11:14:28Z" level=debug msg="checking if pods need to be updated" aerospikecluster=fferreira/as-cluster-0 currentSize=3 desiredSize=3
...processing object..
time="2019-09-06T11:14:30Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-2w4gk" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:14:30Z" level=debug msg="processing object: as-cluster-0-1-as-namespace-0-w7djc" controller=aerospikegarbagecollector
time="2019-09-06T11:14:30Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-nr8tw" controller=aerospikegarbagecollector
time="2019-09-06T11:14:30Z" level=debug msg="processing object: as-cluster-0-2-as-namespace-0-gljzv" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:14:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:14:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-1-as-namespace-0-w7djc
time="2019-09-06T11:14:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-1-as-namespace-0-w7djc'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-nr8tw
time="2019-09-06T11:14:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-nr8tw'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:30Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-cn57t
time="2019-09-06T11:14:30Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-cn57t'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:30Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-bg8jj
time="2019-09-06T11:14:30Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-bg8jj'" controller=aerospikegarbagecollector
time="2019-09-06T11:14:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-2-as-namespace-0-gljzv
time="2019-09-06T11:14:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-2-as-namespace-0-gljzv'" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:15:00Z" level=debug msg="processing object: as-cluster-0-1-as-namespace-0-w7djc" controller=aerospikegarbagecollector
time="2019-09-06T11:15:00Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-nr8tw" controller=aerospikegarbagecollector
time="2019-09-06T11:15:00Z" level=debug msg="processing object: as-cluster-0-2-as-namespace-0-gljzv" controller=aerospikegarbagecollector
time="2019-09-06T11:15:00Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-2w4gk" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:15:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-nr8tw
time="2019-09-06T11:15:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-nr8tw'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:00Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-cn57t
time="2019-09-06T11:15:00Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-cn57t'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:00Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-bg8jj
time="2019-09-06T11:15:00Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-bg8jj'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-2-as-namespace-0-gljzv
time="2019-09-06T11:15:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-2-as-namespace-0-gljzv'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:15:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:00Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-1-as-namespace-0-w7djc
time="2019-09-06T11:15:00Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-1-as-namespace-0-w7djc'" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:15:26Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-2w4gk" controller=aerospikegarbagecollector
time="2019-09-06T11:15:26Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:15:26Z" level=debug msg="no expiration set for pvc" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:15:26Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:26Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
time="2019-09-06T11:15:27Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster

as-cluster-0-0   2/2       Terminating   0         1m

...processing object..
time="2019-09-06T11:15:30Z" level=debug msg="processing object: as-cluster-0-2-as-namespace-0-gljzv" controller=aerospikegarbagecollector
time="2019-09-06T11:15:30Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-2w4gk" controller=aerospikegarbagecollector
time="2019-09-06T11:15:30Z" level=debug msg="processing object: as-cluster-0-1-as-namespace-0-w7djc" controller=aerospikegarbagecollector
...processing object..
time="2019-09-06T11:15:30Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-cn57t
time="2019-09-06T11:15:30Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-cn57t'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:30Z" level=debug msg="checking whether pvc has expired" key=mrunow/offerbiddingserv-as-0-offer-bg8jj
time="2019-09-06T11:15:30Z" level=debug msg="successfully synced 'pvc:mrunow/offerbiddingserv-as-0-offer-bg8jj'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-2-as-namespace-0-gljzv
time="2019-09-06T11:15:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-2-as-namespace-0-gljzv'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:15:30Z" level=debug msg="no expiration set for pvc" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:15:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-1-as-namespace-0-w7djc
time="2019-09-06T11:15:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-1-as-namespace-0-w7djc'" controller=aerospikegarbagecollector
time="2019-09-06T11:15:30Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-nr8tw
time="2019-09-06T11:15:30Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-nr8tw'" controller=aerospikegarbagecollector

E0906 11:15:37.923677       1 generic.go:165] error syncing 'fferreira/as-cluster-0': dial tcp 172.18.10.210:3000: i/o timeout
time="2019-09-06T11:15:37Z" level=info msg="processing cluster" aerospikecluster=fferreira/as-cluster-0
time="2019-09-06T11:15:37Z" level=debug msg="service already exists" aerospikecluster=fferreira/as-cluster-0 service=as-cluster-0
time="2019-09-06T11:15:37Z" level=debug msg="configmap exists and is up to date" aerospikecluster=fferreira/as-cluster-0 configmap=as-cluster-0
time="2019-09-06T11:15:37Z" level=debug msg="networkpolicy already exists" aerospikecluster=fferreira/as-cluster-0
time="2019-09-06T11:15:37Z" level=debug msg="checking if pods need to be updated" aerospikecluster=fferreira/as-cluster-0 currentSize=3 desiredSize=3
time="2019-09-06T11:15:37Z" level=warning msg="pod is in a failure state and will be deleted" aerospikecluster=fferreira/as-cluster-0 pod=fferreira/as-cluster-0-0
time="2019-09-06T11:15:37Z" level=debug msg="processing object: as-cluster-0-0-as-namespace-0-2w4gk" controller=aerospikegarbagecollector
time="2019-09-06T11:15:37Z" level=debug msg="checking whether pvc has expired" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:15:37Z" level=debug msg="no expiration set for pvc" key=fferreira/as-cluster-0-0-as-namespace-0-2w4gk
time="2019-09-06T11:15:37Z" level=debug msg="successfully synced 'pvc:fferreira/as-cluster-0-0-as-namespace-0-2w4gk'" controller=aerospikegarbagecollector

If you need the full log let me know so I can share it in a file.

[pires]

I think the problem may be something else...

(...)

time="2019-09-06T11:14:08Z" level=error msg="failed tip-clear ip on pod \"fferreira/as-cluster-0-2\"" aerospikecluster=as-cluster-0 pod=fferreira/as-cluster-0-0
time="2019-09-06T11:14:08Z" level=error msg="failed alumni-reset on pod \"fferreira/as-cluster-0-2\"" aerospikecluster=as-cluster-0 pod=fferreira/as-cluster-0-0

(...)

time="2019-09-06T11:14:18Z" level=error msg="failed tip-clear ip on pod \"fferreira/as-cluster-0-0\"" aerospikecluster=as-cluster-0 pod=fferreira/as-cluster-0-0
time="2019-09-06T11:14:20Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
time="2019-09-06T11:14:22Z" level=debug msg="processing object: as-cluster-0-0" controller=aerospikecluster
...processing object..
time="2019-09-06T11:14:28Z" level=error msg="failed alumni-reset on pod \"fferreira/as-cluster-0-0\"" aerospikecluster=as-cluster-0 pod=fferreira/as-cluster-0-0
E0906 11:14:28.902207       1 generic.go:165] error syncing 'fferreira/as-cluster-0': detected incorrect cluster size for pod "fferreira/as-cluster-0-0"

(...)

E0906 11:15:37.923677       1 generic.go:165] error syncing 'fferreira/as-cluster-0': dial tcp 172.18.10.210:3000: i/o timeout