Prove syncronisation under the following condition:
-
Keep all in sync and then shut
blue
down and add some records togreen
. Startblue
and execute full sync fromgreen
toblue
. Verify that all records are available inblue
. No records deleted ingreen
.- Start both cluster using class
BlueCluster
andGreenCluster
and MC usinghz-mc start
. Make sure blue and green clusters are available with WAN setup. - Add CLC config
clc config add blue cluster.name=blue cluster.address=localhost:5701
andclc config add green cluster.name=green cluster.address=localhost:5702
- Using CLC add some records to
blue
andgreen
clc -c blue -n replicatedMap map set key1 value1 clc -c blue -n replicatedMap map get key1 clc -c green -n replicatedMap map get key1 clc -c green -n replicatedMap map set key2 value2 clc -c green -n replicatedMap map get key2 clc -c blue -n replicatedMap map get key2
- Now shutdown
blue
cluster and add some records togreen
clc -c green -n replicatedMap map set key3 value3 clc -c green -n replicatedMap map get key3
- Start
blue
cluster and check if all records are available inblue
Results will show thatclc -c blue -n replicatedMap map entry-set
blue
cluster is missingkey1
andkey2
records but haskey3
. This happened becuase the WAN queue ingreen
cluster was holdingkey3
record and whenblue
cluster was unavailale butkey1
andkey2
records were not queued. - Sync from
green
toblue
cluster using MC. Go togreen
cluster in MC -> WAN Replication -> Sync -> and pick the wan rep, EP and Maps and click onSync
button. - Now test if all records are available in
blue
clusterResults will show that all records are available inclc -c blue -n replicatedMap map entry-set
blue
cluster.
- Start both cluster using class
-
Simulate network outage for a prolonged period by breaking the network between
blue
andgreen
cluster. With a WAN queue size of 3, messages start dropping. Once network is restored, manually sync fromgreen
toblue
cluster. Verify that all records are available inblue
cluster.- Start both cluster using class
BlueCluster
andGreenCluster
and MC usinghz-mc start
. Make sure blue and green clusters are available with WAN setup. - If not done, Add CLC config
clc config add blue cluster.name=blue cluster.address=localhost:5701
andclc config add green cluster.name=green cluster.address=localhost:5702
- Using CLC add some records to
blue
andgreen
clc -c green -n replicatedMap map set key1 value1 clc -c green -n replicatedMap map get key1 clc -c blue -n replicatedMap map get key1
- Now simulate network outage by blocking the network between
blue
andgreen
cluster by copying the following command in terminal. CAUTION I use pfctl to block the network but you can use other mechanism. This can mess up your system network settings. Use with caution. CAUTIONsudo cp com.hazelcast.5701.conf /etc/pf.anchors sudo pfctl -ef /etc/pf.anchors/com.hazelcast.5701.conf
- Add some records to
green
clusterQueryingclc -c green -n replicatedMap map set key2 value2 clc -c green -n replicatedMap map set key3 value3 clc -c green -n replicatedMap map set key4 value4 clc -c green -n replicatedMap map set key5 value5 clc -c green -n replicatedMap map set key6 value6 clc -c green -n replicatedMap map remove key1 clc -c green -n replicatedMap map entry-set
blue
cluster will result in timeout or error. Ctrl+C to stop the query.clc -c blue -n replicatedMap map entry-set
- Remove the rule
sudo rm /etc/pf.anchors/com.hazelcast.5701.conf sudo pfctl -F all -f /etc/pf.conf
- Now if you query
blue
cluster, you will see thatkey5
andkey6
are unavailable due to the fact thatGreenCluster
is created with a WAN queue size of 3 we created 4 entries ingreen
cluster.clc -c blue -n replicatedMap map entry-set
- Sync from
green
toblue
cluster using MC. Go togreen
cluster in MC -> WAN Replication -> Sync -> and pick the wan rep, EP and Maps and click onSync
button. - Issue: Note that
key1
exists inblue
cluster but not ingreen
cluster. This is because thekey1
was deleted ingreen
cluster but as replication queue was overflowing, the delete operation was dropped. A full sync does not fix this but can delta fix this? Lets check.
- Start both cluster using class
While enabling Merkle help reduce the amount of data that is transferred but it does not take care of entries which are deleted in source cluster. Once Merkle tree comparision are run and it is observed that there are entries in target cluster which are not there in source cluster, then such a target cluster needs to be restarted and a full manual sync need to be done from source cluster. Merkle tree setup is useful and must be done with care: https://docs.hazelcast.com/hazelcast/5.4/wan/advanced-features#delta-wan-synchronization MC uses Merkle tree if it is configured for the Map but if number of Maps to be repliacted are large it is best to sync Maps individually or in batches. Both MC or Rest API can be used to sync Maps.
Demonstrate client failover by shutting off one cluster and observing the switch over. Use classes Cluster
and Client
.