No way to handle the only semi-sync replica failing

[kklimonda]

(updated topology screenshot to match the question)

I'm testing the following topology:

Single master that has semi-sync replication enabled, with single semi-sync slave running in another DC. The idea here is to gracefully failover to another DC in case master's DC goes offline.

I have a PostFailover hook that makes sure that, in case of master failover, semi-sync replication is enabled on either replica from another DC (preferably) or replica in the same DC as master and disabled on all other replicas.

This seems to be working fine, however I can't figure out how to handle failure of semi-sync replica - there is no hook executed when replica is lost, and so master stops accepting writes as it's waiting for ACK from its remaining replicas (none of which have semi-sync replication enabled).

Is this somehow related to Semi-sync enforcement? I've already raised similar question on the ML, but it's not obvious from documentation and code that this feature is a correct approach here.

One idea I had was to turn that topology into something similar to that:

In this topology semi-sync replica is also intermediate master, which triggers hooks on failure and lets me reconfigure other replica for semi-sync replication.

[shlomi-noach]

how to handle failure of semi-sync replica - there is no hook executed when replica is lost, and so master stops accepting writes as it's waiting for ACK from its remaining replicas (none of which have semi-sync replication enabled).

If I understand correctly, in the post failover hook you try and set up a new replica as semi-sync, and, since that could fail, you'd wish to have some hook, so you can try another replica?

I think of a different solution. Post failover, you can check with orchestrator which replicas are OK, and then choose one to enable semi-sync. Does that make sense?

Regardless, am I correct to understand you never ever want to have a write on a master unless backed by a replica? How about, for the duration of the failover, you set rpl_semi_sync_master_timeout to a reasonably low value, to allow for some recovery mishaps and delays, and eventually set rpl_semi_sync_master_timeout to a high value?

[kklimonda]

Should PostFailover hook be running not only on master failover, but also on failure of replicas? Or perhaps other hook that is executed in that case?

I'm already using orchestrator API to make sure that semi-sync is configured on master failover, and it would be pretty straightforward to do that same when replica failure is detected, but I'm not sure how to detect that failure in the first place.

[shlomi-noach]

but also on failure of replicas?

there is no failover for replicas, and no hooks will run on such failure. This is beyond orchestrator's current interests.

but I'm not sure how to detect that failure in the first place.

try orchestrator-client -c is-replicating -i some.replica

[kklimonda]

but I'm not sure how to detect that failure in the first place.

try orchestrator-client -c is-replicating -i some.replica

Let me rephrase that - I don't see a way to handle a failure of semi-sync replica with orchestrator. When replica failure occurs, master with semi-sync replication enabled will still be waiting on commits for semi-sync replica to ACK them. Without any hooks being executed I can't do any changes to remediate that.

[shlomi-noach]

I'm not sure we're on the same page. How does my suggestion not solve the problem?
PostFailover, you can iterate the replicas, validate which one is functional, pick one that is functional, and enable semi-sync on that replica.

[kklimonda]

Hmm.. We probably aren't :).

In my original post I've pasted two screenshots: first one with mysql-stg-waw1-1 being master (with semi-sync replication enabled) and a number of replicas being replicated from it, with mysql-stg-waw2-2 having semi-sync replication enabled.

In that scenario, failure of mysql-stg-waw2-2 does not result in any hooks being executed, so I can't make changes to the topology.

My other screenshot is an attempt to workaround that - I turn mysql-stg-waw2-2 into an intermediate master at which point orchestrator will execute various hooks and let me make changes to the topology.

[shlomi-noach]

Right. I guess the confusion was one my side, sorry.
You are correct that orchestrator will no let you know should a semi-sync replica fail. If you have infinite wait on the master, then you will have an outage.

I'm curious, though: why do you only set up a single semi-sync replica? In my last production setup where I used semi-sync, we had multiple replicas running as semi-sync. The master only needs to wait for one, but it doesn't care which one. So if one semi-sync replica fails, there's other to ack the master.

[kklimonda]

I want to use semi-sync replication to achieve two connected goals in case of master failover to another DC:

1. make sure that the new master is up-to-date.
2. make sure that the old master stops processing new commits, lowering a chance of a split-brain.

Enabling semi-sync replication on more than one replica in another datacenter would solve a problem of single replica failing, but not entire DC going down - that will still block my master from accepting writes.

On the other hand if I put another semi-sync replica in the same DC as master, this will break the goals I've listed.

[shlomi-noach]

Understood. So the single semi-sync replica is a single point of failure. Let me think about this further. There's no support for this in orchestrator at this time.

[shlomi-noach]

Some foundational work at #1171 is able to analyze a situation where semi sync is enabled on master but not enough semi-sync replicas are available.

[shlomi-noach]

OK, good news and bad news. The good news is that I believe it should be achievable for orchestrator to treat this dead-single-semi-sync-replica-for-semi-sync-master-which-is-now-blocking as a recovery scenario. The point of view will be the master's -- it is the one suffering, not the replica. It will be a failover for the master, of sorts.

The bad news is that the action taken (orchestrator could be the one to take it, actually), which is to set up a new semi-sync replica -- is risky. In a scenario where the original semi-sync replica went down because of network issue (thus, the server is not dead), we will enable semi-sync on a different replica, and then that original replica suddenly reappears -- and we have two semi-sync replicas. The topology is now susceptible to split brain.

I'm not sure how split-brain is 100% solvable via single-semi-sync-replica setup. At least, without being able to shoot the other node in the head.

Thoughts welcome.

[shlomi-noach]

On further thinking, I guess the "bad news" part is reasonably solvable with careful monitoring/orchestration. Possibly orchestrator can enforce an amount of semi-sync replicas, or external scripting can do that.

Either way, the challenge I'm seeing right now is: exactly when to draw the analysis (and run hooks). It might be reasonable for some stalls from time to time, since that's reality. So I probably don't want to fire a hook 1sec into the stall. But yes to fire it some 20sec into the stall.
The difficulty is with measuring the stall.

With (pt-)heartbeat running, and ReplicationLagQuery configured, I can see how long ago a heartbeat was injected on the master. This is an easy solution -- but depends on (pt-)heartbeat to exist. Looking into more options.