[Zen2] Fix test failures in diff-based publishing

[DaveCTurner]

testIncompatibleDiffResendsFullState sometimes makes a 2-node cluster and
then partitions one of the nodes from the leader, which makes the leader stand
down. Then when the partition is removed the cluster re-forms but does so by
sending full cluster states, not diffs, causing the test to fail.

Additionally testDiffBasedPublishing sometimes fails if a publication is
delivered out-of-order, wiping out a fresher last-received cluster state with a
less-fresh one. This change adds a freshness check to avoid this.

[elasticmachine]

Pinging @elastic/es-distributed

[ywelsch]

I would like to avoid throwing a CoordinationStateRejectedException in PublicationTransportHandler. The validation logic should be left to CoordinationState. We can and should improve the logic on when to cache the incoming cluster state though, to avoid old cluster states from poisoning the cache. I wonder if the simplest way to do this would be to only store the new state in lastSeenClusterState after it passes the call to handlePublishRequest.apply. WDYT?

[DaveCTurner]

I wonder if the simplest way to do this would be to only store the new state in lastSeenClusterState after it passes the call to handlePublishRequest.apply. WDYT?

D'oh of course that makes much more sense. Somehow I missed that we were passing it to the coordinator here. I've done that instead.

[ywelsch]

LGTM