[Question] serf initial PushPull messages delivered out of order

[sanimej]

In Docker Overlay network we use serf to exchange the endpoint IP/mac across nodes. On restarting the Docker daemon on a node what I am seeing is that the user events from the initial TCP pushpull sync are delivered out of order. Its verifiable easily with theLTime from serf.UserEvent

As per this comment the client should get the events from EventCh in the right order
https://github.com/hashicorp/serf/blob/master/serf/config.go#L38

Is it guaranteed that the events are delivered in order and it could be bug here or are there factors that can introduce out of order delivery to the client (FWIW, we use a buffered channel for the events)

Since the interface up/down events are order sensitive its bit of a mess when we get the events in incorrect order.

[slackpad]

Hi @sanimej I think it is possible that they end up out of order during a push-pull. The storage of events into the buffer is here:

https://github.com/hashicorp/serf/blob/master/serf/serf.go#L1131-L1144

That idx calculation ends up working kind of like a ring buffer, so you might end up with some newer events at the front if things wrap around. I think to fix this you'd want to change the code here:

https://github.com/hashicorp/serf/blob/master/serf/delegate.go#L262-L274

If you sort by LTime it should process them in the proper temporal order, not in the order they happened to be in the buffer.