Enhancement: Allow watch from "first available index" vs. new updates or updates from a valid start index

[raff]

Currently watch requests only allow watchIndex=0 (new updates) or watchIndex >= start index (start from watchIndex). I would like to be able to get all available updates without having to specify a valid watchIndex.

Something like watchIndex = -1 means "start from first available" or an extra flag to allow watchIndex < startIndex (again, it would start from first available).

With the current implementation I need to parse the error message ("the requested history has been cleared [%v/%v]") and resubmit the request with the current startIndex, but if more updates are published while doing this I may never get to fetch any valid value.

I am happy to submit a pull request if we can agree on the interface (watchIndex < 0 or new flag)

[jonboulle]

I am not entirely sure I understand your use case exactly. What if you simply do watch with stream=true and do not specify a waitIndex - does that achieve what you're after?

[jonboulle]

On rereading, it sounds like you mean you want the "first index in the currently available history", but I'm unclear on what the use case for that would be.

[raff]

Yes, I want to basically be able to fetch the full history from whatever point it is.

The use case is slightly odd. I am thinking of using etcd to add clustering to a simple pubsub system I have, by having my current publishers send message to etcd and having multiple pubsub services watch the message "queue" (the message queue in this case would be a single "key" or multiple keys if I want to support sharding).

When everything is working fine the pubsub services should be able to keep up with updates to the "queue" and process them, but if one of the pubsub services is restarted I would like to be able to process as many messages I can (and I know that I may be missing some, but it's fine in my particular use case). But given that etcd does some "buffering" I would like to be able to process the buffered messages.

[raff]

Just to be clear, the current documentation for watch says "Using the index, we can watch for commands that have happened in the past. This is useful for ensuring you don't miss events between watch commands."

If a client waits "too much" between calls to watch currently it gets an error and the options are:

• start from new messages (completely discarding the history - so it will miss a lot of events)
• try to keep up by issuing new watch commands with an updated index, with the risk of never been able to get any new message (if a writer is updating the key and a similar speed as the reader trying to issue the watch, the reader will keep getting errors and miss all messages)

My proposal is to add an option to say "I am ok loosing some messages, just give me all that is available after this index". I know I'll be losing some, but I know how many messages I have lost (and can decide to abort or continue from there)

[philips]

This is a reasonable feature and I think it should be straightforward to implement. But, I don't think it will make it into the 0.5.0 release so I will put it on 0.6.0-maybe. If you want to submit a PR that would be great.

[raff]

Great! Thanks!

-- Raffaele

On Nov 18, 2014, at 9:10 PM, Brandon Philips [email protected] wrote:

This is a reasonable feature and I think it should be straightforward to implement. But, I don't think it will make it into the 0.5.0 release so I will put it on 0.6.0-maybe. If you want to submit a PR that would be great.

—
Reply to this email directly or view it on GitHub.

[heyitsanthony]

#4495 can do this in v3.

• There's no longer a need to parse an error message; the earliest available revision is given as a WatchResponse in case the revision is too old. To get the earliest revision, Watch on rev=1, and check for a CompactRevision on the first message.
• The missed updates problem shouldn't be an issue now that there's better handling of revision history.