Checkpointing With SimpleConsumer and OffsetCommitContext

[levetm-washpost]

I was working with this sample code: http://kafka-python.readthedocs.org/en/latest/apidoc/kafka.html?highlight=offsetcommitcontext#kafka.context.OffsetCommitContext (adding in consumer.commit() call every 100 marks). I noticed that when I would terminate and restart my Python script, it would read from the beginning of the log file rather than from the last marked offset.

Is there a way to handle checkpointing in the kafka-python library, so if my program unexpectedly terminates, it can begin reading from the last marked offset in the Kafka log?

[dpkp]

Thanks for filing! I actually have never looked at OffsetCommitContext ... not sure if that works as advertised and pretty sure we have no tests for it. needs investigation.

[levetm-washpost]

It seems to be checkpointing, but during the running instance of the program. So I could roll back during a run of my application. It doesn't seem to (as far as I can tell) support picking up close to where I left off if my program terminates and is restarted.

[jeffwidman]

Closing this as "won't fix", since the SimpleConsumer is deprecated and will be removed in #1193 as part of the 2.x release series.