Add Consul-based locking

[alexanderdean]

This lets us run specific jobs as singletons, to prevent overlapping runs.

SQL Runner has an implementation of this that we can borrow.

[BenFradet]

Am I right in assuming this and #20 wouldn't work with --async?

[alexanderdean]

I think we support Consul-locking with --async, but just accept that the user will have to deal with deleting the lock manually (just as they'll have to deal with monitoring the job manually).

I agree it's probably unusual to use both together, but I think let's not introduce the code complexity of banning this.

[BenFradet]

I have given more thoughts to this / started implementing it and I feel weird acquiring a lock never to release it.

I'd rather just not lock in case of async and emit a warning or something like that.

[BenFradet]

@alexanderdean

Also since, there are no names associated with the playbook, we can either lock based on:

1. cluster id + combination of the step names
2. add a name field to the playbook config (which will have nothing to do with EMR) and do cluster id + playbook name

What do you think? I personally like 2 best.

[alexanderdean]

Fine by me on the --async - I suggest we warn or error out, up to you.

On the lock name - what does SQL Runner do?

[BenFradet]

AFAICT, @jbeemster might able to weigh in here, SQL runner uses the path provided by the command line arg -lock.

I was thinking of abstracting this away from the user without any lock-related command line args.

[alexanderdean]

I was thinking of abstracting this away from the user without any lock-related command line args.

I don't think that gives enough control. For example, one user might want to enforce a lock based on just the job name, while another company might want to enforce a single lock across two different jobs which both target the same database (risking a race condition if they run at the same time).

I'd go with @jbeemster's design - it's simple and flexible...