named locks

[BrianHicks]

A named lock would be nice for doing things that affect the whole system, but which Converge currently does in parallel. (Like package management using apk on Alpine)

[BrianHicks]

Maybe syntax:

task "x" {
  check = "..."
  apply = "..."
  lock = ["name1", "name2"]
}

Or should the lock be a global node of some kind?

[ryane]

for certain tasks (package installs like you mentioned), should it be implicit based on the resource type? we might be able to also speed things up by batching into a single transaction.

I kind of asked about this over on the wiki: https://github.com/asteris-llc/converge/wiki/Sample-Elasticsearch-Filebeat-Kibana. Repeating here:

# yum / rpm shell tasks are vulnerable to conflicts (due to yum locks). You will
# see a lot of this in the output:

# ...
# Existing lock /var/run/yum.pid: another copy is running as pid 11771.
# Another app is currently holding the yum lock; waiting for it to exit...
# ...

# yum is smart enough to wait for each individual install to complete and
# converge (so far) eventually is able to install all packages without
# erroring out. But, when we build a package module, we'll probably want to
# handle this so it is more efficient (batch up yum installs in a single
# transaction?). should this (module-specific implicit dependencies/batching) be
# part of the core graph engine?

[BrianHicks]

I think yes, in the long term we can batch things like that. Package
management especially is prone to some rather simple graph
transformations to make things much nicer at execution time. I’m
suggesting locks as a solution for tasks, our “escape hatch” (but
both might work!)

On 13 Sep 2016, at 18:33, Ryan Eschinger wrote:

for certain tasks (package installs like you mentioned), should it be
implicit based on the resource type? we might be able to also speed
things up by batching into a single transaction.

I kind of asked about this over on the wiki:
https://github.com/asteris-llc/converge/wiki/Sample-Elasticsearch-Filebeat-Kibana.
Repeating here:

# yum / rpm shell tasks are vulnerable to conflicts (due to yum 
locks). You will
# see a lot of this in the output:

# ...
# Existing lock /var/run/yum.pid: another copy is running as pid 
11771.
# Another app is currently holding the yum lock; waiting for it to 
exit...
# ...

# yum is smart enough to wait for each individual install to complete 
and
# converge (so far) eventually is able to install all packages without
# erroring out. But, when we build a package module, we'll probably 
want to
# handle this so it is more efficient (batch up yum installs in a 
single
# transaction?). should this (module-specific implicit 
dependencies/batching) be
# part of the core graph engine?

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#249 (comment)

[BrianHicks]

Let's let the implementor worry about exactly how to implement this and change the issues tags around to reflect it's ready. I'd like whoever does this to prefer to store the data in the graph somehow (generated dependencies) if possible. Otherwise, a named mutex probably will work just fine.

[sehqlr]

Could we have the lock be a resource type?

lock "yum" {
  // config on lock
}

task "install nginx" {
  check = ".."
  apply = ".."
  depends = ["lock.yum"]
}

[BrianHicks]

It would probably have to be inside the task, since the goal is to synchronize across multiple modules.

[ryane]

now known as "group" - see #392