Event vs Command race when used as library

[eikenb]

Consul Template version

v0.24.1

Configuration

template {
  contents = "hi "
  destination = "hi.out"
  command = "sh -c 'sleep 3; cat hi.out;'"
  command_timeout = "0s"
}

template {
  contents = "bye\n"
  destination = "bye.out"
  command = "sh -c 'sleep 5; cat bye.out;'"
  command_timeout = "0s"
}

Command

consul-template -config config.hcl  -once

Expected behavior

Templates rendered and commands run before consul-template exits.

Actual behavior

Templates rendered and commands run after consul-template exists.

Details

So the above shows a version of the problem. If you have 0s timeout on a command, consul-template w/ -once will exit before they get run. In cases where consul-template is run as a batch job in a container, the container could well exit before the commands finish (killing them). This is pretty easy to mitigate with the consul-template command by adding a timeout > 0s.

But when using CT as a library, the "standard" way to monitor and know when to exit is to watch for messages on the TemplateRenderedCh(). The problem is that the message is sent down that channel before the commands are run, so even with a timeout > 0s the message has been sent and the program could exit.

The simplest solution is to move the code that sends the message from its current location, right before running the commands, to just after running the commands.

From here...

consul-template/manager/runner.go

Lines 559 to 574 in dc274bc

	// Check if we need to deliver any rendered signals
	if wouldRenderAny || renderedAny {
	// Send the signal that a template got rendered
	select {
	case r.renderedCh <- struct{}{}:
	default:
	}
	}
	// Check if we need to deliver any event signals
	if newRenderEvent {
	select {
	case r.renderEventCh <- struct{}{}:
	default:
	}
	}

To here...

consul-template/manager/runner.go

Line 603 in dc274bc

[eikenb]

The related PR #1370 makes it so that when CT is used as a library, setting the command_timeout to > 0s will block the sending of the render event until after the command is complete. This brings the library behavior in line with the command line version.

It does not directly change the fact that consul-template will exit before the sub-commands are finished if you do set the command_timeout to 0s. As this is not the default behavior I don't think I'm going to change it right now and might just update the documentation to call it out. While I doubt it is in use or that useful, changing it isn't needed to address the CT library issue which was the real driving factor behind this ticket and fixing it would be a much larger change.