More log streamer cleanups

[DrJosh9000]

What

• Use the WaitGroup to count workers instead of pending log uploads, averting any potential races on the WaitGroup
• Signal to workers that work is complete by closing the channel, instead of sending nil several times
• Handle context cancellation in worker with a select
• Add context arg to Process, and a select to handle it there, too
• Track whether the streamer has been stopped, and use that to return an error from Process after Stop
• Reinstate error handling inside streamJobLogsAfterProcessStart and output buffer closing (with explanatory comment)
• Neaten up core of Process loop
• Doc comments
• A test!

Why

Handling context cancellation in a select with the channel operations is, strictly speaking, a bug fix (context should be available as a lever to cancel blocked operations), but practically isn't used here.

Returning an error from Process after Stop is a behaviour change that is similarly theoretical (if the job is always stopped, and Process is never called again before Stop). But it is an improvement: before, if Stop is called then there is the possibility that, if all the workers are stopped and Process is called again, the chunk is left in the channel until the program exits, or potentially the queue fills up and the goroutine managing the streamer blocks forever. Now, Process at least returns an error.

This also means that we can be more explicit about closing the queue and then actually wait for for the workers to finish, rather than waiting for current work to finish before yeeting nil several times into the (buffered) queue channel and not actually waiting them to shut down (this always seemed weird and backwards to me).

Notes

Note that this does not enable log limit enforcement (yet).

[triarius]

Very Nice!