A triggered task can block a flow.

[hjoliver]

A task triggered manually without --reflow has no flow numbers and does not spawn children.

If a task from an ongoing flow catches up to such a task (while it is still running), the two will merge. The merged task will have the ongoing flow number(s) and will spawn children on any subsequent outputs, but:

• any already-completed outputs won't have spawned children
• this effectively blocks the flow from already-completed outputs, potentially causing premature shutdown if those are the only spawning outputs

This is pretty niche; I can't imagine why a user would want to force-trigger a task that was just about to be spawned by the main flow anyway. Still, it's a bug because the main flow would be expected to keep going regardless of a single triggered task out in front of it.

Example:

[scheduling]
    [[graph]]
        R1 = """  # OK to force-trigger bar while foo is still running
           foo => bar => baz
        """
[runtime]
    [[foo, bar, baz]]
        script = sleep 10

If I trigger bar while foo is still running, when foo succeeds it will try to spawn bar and find that it already exists, so the scheduler will merge foo's flow number(s) into bar. So by the time bar finishes it belongs to the flow and will spawn baz as normal.

However, if baz triggers off of an earlier output of bar then under the same scenario the scheduler will shut down without running baz (because baz did not belong to a flow at the time it completed that output):

        R1 = """  # BUG: NOT OK to force-trigger bar while foo is still running
           foo => bar
           bar:start => baz
        """

[hjoliver]

Similarly for parentless tasks, which auto-spawn their own next instance when released from runahead (example from #4640 (comment)):

[scheduling]
    cycling mode = integer
    runahead limit = P1
    [[graph]]  # BUG: NOT OK to trigger 3/foo while 1/foo is still running
        P1 = foo
[runtime]
    [[foo]]
        script = sleep 10

Here, if I force-trigger 3/foo (without --reflow) while 1/foo is still running (and 2/foo exists but is runahead-limited) the scheduler will shut down once all 3 tasks have finished ... because 3/foo, not belonging to a flow, does not spawn 4/foo when it (3/foo) is released from runahead. When 2/foo tries to spawn 3/foo the latter picks up the former's flow number, but too late to continue the flow because the spawning point has already passed.

[hjoliver]

(Note it causes no problem to trigger a future task that finishes before the main flow catches up).

[hjoliver]

To kill both birds with one stone:

[scheduling]
    cycling mode = integer
    runahead limit = P1
    [[graph]] 
        P1 = foo:start => bar
[runtime]
    [[foo]]
        script = sleep 10

Force-triggering 3/foo while 1/foo is still running should result in:

• scheduler does not shut down prematurely
• 3/bar (child of the force-triggered task) spawned with prerequisite satisfied
• so ... workflow continues as normal

I'll base a test on this example.