Task proxy spawn on demand. #3515

hjoliver · 2020-03-09T22:01:14Z

Spawn on Demand implementation as per proposal https://cylc.github.io/cylc-admin/proposal-spawn-on-d.html

See #3474 (superseded) for earlier discussion and some proof of efficiency.

partially addresses spawn on demand / event driven graph #3304
close reload and restart - smart insertion of new tasks #774
close Users should not need to know about insertion (or task proxies generally) #2143
close graph: optional branching #3226
(probably closes other issues too...):
close limit task pool size #987
close Submit job immediately on insertion? #2643
close Release, trigger, insert (etc.) a whole sub-graph #1314
close reset-to-waiting when prerequisite tasks have been removed #2654

This branch:

correctly runs the "complex" test workflow for multiple cycles (last I checked, recently)
handles reflow properly as described in the proposal
seems to work for all manner of manual test cases

What's still to do:

restart: need to record upstream output status for each task (for prerequisites) in the DB, not just task status
~~refine the cycle point based housekeeping to account for inter-cycle dependence~~ (no longer needed)
unit and functional tests (many mods needed)
implement reflow stop mechanisms as described in the proposal
implement cylc spawn for specific individual outputs

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).

Appropriate tests are included (unit and/or functional).
Already covered by existing tests.

Appropriate change log entry included.

(master branch) I have opened a documentation Issue at cylc/cylc-doc#132

hjoliver · 2020-03-24T02:53:28Z

Restart now implemented, and tests/restart (non-remote) functional tests working. (Description updated above)

cylc/flow/task_pool.py

hjoliver · 2020-04-17T07:13:51Z

Latest Travis-CI results on this branch:

All unit tests pass
Failed 2 functional tests out of 1867 (in 2 test scripts out of 577)
- ~~graphql/02-root-queries.t~~ DONE
- ~~offset/04-cycle-offset-chain.t~~ DONE
- ~~flakytests/~~ DONE

dwsutherland · 2020-04-22T01:32:49Z

@hjoliver - Got graphql/02-root-queries.t to pass with:

(flow) sutherlander@cortex-vbox:cylc-flow$ git diff
diff --git a/tests/graphql/02-root-queries.t b/tests/graphql/02-root-queries.t
index 02fc49682..386d769bf 100755
--- a/tests/graphql/02-root-queries.t
+++ b/tests/graphql/02-root-queries.t
@@ -144,18 +144,12 @@ cat > expected << __HERE__
         {
             "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}baa"
         },
-        {
-            "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}bar"
-        },
         {
             "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}foo"
         },
         {
             "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}qar"
         },
-        {
-            "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}qaz"
-        },
         {
             "id": "${USER}${ID_DELIM}${SUITE_NAME}${ID_DELIM}20190101T00${ID_DELIM}qux"
         }

Ran it 3 times to check consistency (but haven't run it on a different/faster machine).

tests/hold-release/05-release.t

hjoliver · 2020-04-22T02:16:40Z

Thanks @dwsutherland - if you're happy that's the correct expected result, I'll make your change and leave the test like that for now.

I see my latest change has caused the unit tests to fail 😬 (I know why though, easy to fix though).

hjoliver · 2020-04-22T03:29:41Z

Unit tests fixed, 3 or 4 "absolute offset" tests broken now, as a result of previous change. Damn it, almost there.

hjoliver · 2020-04-23T11:31:36Z

All tests passing 💥

hjoliver · 2020-07-23T03:53:42Z

So I think we just need to consider absolute triggers in that bit of the validation code. I'll fix it on this branch if easy to do...

Done in latest commit. Other issues raised all addressed except for the "absolute triggers + sequential tasks" one. (I'll comment on that soon...)

tests/functional/api-suite-info/00-get-graph-raw-1/suite.rc

tests/functional/api-suite-info/01-get-graph-raw-2/suite.rc

tests/functional/api-suite-info/02-get-graph-raw-3/suite.rc

tests/functional/api-suite-info/03-get-graph-raw-4/suite.rc

Update tests/functional/api-suite-info/01-get-graph-raw-2/suite.rc Update tests/functional/api-suite-info/02-get-graph-raw-3/suite.rc Update tests/functional/api-suite-info/03-get-graph-raw-4/suite.rc Co-authored-by: Melanie Hall <[email protected]>

hjoliver · 2020-07-28T08:00:11Z

@dwsutherland - feedback addressed. Your examples above work, just need to ~~add a test for the improved absolute trigger handling~~ [done]

dwsutherland

@hjoliver - Great, when trying to run the suite:

[scheduling]
    initial cycle point = 20200101T00
    [[special tasks]]
        sequential = foo
    [[dependencies]]
        #R1 = wiz
        P1M = """
#foo[-P1M] => foo
wiz[^] => foo
"""

I get the following:

(flow) sutherlander@cortex-vbox:wiz$ cylc validate wiz
TaskDefError: No cycling sequences defined for wiz

2020-07-29T12:34:40+12:00 INFO - Cold Start 20200101T0000+12
2020-07-29T12:34:40+12:00 INFO - Suite server: url=tcp://cortex-vbox:43093/ pid=10184
2020-07-29T12:34:40+12:00 INFO - Suite publisher: url=tcp://cortex-vbox:43064
2020-07-29T12:34:40+12:00 INFO - Run: (re)start=0 log=1
2020-07-29T12:34:40+12:00 INFO - Cylc version: 8.0a2
2020-07-29T12:34:40+12:00 INFO - Run mode: live
2020-07-29T12:34:40+12:00 INFO - Initial point: 20200101T0000+12
2020-07-29T12:34:40+12:00 INFO - Final point: None
2020-07-29T12:34:40+12:00 INFO - Suite shutting down - AUTOMATIC
2020-07-29T12:34:41+12:00 INFO - [client-command] identify sutherlander@cortex-vbox:cylc-uiserver
2020-07-29T12:34:41+12:00 INFO - DONE

(branch needs de-conflicted again)

Nice work! 🥇

kinow · 2020-07-29T01:07:07Z

Impressive work @hjoliver !!! 🍾 🍻

TomekTrzeciak · 2020-07-29T07:07:26Z

This one's big! I'm really happy to see this happening already for Cylc 8 🎉 🍾 🚀

arjclark · 2020-07-29T07:53:03Z

cylc/flow/graphnode.py

+            (name, offset, output,
+            offset_is_from_icp, offset_is_irregular, offset_is_absolute)
+
+        TODO: offset_is_from_icp is '^' or None - should be boolean?


@hjoliver - does this need addressing still?

Good spotting. The variable name suggests we thought the value is boolean. It's actually of no consequence as (I checked) the variable is only used in if tests (where '^' vs None behaves just like True vs `False). However, I'll follow-up with a small change to make the code less obtuse.

hjoliver added efficiency For notable efficiency improvements WIP POC Proof of Concept labels Mar 9, 2020

hjoliver added this to the cylc-8.0.0 milestone Mar 9, 2020

hjoliver self-assigned this Mar 9, 2020

hjoliver mentioned this pull request Mar 9, 2020

Spawn-on-demand implementation #3474

Closed

7 tasks

hjoliver force-pushed the spawn-on-demand branch 2 times, most recently from 05ca046 to 5d134a5 Compare March 16, 2020 06:30

hjoliver force-pushed the spawn-on-demand branch 3 times, most recently from 1add368 to 98c7ec3 Compare April 8, 2020 00:07

hjoliver force-pushed the spawn-on-demand branch 2 times, most recently from 9150a0d to 92c293e Compare April 16, 2020 12:56

kinow reviewed Apr 17, 2020

View reviewed changes

cylc/flow/task_pool.py Outdated Show resolved Hide resolved

hjoliver force-pushed the spawn-on-demand branch 2 times, most recently from ee7fb4c to 578107b Compare April 21, 2020 10:51

kinow reviewed Apr 22, 2020

View reviewed changes

tests/hold-release/05-release.t Outdated Show resolved Hide resolved

hjoliver force-pushed the spawn-on-demand branch from 3e84f45 to 873b537 Compare April 22, 2020 02:42

hjoliver force-pushed the spawn-on-demand branch 2 times, most recently from 9bf6097 to 7535e12 Compare April 23, 2020 08:10

hjoliver force-pushed the spawn-on-demand branch from 7535e12 to db3e0b1 Compare April 30, 2020 07:07

hjoliver removed POC Proof of Concept WIP labels Apr 30, 2020

hjoliver marked this pull request as ready for review April 30, 2020 07:09

hjoliver added 2 commits July 25, 2020 01:15

Event-driven suicide triggering.

aa49848

No implicit task creation by suicide triggers.

c7ffec1

datamel reviewed Jul 24, 2020

View reviewed changes

tests/functional/api-suite-info/00-get-graph-raw-1/suite.rc Outdated Show resolved Hide resolved

datamel reviewed Jul 24, 2020

View reviewed changes

tests/functional/api-suite-info/01-get-graph-raw-2/suite.rc Outdated Show resolved Hide resolved

datamel reviewed Jul 24, 2020

View reviewed changes

tests/functional/api-suite-info/02-get-graph-raw-3/suite.rc Outdated Show resolved Hide resolved

datamel reviewed Jul 24, 2020

View reviewed changes

tests/functional/api-suite-info/03-get-graph-raw-4/suite.rc Outdated Show resolved Hide resolved

hjoliver and others added 4 commits July 28, 2020 10:27

Fix handling of abs triggers.

727588a

Style fixes.

66df7fb

Update DB schema test.

e24780f

hjoliver force-pushed the spawn-on-demand branch from 04e1b61 to e24780f Compare July 28, 2020 05:21

hjoliver added 2 commits July 28, 2020 20:00

Remove debug print.

de855b0

New SoD test for abs triggers.

48441ab

dwsutherland approved these changes Jul 29, 2020

View reviewed changes

hjoliver merged commit e0e356a into cylc:master Jul 29, 2020

hjoliver mentioned this pull request Jul 29, 2020

spawn on demand / event driven graph #3304

Open

arjclark reviewed Jul 29, 2020

View reviewed changes

hjoliver mentioned this pull request Jul 29, 2020

Fix ICP offset var type. #3719

Merged

7 tasks

hjoliver modified the milestones: cylc-8.0a3, cylc-8.0b0 Feb 25, 2021

This was referenced Aug 4, 2021

Submit job immediately on insertion? #2643

Closed

Release, trigger, insert (etc.) a whole sub-graph #1314

Closed

reset-to-waiting when prerequisite tasks have been removed #2654

Closed

MetRonnie added the db change Change to the workflow database structure label Jan 21, 2022

oliver-sanders mentioned this pull request Jan 24, 2023

cache Tokens methods #5183

Open

hjoliver deleted the spawn-on-demand branch July 24, 2024 00:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task proxy spawn on demand. #3515

Task proxy spawn on demand. #3515

hjoliver commented Mar 9, 2020 •

edited

Loading

hjoliver commented Mar 24, 2020

hjoliver commented Apr 17, 2020 •

edited

Loading

dwsutherland commented Apr 22, 2020

hjoliver commented Apr 22, 2020

hjoliver commented Apr 22, 2020

hjoliver commented Apr 23, 2020

hjoliver commented Jul 23, 2020

hjoliver commented Jul 28, 2020 •

edited

Loading

dwsutherland left a comment

kinow commented Jul 29, 2020

TomekTrzeciak commented Jul 29, 2020

arjclark Jul 29, 2020

hjoliver Jul 29, 2020

Task proxy spawn on demand. #3515

Task proxy spawn on demand. #3515

Conversation

hjoliver commented Mar 9, 2020 • edited Loading

hjoliver commented Mar 24, 2020

hjoliver commented Apr 17, 2020 • edited Loading

dwsutherland commented Apr 22, 2020

hjoliver commented Apr 22, 2020

hjoliver commented Apr 22, 2020

hjoliver commented Apr 23, 2020

hjoliver commented Jul 23, 2020

hjoliver commented Jul 28, 2020 • edited Loading

dwsutherland left a comment

Choose a reason for hiding this comment

kinow commented Jul 29, 2020

TomekTrzeciak commented Jul 29, 2020

arjclark Jul 29, 2020

Choose a reason for hiding this comment

hjoliver Jul 29, 2020

Choose a reason for hiding this comment

hjoliver commented Mar 9, 2020 •

edited

Loading

hjoliver commented Apr 17, 2020 •

edited

Loading

hjoliver commented Jul 28, 2020 •

edited

Loading