More diagrams (to summarise or replace text content)

[sadielbartholomew]

Perhaps stating the obvious, diagrams are much clearer & intuitive for explaining certain concepts than chunks of text, & they are also great for summarising content. #29 is a lovely example.

There are a lot of great schematics in the docs already, but the diagram to text ratio is still too low for my liking, & there are certain topics I read through & feel are crying out for a summarising, or substitute, diagram.

Bonus points if diagrams can be practicably auto-generated for content that can be & that is likely to change frequently, or at least is editable on an element-level so that PRs would not need to replace a whole image, where changes would be unclear.

[sadielbartholomew]

Here's one I have scribbled down already (from working through the docs for my own understanding). Scribbled is the operative word, sorry, since I didn't want to spend a lot of time creating this electronically until I had could get some feedback, then I added some lines post-scan). It's a diagram to convey task states, how they can be categorised, & the flow of task state transitions, & state "endpoints":

If anyone has time, & thoughts, I would appreciate any comments on anything I have perhaps missed or got incorrect, or how it could be structured more clearly. Otherwise I could get it drawn up using some diagram software.

[kinow]

Ooohh. Thanks a ton for that Sadie! Can't say for certain whether it is correct or not, but it is definitely easier for me to understand the difference that's been discussed on jobs/tasks.

[dwsutherland]

In the job pool (store of job data elements); the creation of an element happens just before job submission, so I added the "ready" state to them..

@sadielbartholomew or others - Thoughts on this?

[sadielbartholomew]

Can't say for certain whether it is correct or not, but it is definitely easier for me to understand the difference that's been discussed on jobs/tasks.

Actually, that is the aspect I am most unsure about 😬 (currently I am just referring to job-submission-related task states in a separate row, & not much more) so please don't take it as truth! This was mainly based on the descriptions in the documentation, & I still need to look at the codebase to determine whether it is correct &/or if there are connections or aspects missed.

As a side note, it would be nice to learn about these aspects together rather than in isolation, so please let me know (via Riot, perhaps) if you ever would like to do some form(s) of joint "learn more about Cylc (or Rose, etc.)" exercise! I'm not sure quite how it would work, but @wxtim I believe used to be a teacher, so he may have some ideas (& may want to join in)?

[sadielbartholomew]

Huge thanks to @matthewrmshin who has just gone over the draft with me & explained on aspects that aren't quite right, a lot of interesting subtleties, & how the task & job distinction influences it. I'll put up another draft to reflect this at some point soon.

[sadielbartholomew]

In the job pool (store of job data elements); the creation of an element happens just before job submission, so I added the "ready" state to them..

Ah, interesting. Thanks for that note @dwsutherland. I do not know at the moment if there are any consequences to consider for that, & also I don't have the most experience to know as such, so if you don't mind I will make this an open question to others in Cylc Core by linking your comment in on an issue regarding the data provision.

[sadielbartholomew]

I've quickly re-drawn (electronically now) the diagram of task states & possible transitions between them & endpoints, not including the rare manual reset from any state to any other, or manual triggering, but notes or alternative (perhaps dashed) arrows could be added to indicate these:

I used the software mermaid to generate the diagram from a source that is easily editable, using the following code (you can generate the diagram, & edit it from the code, via the live editor here):

graph LR

subgraph "Not ready to run"
A(WAITING)
B[QUEUED]
C[RUNAHEAD]
D[HELD]
end

E[READY]
G[RUNNING]
H[RETRYING]
I[SUBMIT-RETRYING]
F[SUBMITTED]

subgraph "Final states"
J(SUCCEEDED)
K(FAILED)
L(SUBMIT-FAILED)
M(EXPIRED)
end

A-->B
A-->D
A-->E
A-->M
B-->E
B-->M
C-->A
C-->M
D-->E
D-->M
E-->F
E-->I
E-->L
E-->M
F-->G
F-->I
F-->L
G-->H
G-->I
G-->J
G-->K
G-->L
H-->E
H-->J
H-->K
I-->E
I-->L

(Eventually, it might be nice to add brief labels to indicate why a task would transition from one to another, i.e. for each arrow, but it would be a challenge to do that without making the diagram too dense).

Anyway, from recent thoughts culminating (at least partly) in cylc/cylc-admin#47, the task states & meanings will probably change, so there is not much point adding this diagram into the documentation now as it will become outdated quickly. But, I do think a diagram such as, or similar to, this could help to guide & describe thoughts on the future of task states & meanings. @hjoliver, would you like me to convert your thoughts from #47 into an equivalent diagram, this time with labelled arrows, to summarise the new proposal?

[kinow]

I do think a diagram such as, or similar to, this could help to guide & describe thoughts on the future of task states & meanings.

+1, and once reviewed/approved, I'm printing it and attaching next to the design document and the vue component lifecycle diagrams in my cubicle.

[hjoliver]

@hjoliver, would you like me to convert your thoughts from #47 into an equivalent diagram, this time with labelled arrows, to summarise the new proposal?

@sadielbartholomew - nice diagram. Let's just park this (only briefly) until #47 is settled, because it may be impacted by cylc/cylc-ui#201 as well.

[wxtim]

Thank you Sadie - I found this really helpful - Like Bruno I think that this might get printed a stuck up above my desk.

How about the following?

graph LR

subgraph "Not ready to run"
A(WAITING)
B[QUEUED]
C[RUNAHEAD]
D[HELD]
end

E[READY]
G[RUNNING]
H[RETRYING]
I[SUBMIT-RETRYING]
F[SUBMITTED]

subgraph "Final states"
J(SUCCEEDED)
K(FAILED)
L(SUBMIT-FAILED)
M(EXPIRED)
end


A-->B
A-->D
A-->E
A-->M
B-->E
B-->M
C-->A
C-->M
D-->E
D-->M
E-->F
E-->I
E-->L
E-->M
F-->G
F-->I
F-->L
G-->H
G-->I
G-->J
G-->K
G-->L
H-->E
H-->J
H-->K
I-->E
I-->L

linkStyle 10 stroke:#0a0,stroke-width:3px;
linkStyle 11 stroke:#fa0,stroke-width:2px;
linkStyle 12 stroke:#f00,stroke-width:2px;
linkStyle 14 stroke:#0A0,stroke-width:3px;
linkStyle 15 stroke:#fa0,stroke-width:2px;
linkStyle 16 stroke:#f00,stroke-width:2px;
linkStyle 17 stroke:#fa0,stroke-width:2px;
linkStyle 18 stroke:#fa0,stroke-width:2px;
linkStyle 19 stroke:#0a0,stroke-width:3px;
linkStyle 20 stroke:#f00,stroke-width:2px;
linkStyle 21 stroke:#f00,stroke-width:2px;
linkStyle 22 stroke:#fa0,stroke-width:2px,stroke-dasharray: 8, 4;
linkStyle 23 stroke:#0a0,stroke-width:3px;
linkStyle 24 stroke:#f00,stroke-width:2px;
linkStyle 25 stroke:#fa0,stroke-width:2px,stroke-dasharray: 8, 4;
linkStyle 26 stroke:#f00,stroke-width:2px;

[oliver-sanders]

Some corrections:

• Held can actually apply to most task states and there is complicated retry/held interaction so there are a lot more arrows from/to held BUT ...
• At Cylc8 Held is no longer a task state so can be removed from the diagram all-together.
• We may do something similar with the retry states too which would simplify things further.
• Tasks briefly pass through the Queued state before running* so I think:
  • The Waiting - Ready arrow should be removed
  • The Queued state should be removed from the "not ready to run" box (as they are definitely ready to run)
  • (*) Normally you don't notice tasks passing through the Queued state but if the user has configured internal Cylc queues then tasks get stuck in the Queued state for a while.

There are also user requested state changes which are best left off the diagram for simplicity but for reference you can reset to: Submitted, Submit-Failed, Running, Waiting, Expired, Succeeded and Failed.

For more info there are a bunch of sets at the top of cylc.flow.task_state which give you the various groupings of task statuses.

[wxtim]

I believe that user requested state changes have been deliberately left off to keep it simple.

[sadielbartholomew]

Thanks @oliver-sanders, when I go back to this after the states have stablised for Cylc 8, I will amend the diagram to address some aspects you have raised.

For more info there are a bunch of sets at the top of cylc.flow.task_state which give you the various groupings of task statuses.

Thanks for the pointer, going to look at relevant code in detail was my next step, but I am putting that on hold until decisions have been made on the future of the states.

There are also user requested state changes

Yes, FYI, I was aware & noted that:

not including the rare manual reset from any state to any other, or manual triggering

I think for an ultimate diagram such user-requested state changes would be best as a brief side note or as detail in caption, as it would clutter the diagram far too much to add those in as arrows (certainly for the any-to-any case).