suite database: task and job status

[matthewrmshin]

Cylc has changed a lot since we implemented the task states and events database in #198. It is about time that we rethink the data model. This is a list of things I can think of, roughly arranged in possible tables.

Job status:

• time created/updated?
• task name
• cycle
• job (i.e. submit) number
• job type (normal, submit retry, run retry, manual trigger originator)
• submit status
• run status
• run signal
• time of submit
• time run init
• time run exit
• time disappeared
• user@host of submit
• batch system name
• batch system job ID
• cost (e.g. memory, CPUs, wall clock, if possible or relevant)

Task pool/status:

• time created/updated
• task name
• cycle
• n-jobs
• n-submit-retries
• n-execute-retries
• has spawned its successor? (Or store successor cycle?)
• status (improved?)
  • held:
    • held by prerequisites
    • held by runahead
    • held by queue
    • held by user?
    • held by shutdown?
    • (anything else?)
  • active
    • job submitting
    • job submitted
    • job running
    • waiting to re-submit a new job
  • complete (state of final job)
    • submit-failed
    • disappeared
    • run-ok
    • run-failed

Task status change:

• time
• task name
• cycle
• old value
• new value
• change originator (if change is the result of a user request)

Do we need to store task prerequisites? If so:

Task prerequisites and outputs:

• task name
• cycle
• prerequisite/output
• time when this prerequisite is met or output is done

See also #421, #705, #1032.

[arjclark]

The task status entries will probably need an in_pool? flag so we can replace the suite state files.

[matthewrmshin]

The task status entries will probably need an in_pool? flag so we can replace the suite state files.

I would have thought that it is not the task's concern to know where it sits? All we need to do is put the task in a table that represents the current pool?

[matthewrmshin]

More considerations... Users want to query:

• List all the failed jobs ((in a family) in a cycle).
• How long does it take to run (a family in) a cycle? (Elapsed time as well as total time?)

[matthewrmshin]

Job status and log files table implemented in #1493.

[benfitzpatrick]

[meeting] @matthewrmshin, please strike through stuff that's done.

[matthewrmshin]

A user mentioned that it would be useful to be able to easily find out if a task's status has been reset by a user command or not via Rose Bush. Currently, we can only find this information via the suite log. If we have a table to record task status change time and originator in the database, we should be able to display the information via Rose Bush (or successor).

[oliver-sanders]

Most of this is now done. We are considering a DB refactor, closing this as superseded by #3872.