Formalise Suite Parameters

[oliver-sanders]

Note: suite parameters refers to a suite's runtime values (e.g. is_held) rather than cylc Parameters (as used in the suite.rc file).

The Issue

At present Cylc handles a subset of a suite's "runtime parameters" which are loaded from the database when a suite is restarted:

• is_held
• start_point
• stop_point
• initial_point
• warm_point
• uuid_str
• template_vars

The remaining parameters (which I've identified) should also benefit from the same treatment:

• can_auto_stop
• final_point
• pool_hold_point
• run_mode
• stop_clock_time
• stop_mode
• stop_task
• reloading (doesn't make sense across restart but is still a suite param)?

As these remaining parameters are not restart safe this is effectively a blocker for a universal implementation of #988.

The issue cannot be circumnavigated using the [current] CLI, e.g. cylc stop SUITE POINT sets the stop_point whereas cylc restart SUITE --until POINT sets the final_point

Proposal

• Lump together the suite parameters in a single place where they can be easily loaded / dumped.
• Automatically update the corresponding item in the database when any value changes.
• Start with default parameters (None for all values) then apply in order:
  1. The suite config.
  2. if is_restart: State from the prior run (First subtracting any item(s) which caused the suite to shutdown e.g. custom stop_point).
  3. CLI options.
  4. User interaction at runtime.

@cylc/core I've made an initial attempt at this:

• Does this approach seem sensible?
• Would there be any changes in behaviour caused by the "suite parameter inheritance order" I haven't thought of?
• Is the parameter list complete?

[matthewrmshin]

See also #1032.

[matthewrmshin]

Sounds sensible - as long as the solution is contained (which I believe would be). Caveats, probably to solve beyond the upcoming release:

• The restart command line options may also override settings - and may not match previous run/restarts.
• Restart currently reloads - and disregards/overrides any settings that may be in the previous run/restarts.

[hjoliver]

Yep, sounds good. Ditto Matt's comments.

[oliver-sanders]

On further investigation, this change has the potential to be quite dangerous and needs a lot of time for testing and stabilisation to ensure we don't release anything faulty so bumping this issue back.

The approach I have employed in my attempts so far is:

• Create a SuiteParameters object belonging to the Scheduler which updates the database whenever any of the parameters change.
• On TaskPool.STOP_AUTO reset the parameter that caused the suite to enter this state so as to allow the suite to restart.

[oliver-sanders]

Currently there are four main cycle point parameters:

• initial_cycle
• start_cycle
• final_cycle
• stop_cycle

My interpretation of this is that initial and final relate to the suite configuration whilst start and stop relate to the scheduler instance.

Unfortunately work on #2809 has revealed that the current usage of these parameters is mixed with their purpose becoming blurred.

E.g. when restarting a suite cylc restart converts the final_cycle from the previous run into stop_cycle in the new one.

For consistency between cylc run and cylc restart this discrepancy will have to be addressed.