Intelligent Host Selection from Platform Group #4329

wxtim · 2021-07-29T08:28:50Z

These changes close #3827

~~Put up as draft - Functionally tested and probably working. Unit tests to be written.~~

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Appropriate tests are included.
- Functional
- Unit (For the get_platform_from_group function)
Appropriate change log entry included.
(master branch) I have opened a documentation PR at added a guide to writing a platform configuration. cylc-doc#289, covering the admin ramifications of this change.
(master branch) I have opened a documentation PR at Selection of hosts and platforms cylc-doc#299 (very much still wip)

added selection method to config for platform groups functional tests for platform from group

…rning boolean.

hjoliver

(Had a quick look, looks good 👍 )

hjoliver · 2021-08-31T02:47:11Z

(Resolved a small conflict on this branch via GH UI)

cylc/flow/exceptions.py

cylc/flow/task_job_mgr.py

hjoliver · 2021-08-31T03:19:59Z

Could get coverage up a bit?

Co-authored-by: Hilary James Oliver <[email protected]>

wxtim · 2021-08-31T07:11:38Z

Could get coverage up a bit?

Always happy to try to do this.

But there may have been reasons why I haven't.

cylc/flow/cfgspec/globalcfg.py

…ihs.from-platform-group * 'ihs.from-platform-group' of github.com:wxtim/cylc: (75 commits) Update cylc/flow/task_job_mgr.py Update cylc/flow/exceptions.py Add unit test. update cfgspec doc Cycle point format colon restriction Codecov: use flags for fast & functional tests Remove :cylc:conf: Attempt to infer run number if there is no runN symlink. (cylc#4373) Update cylc/flow/cfgspec/globalcfg.py Update cylc/flow/cfgspec/globalcfg.py Added ..attention:: to localhost == scheduler host NOT ROSE_ORIG_HOST Add `versionchanged` admonitions to certain config items Add selection to global.cylc[platforms][localhost] Prevents documentation build failure. GH Actions: Update action Fix test codecov: fiddle configuration Update changelog Change 'ignore' to 'reload' for cycle point cli opts Tidy Run the container after pulling the image from GHCR ...

…ihs.from-platform-group * 'ihs.from-platform-group' of github.com:wxtim/cylc: GH Actions: Add test workflows badges to release PRs Hot fix line length. response to review. Apply suggestions from code review Apply suggestions from code review Update cylc/flow/cfgspec/globalcfg.py Added deprecation warnings in platforms section platform documentation changes cli: change list_plugins to return a string cli: remove gui from command blacklist schema: correct faulty enum

cylc/flow/task_job_mgr.py

wxtim · 2021-09-01T10:31:57Z

@hjoliver

Could get coverage up a bit?

Probably not. As far as I can see the failed lines are all covered by /tests/functional/intelligent_host_selection/06...
I could always add a coverage ignore.

And now it says patch is 100%. I'm confused.

MetRonnie · 2021-09-01T11:02:55Z

Note that Codecov starts reporting coverage as soon as it receives the first uploaded report, so it will initially have the (low) fast-tests coverage only until the functional tests finish

oliver-sanders

Looks good, need to do some heavy testing.

cylc/flow/exceptions.py

…HostsError

hjoliver

(review in process...)

cylc/flow/cfgspec/globalcfg.py

hjoliver

(Releasing some pending review comments from a few days ago...)

cylc/flow/platforms.py

hjoliver

Looks good, tests as working, nice 👍 No real problems found.

Co-authored-by: Hilary James Oliver <[email protected]>

…latform-group * 'master' of https://github.com/cylc/cylc: (101 commits) Update cylc/flow/cfgspec/workflow.py clean up interim obselete message fix flake8-simplify failures caused by recently added checks. (cylc#4420) Fix Change log Add test and raise error on src dir move change log Source symlink fix publish pending flag handle orphan tasks post reload Create window elements & load jobs from DB Deltas batched post reload Fix a doctest. Add colour to GH Actions pytest log Address review feedback. Type annotations for timer. Correct func test timeouts. Fix test. Add Cylc 7 back-compat func tests. Add stall restart func test. Func test for workflow timeout. ...

wxtim · 2021-09-23T07:02:58Z

Strongly suggest that commits are squashed on merge.

oliver-sanders

A few small docs comments.

cylc/flow/cfgspec/globalcfg.py

cylc/flow/platforms.py

…m doc

oliver-sanders · 2021-09-27T12:08:00Z

tests/functional/intelligent-host-selection/06-from-platform-group-fails.t

+for host in {1..4}; do
+    named_grep_ok "job submit fails for bad_host${host}"\
+        "\"remote-init\" failed.*\"bad_host${host}\"" \
+        "$logfile"
+done
+
+# Look for message indicating that remote init has failed on both bad platforms
+# in the platform group.
+for platform in {1..2}; do
+    named_grep_ok "job submit fails for badplatform${platform}"\
+    "badhostplatform${platform}: Tried all the hosts"\
+    "$logfile"
+done


oliver-sanders

🍾

wxtim added 2 commits July 29, 2021 08:38

created a test workflow for platform from groups

312309a

added selection method to config for platform groups functional tests for platform from group

flake 8 fixes

547a172

wxtim changed the title ~~Ihs.from platform group~~ Intelligent Host Selection from Platform Group Jul 29, 2021

wxtim requested a review from oliver-sanders July 29, 2021 08:31

wxtim marked this pull request as draft July 29, 2021 08:31

wxtim added the intelligent-host-selection label Jul 29, 2021

wxtim added this to the cylc-8.0b3 milestone Jul 29, 2021

wxtim added 3 commits July 29, 2021 10:34

re-arrange failure of get_platforms to use exception rather than retu…

c4bbed2

…rning boolean.

fix existing unit test

f5e2e31

unit tests for get_platform_from_group

6ac240d

wxtim self-assigned this Jul 29, 2021

wxtim added 2 commits July 30, 2021 11:36

fixed test broken by logging changes.

b00f135

fix bug introduced to get_platform usage

9b2aeca

wxtim marked this pull request as ready for review July 31, 2021 08:42

hjoliver reviewed Aug 3, 2021

View reviewed changes

Merge branch 'master' into ihs.from-platform-group

849fe76

hjoliver reviewed Aug 31, 2021

View reviewed changes

cylc/flow/exceptions.py Outdated Show resolved Hide resolved

hjoliver reviewed Aug 31, 2021

View reviewed changes

cylc/flow/task_job_mgr.py Outdated Show resolved Hide resolved

wxtim and others added 2 commits August 31, 2021 08:05

Update cylc/flow/exceptions.py

52a2640

Co-authored-by: Hilary James Oliver <[email protected]>

Update cylc/flow/task_job_mgr.py

c6df734

wxtim commented Sep 1, 2021

View reviewed changes

cylc/flow/cfgspec/globalcfg.py Show resolved Hide resolved

wxtim added 5 commits September 1, 2021 10:44

reformat comments to avoid codecov thinking that they are untested code

499d390

Merge branch 'master' into ihs.from-platform-group

444cc27

fix type

e21bcd0

wxtim commented Sep 1, 2021

View reviewed changes

cylc/flow/task_job_mgr.py Show resolved Hide resolved

wxtim requested a review from hjoliver September 1, 2021 10:25

wxtim force-pushed the ihs.from-platform-group branch from 06d2672 to e4494a4 Compare September 2, 2021 11:45

oliver-sanders reviewed Sep 6, 2021

View reviewed changes

cylc/flow/exceptions.py Outdated Show resolved Hide resolved

cylc/flow/exceptions.py Outdated Show resolved Hide resolved

refactored exceptions in response to review; - NoPlatformsError; - No…

46a2cd2

…HostsError

wxtim requested a review from oliver-sanders September 9, 2021 07:41

wxtim added 2 commits September 9, 2021 12:36

fixed group name issue

9adb720

fixed tests accidentally broken

5ef6254

hjoliver reviewed Sep 16, 2021

View reviewed changes

cylc/flow/cfgspec/globalcfg.py Show resolved Hide resolved

wxtim requested a review from hjoliver September 21, 2021 14:38

hjoliver reviewed Sep 22, 2021

View reviewed changes

cylc/flow/platforms.py Outdated Show resolved Hide resolved

cylc/flow/platforms.py Outdated Show resolved Hide resolved

cylc/flow/platforms.py Outdated Show resolved Hide resolved

hjoliver approved these changes Sep 22, 2021

View reviewed changes

Update cylc/flow/platforms.py

e749c26

Co-authored-by: Hilary James Oliver <[email protected]>

wxtim force-pushed the ihs.from-platform-group branch from 8722f09 to e749c26 Compare September 22, 2021 06:56

wxtim added 4 commits September 22, 2021 07:57

added changelog entry

c3d71e2

small test fix

65472a8

fix test broken by timeout change

cdc7a86

Merge branch 'master' into ihs.from-platform-group

a64248b

oliver-sanders reviewed Sep 27, 2021

View reviewed changes

cylc/flow/cfgspec/globalcfg.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/globalcfg.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/globalcfg.py Outdated Show resolved Hide resolved

cylc/flow/platforms.py Show resolved Hide resolved

wxtim added 2 commits September 27, 2021 11:48

Fixed platform group documentation which had been copied from platfor…

9b24c48

…m doc

Fixed platform group documentation which had been copied from platfor…

2d67390

…m doc

wxtim requested a review from oliver-sanders September 27, 2021 10:55

Merge branch 'master' into ihs.from-platform-group

a720456

oliver-sanders reviewed Oct 4, 2021

View reviewed changes

oliver-sanders approved these changes Oct 4, 2021

View reviewed changes

oliver-sanders merged commit 1a1341c into cylc:master Oct 4, 2021

wxtim deleted the ihs.from-platform-group branch October 5, 2021 07:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intelligent Host Selection from Platform Group #4329

Intelligent Host Selection from Platform Group #4329

wxtim commented Jul 29, 2021 •

edited by oliver-sanders

Loading

hjoliver left a comment

hjoliver commented Aug 31, 2021

hjoliver commented Aug 31, 2021

wxtim commented Aug 31, 2021

wxtim commented Sep 1, 2021

MetRonnie commented Sep 1, 2021 •

edited

Loading

oliver-sanders left a comment

hjoliver left a comment

hjoliver left a comment

hjoliver left a comment

wxtim commented Sep 23, 2021

oliver-sanders left a comment

oliver-sanders Sep 27, 2021

oliver-sanders left a comment

Intelligent Host Selection from Platform Group #4329

Intelligent Host Selection from Platform Group #4329

Conversation

wxtim commented Jul 29, 2021 • edited by oliver-sanders Loading

hjoliver left a comment

Choose a reason for hiding this comment

hjoliver commented Aug 31, 2021

hjoliver commented Aug 31, 2021

wxtim commented Aug 31, 2021

wxtim commented Sep 1, 2021

MetRonnie commented Sep 1, 2021 • edited Loading

oliver-sanders left a comment

Choose a reason for hiding this comment

hjoliver left a comment

Choose a reason for hiding this comment

hjoliver left a comment

Choose a reason for hiding this comment

hjoliver left a comment

Choose a reason for hiding this comment

wxtim commented Sep 23, 2021

oliver-sanders left a comment

Choose a reason for hiding this comment

oliver-sanders Sep 27, 2021

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

wxtim commented Jul 29, 2021 •

edited by oliver-sanders

Loading

MetRonnie commented Sep 1, 2021 •

edited

Loading