Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added a guide to writing a platform configuration. #289

Merged
merged 21 commits into from
Oct 13, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
69aaaba
added a guide to writing a platform configuration.
wxtim Aug 26, 2021
c9c9e6f
Apply suggestions from code review
wxtim Sep 3, 2021
e121f80
Update src/admin-guide/writing-global-configurations/platforms.rst
wxtim Sep 6, 2021
a7575a9
Update src/admin-guide/writing-global-configurations/platforms.rst
wxtim Sep 6, 2021
f4ff67c
reconfigured to be in the reference section
wxtim Sep 7, 2021
fd4cc17
Merge branch 'document.platform.setup' of github.com:wxtim/cylc-doc i…
wxtim Sep 7, 2021
83c9e75
Apply suggestions from code review
wxtim Sep 9, 2021
bede7eb
separted localhost and submit from localhost
wxtim Sep 9, 2021
c6fa66e
refactor platforms how to write admin guide
wxtim Sep 9, 2021
e5d7881
Update src/reference/config/writing-platform-configs.rst
wxtim Sep 10, 2021
442b59e
added a reference to the new writing platforms
wxtim Sep 10, 2021
4e0ca1b
Merge branch 'document.platform.setup' of github.com:wxtim/cylc-doc i…
wxtim Sep 10, 2021
dc8605e
document graph line continuation on boolean
wxtim Sep 10, 2021
e29dbd9
work in progress
wxtim Sep 22, 2021
a7a01f9
added note about not using platforms and groups with the same names; …
wxtim Oct 1, 2021
de2a11d
fix error in glossary
wxtim Oct 5, 2021
94474a5
Merge branch 'document.platform.setup' into selection-of-hosts-and-pl…
wxtim Oct 5, 2021
808ad40
Merge pull request #2 from wxtim/selection-of-hosts-and-platforms
wxtim Oct 5, 2021
2d30c25
Apply suggestions from code review
wxtim Oct 6, 2021
24c8574
Added really simple platform
wxtim Oct 6, 2021
8474281
Added simple server with install target
wxtim Oct 6, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 52 additions & 18 deletions src/7-to-8/major-changes/platforms.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ Platforms
.. code-block:: cylc

[runtime]
[[task_name]]
[[[job]]]
batch system = slurm
[[[remote]]]
host = my_supercomputer
[[task_name]]
[[[job]]]
batch system = slurm
[[[remote]]]
host = my_supercomputer

Read this section if your workflow's jobs run on a remote computer or if
you see the following warning on running ``cylc validate``:
Expand All @@ -29,6 +29,21 @@ Platforms

WARNING - Task <task>: deprecated "host" and "batch system" will be removed at Cylc 9

If you currently use the ``rose host-select`` utility or a similar host
selection or load balancing utility the intelligent host selection
functionality of Cylc 8 may be used instead:

.. code-block:: cylc

[runtime]
[[task_name]]
[[[remote]]]
host = $(rose host-select my-computer)
[[another_task]]
# An example of a home-rolled host selector
[[[remote]]]
host = $(test $((RANDOM%2)) -eq 0 && echo "host_a" || echo "host_b")


Overview
--------
Expand All @@ -41,6 +56,22 @@ Cylc 8 allows site administrators (and users) to configure
multiple hosts with associated platform-specific settings. Users only need to
select the platform for their task jobs.

Platforms also define how hosts are selected from each platform:

- Randomly (default)
- By definition order

There may be cases where sets of platforms (for example a group of
standalone compute servers, or a pair of mirrored HPC's) might be equally
suitable for a task, but not share files systems to allow them to constitute
a single platform. Such platforms can be set up to be ``platform groups``


.. seealso::

:ref:`AdminGuide.PlatformConfigs` for detailed examples of platform
configurations.

.. warning::

Cylc 8 contains upgrade logic which handles Cylc 7
Expand All @@ -52,8 +83,10 @@ Cylc 8 allows site administrators (and users) to configure
Examples
--------

See :cylc:conf:`global.cylc[platforms]` for a detailed explanation of how
platforms are defined.
.. seealso::

:cylc:conf:`global.cylc[platforms]` has a detailed explanation of how
platforms and platform groups are defined.

Showing how the global config changes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -127,16 +160,17 @@ In Cylc 8 the equivalent might be:
.. code-block:: cylc

[runtime]
[[mytask_cylc_server]]
[[mytask_cylc_server]]

[[mytask_big_server]]
platform = linuxbox42
[[mytask_big_server]]
platform = linuxbox42

[[mytask_submit_local_to_remote_computer]]
platform = pbs_local
[[mytask_submit_local_to_remote_computer]]
platform = pbs_local

[[mytask_login_to_hpc_and_submit]]
platform = $(supercomputer_login_node_selector_script)
[[mytask_login_to_hpc_and_submit]]
# This is still legal, but you could also use host selection.
platform = $(supercomputer_login_node_selector_script)

The platform settings for these examples might be:

Expand All @@ -147,9 +181,9 @@ The platform settings for these examples might be:
# Without a hosts, platform name is used as a single host.

[[pbs_local]]
job runner = pbs
hosts = localhost
job runner = pbs
hosts = localhost

[[slurm_supercomputer]]
hosts = login_node01, login_node02 # Cylc will pick a host.
job runner = slurm
hosts = login_node01, login_node02 # Cylc will pick a host.
job runner = slurm
3 changes: 2 additions & 1 deletion src/7-to-8/summary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,8 @@ Platform Awareness

.. seealso::

:ref:`Platforms at Cylc 8.<majorchangesplatforms>`
- :ref:`Platforms at Cylc 8. <majorchangesplatforms>`
- :ref:`System admin's guide to writing platforms. <AdminGuide.PlatformConfigs>`

Cylc 7 was aware of individual job hosts.

Expand Down
5 changes: 5 additions & 0 deletions src/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -656,6 +656,11 @@ Glossary

* :term:`job host`
* :term:`job runner`
* :term:`platform group`

platform group

A set of :term:`platforms <platform>`

scheduler
When we say that a :term:`workflow` is "running" we mean that the cylc
Expand Down
1 change: 1 addition & 0 deletions src/reference/config/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ Configuration
workflow
global
types
writing-platform-configs
Loading