Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.1.9 Release #398

Merged
merged 150 commits into from
May 29, 2022
Merged
Changes from 1 commit
Commits
Show all changes
150 commits
Select commit Hold shift + click to select a range
22ab603
Addition of user enabled workspace hashing (#145)
FrankD412 Sep 30, 2018
7b9bee0
Update setup.py to 1.1.4dev
FrankD412 Oct 5, 2018
026e7e6
More generalized FluxScriptAdapter (#149)
FrankD412 Oct 12, 2018
8e4e5de
README tweak to update quickstart link. (#139)
FrankD412 Oct 21, 2018
4fc2f30
typos. fixes #141 (#154)
gonsie Oct 23, 2018
f34d055
Update to setup.py to reflect dev version 1.0
Oct 31, 2018
29ce169
Correction to safe pathing for missed cases and make_safe_path enhanc…
FrankD412 Nov 3, 2018
cd90a8c
Correction to fix the format of output status time to avoid a comma t…
FrankD412 Nov 5, 2018
dce0701
Removal of _stage_linear since it is now not needed. (#156)
FrankD412 Nov 6, 2018
489a11c
Addition of pargs for passing parameters to custom parameter generati…
FrankD412 Nov 8, 2018
07e056e
Added confirmation message after launching a study (#163)
jsemler Nov 15, 2018
852b09a
Addition of tag to LULESH git dependency. (#169)
FrankD412 Jan 8, 2019
72c55d9
Script Adapter Plugin (#167) (#170)
kcathey Jan 18, 2019
1a2fd12
PyYAML vulnerability fix (#171)
FrankD412 Feb 2, 2019
1e4eb12
Minor tweak to indentation for flake8 failure.
Feb 6, 2019
0a05230
fixed pyyaml to requirements (#172)
kcathey Mar 1, 2019
f684154
Addition of a loader to the yaml load call. (#174)
FrankD412 Mar 8, 2019
25fe6c1
Correction to install enum34 for Python versions < 3.4 (#176)
FrankD412 Mar 19, 2019
09b4646
Addition of a Dockerfile for tutorials and ease of trying out. (#178)
FrankD412 Mar 24, 2019
959b525
Take out shebang from shell definition and add it when script is writ…
koning Apr 18, 2019
36baee1
Correction to message when stating no to launch.
Apr 19, 2019
af02c7d
Enhance shell batch setting to apply to scheduler scripts. (#183)
Apr 25, 2019
54d3c0d
Fixes the addition of the shebang header for SLURM (#184)
Apr 26, 2019
32ad628
Correction to an accidental reassignment of cmd.
FrankD412 May 1, 2019
5109b28
Removal of an assignment of self._exec in SLURM adapter.
May 1, 2019
b758662
Change to transition adapter returns to Record objects. (#177)
May 1, 2019
8631738
Addition of a SLURM enabled LULESH sample specification.
May 8, 2019
2aa0113
Addition of output for stdout and stderr for Local adapter.
May 9, 2019
56c1a19
Correction of file to open.
May 9, 2019
a4fed39
Addition of 3.7 to testing stack.
May 9, 2019
213ee04
Added 3.7 to tox.ini.
May 9, 2019
895518e
Removal of py37 in testing.
May 9, 2019
83fdc21
Addition of build status badge.
May 9, 2019
b97b3b8
Update SLURM sample spec to add missing walltime.
Jun 12, 2019
7d4cb56
Addition of documentation that covers the set up of a simple study (#…
Jun 26, 2019
93efe88
Correction of link to examples.
Jun 26, 2019
c26d566
Correction of link to examples (again).
Jun 26, 2019
40c6dc6
Removal of Pipfile.lock.
Jul 2, 2019
cfae982
Additions to gitignore for vscode and pipenv.
Jul 2, 2019
1fed3b7
Marking for v1.1.4 release.
Jul 14, 2019
32f5ff7
Corrected a missed merge for release v1.1.4
Jul 15, 2019
281158a
Extend the Specification interface to break out loading from streams.…
Sep 26, 2019
e4ddd9e
Quickfix: Addition of the accidental removal of the path member varia…
Sep 26, 2019
05445fb
Updating the version to 1.1.5dev (forgotten previously).
Sep 26, 2019
87b598c
Correction to versioning for install.
Oct 8, 2019
9e07e4f
Addition of version information to package and command line (#205)
Dec 3, 2019
ff2504f
Pinning version for release 1.1.5
Dec 5, 2019
3d07372
Addition of 1.1.5a to line up with PyPi labeling.
Dec 6, 2019
97e4f9c
Increment up to get rid of a0
Dec 6, 2019
1ee11a4
Addition of a simple example and logo to the README. (#208)
Dec 11, 2019
53bac8e
Update __init__.py to tick version to dev version.
Dec 14, 2019
aa3de08
Enhances pgen to have access to environment specified in a study. (#209)
Dec 17, 2019
5d96c47
Updates to add scheduled workflows and reorganize.
Dec 18, 2019
70c9a5d
Correction to HPC wikipedia link.
Dec 18, 2019
caf580f
Updates to the classifiers for setup.py
Dec 19, 2019
528cbf0
Addition of long text setting.
Dec 19, 2019
c069827
Correction of missing quote.
Dec 19, 2019
925447c
Drop support for Python 3.4 (#218)
Jan 17, 2020
0316f78
Applies workspace substitution to the restart command. (#219)
Jan 22, 2020
1c62c90
Test/interfaces/lsf (#215)
Jan 22, 2020
1dc94c9
Sets log and err out for SLURM to parameterized step name (#220)
Jan 22, 2020
015dab9
Update to dev0 to differentiate for pre-release.
Jan 22, 2020
e465856
Ticked up version to 1.1.7dev1.
Jan 22, 2020
4cce6b2
Update Maestro logo link to full link for PyPi.
Jan 22, 2020
b49362f
An update to Maestro's description.
Jan 30, 2020
0c10839
Adding Neptune to the list on Planets (#222)
adrienbernede Feb 5, 2020
ee83752
Small README tweak.
Feb 22, 2020
0e7d580
Updated the study step name in the README.md file (#227)
jsemler Feb 25, 2020
c7598aa
Updated the package version in the Sphinx docs (#229)
jsemler Feb 26, 2020
1726b6d
Added a link to Maestro's documentation (#231)
Mar 5, 2020
4fdbe93
Improve the performance of expansion (#232)
Mar 9, 2020
50099e2
Updated the description in the setup.py file (#233)
jsemler Mar 12, 2020
01746dd
Added dill to the requirements.txt file (#235)
jsemler Mar 12, 2020
5d9233c
Fix to add PID to local log names. (#241)
Apr 3, 2020
8a7bb10
Refactor to move DAG operations to the back end Conductor (#242)
Apr 8, 2020
421e038
Added pre-commit to enable flake8 checks (#244)
kcathey Apr 9, 2020
7a5a497
Bugfix for logging that didn't appear in submodules (#247)
Apr 14, 2020
6446fb4
Added spec verification via jsonschema
ben-bay Apr 9, 2020
803b392
Reformatted and added color to logger messages. (#252)
ben-bay Apr 24, 2020
4426e63
Bug fix for unintentional variable section requirement. (#256)
ben-bay Apr 30, 2020
36657dd
Update to broken venv documentation link.
May 7, 2020
256ba37
Addition of a simple dry-run capability. (#259)
May 13, 2020
b798b20
enabled raw sbatch errors to be logged (#262)
crkrenn May 16, 2020
6759543
Tweaks and fixes to the SLURM header. (#263)
May 17, 2020
8341c8a
modified executiongraph to round datetimes to nearest second
May 17, 2020
189afe7
Adds a check for UNKNOWN state. (#266)
May 17, 2020
c70d131
Official 1.1.7 release.
FrankD412 May 17, 2020
af4a6f4
Official start to 1.1.8dev0.
May 17, 2020
d75d498
Addition of README as long description. (#269)
May 18, 2020
b795114
Pgen docs (#275)
jwhite242 May 27, 2020
c3de932
Replace unicode quote with ascii quote (#277)
jwhite242 May 29, 2020
64d8711
Updated the docs release version to 1.1.7 (#278)
jsemler May 29, 2020
08aa1ea
Modified validation logic to skip over variable tokens. (#279)
ben-bay Jun 2, 2020
88d7b71
Inheritable validation and spec module (#280)
ben-bay Jun 4, 2020
180307d
Tick up to 1.1.8 patch version.
FrankD412 Jun 11, 2020
7be2647
Correction of package data path to schema.
FrankD412 Jun 11, 2020
b0be44d
whitespace fixes from pre-commit checks
FrankD412 Jun 11, 2020
bf377e8
Start of version to 1.1.9dev0
Jun 14, 2020
438d14a
Updated the docs version to 1.1.8 (#288)
jsemler Jun 19, 2020
4e8d42f
Added a link to the maestro sheetmusic repo and added descriptions fo…
jsemler Jun 19, 2020
fbe2184
Added introduction documentation from the README.md file (#289)
jsemler Jun 23, 2020
6bf6fb2
Docs/formatting (#296)
crkrenn Jul 9, 2020
6447d06
Update parameters.rst (#298)
tanimislam Jul 21, 2020
8dd66b1
Correct documentation compilation warnings (#299)
Jul 21, 2020
bd73003
Bugfix: Division by string instead of number (#302)
kennyweiss Aug 7, 2020
9f37a1f
Addition of processing multiple directories and globs for status (#301)
youngjeffrey Aug 19, 2020
bd75e6d
Updated the documentation to pull the recent version (#306)
jsemler Aug 21, 2020
55d2d84
Updated the documentation for installing Maestro (#308)
jsemler Aug 21, 2020
1fc3274
Updated the Dockerfile and documentation (#312)
jsemler Sep 5, 2020
64e07a0
Removal of Python 2.7 and addition of newer Python 3 versions (#315)
Sep 12, 2020
37d0996
simple bug fix in the YAMLSpec description setter (#319)
kcathey Sep 14, 2020
e7555a3
Conversion to Poetry Build System (#316)
kcathey Sep 14, 2020
97a5f16
YAMLSpecification Testing (#317)
kcathey Sep 16, 2020
d911ba1
Add name and real_name StudyStep properties. (#314)
Sep 16, 2020
96a3364
Re-add setup.py to retain standard editable install.
FrankD412 Sep 18, 2020
4489d0e
Slurm Script Adapter Bug Fix (#305)
youngjeffrey Sep 21, 2020
8603f21
WIP: Add job identifier to status command (#323)
youngjeffrey Oct 5, 2020
27790c9
Fixes #324 (#325)
scottwedge Oct 9, 2020
9bad73f
A fix to use communication instead of wait. (#328)
Oct 14, 2020
7b96620
Updated README to correct badges.
Nov 10, 2020
5f62435
Corrected badge colors for PePy
Nov 10, 2020
9c03009
Removal of Python3.5 from testing (#339)
Jan 30, 2021
c8973da
Updates to the FluxScriptAdapter to use the latest Flux. (#251)
Feb 1, 2021
c3aff33
Addition of cancellation for multiple directories and globs. (#338)
Feb 23, 2021
6c7d56d
update .coveragerc to ignore tox and venvs (#346)
kcathey Feb 23, 2021
08e27e9
Spectrum adapter remove/lingering in-class import. (#352)
Mar 9, 2021
afa8bb8
Limit to cryptography<3 for build (#353)
Mar 18, 2021
076cf87
Update to python versions >=3.6
FrankD412 May 7, 2021
f6ecffe
Add support for Flux core 0.26.0 (#357)
SteVwonder May 13, 2021
83e703c
Ticked up the version to 1.1.9dev1
FrankD412 May 13, 2021
52da01d
Flux update (#359)
SteVwonder May 25, 2021
1799774
Update to fix the addition of -g erroneously (#361)
May 26, 2021
b38443b
Removal of TravisCI due to transition to GitHub Actions.
FrankD412 Jun 1, 2021
d2fd935
Update pyproject version to match setup.py
FrankD412 Jun 1, 2021
ea136eb
Addition of a first pass GH Action linter.
FrankD412 Jun 3, 2021
3e4e4a7
Status tweaks (#358)
jwhite242 Jun 5, 2021
e64cbce
Slurm QOS specification. (#365)
youngjeffrey Jun 21, 2021
f19ecdf
Addition of reservation to bsub headers. (#367)
Jul 14, 2021
cf18f42
Additional flexibility to walltime parsing for Flux (#369)
Jul 24, 2021
5747ff8
Addition of Flux connection to try/catch (#375)
Aug 19, 2021
90b4991
Avoid overriding previous lines in Lulesh example (#382)
TobiasDuswald Feb 21, 2022
0ce6599
Flux Adapter Clean up and Addition of Job Priority (#379)
FrankD412 Mar 12, 2022
2be47aa
Addition of Workflow Community Initiative information. (#385)
FrankD412 Mar 12, 2022
1e80072
Bump paramiko from 2.7.2 to 2.10.1 (#387)
dependabot[bot] Mar 31, 2022
564483d
Add black and update versioning. (#389)
FrankD412 Apr 4, 2022
0afcbdc
Enable more complete configurability of jsrun launcher (#384)
jwhite242 Apr 30, 2022
e8a34a5
Updating dependency versions. (#392)
FrankD412 Apr 30, 2022
0b33f20
Correct conditional to correct empty string catch. (#393)
FrankD412 May 7, 2022
714baa4
Move to purely poetry install (#394)
FrankD412 May 16, 2022
9db08af
Ci/test and release (#396)
jwhite242 May 17, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Slurm QOS specification. (#365)
* qos feature added for Slurm

* revert naming of this file
youngjeffrey authored and FrankD412 committed May 28, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit e64cbceace93d8323feaac6cb4b4af794437c4a2
6 changes: 6 additions & 0 deletions maestrowf/interfaces/script/slurmscriptadapter.py
Original file line number Diff line number Diff line change
@@ -73,6 +73,7 @@ def __init__(self, **kwargs):
self.add_batch_parameter("bank", kwargs.pop("bank"))
self.add_batch_parameter("queue", kwargs.pop("queue"))
self.add_batch_parameter("reservation", kwargs.pop("reservation", ""))
self.add_batch_parameter("qos", kwargs.get("qos"))

# Check for procs separately, as we don't want it in the header if it's
# not present.
@@ -96,6 +97,7 @@ def __init__(self, **kwargs):

self._ntask_header = "#SBATCH --ntasks={procs}"
self._exclusive = "#SBATCH --exclusive"
self._qos = "#SBATCH --qos={qos}"

self._cmd_flags = {
"cmd": "srun",
@@ -156,6 +158,10 @@ def get_header(self, step):
if exclusive:
modified_header.append(self._exclusive)

qos = resources.get("qos")
if qos:
modified_header.append(self._qos.format(qos=qos))

return "\n".join(modified_header)

def get_parallelize_command(self, procs, nodes=None, **kwargs):
3 changes: 2 additions & 1 deletion maestrowf/specification/schemas/yamlspecification.json
Original file line number Diff line number Diff line change
@@ -65,7 +65,8 @@
{"type": "boolean"},
{"type": "string", "pattern": "^\\$\\(\\w+\\)$"}
]},
"use_broker": {"type": "boolean"}
"use_broker": {"type": "boolean"},
"qos": {"type": "string", "minLength": 1}
},
"additionalProperties": false,
"required": [