sync with LLNL/develop #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Addition of user enabled workspace hashing (#145) * Addition of hashing to Study parameterization. * Addition of the hashws option to argparse. * Addition of a warning note for users who use labels in steps. * Update setup.py to 1.1.4dev * More generalized FluxScriptAdapter (#149) * Addition of a more general flux ScriptAdapter. * Addition of some casting from int to str * Corrected "gpus" to "ngpus" * Rework jobspec construction to make a valid jobspec. * Check for empty value for cores per task. * README tweak to update quickstart link. (#139) * typos. fixes #141 (#154) * Correction of flake8 style errors [new version of flake8]. * Update to setup.py to reflect dev version 1.0 * Correction to safe pathing for missed cases and make_safe_path enhancements. (#157) * Made pickle and log path string safe for pathing. * Tweaks to make_safe_path to include a base path. * Updates to make_safe_path usage * Correction to not modify the iterator copy. * Correction to fix the format of output status time to avoid a comma that breaks printing. (#160) * Addition of a utility function for formatting times to H:M:S * _StepRecord time methods now call the new utility function. * Tweaks to add days to the format to avoid 3 digit hours. * Tweak to formatting. * Made the day format more parsable. * Removal of _stage_linear since it is now not needed. (#156) * Removal of _stage_linear since it is now not needed. * Addition of linear LULESH samples. * Update the dev to 1.1. * Addition of pargs for passing parameters to custom parameter generation (#152) * Addition of a utility method to create a dictionary from a list of key-value pairs. * Addition of the pargs interface for passing parameters to custom parameter generation. * Addition of a Monte Carlo example that accepts pargs. * Addition of pargs check for dependency on pgen. * Addition of clearer error message for malformed parameters. * Update setup.py * do not overwrite log file. (#162) Signed-off-by: Peter Robinson <[email protected]> * Added confirmation message after launching a study (#163) * Enhancements to store relative pathing in the metadata file. (#165) * Changes to make workspaces reflect relative pathing based on step names. * Addition of an alternative output format based on step combinations. * Addition of tag to LULESH git dependency. (#169) * Script Adapter Plugin (#167) (#170) Fixes #167 * added pytest to requirements added Pipfile and pipenv settings * Added property key to Abstract.ScriptAdapter (#167) Also added impementation and tests to verify that existing functionality isn't changed * updated factory to use key when registering adapters(#167) * cleanedup linelength * cleaned up imports to be specific to module (#167) * added tests to verify exception for unknown adapter * moved adapters tests to individual files * added test to verify scriptadapter functionality (#167) updated gitignore to have testing and pycharm ignores testing existing adapters in factory (#167) added test to verify factories.keys matches get_valid_adapters (#167) added copyright to file * updated __init__ modules to do dynamic includes * removed unneeeded imports * updated dependency versions * fixed all flake8 errors * updated to run flake8 and pytest when run locally * updated tests to have documentation about purpose and function as requested in #170 * fixed line length * Removal of nose from requirements. * updated to remove nose from the requirements * PyYAML vulnerability fix (#171) * Locking the version of PyYAML to be above 2.1 because of an arbitrary code execution vulnerability. * Addition of a version condition to pyyaml to patch a vulnerability. * Update of Pipfile.lock to match Pipefile. * Minor tweak to indentation for flake8 failure. * fixed pyyaml to requirements (#172) * Addition of a loader to the yaml load call. (#174) Fixes #173 * Addition of a loader to the yaml load call. * Addition of a catch if the loader attribute is missing. * Correction to install enum34 for Python versions < 3.4 (#176) * Moved enum34 to condition dependent on Python<3.4. * Addition of conditional enum34 install for requirements.txt. * Correction of requirements.txt syntax for python version. * Addition of a Dockerfile for tutorials and ease of trying out. (#178) * Addition of a Dockerfile for quick tutorials. * Tweaks for Docker and addition of git. * Tweak to Docker file for caching. * Addition of Docker documentation. * Tweaks to Docker documentation. * Removal of markdown ## * Take out shebang from shell definition and add it when script is written. (#181) * Take out shebang from shell definiton and at it when script is written. * Include shebang in cmd and fix format of string written to file. * Tweaks to fix malformed log statements. (#182) * Correction to message when stating no to launch. * Enhance shell batch setting to apply to scheduler scripts. (#183) * Extension of shebang feature to allow users to specify shells. * Addition of debug message to print kwargs. * Addition of kwargs. * Addition of basic batch settings to LULESH sample. * Addition of kwargs to Flux adapters. * Docstring tweaks. * Docstring update. * Fixes the addition of the shebang header for SLURM (#184) * Docstring correction for LocalAdapter. * Correction to addition of exec line at top of scripts. * Correction to an accidental reassignment of cmd. * Removal of an assignment of self._exec in SLURM adapter. * Change to transition adapter returns to Record objects. (#177) * Addition of a Record class for storing general data. * Addition of SubmissionRecord type. * Update to the order of for record parameters. * Changes to StepRecord to expect SubmissionRecord returns. * Updates to SLURM and local adapters to use SubmissionRecords. * Slight tweak to LocalAdapter docstring. * Tweak to have SubmissionRecord initialize its base. * Addition of CancellationRecord class. * Changes to CancellationRecord to map based on status. * Additional interface additions and tweaks. * Changes to have cancel use CancellationRecords. * Update to ExecutionGraph to use records. * Updates to SLURM and local adapters to use SubmissionRecords. * Slight tweak to LocalAdapter docstring. * Addition of CancellationRecord class. * Additional interface additions and tweaks. * Changes to have cancel use CancellationRecords. * Cherry pick of execution commit. * Removal of redundant "get" definiton. * Addition of a SLURM enabled LULESH sample specification. * Addition of output for stdout and stderr for Local adapter. * Correction of file to open. * Addition of 3.7 to testing stack. * Added 3.7 to tox.ini. * Removal of py37 in testing. * Addition of py37 to travisCI (#187) * Correction to 3.7 specification. * Removed py37 from tox testing. * Readded py37 to tox and removed duplicate from travis. * Addition of build status badge. * Update SLURM sample spec to add missing walltime. * Addition of dumping the environment to a YAML file. (#190) * Addition of documentation that covers the set up of a simple study (#168) * Addition of simple Hello World spec. * Addition of basics page to index. * Addition of hello_world documentation. * Additions to hello_world. * More documentation in single step section. * Continued edits to Hello World. * Addition of parameter section. * Addition of a note about %% token. * Addition of directory structure. * Continuation of parameter documentation. * Removal of the depends key. * Addition of the env section description. * Addition of a link to Docker documentation for Dockerfiles. * Addition of single parameter hello world. * Correction of double colons. * Correction of indentation. * Addition of print out to verify output. * Addition of sample specifications for multi and single params. * Addition of more documentation for single param. * Additional output to show parameter results. * Correction to formatting. * Addition of samples. * Addition of simple Hello World spec. * Addition of basics page to index. * Addition of hello_world documentation. * Additions to hello_world. * More documentation in single step section. * Continued edits to Hello World. * Addition of parameter section. * Addition of a note about %% token. * Addition of directory structure. * Continuation of parameter documentation. * Removal of the depends key. * Addition of the env section description. * Addition of a link to Docker documentation for Dockerfiles. * Addition of single parameter hello world. * Correction of double colons. * Correction of indentation. * Addition of print out to verify output. * Addition of sample specifications for multi and single params. * Addition of more documentation for single param. * Additional output to show parameter results. * Correction to formatting. * Updates to docstrings for data structures. * Updates to clear Sphinx warnings. * Removal of escape on the *args becuase of flake8 failure. * Clean up of existing hello world specs. * Addition of multistep example spec. * Removal of * to fix sphinx errors. * Correction to some docstrings. * Tweaks to specs for consistent naming. * Finished multi-step parameterized example. * Tweaks to hello world docs. * Addition of link to examples on GitHub. * Correction of link to examples. * Correction of link to examples (again). * Removal of Pipfile.lock. * Additions to gitignore for vscode and pipenv. * Marking for v1.1.4 release. * Corrected a missed merge for release v1.1.4 * Extend the Specification interface to break out loading from streams. (#198) * Closes #198 * Addition of loading specification "from_str". * Updates to Specification docstrings. * Updates to abstract Specification to change from str to stream. * Updates to YAMLSpecification to use the new stream API. * Removal of IOString * Update to the YAMLSpecification load stream method. * Quickfix: Addition of the accidental removal of the path member variable. * Updating the version to 1.1.5dev (forgotten previously). * Correction to versioning for install. * Moved SLURM parameters to be placed in front of the submitted script. (#202) Fixes #201 * Addition of version information to package and command line (#205) * Addition of version information. * Tweak to have setup.py pull from __version__ * Addition of command line arg to print version. * Pinning version for release 1.1.5 * Addition of 1.1.5a to line up with PyPi labeling. * Increment up to get rid of a0
* Tweaks and fixes to the SLURM header. Adds the ability to specify GPUs, fixes reservation pairing with accounts, and now uses a ChainMap to handle the internal key conflicts between the batch and step settings. Also introduces the exclusive key. Changed to full parameter names for clarity. * Addition of chainmap package for python2.7 * A check to see if procs is included in the header. Fixes #234 and includes ntasks in the header if the batch section includes the key.
Fixed #264 -- When testing #263, SLURM ran out of memory during the completing stage and aborted the jobs and left the job in an unknown state. This PR fixes this issue by defaulting to a failure when the status is found to be UNKNOWN. * Adds a check for UNKNOWN state. * Correction of bad variable name "state" * Tweak to treat UNKNOWN as failed. * Change marking of failed state to unknown. * Some fixes for style and credit from #265
* Addition of README as long description. * Dropping encoding as it's not supported in 2.7
* Initial pgen docs with itertools example * Add pargs example * Add pgen using numpy plus helper function for 1D distribution * Fix typos, update image * Initial port of complete Parameters documentation * Fix up api docs warnings, add missing script adapters * Add subsection on accessing env block variables inside pgen * Literal imports and some style tweaks. * Some minor title and header tweaks. * Remove out of date notes * Renamed itertools_pgen to reference LULESH. * Update doc strings on parameter generator samples * Make flake8 happy * Misc cleanup and formatting, adding more links and internal references * Fix up section listing, in-text moniker/function formatting Co-authored-by: Frank Di Natale <[email protected]>
Co-authored-by: Francesco Di Natale <[email protected]>
* added checks for valid keys * branch updates * working on validation * added schema file * updates * fixed spec * fixed spec * refactor updates * verification logic * removed test field * made specification module * removed TODO * fixed flake8 style * schema name is now hardcoded * updated MANIFEST * adjusted imports * tuples -> lists Co-authored-by: Francesco Di Natale <[email protected]>
…r the (#287) documentation links
* added some missing spaces in multiline text strings * added some missing spaces in multiline text strings
* Update parameters.rst Make as explicit as possible that if you run with PGEN, then just comment out the global.parameters block in your YAML spec file. * Update parameters.rst * Update parameters.rst * Update parameters.rst Made changes according to Francesco DiNatale's suggestion.
* Updates to correct indentation errors. * Addition of newly compile docs. * Indendations added to Record.get docstring.
When computing CPUs per node on LSF.
* Addition of processing multiple directories and globs for status * Minimal edits after first commit * Shortened code to allow more glob functionality * Removed glob, added nargs+ to conductor.py, rearranged code in maestro.py * Removed some redundant code * Reduced more code. Works for multiple directory inputs (* included). Absolute path added * Added refinement features to allow ease of interpreting results * Additional refinement. No more logging for status command. * Flake8 and minor style tweaking
* Updated the Sphinx documentation to pull Maestro’s version from the package * Updated formatting for flake8 check
* Updated the documentation for building the Docker image and updated the Dockerfile to use Python 3 * Updated the Dockerfile format
* Removal of py27 and additions of py37+8 * Removal of 3.7 from travis * Tweak to remove py37 matrix
* added pytest and coverage settings * initial change to poetry * remove setup.py and convert to pyproject.toml * fixes for tox and travis pipelines * Changes to support python >3.4 and formatting updates. * adding athey1 as a maintainer * fixed tab spacing * fixed spacing * added jessica as a maintainer * added jeremy as a maintainer
* initial testing on loading spec * black settings added * reformatted by black * added test for validation errors * added test to check for missing key in step * updated schema to add check for non-null variables * added check for study steps * added test for multiple dependencies of the same name * changed yaml spec testing for easier reading * added test of global parameters * fixed test execution to verify error thrown * added output_path test * formatting fix * added tests for steps generation and params generation * added tests for get_study_env * initial testing on loading spec * black settings added * reformatted by black * added test for validation errors * added test to check for missing key in step * updated schema to add check for non-null variables * added check for study steps * added test for multiple dependencies of the same name * changed yaml spec testing for easier reading * added test of global parameters * fixed test execution to verify error thrown * added output_path test * formatting fix * added tests for steps generation and params generation * added tests for get_study_env * updated to add testing for description setter * added test for spec.name setter * pulled spec_path code into a pytest fixture * added pytest and coverage config * added pytest-cov as a tox requirement * added report.xml to gitignore * changed line length for black * added coverage and junit reports to tox * Added coverage appending for test runs
* Add name and real_name StudyStep properties. There now needs to be an aliasing of nickname with a step name when requested because adapters expect to use the name. StudyStep objects now return their nickname if the nickname is set. Objects such as the ExecutionGraph must now use the real_name for logistic tracking, while adapters can continue to use name. * Addition of extension property for Flux. * Some typo and string style tweaks.
* Testing by adding comment * Initial commit, reformatted checking for Nodes and Procs in slurmscriptadapter.py, added extra conditional * Removed condition for checking Node, moved condition for raising exception to get_header * Changed code for condition that handles no nodes or procs * Tweaked the error message * Minor variable mispelling * Handled additional case of nodes absent and procs present to add procs to SBATCH * In the case of Nodes absence and Procs presence, it will be added to SBATCH * Nodes now appear in SBATCH header even if uninitialized in a step * Changed Chain Map to regular Dictionary * Remove extra loop and change resource construction. * Cleared styling issues, removed time stamped outputs from Samples directory * Some refactor clean up. Co-authored-by: Frank Di Natale <[email protected]>
* Job ID shows up in Maestro Status, edited executiongraph.py write_status function * Changed sleep time back to original. Job ID gets last index. * Fixed a casting issue (int to str). + style Co-authored-by: Frank Di Natale <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.