Feature/enable aggregation #364
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Changes the submission id generation to prepare for aggregation. The `__str__` method of `_JobOperation` is also changed to better handle job aggregation. The unique id generated using a hash of all the `_JobOperation` object's data is still the same, so the uniqueness should not be in question; the only change is to the readable part of the id. The change to the `str` implementation mimics NumPy, and the submission id favors brevity.
Add a new decorator by which operations can be tagged for various types of aggregation, along with classes to store actual aggregates once they have been generated.
…ing (#351) * Use sort_ascending instead of reverse_order. * Fix and simplify hash methods. * Clarify docstrings. * Initialize list/dict with literals. * Use one-line function instead of defining a separate method. * Don't wrap line. * class -> object. * use itertools reference while using groupby * Use reference for zip_longest too * Document _get_unique_function_id Co-authored-by: Hardik Ojha <[email protected]>
…erlab/signac-flow into feature/enable-aggregation
…erlab/signac-flow into feature/enable-aggregation
Change the internals of flow so that everything operates on default aggregates (aggregates of size 1) rather than individual jobs.
vyasr
requested review from
atravitz and
vishav1771
and removed request for
a team,
atravitz and
vishav1771
|
No review necessary here. I'll merge into next as soon as the tests on enable-aggregation pass (from merging in the #335). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add infrastructure for aggregation into flow. All the necessary classes are in place and the code has been modified where appropriate to handle aggregates instead of jobs. However, no actual aggregation occurs yet (everything is an aggregate of size 1). We plan to refactor and optimize the existing code paths before implementing that feature.