Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Aggregation Feature to Flow #336

Closed
wants to merge 96 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
4eb97a0
Add the private Aggregate class having fundamental concepts
kidrahahjo Jul 15, 2020
07c5e16
Tests for _Aggregate class added
kidrahahjo Jul 15, 2020
5cc7912
Logic for _condition, BaseFlowOperation, FlowGroup added
kidrahahjo Jul 15, 2020
b9c4655
Logic extended for exec command
kidrahahjo Jul 15, 2020
9a7ca7e
Logic extended for next command
kidrahahjo Jul 15, 2020
15b79cf
Logic for run command
kidrahahjo Jul 15, 2020
df99e50
Logic for script command added
kidrahahjo Jul 15, 2020
5f1e060
Logic for submit and status enabled
kidrahahjo Jul 16, 2020
0a33a27
Make tests pass, replaced by-job grouping with by-op grouping
kidrahahjo Jul 16, 2020
6a37920
Style Fix
kidrahahjo Jul 16, 2020
39b4a3c
Remove print statement
kidrahahjo Jul 16, 2020
7478f4d
Suggested changes implemented with refactoring of Aggregate class and…
kidrahahjo Jul 16, 2020
b5ba89f
Documentation Edit and implement suggested changes
kidrahahjo Jul 17, 2020
1b07f7b
Improve status fetching performance
kidrahahjo Jul 18, 2020
6da1cf1
Use get_id() instead of the id property
kidrahahjo Jul 18, 2020
eadb031
Remove the use of Aggregate class and refactored
kidrahahjo Jul 23, 2020
a2446e5
Resolve merge conflicts
kidrahahjo Jul 23, 2020
b937d46
Minute document change
kidrahahjo Jul 23, 2020
be56b51
Apply suggested changes
kidrahahjo Jul 26, 2020
f266987
Merge upstream/master
kidrahahjo Jul 26, 2020
a4f51e2
Deprecate eligible and complete methods of Flow{Group|Operation}
kidrahahjo Jul 27, 2020
729dca0
Add support with private class _JobOperation
kidrahahjo Jul 27, 2020
0fd78b5
Merge remote-tracking branch 'upstream/master' into enable-aggregate-…
kidrahahjo Jul 27, 2020
fab2f68
Move property method job to JobOperation only
kidrahahjo Jul 28, 2020
fcb29de
Change the submission ID for aggregate support
kidrahahjo Jul 29, 2020
abbd6d1
Indentation fix
kidrahahjo Jul 29, 2020
26928dc
Merge remote-tracking branch 'upstream/master' into enable-submission…
kidrahahjo Jul 30, 2020
14f9fa9
Merge remote-tracking branch 'upstream/master' into enable-aggregate-…
kidrahahjo Jul 30, 2020
c11f40b
Merge branch 'enable-aggregate-logic' into enable-submission-aggregat…
kidrahahjo Jul 30, 2020
8dc5a95
Apply suggested changes
kidrahahjo Jul 31, 2020
4efa480
Merge remote-tracking branch 'origin/enable-aggregate-logic' into ena…
kidrahahjo Jul 31, 2020
bd26aa8
Apply suggested changes
kidrahahjo Aug 3, 2020
39ec72b
Merge branch 'enable-aggregate-logic' into enable-submission-aggregat…
kidrahahjo Aug 3, 2020
702ae4e
Enable storing aggregate information while submission and fetch befor…
kidrahahjo Jul 29, 2020
05855af
Make a method for fetching aggregates
kidrahahjo Aug 3, 2020
fd2aaee
Implementation of _aggregate and _select classes
kidrahahjo Jun 30, 2020
857b54c
Tests added for aggregate and select class with refactoring
kidrahahjo Jun 30, 2020
c3154f1
Indentation fix
kidrahahjo Jun 30, 2020
2cca72a
Merged select and aggregate class into one aggregate class
kidrahahjo Jul 7, 2020
7e66463
Improved visual indentation
kidrahahjo Jul 7, 2020
355a9bf
Used autopep8 for indentation fix
kidrahahjo Jul 7, 2020
35e8295
Bad Lint
kidrahahjo Jul 8, 2020
3b41151
Aggregate class now also handles making of aggregated while calling
kidrahahjo Jul 9, 2020
c1e69cc
Minor edit
kidrahahjo Jul 9, 2020
760de94
Support for JobCursor added
kidrahahjo Jul 10, 2020
8a7f630
Style fix
kidrahahjo Jul 10, 2020
417ff18
Suggested changes made
kidrahahjo Jul 16, 2020
b720363
Minor change
kidrahahjo Jul 16, 2020
e460b70
Make a seperate function to create MakeAggregate instance
kidrahahjo Jul 23, 2020
441db26
Introduce Aggregator class in flow
kidrahahjo Aug 3, 2020
c300013
Refactor fetching aggregates
kidrahahjo Aug 3, 2020
810850a
Refactor status check
kidrahahjo Aug 3, 2020
2ff36c0
Style fix
kidrahahjo Aug 3, 2020
cc3d42a
Style fix
kidrahahjo Aug 4, 2020
fe17853
Merge branch 'feature/enable-aggregate-status-logic' into feature/int…
kidrahahjo Aug 4, 2020
1f3c268
Refactor status check for deleted aggregates
kidrahahjo Aug 4, 2020
024921a
Add make_aggregates method for signac-examples support
kidrahahjo Aug 6, 2020
86d784f
Change message of progress bar
kidrahahjo Aug 6, 2020
e64c09c
Add aggregate CLA for status which tells whether to fetch previously …
kidrahahjo Aug 6, 2020
b2e21b7
Refactor all the jinja templates for status print to support aggregation
kidrahahjo Aug 8, 2020
4322323
raise ValueError if num <=0
kidrahahjo Aug 8, 2020
1d5f6cc
Changes made
kidrahahjo Aug 12, 2020
a9d5b1a
Resolve merge conflicts
kidrahahjo Aug 12, 2020
d74b3a0
Enable storing aggregate information while submission and fetch befor…
kidrahahjo Jul 29, 2020
bc75900
Make a method for fetching aggregates
kidrahahjo Aug 3, 2020
f05cb70
Refactor fetching aggregates
kidrahahjo Aug 3, 2020
6c3c21b
Style fix
kidrahahjo Aug 4, 2020
339345f
Apply suggestions from code review
kidrahahjo Aug 4, 2020
9bbbcfc
Merge remote-tracking branch 'origin/feature/enable-submission-aggreg…
kidrahahjo Aug 12, 2020
d5e558c
Add PersmissionError check
kidrahahjo Aug 12, 2020
6598004
Apply suggested changes
kidrahahjo Aug 13, 2020
49007b6
Merge feature/enable-aggregation
kidrahahjo Aug 13, 2020
66d78eb
Fixed _fn_stored method
kidrahahjo Aug 13, 2020
f196cdd
Refactor _fetch_status
kidrahahjo Aug 13, 2020
e0d1483
Update this branch and add tests in test_project.py
kidrahahjo Aug 14, 2020
cfbc133
Add template tests
kidrahahjo Aug 14, 2020
5881a69
Refactor templates
kidrahahjo Aug 14, 2020
07dd725
Remove template test for aggregates
kidrahahjo Aug 14, 2020
e1f815a
Apply suggested change
kidrahahjo Aug 14, 2020
4cbe754
Generate aggregates on initialization
kidrahahjo Aug 17, 2020
5f4f09d
Change expected number of submit steps
kidrahahjo Aug 17, 2020
63d6810
Refactor creation of aggregates
kidrahahjo Aug 17, 2020
cd3def4
Unable static aggregation and some refactoring
kidrahahjo Aug 18, 2020
ec8a04b
Minor refactoring, apply suggested changes
kidrahahjo Aug 18, 2020
d16edeb
Merge feature/enable-aggregate-status-logic, Refactor hashing, cmd ar…
kidrahahjo Aug 18, 2020
ef70af7
Fix execution command
kidrahahjo Aug 18, 2020
ea4bccb
Parallelize status update completely, apply suggested change
kidrahahjo Aug 19, 2020
a76c5b2
Merge #335 into this branch, and refactor
kidrahahjo Aug 20, 2020
8925ad4
Fix issue with dummy operations
kidrahahjo Aug 20, 2020
8d4313b
Implement by-job order option
kidrahahjo Aug 20, 2020
35b8bda
use get_id instead of id
kidrahahjo Aug 20, 2020
97714c9
Implement by-job order option
kidrahahjo Aug 20, 2020
fbb9e15
Apply suggested change of merging if else block
kidrahahjo Aug 20, 2020
92b425b
Apply suggestions from code review and revert the deprecation of get_…
kidrahahjo Aug 21, 2020
ad2da19
Change the --aggregate CLI option for status check to --orphan and me…
kidrahahjo Aug 21, 2020
c7e463d
Apply suggestions from code review
bdice Nov 23, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Documentation Edit and implement suggested changes
kidrahahjo committed Jul 17, 2020
commit b5ba89fcc826d1267181b465ea9c44102f5a8197
19 changes: 10 additions & 9 deletions flow/aggregate.py
Original file line number Diff line number Diff line change
@@ -43,14 +43,14 @@ def aggregator(jobs):

return cls(aggregator)

def __call__(self, obj=None):
if callable(obj):
setattr(obj, '_flow_aggregate', _MakeAggregate(self))
return obj
def __call__(self, func=None):
if callable(func):
setattr(func, '_flow_aggregate', _MakeAggregate(self))
return func
else:
raise TypeError('Invalid argument passed while calling '
'the aggregate instance. Expected a callable, '
'got {}.'.format(type(obj)))
'got {}.'.format(type(func)))


class _MakeAggregate:
@@ -67,12 +67,13 @@ class _MakeAggregate:
def __init__(self, _aggregate=Aggregate()):
self._aggregate = _aggregate

def __call__(self, obj):
def __call__(self, list_of_jobs):
"Return aggregated jobs"
aggregated_jobs = list(obj)
aggregated_jobs = self._aggregate._aggregator([job for job in aggregated_jobs])
# Ensuring list
list_of_jobs = list(list_of_jobs)
aggregated_jobs = self._aggregate._aggregator([job for job in list_of_jobs])
aggregated_jobs = [[job for job in aggregate] for aggregate in aggregated_jobs]
if not len(aggregated_jobs):
if len(aggregated_jobs) == 0:
return []
for i, job in enumerate(aggregated_jobs[-1]):
if job is None:
Loading