-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix and improve map reduce #232
Conversation
It was only used in one place and did not make sense in the context of parallelization
…thereof is passed
@mdbenito I have tried using |
Co-authored-by: Miguel de Benito Delgado <[email protected]>
…r and get_shapley_coordinator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking nice :) I still have my previous question about putting everything to the backend, though. And a few comments here and there
Co-authored-by: Miguel de Benito Delgado <[email protected]>
Co-authored-by: Miguel de Benito Delgado <[email protected]>
…alue and using partial instead of map_kwargs and reduce_kwargs
@mdbenito I have made the changes we talked about:
I also added two tests for |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this looks rather nice now :) I suggested a couple of cosmetic changes, ditching redundant elses, but other than that I'm 100% fine with merging. Go ahead!
if map_kwargs is None: | ||
self.map_kwargs = dict() | ||
else: | ||
self.map_kwargs = { | ||
k: self.parallel_backend.put(v) for k, v in map_kwargs.items() | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if map_kwargs is None: | |
self.map_kwargs = dict() | |
else: | |
self.map_kwargs = { | |
k: self.parallel_backend.put(v) for k, v in map_kwargs.items() | |
} | |
self.map_kwargs = { | |
k: self.parallel_backend.put(v) for k, v in (map_kwargs or {}).items() | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this makes it harder to read since this is not a common pattern.
if reduce_kwargs is None: | ||
self.reduce_kwargs = dict() | ||
else: | ||
self.reduce_kwargs = { | ||
k: self.parallel_backend.put(v) for k, v in reduce_kwargs.items() | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if reduce_kwargs is None: | |
self.reduce_kwargs = dict() | |
else: | |
self.reduce_kwargs = { | |
k: self.parallel_backend.put(v) for k, v in reduce_kwargs.items() | |
} | |
self.reduce_kwargs = { | |
k: self.parallel_backend.put(v) for k, v in (reduce_kwargs or {}).items() | |
} |
|
||
import numpy as np | ||
from numpy.typing import NDArray |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why don't we always directly import like this instead of using the if TYPING guard and then quoting all types? (that's a bit of a PITA, tbh)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it was a leftover from when we didn't pin the minimum numpy version to 1.20.
The typing module was added to numpy in version 1.20.
So, yes I think we should just import it directly.
Co-authored-by: Miguel de Benito Delgado <[email protected]>
Description
This PR closes #193 and closes #165
Changes
SequentialParallelBackend
class that runs all jobs in the current thread.BaseParallelBackend
abstract class.AbstractNoPublicConstructor
abstract metaclass to disallow directly instantiating parallel backends.MapReduceJob
at initialization.chunkify_inputs
parameter fromMapReduceJob
.n_runs
parameter fromMapReduceJob
.put()
method for each generated chunk in_chunkify()
.MapReduceJob
ifn_runs
>=n_jobs
.singledispatch
andsingledispatchmethod
to handle different input types inchunkify
and_get_value
.__repr__
method toValuationResult
.parallel_backend.put()
on the input toMapReduceJob
's__call__
method if it is a sequence ofUtility
objects.ParallelConfig
'snum_workers
attribute ton_local_workers
.MapReduceJob
's docstring.Checklist
"nbsphinx":"hidden"