Bnb/ddh norm #180
Bnb/ddh norm #180
Conversation
| else: | ||
| with ThreadPoolExecutor(max_workers=max_workers) as exe: | ||
| futures = {} | ||
| now = dt.now() | ||
| for idh, dh in enumerate(self.data_handlers): | ||
| future = exe.submit(dh.normalize, self.means, self.stds, | ||
| max_workers=1) | ||
| max_workers=dh.norm_workers) |
There was a problem hiding this comment.
Is there a specific reason you would want a worker to spin up more threads within a thread pool? Historically i've been very cautious of this. I don't want my threads/processes to be forking themselves.
There was a problem hiding this comment.
Yeah that's fair. Makes sense to go back to 1
| self._raw_data.pop(t) | ||
|
|
||
| def parallel_data_fill(self, shifted_time_chunks, max_workers=None): | ||
| def data_fill(self, shifted_time_chunks, max_workers=None): |
There was a problem hiding this comment.
Not a huge fan of this naming convention. I always find it confusing when there is a .method and a ._method. I think you're trying to hide the serial method from the "public" view but it makes the developer's job more confusing. I'd suggest _single_data_fill() and _data_fill()
There was a problem hiding this comment.
I renamed this just bc it includes the serial method also so its not strictly "parallel"
There was a problem hiding this comment.
Yeah understood but I think _single*() and _data_fill() would be the most clear, dont you think?
No description provided.