Code

Fix and improve map reduce#232

Code

Merged

AnesBenmerzoug merged 43 commits intodevelopaai-institute/pyDVL:developfrom fix-and-improve-map-reduceaai-institute/pyDVL:fix-and-improve-map-reduceCopy head branch name to clipboardJan 1, 2023

+471-336

Commits

Commits on Dec 9, 2022

• Add SequentialParallelBackend to run tasks without parallelism

  AnesBenmerzoug

  committed
  37f84d2

  Copy full SHA for 37f84d2

Commits on Dec 11, 2022

• Fix map_reduce test for non chunkified inputs

  AnesBenmerzoug

  committed
  949a3bf

  Copy full SHA for 949a3bf

• Use expit from scipy to avoid overflow warnings

  AnesBenmerzoug

  committed
  74c4cdd

  Copy full SHA for 74c4cdd

• Remove the chunkify_inputs argument of MapReduceJob

  Show description for 58d2149
  AnesBenmerzoug

  committed
  58d2149

  Copy full SHA for 58d2149

• Rename num_chunks to n_chunks for consistency

  AnesBenmerzoug

  committed
  9a1ad1a

  Copy full SHA for 9a1ad1a

• Remove all optional arguments from MapReduceJob's __call__ method

  AnesBenmerzoug

  committed
  80dec53

  Copy full SHA for 80dec53

• Call parallel backend inside MapReduceJob when a Utility or sequence thereof is passed

  AnesBenmerzoug

  committed
  31be0b6

  Copy full SHA for 31be0b6

• Rename num_workers in ParallelConfig to n_local_workers

  AnesBenmerzoug

  committed
  7c72d2a

  Copy full SHA for 7c72d2a

• Add a __repr__ to ValuationResults

  AnesBenmerzoug

  committed
  219403f

  Copy full SHA for 219403f

• If n_runs >= n_jobs, do not chunkify inputs

  AnesBenmerzoug

  committed
  9bc96d1

  Copy full SHA for 9bc96d1

Commits on Dec 19, 2022

• Use older version of tox

  AnesBenmerzoug

  committed
  6fcdc5f

  Copy full SHA for 6fcdc5f

• Update changelog

  AnesBenmerzoug

  committed
  0c2454d

  Copy full SHA for 0c2454d

• Apply suggestions from code review

  Show description for c77a474
  

  AnesBenmerzoug

  and

  mdbenito

  authored
  c77a474

  Copy full SHA for c77a474

• Set explicit type for result of expit

  AnesBenmerzoug

  committed
  4aa6450

  Copy full SHA for 4aa6450

• Explicitly use RayParallelBackend in RayActorWrapper's docstring example

  AnesBenmerzoug

  committed
  8a15206

  Copy full SHA for 8a15206

• Add support for sequential parallel backend in get_shapley_coordinator and get_shapley_coordinator

  AnesBenmerzoug

  committed
  2f4be67

  Copy full SHA for 2f4be67

• Use singledispatchmethod and singledispatch decorators in map_reduce

  AnesBenmerzoug

  committed
  a75a4c9

  Copy full SHA for a75a4c9

Commits on Dec 20, 2022

• Use Any as type of default _get_value function

  AnesBenmerzoug

  committed
  9e68790

  Copy full SHA for 9e68790

• Use Iterable from typing for ReduceFunction type

  AnesBenmerzoug

  committed
  89b67b3

  Copy full SHA for 89b67b3

• Fixes

  AnesBenmerzoug

  committed
  cd1ced5

  Copy full SHA for cd1ced5

• Another fix

  AnesBenmerzoug

  committed
  dc491ec

  Copy full SHA for dc491ec

Commits on Dec 21, 2022

• Set ignore_reinit_error to True when using local ray cluster

  AnesBenmerzoug

  committed
  d06389f

  Copy full SHA for d06389f

• Fix check in RayParallelBackend's init

  AnesBenmerzoug

  committed
  f35de7a

  Copy full SHA for f35de7a

Commits on Dec 28, 2022

• Do not expose available_cpus at the package level

  AnesBenmerzoug

  committed
  7d25f14

  Copy full SHA for 7d25f14

• Disallow instantiating parallel backend classes directly

  AnesBenmerzoug

  committed
  8ed2806

  Copy full SHA for 8ed2806

• Remove unused import

  AnesBenmerzoug

  committed
  cbe1ca9

  Copy full SHA for cbe1ca9

• Fix RayActorWrapper's docstring

  AnesBenmerzoug

  committed
  9cdcde8

  Copy full SHA for 9cdcde8

• Pass inputs to MapReduceJob at initialization

  AnesBenmerzoug

  committed
  fd8f6bd

  Copy full SHA for fd8f6bd

Commits on Dec 29, 2022

• Apply suggestions from code review

  Show description for 0ce2d59
  

  AnesBenmerzoug

  and

  mdbenito

  authored
  0ce2d59

  Copy full SHA for 0ce2d59

• Fix indentation error

  AnesBenmerzoug

  committed
  59534d6

  Copy full SHA for 59534d6

• Improve effective_n_jobs interface

  AnesBenmerzoug

  committed
  ff247df

  Copy full SHA for ff247df

• Set default n_jobs in shapley methods to 1

  AnesBenmerzoug

  committed
  bbaa07b

  Copy full SHA for bbaa07b

• Call put on map_kwargs' and reduce_kwargs' items inside MapReduceJob's __init__

  AnesBenmerzoug

  committed
  4dffe03

  Copy full SHA for 4dffe03

• Update src/pydvl/utils/parallel/backend.py

  Show description for e9ccbc2
  

  AnesBenmerzoug

  and

  mdbenito

  authored
  e9ccbc2

  Copy full SHA for e9ccbc2

• Remove usage of singledispatchmethod in MapReduceJob

  AnesBenmerzoug

  committed
  fe2bab6

  Copy full SHA for fe2bab6

• Fix type hint

  AnesBenmerzoug

  committed
  f8e3aba

  Copy full SHA for f8e3aba

Commits on Dec 30, 2022

• Call put() on chunks inside _chunkify

  AnesBenmerzoug

  committed
  0cd9945

  Copy full SHA for 0cd9945

• Remove n_runs from MapReduceJobs, fix bug in _chunkify when passing numpy arrays

  AnesBenmerzoug

  committed
  12cfb0c

  Copy full SHA for 12cfb0c

• Set default value of max_parallel_tasks to None, add tests for _get_value and using partial instead of map_kwargs and reduce_kwargs

  AnesBenmerzoug

  committed
  03c690f

  Copy full SHA for 03c690f

• Merge branch 'develop' into fix-and-improve-map-reduce

  AnesBenmerzoug

  committed
  f698124

  Copy full SHA for f698124

• Update changelog

  AnesBenmerzoug

  committed
  faa582e

  Copy full SHA for faa582e

• Update import path for available_cpus helper function

  AnesBenmerzoug

  committed
  cdabd2f

  Copy full SHA for cdabd2f

Commits on Jan 1, 2023

• Apply suggestions from code review

  Show description for cdfe9eb
  

  AnesBenmerzoug

  and

  mdbenito

  authored
  cdfe9eb

  Copy full SHA for cdfe9eb