Code

Add TaskState to worker.py#4107

Code

Merged

mrocklin merged 91 commits intodask:masterdask/distributed:masterfrom gforsyth:taskstategforsyth/distributed:taskstateCopy head branch name to clipboardOct 15, 2020

+648-704

Commits

Commits on Sep 10, 2020

• Add TaskState for some (but not all) relevant dicts

  Show description for 1eddc52
  gforsyth

  committed
  1eddc52

  Copy full SHA for 1eddc52

• Fixup some more task_state

  gforsyth

  committed
  eb0d87d

  Copy full SHA for eb0d87d

• Fix up validation checks

  gforsyth

  committed
  f4ae9ac

  Copy full SHA for f4ae9ac

• Make sure key exists

  Show description for 7a36655
  gforsyth

  committed
  7a36655

  Copy full SHA for 7a36655

• Fix up some missing things and fix a few tests

  gforsyth

  committed
  59f2ce2

  Copy full SHA for 59f2ce2

Commits on Sep 14, 2020

• Fix up a bunch of failing tests

  gforsyth

  committed
  5e49258

  Copy full SHA for 5e49258

• First cut at removing dep_state, dependencies and dependents dicts

  Show description for 604f23c
  gforsyth

  committed
  604f23c

  Copy full SHA for 604f23c

• Add new attributes to init

  Show description for 3ece01f
  gforsyth

  committed
  3ece01f

  Copy full SHA for 3ece01f

• who_has should be a set

  gforsyth

  committed
  fff2281

  Copy full SHA for fff2281

Commits on Sep 16, 2020

• Check key not instance

  gforsyth

  committed
  a7af14d

  Copy full SHA for a7af14d

• deadlocking

  gforsyth

  committed
  a5d08ff

  Copy full SHA for a5d08ff

Commits on Sep 17, 2020

• Abandon parent language, just makes it more confusing

  gforsyth

  committed
  69bac91

  Copy full SHA for 69bac91

• Add waiting_for_data to TaskState

  gforsyth

  committed
  e380517

  Copy full SHA for e380517

• Fix up state checks for waiting_for_data

  gforsyth

  committed
  a35bddc

  Copy full SHA for a35bddc

• Deps -> memory working (but a mess)

  gforsyth

  committed
  1180f71

  Copy full SHA for 1180f71

• Remove breakpoints

  gforsyth

  committed
  2f05334

  Copy full SHA for 2f05334

• Remove old notes

  gforsyth

  committed
  81150e5

  Copy full SHA for 81150e5

• Discard instead of is in then remove

  gforsyth

  committed
  249b196

  Copy full SHA for 249b196

Commits on Sep 18, 2020

• Remove attribute from Worker repr

  gforsyth

  committed
  2012784

  Copy full SHA for 2012784

• Remove some commented out code and remove "parent" language

  gforsyth

  committed
  e441e56

  Copy full SHA for e441e56

• Set state in fewer places, handle dep releases a little better

  gforsyth

  committed
  0486146

  Copy full SHA for 0486146

Commits on Sep 19, 2020

• Fix assertion in key_waiting

  gforsyth

  committed
  b345125

  Copy full SHA for b345125

Commits on Sep 21, 2020

• Move some transitions into _transitions

  gforsyth

  committed
  2bf3259

  Copy full SHA for 2bf3259

• Clean up suspicious deps in release_key

  gforsyth

  committed
  112823f

  Copy full SHA for 112823f

• Remove data if no dependents left

  gforsyth

  committed
  dc725bb

  Copy full SHA for dc725bb

• Pop dependencies out of TS when transition->done

  gforsyth

  committed
  ef4eea4

  Copy full SHA for ef4eea4

• Clear Task dependencies when transition -> done

  gforsyth

  committed
  6156e9c

  Copy full SHA for 6156e9c

• Swap out all dep-specific method calls

  gforsyth

  committed
  a27ed18

  Copy full SHA for a27ed18

• More explicit bookkeeping for dependency keys

  gforsyth

  committed
  d0ffad2

  Copy full SHA for d0ffad2

• Remove unneeded xfail

  gforsyth

  committed
  b98b199

  Copy full SHA for b98b199

• Smol cleanup

  gforsyth

  committed
  f77b33c

  Copy full SHA for f77b33c

• black

  gforsyth

  committed
  a9b2ff9

  Copy full SHA for a9b2ff9

• Merge _dep_transitions and _transitions and remove _dep checks

  Show description for 9f7cf89
  gforsyth

  committed
  9f7cf89

  Copy full SHA for 9f7cf89

• Add flight -> ready entry in `transitions

  Show description for 2ae9a87
  gforsyth

  committed
  2ae9a87

  Copy full SHA for 2ae9a87

• Fix initial state for tasks coming off of ready heap

  gforsyth

  committed
  9446af9

  Copy full SHA for 9446af9

Commits on Sep 22, 2020

• Add Worker-level count of waiting for data tasks

  Show description for aee4bae
  gforsyth

  committed
  aee4bae

  Copy full SHA for aee4bae

• Test fix

  gforsyth

  committed
  ff05783

  Copy full SHA for ff05783

• remove leftover breakpoint

  gforsyth

  committed
  0b587d6

  Copy full SHA for 0b587d6

• Set state to waiting, handle direct calls to put_key_in_memory

  gforsyth

  committed
  eda1bde

  Copy full SHA for eda1bde

• Update failed worker tests

  gforsyth

  committed
  4f47d02

  Copy full SHA for 4f47d02

• black again

  gforsyth

  committed
  e1dac2f

  Copy full SHA for e1dac2f

• Sloppy fix for test_broken_worker_during_computation

  gforsyth

  committed
  64f2af3

  Copy full SHA for 64f2af3

• Restore and tweak validate_state

  gforsyth

  committed
  4d6f191

  Copy full SHA for 4d6f191

• Remove unneccessary checks

  Show description for c5eaf37
  gforsyth

  committed
  c5eaf37

  Copy full SHA for c5eaf37

• Changes to validate_state

  Show description for 4f94290
  gforsyth

  committed
  4f94290

  Copy full SHA for 4f94290

• Don't overwork the garbage collector

  gforsyth

  committed
  28e5694

  Copy full SHA for 28e5694

• Remove old skip

  gforsyth

  committed
  5ab969c

  Copy full SHA for 5ab969c

• Merge branch 'master' into taskstate

  gforsyth

  committed
  b32629c

  Copy full SHA for b32629c

Commits on Sep 23, 2020

• Update tests to not call put_key_in_memory directly

  Show description for d8ce2d0
  gforsyth

  committed
  d8ce2d0

  Copy full SHA for d8ce2d0

• continue if there are missing tasks on ready

  Show description for c3c1110
  gforsyth

  committed
  c3c1110

  Copy full SHA for c3c1110

Commits on Sep 24, 2020

• Use the task key

  gforsyth

  committed
  de950fd

  Copy full SHA for de950fd

• Fixes for missing deps / tasks associated with missing workers

  gforsyth

  committed
  24e6ed4

  Copy full SHA for 24e6ed4

• Add ready -> waiting transition

  Show description for 44d761c
  gforsyth

  committed
  44d761c

  Copy full SHA for 44d761c

Commits on Sep 25, 2020

• Transition and validate TaskState instead of keys

  gforsyth

  committed
  ad3c248

  Copy full SHA for ad3c248

• Don't transition from ready -> waiting

  gforsyth

  committed
  046360e

  Copy full SHA for 046360e

• Put key in memory takes TaskState

  gforsyth

  committed
  4550780

  Copy full SHA for 4550780

• Maybe an extraneous transition?

  gforsyth

  committed
  8f1892b

  Copy full SHA for 8f1892b

• Send task state in callback, not key

  gforsyth

  committed
  3a24aa8

  Copy full SHA for 3a24aa8

• Handle missing deps with TaskState

  Show description for 24108f2
  gforsyth

  committed
  24108f2

  Copy full SHA for 24108f2

• Remove a few unnecessary key lookups

  gforsyth

  committed
  887081e

  Copy full SHA for 887081e

• Add comment about data check in add_task

  gforsyth

  committed
  504a51f

  Copy full SHA for 504a51f

Commits on Sep 28, 2020

• Start adding docstring for TaskState class

  gforsyth

  committed
  7b85334

  Copy full SHA for 7b85334

Commits on Sep 29, 2020

• Fix-up release-key to avoid extra in add_task

  gforsyth

  committed
  fbf776a

  Copy full SHA for fbf776a

• Add type_ to TaskState

  Show description for 349fe8e
  gforsyth

  committed
  349fe8e

  Copy full SHA for 349fe8e

• Move suspicious count to TaskState

  gforsyth

  committed
  0cc88f8

  Copy full SHA for 0cc88f8

• Test fix and remove unneeded check

  gforsyth

  committed
  992fbbd

  Copy full SHA for 992fbbd

• Move startstops to TaskState

  gforsyth

  committed
  b60d4e5

  Copy full SHA for b60d4e5

• Remove some older comments

  gforsyth

  committed
  ced3fd4

  Copy full SHA for ced3fd4

• Check if state is waiting

  gforsyth

  committed
  2ef35ff

  Copy full SHA for 2ef35ff

• Add TaskState to api docs

  gforsyth

  committed
  f67353b

  Copy full SHA for f67353b

• Restore ts check in transition

  Show description for 173c28d
  gforsyth

  committed
  173c28d

  Copy full SHA for 173c28d

Commits on Sep 30, 2020

• Restore self.executing check

  gforsyth

  committed
  44e4404

  Copy full SHA for 44e4404

• Use clear instead of creating a new empty set

  gforsyth

  committed
  3ad8972

  Copy full SHA for 3ad8972

• Format docstring for gather_dep

  gforsyth

  committed
  1a2c591

  Copy full SHA for 1a2c591

• Change horrible variable name

  gforsyth

  committed
  5c4aa90

  Copy full SHA for 5c4aa90

• Drop assertion in flight -> memory transition

  gforsyth

  committed
  1a3b189

  Copy full SHA for 1a3b189

• Remove extra return and update dep check

  gforsyth

  committed
  c2ae508

  Copy full SHA for c2ae508

• Make task existence check more explicit

  gforsyth

  committed
  5aaac47

  Copy full SHA for 5aaac47

• Add validate check for "flight"

  Show description for fde532c
  gforsyth

  committed
  fde532c

  Copy full SHA for fde532c

• Fix counter for waiting_for_data and use clear instead of new set

  gforsyth

  committed
  44135db

  Copy full SHA for 44135db

Commits on Oct 9, 2020

• Add docstring, change var name for waiting task count

  Show description for 5135c1c
  gforsyth

  committed
  5135c1c

  Copy full SHA for 5135c1c

• Apply suggestions from code review

  Show description for f24365e
  

  gforsyth

  and

  jrbourbeau

  authored
  f24365e

  Copy full SHA for f24365e

• Couple more waiting_for_data -> waiting_for_data_count

  jrbourbeau

  committed
  ccfd9eb

  Copy full SHA for ccfd9eb

Commits on Oct 13, 2020

• Remove inadvisable re-addition of tasks to self.executing

  gforsyth

  committed
  0877311

  Copy full SHA for 0877311

• Replace self.executing with self.executing_count

  Show description for f49fbe8
  gforsyth

  committed
  f49fbe8

  Copy full SHA for f49fbe8

• Change type_ to type

  gforsyth

  committed
  508927f

  Copy full SHA for 508927f

• Fixup Worker.executing_count in tests and dashboard

  gforsyth

  committed
  abce5f6

  Copy full SHA for abce5f6

• Fix test

  gforsyth

  committed
  6f9e6a9

  Copy full SHA for 6f9e6a9

• Test fix?

  gforsyth

  committed
  d15e443

  Copy full SHA for d15e443

Commits on Oct 15, 2020

• Change out in_flight_tasks dict for counter

  gforsyth

  committed
  1e7de67

  Copy full SHA for 1e7de67

• Update distributed/diagnostics/tests/test_worker_plugin.py

  Show description for 403ca03
  

  gforsyth

  and

  jrbourbeau

  authored
  403ca03

  Copy full SHA for 403ca03