Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrap with dask only after dependency creation #7

Closed
SimonHeybrock opened this issue Jul 12, 2023 · 1 comment
Closed

Wrap with dask only after dependency creation #7

SimonHeybrock opened this issue Jul 12, 2023 · 1 comment

Comments

@SimonHeybrock
Copy link
Member

Currently we wrap callables in Delayed, before storing them as "providers". Before dropping injector (in #6) this was necessary. Now we find dependencies manually in Container, so we could delay tasks later. This would have some advantages:

  • Better type hints. Delayed is not generic yet, so currently internals of Delayed are all hinted with a plain Delayed.
  • No hack using @wraps instead of using dask.delayed directly.
  • More flexibility for doing things other than building a dask graph (without hacks). For example, we could build a custom visualization more easily.

To make this change, Container needs to switch to recursively building a dict of functions and their dependencies, without calling the functions directly. This may be similar to a task dict that dask uses for defining task graphs.

@SimonHeybrock
Copy link
Member Author

This is obsolete after recent changes/improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant