Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New class hierarchy and better readiness checking #729

Merged
merged 106 commits into from
Jul 12, 2023
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
106 commits
Select commit Hold shift + click to select a range
3a71a5a
Only give the final warning
samwaseda Jun 20, 2023
fc59f88
Streamline strict naming flag
samwaseda Jun 20, 2023
b8c732a
Merge branch 'main' into is_nodal_mixin
liamhuber Jun 20, 2023
160b77c
Rolled back too far
liamhuber Jun 20, 2023
4179a71
Merge branch 'has_nodes_mixin' into is_nodal_mixin
liamhuber Jun 20, 2023
d524c3b
Merge branch 'has_nodes_mixin' into is_nodal_mixin
liamhuber Jun 20, 2023
c18f693
Merge remote-tracking branch 'origin/is_nodal_mixin' into is_nodal_mixin
liamhuber Jun 20, 2023
ae710f4
Merge branch 'main' into is_nodal_mixin
liamhuber Jun 21, 2023
76df650
Extract a new parent class for nodal objects and rebase node onto it
liamhuber Jun 21, 2023
b326d66
:bug: call super in has_nodes mixin
liamhuber Jun 21, 2023
96a6f74
Move initialization of signals up into the parent class
liamhuber Jun 21, 2023
98486ce
Rebase workflow onto IsNodal mixin
liamhuber Jun 21, 2023
bbe6988
Remove unused imports
liamhuber Jun 21, 2023
2a4effb
Format black
pyiron-runner Jun 21, 2023
b6d2cd3
Add module summary
liamhuber Jun 22, 2023
04d748e
Add docstrings
liamhuber Jun 22, 2023
b4b53d5
Merge branch 'main' into is_nodal_mixin
liamhuber Jun 22, 2023
05ac875
Make Workflow conform to IsNodal type hints for data IO
liamhuber Jun 22, 2023
b2e5b22
Pull run method up to the parent class
liamhuber Jun 22, 2023
0ccc490
Merge branch 'main' into is_nodal_mixin
liamhuber Jun 26, 2023
8bd563d
Merge branch 'main' into is_nodal_mixin
liamhuber Jun 26, 2023
1deee14
Finish merging Sam's with-self functionality
liamhuber Jun 26, 2023
1c62b71
Remove todo -- it is done in finish_run
liamhuber Jun 26, 2023
fe9b55e
Remove todo -- it is done in finish_run
liamhuber Jun 26, 2023
7cab7bf
Merge branch 'main' into is_nodal_mixin
liamhuber Jun 27, 2023
e52d0a2
Remove ununsed import
liamhuber Jun 27, 2023
7a1bc23
Move _working_directory up into the parent class
liamhuber Jun 27, 2023
91dc898
Refactor: pull parent attribute into base class
liamhuber Jun 27, 2023
2959fe7
Pull working_directory up into base class and allow nodes to have the…
liamhuber Jun 27, 2023
62d36bc
Oops, add the parent kwarg back to nodes
liamhuber Jun 27, 2023
5bbf4dd
Conform to and test the spec
liamhuber Jun 28, 2023
5a0d3e5
Modify node tests to conform to new spec
liamhuber Jun 28, 2023
738ac1b
Order imports alphabetically
liamhuber Jun 28, 2023
2654061
PEP8
liamhuber Jun 28, 2023
7e8827d
Update pyiron_contrib/workflow/workflow.py
liamhuber Jun 28, 2023
f774a13
Better clarify intent to future devs
liamhuber Jun 28, 2023
be0bdfa
PEP8 newline
liamhuber Jun 28, 2023
471ed49
Merge pull request #745 from pyiron/centralize_working_directory
liamhuber Jun 28, 2023
9c5d2e9
Make all nodes have to_dict
liamhuber Jun 28, 2023
0738112
Pull update up to the parent class
liamhuber Jun 28, 2023
0068072
Replace HasNodes mix-in behaviour with Composite inheritance behaviour
liamhuber Jun 28, 2023
f9b0448
Allow workflow to inherit on_run and update behaviour from Composite
liamhuber Jun 28, 2023
f727e74
Add run_on_updates docs to parent class
liamhuber Jun 28, 2023
856a152
Fix typo
liamhuber Jun 28, 2023
4cfaa8a
Allow composite nodes to specify what nodes should start runs
liamhuber Jun 28, 2023
8bf8ef3
Update docs
liamhuber Jun 28, 2023
285d13d
Remove variable that's not used any more
samwaseda Jun 28, 2023
ae7af12
Merge branch 'is_nodal_mixin' into composition_to_inheritance
liamhuber Jun 28, 2023
506735a
Update type hints to use base class
liamhuber Jun 28, 2023
03f6cef
Fix workflow tests
liamhuber Jun 28, 2023
6d05ff4
Fix some more hints
liamhuber Jun 28, 2023
aad1c1a
Fix some more hints
liamhuber Jun 28, 2023
b77952d
Rename Node to Function
liamhuber Jun 28, 2023
cf03a52
Rename FastNode to Fast
liamhuber Jun 28, 2023
8be1263
Rename SingleValueNode to SingleValue
liamhuber Jun 28, 2023
3751e2d
Update test class names
liamhuber Jun 28, 2023
1b049f7
Rename the decorator
liamhuber Jun 28, 2023
0a08a9f
Update docstrings
liamhuber Jun 28, 2023
2c44a10
Rename IsNodal to Node
liamhuber Jun 28, 2023
3fb23b3
Update the demo notebook
liamhuber Jun 28, 2023
eceb817
Format black
pyiron-runner Jun 28, 2023
2d3cf9b
Merge branch 'is_nodal_mixin' into fix_type_hints
liamhuber Jun 28, 2023
c9c4f08
Merge branch 'is_nodal_mixin' into composition_to_inheritance
liamhuber Jun 28, 2023
3aa26ac
Merge remote-tracking branch 'origin/composition_to_inheritance' into…
liamhuber Jun 28, 2023
50062d5
Merge branch 'fix_type_hints' into rename_classes
liamhuber Jun 28, 2023
ab8de4b
Black
liamhuber Jun 28, 2023
a98f4dd
Update docs
liamhuber Jun 28, 2023
f11424c
Introduce NotData as a default for data channels
liamhuber Jun 29, 2023
622a6a5
Update node tests that were expecting a None default
liamhuber Jun 29, 2023
bb4ff2f
Don't override the data channel default with None
liamhuber Jun 29, 2023
86173f6
Test more than just implementation
liamhuber Jun 29, 2023
17f9020
Update the example notebook
liamhuber Jun 29, 2023
b6d16b8
Update docstring
liamhuber Jun 29, 2023
3608cd6
Fix typo
liamhuber Jun 29, 2023
c85afdd
Make fast the default behaviour for function nodes
liamhuber Jun 29, 2023
d23a83e
Update the docstring
liamhuber Jun 29, 2023
76c5ab5
Remove unnecessary specification of the defaults
liamhuber Jun 29, 2023
cdd8cd9
Reparent SingleValue directly onto Function
liamhuber Jun 29, 2023
d36060c
Replace the Fast node with a Slow node now that fast is default
liamhuber Jun 29, 2023
4158fd3
Make atomistic calculation nodes slow
liamhuber Jun 29, 2023
03a011f
Expose the other functions on the node adder
liamhuber Jun 29, 2023
95eb3f5
Update the example notebook
liamhuber Jun 29, 2023
f825b67
Merge pull request #753 from pyiron/move_docstrings
samwaseda Jun 30, 2023
e44feab
Make NotData checking property private
Jun 30, 2023
357592e
Merge pull request #755 from pyiron/fast_by_default
liamhuber Jun 30, 2023
bdbbe4f
Merge pull request #754 from pyiron/data_channel_not_data
liamhuber Jun 30, 2023
6d897e9
Merge pull request #751 from pyiron/rename_classes
liamhuber Jun 30, 2023
454067b
Merge pull request #750 from pyiron/fix_type_hints
liamhuber Jun 30, 2023
24c62a1
Merge pull request #748 from pyiron/composition_to_inheritance
liamhuber Jun 30, 2023
82fdcfd
Format black
pyiron-runner Jun 30, 2023
ffd7a2b
Introduce running on an executor
liamhuber Jul 5, 2023
50efd4a
Reorder when and where the result processing/ran signal happens
liamhuber Jul 6, 2023
539eb89
Add an integration test for cyclic graphs
liamhuber Jul 6, 2023
a1efbd3
Clean the text output in integration test by avoiding 0th rand call
liamhuber Jul 7, 2023
9f00bf8
Update docs
liamhuber Jul 10, 2023
86623f7
Update pyiron_contrib/workflow/node.py
liamhuber Jul 10, 2023
045cc53
Update pyiron_contrib/workflow/node.py
liamhuber Jul 10, 2023
d0dac28
Update pyiron_contrib/workflow/node.py
liamhuber Jul 10, 2023
491d855
Merge pull request #761 from pyiron/order_things_for_circular
liamhuber Jul 11, 2023
57328b7
Merge branch 'main' into is_nodal_mixin
liamhuber Jul 12, 2023
7839d66
Merge branch 'is_nodal_mixin' into node_executor
liamhuber Jul 12, 2023
b488fa5
Use the cloudpickle executor
liamhuber Jul 12, 2023
44c9ffb
Give access to the creator right from composite
liamhuber Jul 12, 2023
d23f8d1
Test parallel process execution
liamhuber Jul 12, 2023
60c34f5
Update node docs
liamhuber Jul 12, 2023
9b5368b
Merge pull request #760 from pyiron/node_executor
liamhuber Jul 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 16 additions & 13 deletions pyiron_contrib/workflow/has_nodes.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ class HasNodes(ABC):
"""

def __init__(self, *args, strict_naming=True, **kwargs):
super().__init__(*args, **kwargs)
self.nodes: DotDict = DotDict()
self.add: NodeAdder = NodeAdder(self)
self.strict_naming: bool = strict_naming
Expand All @@ -39,16 +40,16 @@ def add_node(self, node: Node, label: Optional[str] = None) -> None:
raise TypeError(
f"Only new node instances may be added, but got {type(node)}."
)

label = self._ensure_label_is_unique(node.label if label is None else label)
self._ensure_node_has_no_other_parent(node, label)
self._ensure_node_has_no_other_parent(node)
label = self._get_unique_label(node.label if label is None else label)
self._ensure_node_is_not_duplicated(node, label)

self.nodes[label] = node
node.label = label
node.parent = self
return node

def _ensure_label_is_unique(self, label):
def _get_unique_label(self, label):
if label in self.__dir__():
if isinstance(getattr(self, label), Node):
if self.strict_naming:
Expand Down Expand Up @@ -78,23 +79,25 @@ def _add_suffix_to_label(self, label):
)
return new_label

def _ensure_node_has_no_other_parent(self, node: Node, label: str):
def _ensure_node_has_no_other_parent(self, node: Node):
if node.parent is not None and node.parent is not self:
raise ValueError(
f"The node ({node.label}) already belongs to the parent "
f"{node.parent.label}. Please remove it there before trying to "
f"add it to this parent ({self.label})."
)

def _ensure_node_is_not_duplicated(self, node: Node, label: str):
if (
node.parent is self # This should guarantee the node is in self.nodes
node.parent is self
and label != node.label
and self.nodes[node.label] is node
):
assert self.nodes[node.label] is node # Should be unreachable by users
warn(
f"Reassigning the node {node.label} to the label {label} when "
f"adding it to the parent {self.label}."
)
del self.nodes[node.label]
elif node.parent is not None:
raise ValueError(
f"The node ({node.label}) already belongs to the parent "
f"{node.parent.label}. Please remove it there before trying to "
f"add it to this parent ({self.label})."
)

def remove(self, node: Node | str):
if isinstance(node, Node):
Expand Down
81 changes: 81 additions & 0 deletions pyiron_contrib/workflow/is_nodal.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
from __future__ import annotations

from abc import ABC, abstractmethod
from typing import TYPE_CHECKING

from pyiron_contrib.workflow.io import Signals, InputSignal, OutputSignal

if TYPE_CHECKING:
from pyiron_base.jobs.job.extension.server.generic import Server

from pyiron_contrib.workflow.io import Inputs, Outputs


class IsNodal(ABC):
"""
A mixin class for classes that can represent nodes on a computation graph.
"""

def __init__(self, label: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.label: str = label
self.running = False
self.failed = False
# TODO: Replace running and failed with a state object
self._server: Server | None = (
None # Or "task_manager" or "executor" -- we'll see what's best
)
self.signals = self._build_signal_channels()

@property
@abstractmethod
def inputs(self) -> Inputs:
pass

@property
@abstractmethod
def outputs(self) -> Outputs:
pass

@abstractmethod
def update(self):
pass

@abstractmethod
def run(self):
pass

def _build_signal_channels(self) -> Signals:
signals = Signals()
signals.input.run = InputSignal("run", self, self.run)
signals.output.ran = OutputSignal("ran", self)
return signals

@property
def server(self) -> Server | None:
return self._server

@server.setter
def server(self, server: Server | None):
self._server = server

def disconnect(self):
self.inputs.disconnect()
self.outputs.disconnect()
self.signals.disconnect()

@property
def ready(self) -> bool:
return not (self.running or self.failed) and self.inputs.ready

@property
def connected(self) -> bool:
return self.inputs.connected or self.outputs.connected or self.signals.connected
samwaseda marked this conversation as resolved.
Show resolved Hide resolved

@property
def fully_connected(self):
return (
self.inputs.fully_connected
and self.outputs.fully_connected
and self.signals.fully_connected
)
66 changes: 20 additions & 46 deletions pyiron_contrib/workflow/node.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,17 @@
from functools import partialmethod
from typing import get_args, get_type_hints, Optional, TYPE_CHECKING

from pyiron_contrib.workflow.channels import (
InputData,
OutputData,
InputSignal,
OutputSignal,
)
from pyiron_contrib.workflow.channels import InputData, OutputData
from pyiron_contrib.workflow.has_channel import HasChannel
from pyiron_contrib.workflow.has_to_dict import HasToDict
from pyiron_contrib.workflow.io import Inputs, Outputs, Signals
from pyiron_contrib.workflow.is_nodal import IsNodal

if TYPE_CHECKING:
from pyiron_contrib.workflow.workflow import Workflow


class Node(HasToDict):
class Node(IsNodal, HasToDict):
"""
Nodes have input and output data channels that interface with the outside world, and
a callable that determines what they actually compute. After running, their output
Expand Down Expand Up @@ -323,28 +319,25 @@ def __init__(
parent: Optional[Workflow] = None,
**kwargs,
):
super().__init__(
label=label if label is not None else node_function.__name__,
# **kwargs,
)
self.parent = parent
if parent is not None:
parent.add(self)
if len(output_labels) == 0:
raise ValueError("Nodes must have at least one output label.")

self.running = False
self.failed = False
self.server = None # Or "task_manager" or "executor" -- we'll see what's best
self.node_function = node_function
self.label = label if label is not None else node_function.__name__

self.parent = None
if parent is not None:
parent.add(self)

input_channels = self._build_input_channels(input_storage_priority)
self.inputs = Inputs(*input_channels)
self._inputs = Inputs(*input_channels)

output_channels = self._build_output_channels(
*output_labels, storage_priority=output_storage_priority
)
self.outputs = Outputs(*output_channels)

self.signals = self._build_signal_channels()
self._outputs = Outputs(*output_channels)

self.channels_requiring_update_after_run = (
[]
Expand All @@ -364,6 +357,14 @@ def __init__(
if update_on_instantiation:
self.update()

@property
def inputs(self) -> Inputs:
return self._inputs

@property
def outputs(self) -> Outputs:
return self._outputs

def _build_input_channels(self, storage_priority: dict[str:int]):
channels = []
type_hints = get_type_hints(self.node_function)
Expand Down Expand Up @@ -451,12 +452,6 @@ def _build_output_channels(

return channels

def _build_signal_channels(self) -> Signals:
signals = Signals()
signals.input.run = InputSignal("run", self, self.run)
signals.output.ran = OutputSignal("ran", self)
return signals

def _verify_that_channels_requiring_update_all_exist(self):
if not all(
channel_name in self.inputs.labels
Expand Down Expand Up @@ -520,27 +515,6 @@ def process_output(self, function_output):
def __call__(self) -> None:
self.run()

def disconnect(self):
self.inputs.disconnect()
self.outputs.disconnect()
self.signals.disconnect()

@property
def ready(self) -> bool:
return not (self.running or self.failed) and self.inputs.ready

@property
def connected(self) -> bool:
return self.inputs.connected or self.outputs.connected or self.signals.connected

@property
def fully_connected(self):
return (
self.inputs.fully_connected
and self.outputs.fully_connected
and self.signals.fully_connected
)

def set_storage_priority(self, priority: int):
self.inputs.set_storage_priority(priority)
self.outputs.set_storage_priority(priority)
Expand Down
6 changes: 3 additions & 3 deletions pyiron_contrib/workflow/workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

from pyiron_contrib.workflow.has_nodes import HasNodes
from pyiron_contrib.workflow.has_to_dict import HasToDict
from pyiron_contrib.workflow.is_nodal import IsNodal
from pyiron_contrib.workflow.node import Node, node, fast_node, single_value_node
from pyiron_contrib.workflow.util import DotDict

Expand All @@ -14,7 +15,7 @@ class _NodeDecoratorAccess:
single_value_node = single_value_node


class Workflow(HasToDict, HasNodes):
class Workflow(IsNodal, HasToDict, HasNodes):
"""
Workflows are an abstraction for holding a collection of related nodes.

Expand Down Expand Up @@ -119,8 +120,7 @@ class Workflow(HasToDict, HasNodes):
wrap_as = _NodeDecoratorAccess

def __init__(self, label: str, *nodes: Node, strict_naming=True):
super().__init__(strict_naming=strict_naming)
self.label = label
super().__init__(label=label, strict_naming=strict_naming)

for node in nodes:
self.add_node(node)
Expand Down
10 changes: 8 additions & 2 deletions tests/unit/workflow/test_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,14 @@ def test_node_addition(self):
wf.add(Node(fnc, "x", label="foo"))
wf.add.Node(fnc, "y", label="bar")
wf.baz = Node(fnc, "y", label="whatever_baz_gets_used")
Node(fnc, "x", label="boa", parent=wf)
self.assertListEqual(list(wf.nodes.keys()), ["foo", "bar", "baz", "boa"])
Node(fnc, "x", label="qux", parent=wf)
self.assertListEqual(list(wf.nodes.keys()), ["foo", "bar", "baz", "qux"])
wf.boa = wf.qux
self.assertListEqual(
list(wf.nodes.keys()),
["foo", "bar", "baz", "boa"],
msg="Reassignment should remove the original instance"
)

wf.strict_naming = False
# Validate name incrementation
Expand Down