Skip to content

v0.3.5
Compare
Choose a tag to compare
@haixuanTao haixuanTao released this 03 Jul 16:46
· 578 commits to main since this release
v0.3.5
110ce99
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

dora 0.3.5 Release

Better Error Logging and Graceful Stopping

Before

018ffe79-1c0c-7b68-8886-d8cff079c232
  2024-06-09T19:28:22.877741Z ERROR dora_daemon::spawn: 018ffe79-1c0c-7b68-8886-d8cff079c232/webcam: 
Traceback (most recent call last):
  File "/home/peter/Documents/work/dora/examples/python-dataflow/webcam.py", line 11, in <module>
    node = Node()
RuntimeError: failed to init event stream

Caused by:
    subscribe failed: Some nodes exited before subscribing to dora: {NodeId("plot")}

    This is typically happens when an initialization error occurs
                    in the node or operator. To check the output of the failed
                    nodes, run `dora logs 018ffe79-1c0c-7b68-8886-d8cff079c232 plot`.

Location:
    apis/rust/node/src/event_stream/mod.rs:90:17

    at binaries/daemon/src/spawn.rs:371


  2024-06-09T19:28:23.263843Z ERROR dora_daemon: 
    018ffe79-1c0c-7b68-8886-d8cff079c232/object_detection failed with exit code 1.

    Check logs using: dora logs 018ffe79-1c0c-7b68-8886-d8cff079c232 object_detection
                            
    at binaries/daemon/src/lib.rs:1077

  2024-06-09T19:28:23.264003Z ERROR dora_coordinator: some nodes failed:
                            
  - webcam: 
    018ffe79-1c0c-7b68-8886-d8cff079c232/webcam failed with exit code 1.

    Check logs using: dora logs 018ffe79-1c0c-7b68-8886-d8cff079c232 webcam
                            

Location:
    /home/peter/Documents/work/dora/binaries/coordinator/src/listener.rs:90:56
    at binaries/coordinator/src/lib.rs:279

dataflow failed: errors occurred in dataflow 018ffe79-1c0c-7b68-8886-d8cff079c232:
- machine ``:
    some nodes failed:
      - webcam: 
        018ffe79-1c0c-7b68-8886-d8cff079c232/webcam failed with exit code 1.
    
        Check logs using: dora logs 018ffe79-1c0c-7b68-8886-d8cff079c232 webcam
                                
    
    Location:
        /home/peter/Documents/work/dora/binaries/coordinator/src/listener.rs:90:56

After

01907807-9461-738c-a503-c92b82e0d4c2
attaching to dataflow (use `--detach` to run in background)
ERROR webcam: exited with code 1 with stderr output:
---------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/peter/Documents/work/dora/examples/python-dataflow/webcam.py", line 12, in <module>
    assert False, "could not start dataflow"
AssertionError: could not start dataflow
---------------------------------------------------------------------------------

INFO  plot: node finished successfully
INFO  object_detection: node finished successfully


[ERROR]
Dataflow 01907807-9461-738c-a503-c92b82e0d4c2 failed:

Node `webcam` failed: exited with code 1 with stderr output:
---------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/peter/Documents/work/dora/examples/python-dataflow/webcam.py", line 12, in <module>
    assert False, "could not start dataflow"
AssertionError: could not start dataflow
---------------------------------------------------------------------------------

Dynamic Node

We're introducing dynamic node, to run node in a native way, whether it is by calling python, jupyter or cargo. All you have to do is specify that the node is dynamic in the dataflow and specify the node id within the node code and you will be able to run the node natively without using dora start.

Warning

We have noticed some bugs on MacOS/Windows with dynamic node: ##575

Getting Started

You can try minimal example:

cd examples/python-dataflow
dora start dataflow_dynamic.yml 
# In another terminal
python plot_dynamic.py
  • Then starting the dataflow
dora start dataflow.yaml
  • Then using one of the following method to start the node:

Using interactive python:

$ python
>>> from dora import Node
>>> node = Node("node_0")
>>> event = node.next()
>>> event
{'id': 'tick', 'type': 'INPUT', 'value': <pyarrow.lib.NullArray object at 0x7bda86924460>
0 nulls, 'kind': 'dora', 'metadata': {'open_telemetry_context': ''}}

Using Python

$ python my_script.py # <-- node = Node("node_0")
# Log will appear in your terminal

Using Rust

$ cargo run my_node # <-- DoraNode::init_from_node_id(node_id);
# Log will appear in your terminal

Using a jupyter notebook

python

Made all dora python object representable in python

All dora-rs now implements: __str__, __repr__, __dir__, __dict__

Meaning that from Python, you will always be able to represent dora object as a Python Object.

>>> node
Node()
>>> event = node.next()
>>> event
{'id': 'tick', 'type': 'INPUT', 'value': <pyarrow.lib.NullArray object at 0x7bda86924460>
0 nulls, 'kind': 'dora', 'metadata': {'open_telemetry_context': ''}}
>>>
>>> from dora import Ros2NodeOptions, Ros2QosPolicies
>>>
>>> policies = Ros2QosPolicies()
>>> policies
Ros2QosPolicies(durability=Ros2Durability.Volatile, liveliness=Ros2Liveliness.Automatic, lease_duration=inf, reliable=false, max_blocking_time=0.0, keep_all=false, keep_last=1)
>>> policies.__dict__
{'keep_all': False, 'keep_last': 1, 'liveliness': Ros2Liveliness.Automatic, 'durability': Ros2Durability.Volatile, 'reliable': False, 'max_blocking_time': 0.0, 'lease_duration': inf}

Thanks a million to @EricLBuehler!

Simplified the dataflow

We removed one level of hierarchy by removing custom that was referencing custom node.

We also use the keyword path instead of source that might be a bit too confusing.

This change is backward compatible and you can keep the state of your previous dataflow.

Before:

  - id: idefics2
    custom:
      source: ../nodes/idefics2_node_demo.py
      inputs:
        text: whisper/text_llm
      outputs:
        - speak
        - control

After:

  - id: idefics2
    path: ../nodes/idefics2_node_demo.py
    inputs:
      text: whisper/text_llm
    outputs:
      - speak
      - control

Enabled dora on multiple machines

We can now experiment dora on different machine!

You can getting started with our multi-daemon example

A very minimalist setup would look like this

# On machine A
dora coordinator
dora daemon --machine-id A

# On machine B
dora daemon --machine-id B --coordinator-addr <COORDINATOR_IP>:53290

# On machine A or B
dora start ./examples/multiple-daemons/dataflow.yml
nodes:
  - id: rust-node
    _unstable_deploy:
      machine: A
    path: abs/path/to/target/debug/rust-node
    inputs:
      tick: dora/timer/millis/10
    outputs:
      - random

  - id: rust-sink
    _unstable_deploy:
      machine: B
    path: abs/path/to/target/debug/rust-sink
    inputs:
      message: rust-node/random

Thanks a million to @Gege-Wang and @XxChang!

I would also thanks @Michael-J-Ward and @LyonRust for their help on this release as well!!!

Full Changelog

New Contributors

Full Changelog: v0.3.4...v0.3.5

Contributors

phil-opp, LeonRust, and 6 other contributors
Assets 17
Loading