Fix: poll DHT in background when worker runs up a workflow + dual-stack webserver #590

zeeshanlakhani · 2024-02-28T19:18:25Z

Completes:

Includes:

We no longer block DHT polling of receipts, and, instead, purposefully race execution and DHT finding/polling.
This moves us toward the goal of resolving outside-of-workflow awaited (CIDs), but we'll need another background process to re-run a workflow once known-awaits are resolved (separate ticket). The next set of work will also validate that those outside-of-workflow promises exist, i.e. currently running in another workflow or have already run.
v4/v6 host address settings for the Json-RPC web server so that the web server can handle dual v4/v6 requests on the same port.
Better / ordered error handling around CIDs that fail to resolve.
Workflow status field in the DB (+ migration) (SQLite enum, with special patching).
Workflow retries field in the DB (in same migration).
A new poller implementation that's general purpose.
Removes async_trait except for wasmtime impls (due to 1.73).

…ck webserver Closes #278. Closes #589. Includes: * We no longer block on DHT polling of receipts, and, instead, purposefully race execution and DHT finding/polling. * This moves us toward the goal of resolving outside-of-workflow awaited (cids), but we'll need another background process to re-run a workflow once known-awaits are resolved (separte ticket). The next set of work will also validate that those outside-of-workflow promises exist, i.e. currently running in another workflow or have already run. * v4/v6 host addr settings for the Json-RPC webserver so that the webserver can handle dual v4/v6 requests on the same port. * Better / ordered error handling around Cids that fail to resolve. * Workflow status field in the DB (+ migration) (sqlite enum, with special patching). * Workflow retries field in the DB (in same migration). * A new poller implementation that's general purpose.

codecov · 2024-02-28T19:30:43Z

Codecov Report

Attention: Patch coverage is 83.17757% with 72 lines in your changes are missing coverage. Please review.

Project coverage is 72.42%. Comparing base (81749c7) to head (3d0bb28).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #590      +/-   ##
==========================================
+ Coverage   70.55%   72.42%   +1.87%     
==========================================
  Files          92       95       +3     
  Lines       12311    12473     +162     
==========================================
+ Hits         8686     9034     +348     
+ Misses       3625     3439     -186

Files	Coverage Δ
homestar-invocation/src/receipt.rs	`55.94% <ø> (ø)`
homestar-runtime/src/db.rs	`88.60% <100.00%> (+1.43%)`	⬆️
homestar-runtime/src/db/schema.rs	`90.00% <100.00%> (+0.71%)`	⬆️
homestar-runtime/src/event_handler.rs	`100.00% <ø> (ø)`
homestar-runtime/src/event_handler/event.rs	`67.42% <ø> (+22.24%)`	⬆️
homestar-runtime/src/runner.rs	`92.55% <100.00%> (ø)`
homestar-runtime/src/runner/response.rs	`69.67% <100.00%> (ø)`
homestar-runtime/src/settings.rs	`98.18% <100.00%> (+<0.01%)`	⬆️
homestar-runtime/src/settings/libp2p_config.rs	`100.00% <100.00%> (ø)`
homestar-runtime/src/tasks.rs	`100.00% <ø> (ø)`
... and 13 more

... and 3 files with indirect coverage changes

homestar-runtime/src/event_handler/swarm_event.rs

homestar-runtime/src/scheduler.rs

homestar-runtime/src/workflow.rs

homestar-runtime/src/worker.rs

homestar-runtime/src/worker/resolver.rs

homestar-runtime/tests/network/dht.rs

bgins

Looks good! One test was a bit flaky for me, left a note about it.

homestar-runtime/src/workflow.rs

homestar-runtime/src/worker/poller.rs

bgins · 2024-02-28T22:54:10Z

homestar-runtime/src/workflow/info.rs

+#[derive(Debug, Clone, PartialEq, diesel_derive_enum::DbEnum)]
+pub enum Status {
+    /// Workflow is pending - default case.
+    Pending,
+    /// Workflow is currently running.
+    Running,
+    /// Workflow has been completed.
+    Completed,
+    /// Workflow is stuck, awaiting CIDs we can't find on the network.
+    Stuck,
+}


Nice, great to have this. 💯

## 🤖 New release * `homestar-runtime`: 0.2.0 -> 0.3.0 (⚠️ API breaking changes) * `homestar-invocation`: 0.2.0 -> 0.3.0 (⚠️ API breaking changes) * `homestar-wasm`: 0.2.0 -> 0.3.0 (⚠️ API breaking changes) * `homestar-workflow`: 0.2.0 -> 0.3.0 (✓ API compatible changes) ### ⚠️ `homestar-runtime` breaking changes ``` --- failure enum_variant_added: enum variant added on exhaustive enum --- Description: A publicly-visible enum without #[non_exhaustive] has a new variant. ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.29.1/src/lints/enum_variant_added.ron Failed in: variant Command:Init in /tmp/.tmpgzntbb/homestar/homestar-runtime/src/cli.rs:152 ``` ### ⚠️ `homestar-invocation` breaking changes ``` --- failure enum_tuple_variant_field_added: pub enum tuple variant field added --- Description: An enum's exhaustive tuple variant has a new field, which has to be included when constructing or matching on this variant. ref: https://doc.rust-lang.org/reference/attributes/type_system.html#the-non_exhaustive-attribute impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.29.1/src/lints/enum_tuple_variant_field_added.ron Failed in: field 1 of variant Nonce::Nonce96 in /tmp/.tmpgzntbb/homestar/homestar-invocation/src/task/instruction/nonce.rs:39 field 1 of variant Nonce::Nonce128 in /tmp/.tmpgzntbb/homestar/homestar-invocation/src/task/instruction/nonce.rs:41 --- failure enum_variant_added: enum variant added on exhaustive enum --- Description: A publicly-visible enum without #[non_exhaustive] has a new variant. ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.29.1/src/lints/enum_variant_added.ron Failed in: variant Error:FromMultibase in /tmp/.tmpgzntbb/homestar/homestar-invocation/src/error.rs:38 variant Error:FromMultibase in /tmp/.tmpgzntbb/homestar/homestar-invocation/src/error.rs:38 ``` ### ⚠️ `homestar-wasm` breaking changes ``` --- failure auto_trait_impl_removed: auto trait no longer implemented --- Description: A public type has stopped implementing one or more auto traits. This can break downstream code that depends on the traits being implemented. ref: https://doc.rust-lang.org/reference/special-types-and-traits.html#auto-traits impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.29.1/src/lints/auto_trait_impl_removed.ron Failed in: type State is no longer Sync, in /tmp/.tmpgzntbb/homestar/homestar-wasm/src/wasmtime/world.rs:41 type State is no longer Sync, in /tmp/.tmpgzntbb/homestar/homestar-wasm/src/wasmtime/world.rs:41 --- failure enum_variant_missing: pub enum variant removed or renamed --- Description: A publicly-visible enum has at least one variant that is no longer available under its prior name. It may have been renamed or removed entirely. ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.29.1/src/lints/enum_variant_missing.ron Failed in: variant InterpreterError::MapType, previously in file /tmp/.tmpEeUFOH/homestar-wasm/src/error.rs:36 ``` <details><summary>Changelog</summary> ## `homestar-runtime` <blockquote> ## [0.3.0](homestar-runtime-v0.2.0...homestar-runtime-v0.3.0) - 2024-03-13 ### Added - log the creation of the key file in `init` - default `key-file` path to output directory - support generating PEM encoded ed25519 keys from `init` - translate `InquireError` to `miette::Result` in `init` - recursively create config directory on `init` - output a cleaner error when an invalid seed is passed to `init` - support configuring key using `init` command - run `init` command non-interactively if a TTY isn't detected - add `--no-input` to `init` command - add `--force` to `init` command for forcing destructive operations - add `--quiet` flag to `init` command - support `--dry-run` for `init` command - add `init` command for initializing a settings file - load settings from a well-known config file - finish interpreter ([#595](#595)) ### Fixed - cleanup empty key file when bailing out of generating secp256k1 - parse ed25519 keys using the old strategy as a fallback - use `ed25519-dalek` for parsing PEM-encoded PKCS[#8](#8) ed25519 keys - only constrain `inquire` and `derive_builder` by minor version - hide `IpfsSettings` behind "ipfs" feature - set `truncate(true)` when forcefully overwriting the config - Update nonce schema with IPLD bytes ([#593](#593)) ### Other - Add workflow spans and every cli logging ([#603](#603)) - *(schemas)* update OpenRPC API doc and JSON schemas - handle nonce as incoming string/arraybuf ([#611](#611)) - [chore(cargo)](deps): Bump toml from 0.8.10 to 0.8.11 ([#612](#612)) - document that a random seed will be chosen if `key-seed` is unset - document that if unset, a default path is used with `key-file` - update help text for `key-file` to say it'll generate a key - prompt for the key file as a `String` instead of `PathBuf` - add a test for writing the generated config file + key - split `force` field out of `OutputMode::File` - remove `KeyTypeArg` in favor of using `KeyType` - remove unneeded `defaults.toml` - add simple tests for `init` command - remove out of date TODO in `init.rs` - remove unneeded `#[allow(dead_code)]` in `settings.rs` - wrap all `init` args in `InitArgs` and consolidate handling - sort imports in `cli/init.rs` - remove docs link to private `homestar_runtime::db::pool` - improve error for passing `--no-input` to `init` with no key - remove extraneous `...` destructuring of `Command::Init` - change `--config` flag to `--output` for `init` command - move handling of `init` command to `init.rs` - fix comments listing supported public key types - *(schemas)* update OpenRPC API doc and JSON schemas - *(schemas)* update OpenRPC API doc and JSON schemas - poll DHT in background when worker runs up a workflow + dual-stack webserver ([#590](#590)) - [chore(cargo)](deps): Bump config from 0.13.4 to 0.14.0 ([#588](#588)) - [chore(cargo)](deps): Bump nix from 0.27.1 to 0.28.0 ([#587](#587)) </blockquote> ## `homestar-invocation` <blockquote> ## [0.3.0](homestar-invocation-v0.2.0...homestar-invocation-v0.3.0) - 2024-03-13 ### Added - finish interpreter ([#595](#595)) ### Fixed - Update nonce schema with IPLD bytes ([#593](#593)) ### Other - handle nonce as incoming string/arraybuf ([#611](#611)) - test json/ipld/nonce ([#610](#610)) - poll DHT in background when worker runs up a workflow + dual-stack webserver ([#590](#590)) </blockquote> ## `homestar-wasm` <blockquote> ## [0.3.0](homestar-wasm-v0.2.0...homestar-wasm-v0.3.0) - 2024-03-13 ### Added - finish interpreter ([#595](#595)) ### Other - Add workflow spans and every cli logging ([#603](#603)) - handle incoming as Wit integer, but argument is float ([#609](#609)) - handle ref/non-ref case with rigor ([#608](#608)) - doc interpreter ([#607](#607)) - poll DHT in background when worker runs up a workflow + dual-stack webserver ([#590](#590)) - wasmtime 17->18 ([#585](#585)) </blockquote> ## `homestar-workflow` <blockquote> ## [0.3.0](homestar-workflow-v0.2.0...homestar-workflow-v0.3.0) - 2024-03-13 ### Other - handle nonce as incoming string/arraybuf ([#611](#611)) </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/MarcoIeni/release-plz/). --------- Signed-off-by: release-plz-ipvm-wg[bot] <144082651+release-plz-ipvm-wg[bot]@users.noreply.github.com> Co-authored-by: release-plz-ipvm-wg[bot] <144082651+release-plz-ipvm-wg[bot]@users.noreply.github.com> Co-authored-by: release-plz-ipvm-wg[bot] <release-plz-ipvm-wg[bot]@users.noreply.github.com>

zeeshanlakhani requested a review from a team as a code owner February 28, 2024 19:18

zeeshanlakhani added 3 commits February 28, 2024 14:33

chore: ..

95ec228

chore: ..

7308e7f

chore: ..

1274d64