Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compute: Move DHT retrieval of receipts to the background #278

Closed
zeeshanlakhani opened this issue Aug 29, 2023 · 0 comments · Fixed by #590
Closed

Compute: Move DHT retrieval of receipts to the background #278

zeeshanlakhani opened this issue Aug 29, 2023 · 0 comments · Fixed by #590
Assignees
@zeeshanlakhani
Labels
networking Features, functionality involving networking

Comments

@zeeshanlakhani
Copy link
Contributor

zeeshanlakhani commented Aug 29, 2023 •
edited
Loading

Summary

Currently, when a worker runs, we resolve promises linked to an instruction (which points to a receipt) by first looking in the local database and then, if that returns nothing, asking the "network"/DHT, i.e. libp2p, for the data within a deadline/timeout. If the receipt is not gathered, we then fully execute the task/wasm on the homestar node.

Instead of waiting for the libp2p get/check, we should default to executing inlined instructions/tasks simultaneously while also trying to resolve those promises within an entire workflow for later executions. Again, this only works for inlined instructions/tasks, not those that point to another workflow running elsewhere.

ETA: 2023-11-10
@zeeshanlakhani zeeshanlakhani added enhancement New feature or request and removed enhancement New feature or request labels Sep 26, 2023
@zeeshanlakhani zeeshanlakhani changed the title Run inlined promises that are executable simultaneously with libp2p check for receipt(s) Compute: Run inlined promises that are executable simultaneously with libp2p check for receipt(s) Sep 29, 2023
@expede expede mentioned this issue Oct 31, 2023
Closed
@zeeshanlakhani zeeshanlakhani self-assigned this Feb 7, 2024
@zeeshanlakhani zeeshanlakhani added networking Features, functionality involving networking and removed enhancement New feature or request labels Feb 8, 2024
@zeeshanlakhani zeeshanlakhani changed the title Compute: Run inlined promises that are executable simultaneously with libp2p check for receipt(s) Compute: Move DHT retrieval of receipts to the background Feb 8, 2024
zeeshanlakhani added a commit that referenced this issue Feb 28, 2024
@zeeshanlakhani
Fix: poll DHT in background when worker runs up a workflow + dual-sta…
9a30746 
…ck webserver

Closes #278.
Closes #589.

Includes:
    * We no longer block on DHT polling of receipts, and, instead, purposefully
      race execution and DHT finding/polling.
    * This moves us toward the goal of resolving outside-of-workflow awaited (cids),
      but we'll need another background process to re-run a workflow once
      known-awaits are resolved (separte ticket).
      The next set of work will also validate that those outside-of-workflow
      promises exist, i.e. currently running in another workflow or have already run.
    * v4/v6 host addr settings for the Json-RPC webserver so that the webserver
      can handle dual v4/v6 requests on the same port.
    * Better / ordered error handling around Cids that fail to resolve.
    * Workflow status field in the DB (+ migration) (sqlite enum, with special patching).
    * Workflow retries field in the DB (in same migration).
    * A new poller implementation that's general purpose.
@zeeshanlakhani zeeshanlakhani mentioned this issue Feb 28, 2024
Merged
zeeshanlakhani pushed a commit that referenced this issue Feb 29, 2024
Fix: poll DHT in background when worker runs up a workflow + dual-sta…
2d77b0a 
…ck webserver (#590)

Completes:

- Closes #278. 
- Closes #589.

Includes:
* We no longer block DHT polling of receipts, and, instead, purposefully
race execution and DHT finding/polling.
* This moves us toward the goal of resolving outside-of-workflow awaited
(CIDs), but we'll need another background process to re-run a workflow
once known-awaits are resolved (separate ticket). The next set of work
will also validate that those outside-of-workflow promises exist, i.e.
currently running in another workflow or have already run.
* v4/v6 host address settings for the Json-RPC web server so that the
web server can handle dual v4/v6 requests on the same port.
  * Better / ordered error handling around CIDs that fail to resolve.
* Workflow status field in the DB (+ migration) (SQLite enum, with
special patching).
  * Workflow retries field in the DB (in same migration).
  * A new poller implementation that's general purpose.
  * Removes `async_trait` except for wasmtime impls (due to 1.73).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zeeshanlakhani zeeshanlakhani

Labels
networking Features, functionality involving networking
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix: poll DHT in background when worker runs up a workflow + dual-stack webserver ipvm-wg/homestar
1 participant
@zeeshanlakhani