Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A Zinnia module to perform the retrieval checks #1

Closed
10 tasks done
juliangruber opened this issue May 16, 2023 · 2 comments · Fixed by #2 or filecoin-station/core#142
Closed
10 tasks done

A Zinnia module to perform the retrieval checks #1

juliangruber opened this issue May 16, 2023 · 2 comments · Fixed by #2 or filecoin-station/core#142
Assignees

Comments

@juliangruber
Copy link
Member

juliangruber commented May 16, 2023

  • We can use HTTP API to ask the Orchestrator about the job to perform.
  • Retrieve the CID using the IPFS Gateway or Saturn CDN, making HTTP requests via the Fetch API. (We will rework this in the following iterations.)
  • Submit retrieval logs & metrics to Ingester
    • Look at what Saturn and Rhea/Lassie is including in their logs.
    • TTFB, TTLB, error rates, retry count, download speed, etc.
    • Talk to Will and Lauren what data will be useful to them. Maybe we can make this dynamically configurable?
    • Let’s start with the easy metrics we can get from the Fetch API only.
    • When Station reports the job outcome to the Ingester, it includes JobId and WalletAddress in the payload. Add submit wallet address spark-api#14
    • We will need to mock/stub Orchestrator and Ingester for testing.
  • The module will be deployed to Filecoin Stations, it will replace the placeholder peer-checker module remove peer-checker, add spark core#142

For now, no verification of the retrieval content, as this will be covered with Lassie later

Estimate: 5 days

@juliangruber juliangruber assigned juliangruber and unassigned bajtos May 23, 2023
juliangruber added a commit that referenced this issue May 31, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
* docs

* add get job from orchestrator

* style

* add fetch car via saturn gateway

* add submit success to ingestor

* fix submit retrieval

* log

* refactor

* add loop

* more stable loop

* call `Zinnia.jobCompleted()`

* collect ttfb

* add response status

* abort early if res not ok

* refactor

* add read timeout

* report error activity

* clean up

* refactor

* refactor

* start test suite

* add ci

* refactor

* test

* add retries to integration test

* fix ci

* add unit test

* simplify, add unit test

* add unit test

* todo

* todo

* more tests

* add submit more fields

* fmt

* module.js -> main.js

* ci: add lint

* docs

* add receive headers timeout

* add `ActivityState`

* move `test/all.js` to `test.js`

* add missing throw in integration test

* refactor using async iterator syntax

* fix lint

* allow more attempts for integration tests

* add test order, arguments and length of `fetch` calls

* fix submitRetrieval + test

* fix lint

* fix tests and condition

* update `spark-api` schema

* add check retrieval from api

* include status code in error message

* `startTimeout` -> `resetTimeout`

* make `stats` also accessible if retrieval fails

* use activity class

* add assert res.ok

* fix lint
@juliangruber juliangruber reopened this May 31, 2023
@juliangruber
Copy link
Member Author

Station integration is missing

@bajtos
Copy link
Member

bajtos commented Jun 1, 2023

Maybe we should keep this issue open until the change is released in Station Desktop?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants