-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Reframe protocol server #251
Comments
@davidd8 @willscott : wanted to make sure that you saw I created this placeholder issue. It's linked from https://www.notion.so/pl-strflt/migrate-store-the-index-hydra-comms-to-the-delegated-routing-protocol-6c97dbc25a084a35b60fed1c3ef715a4 . I wanted a GitHub issue I could link to from other GitHub issues. |
Is there an expected time frame that the protocol will be defined. |
Good question. Lets discuss timeline for the interface/protocol definition in ipfs/go-delegated-routing#8
A couple of things:
|
updating this to cover: "Add Reframe protocol server", let me know if that reflects the current ask, @BigLep |
@willscott : yes, that's accurate. Thanks for updating. |
@willscott Delegated Routing is about to be merged for production (ipfs/go-delegated-routing#11). This is a good time to take care of this issue and integrate it in storetheindex. Can I help with this by preparing an integration PR? |
sure. The main point of complexity I foresee is how we handle the metadata fields of records. Right now, the storetheindex server node does not attempt parsing them at all - it just passes them through as bytes. We can have a limiting parsing based on currently understood fields to allow the current bitswap and graphsync metadata records to be converted into their reframe equivalents, but i'm not sure there's a good open-ended way for storetheindex to fail-open and be future-proofed against other metadata protocols currently since it doesn't try to enforce any specific type of encoding of the bytes. |
@willscott I want to make sure I understand the context of your concern with metadata. I am working under the assumption that we are integrating the Reframe API server in STI purely for the purposes of STI talking to Hydra. In particular, STI would continue to serve its other users using the specialized API that you have developed for them. If this is the case, the only data that the Reframe API understands and cares about currently is provider records for bitswap-capable nodes. Therefore, the integration would involve parsing out bitswap records from the metadata on the STI side and feeding them into the appropriate structures of the Reframe API. If in the future Reframe API supports more types of records, we will have to upgrade the integration accordingly. |
Right, it's that we can't support the Filtering to bitswap records on storetheindex is fine. The point of coordination that's worth a heads up to @aschmahmann is that if we end up wanting to extend bitswap records to have auth, this parsing will mean we can't just flow the additional record data to clients. I don't think we need to figure out the longer term metadata format in this iteration, but it's something we're setting ourselves up to want pretty quickly. |
Reframe integration in STI is in the PR: #464 |
@gammazero : I'm new on this project and its deployments. Is it self-service to determine whether this has been deployed? |
we are trialing gitops as a preferred deployment strategy. commits merged to the repo will trigger a bot PR that can be merged to trigger dev deployment. tags will trigger PRs that merge to trigger prod deployments. to determine what's been deployed you can either look at the .deploy directory, which includes the active images, or use the https://cid.contact/health / https://dev.cid.contact/health endpoint to see the current deployed git commit. |
I'm reopening because of the need to add an operational dasbhoard like there is for the existing providers endpoint: https://protocollabs.grafana.net/d/pPROWOD7z/providers?orgId=1&var-Source=storetheindex-prod&var-Jobs=storetheindex%2Findexer&var-pod=indexer-0 |
I also updated the done criteria to make it clear we need this to be working in production. |
All: I added this note to the description "For point number 2 above, EngRes IPFS team will take on there are metrics available. storetheindex will own creating a dashboard that uses them." @willscott : will a promotion to prod deploy occur automatically or does a manual step need to be taken? I want to know when we can move forward on the Hydra production deployment. |
productions to prod are happening a couple times a week. they're visible as active PRs in this storetheindex repo.
|
@BigLep The current in-production commit for STI includes the latest Reframe code. Therefore, I will initiate deployment on the Hydra end. |
@BigLep is this a duplicated item from the past, should we close this? |
@honghaoq : I don't know if it was duplicated, but it looks like the done criteria were met and thus can be closed. Go for it! |
Done. |
Done Criteria
Why Important
go-ipfs isn't planning to add support for the current version of the storetheindex client. It is planning to add support for delegated-routing though as part of ipfs/kubo#8775 . Having an easy self-service configuration option for go-ipfs to query storetheindex directly will ensure 100% content discovery for any go-ipfs node that has storetheindex as a delegated-routing server for CIDs that exist in storetheindex only. The most immediate usecase is to configure the ipfs.io gateway in this way.
Notes
Before this work can be done:
Once this change is deployed to storetheindex, the Hydras can be updated to use delegated routing: libp2p/hydra-booster#162
For point number 2 above, EngRes IPFS team will take on there are metrics available. storetheindex will own creating a dashboard that uses them.
The text was updated successfully, but these errors were encountered: