-
-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retrieve metadata by slot-no #68
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… point. This is pretty rudimentary but _works_. This assumes that the caller has ways to retrieve any checkpoint necessary for them to find blocks. In the case of kupo, this is now trivially done since every on-chain point is stored in the database, with proper indexing allowing to navigate backward from any point. The Ogmios client is still to be written but will be done in a similar fashion.
Going for something simple, allowing to retrieve metadata behind GET /metadata/{slot-no} Possibly, authorizing a filter by transaction-id. Note that I am tempting to change the response already to move the 'created_at' point to the top-level of the response -- since it'll be identical for all items in the array in case where there are multiple metadata object. Plus, it'll make it easier for clients to verify that the points they're getting is indeed matching whatever they expect (and we should encourage them to do in the documentation!) because fetching by slot is "unsafe" for blocks that aren't at least k=2160 blocks deep in the chain.
Retrieving these from Ogmios isn't implemented yet. It isn't trivial as it requires to parse the detailed JSON metadata description. It't not essential to move on with testing and the rest of the implementation so I'll leave it for later.
…h-block client. This was a bit more involved than I originally envisionned, because of the state-machine testing mainly which obliges the application logic to somewhat abstract the fetch-block client too. This is fortunately relatively easy to do, and has the nice benefits of making it testable too through the state-machine machinery. I am starting off a simple generator for metadata (though I'll make it more stuffed as soon as I begin looking into testing serialization roundtrips when implementing the Ogmios' part). So far, everything compiles (without +production) and tests pass; so this isn't breaking any existing functionalities. I can now extend the test-suite to demonstrate that the new functionality work properly. Eventually, an end-to-end scenario will confirm that the integration with actual chain producers also work (since the state-machine tests mocks them away).
And fix various little things accordingly, including the missing chain-sync exception handler around the fetch-block client.
```console $ time curl -v http://localhost:1442/metadata/70146054 | jq > GET /metadata/70146054 HTTP/1.1 > Host: localhost:1442 > User-Agent: curl/7.77.0 > Accept: */* < HTTP/1.1 200 OK < Transfer-Encoding: chunked < Date: Sat, 01 Oct 2022 12:15:25 GMT < Server: kupo < X-Block-Header-Hash: 6c735d4883068bb7d3c5b22b1b8166490317480bab3454e2fe227a351448bdb8 < X-Most-Recent-Checkpoint: 70257131 < Content-Type: application/json;charset=utf-8 < [ { "hash": "04e6464091d5b8b5555fc4c3e2240c8676ece51ebd7c090115013f8fe74f07dd", "raw": "a11902a2a1636d736781687465737420313233", "schema": { "674": { "map": [ { "k": { "string": "msg" }, "v": { "list": [ { "string": "test 123" } ] } } ] } } } ] 0.00s user 0.01s system 57% cpu 0.022 total ```
And roundtrip test the heck out of it. This is THE missing piece for the Ogmios' implementation. Now that this is done, rest should be a piece of cake.
Effectively enabling metadata retrieving through Ogmios as well.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #51
📍 Define a new ouroboros local client for fetching blocks by previous point.
This is pretty rudimentary but works. This assumes that the caller has ways to retrieve any checkpoint necessary for them to find blocks.
In the case of kupo, this is now trivially done since every on-chain point is stored in the database, with proper indexing allowing to navigate backward from any point.
The Ogmios client is still to be written but will be done in a similar fashion.
📍 Define the API extension for metadata.
Going for something simple, allowing to retrieve metadata behind
GET /metadata/{slot-no}
Possibly, authorizing a filter by transaction-id. Note that I am tempted to change the response already to move the 'created_at' point to the top-level of the response -- since it'll be identical for all items in the array in case where there are multiple metadata object.
Plus, it'll make it easier for clients to verify that the points they're getting is indeed matching whatever they expect (and we should encourage them to do in the documentation!) because fetching by slot is "unsafe" for blocks that aren't at least k=2160 blocks deep in the chain.
📍 Move few common JSON encoding helpers to the Prelude.
📍 Define supporting code for Metadata/ MetadataHash
Retrieving these from Ogmios isn't implemented yet. It isn't trivial as it requires to parse the detailed JSON metadata description. It't not essential to move on with testing and the rest of the implementation so I'll leave it for later.
📍 Implement HTTP server logic for retrieving metadata and wire the fetch-block client.
This was a bit more involved than I originally envisionned, because of the state-machine testing mainly which obliges the application logic to somewhat abstract the fetch-block client too. This is fortunately relatively easy to do, and has the nice benefits of making it testable too through the state-machine machinery. I am starting off a simple generator for metadata (though I'll make it more stuffed as soon as I begin looking into testing serialization roundtrips when implementing the Ogmios' part).
So far, everything compiles (without +production) and tests pass; so this isn't breaking any existing functionalities. I can now extend the test-suite to demonstrate that the new functionality work properly. Eventually, an end-to-end scenario will confirm that the integration with actual chain producers also work (since the state-machine tests mocks them away).
📍 Incude metadata to state-machine & http test benchs.
And fix various little things accordingly, including the missing chain-sync exception handler around the fetch-block client.
📍 Add end-to-end scenario illustrating fetching metadata by slot-no.
📍 Write more elaborate Metadata generator, going through all primitives.
📍 Implement JSON parser for metadata detailed JSON
And roundtrip test the heck out of it. This is THE missing piece for the Ogmios' implementation. Now that this is done, rest should be a piece of cake.
📍 Implement FetchBlockClient for Ogmios.
Effectively enabling metadata retrieving through Ogmios as well.
📍 Add optional filter by transaction id on metadata retrieving.