Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrieve metadata by slot-no #68

Merged
merged 11 commits into from
Oct 1, 2022
Merged

Retrieve metadata by slot-no #68

merged 11 commits into from
Oct 1, 2022

Conversation

KtorZ
Copy link
Member

@KtorZ KtorZ commented Sep 30, 2022

Fixes #51


  • 📍 Define a new ouroboros local client for fetching blocks by previous point.
    This is pretty rudimentary but works. This assumes that the caller has ways to retrieve any checkpoint necessary for them to find blocks.

    In the case of kupo, this is now trivially done since every on-chain point is stored in the database, with proper indexing allowing to navigate backward from any point.

    The Ogmios client is still to be written but will be done in a similar fashion.

  • 📍 Define the API extension for metadata.
    Going for something simple, allowing to retrieve metadata behind

    GET /metadata/{slot-no}

    Possibly, authorizing a filter by transaction-id. Note that I am tempted to change the response already to move the 'created_at' point to the top-level of the response -- since it'll be identical for all items in the array in case where there are multiple metadata object.

    Plus, it'll make it easier for clients to verify that the points they're getting is indeed matching whatever they expect (and we should encourage them to do in the documentation!) because fetching by slot is "unsafe" for blocks that aren't at least k=2160 blocks deep in the chain.

  • 📍 Move few common JSON encoding helpers to the Prelude.

  • 📍 Define supporting code for Metadata/ MetadataHash
    Retrieving these from Ogmios isn't implemented yet. It isn't trivial as it requires to parse the detailed JSON metadata description. It't not essential to move on with testing and the rest of the implementation so I'll leave it for later.

  • 📍 Implement HTTP server logic for retrieving metadata and wire the fetch-block client.
    This was a bit more involved than I originally envisionned, because of the state-machine testing mainly which obliges the application logic to somewhat abstract the fetch-block client too. This is fortunately relatively easy to do, and has the nice benefits of making it testable too through the state-machine machinery. I am starting off a simple generator for metadata (though I'll make it more stuffed as soon as I begin looking into testing serialization roundtrips when implementing the Ogmios' part).

    So far, everything compiles (without +production) and tests pass; so this isn't breaking any existing functionalities. I can now extend the test-suite to demonstrate that the new functionality work properly. Eventually, an end-to-end scenario will confirm that the integration with actual chain producers also work (since the state-machine tests mocks them away).

  • 📍 Incude metadata to state-machine & http test benchs.
    And fix various little things accordingly, including the missing chain-sync exception handler around the fetch-block client.

  • 📍 Add end-to-end scenario illustrating fetching metadata by slot-no.

    $ time curl -v http://localhost:1442/metadata/70146054 | jq
    > GET /metadata/70146054 HTTP/1.1
    > Host: localhost:1442
    > User-Agent: curl/7.77.0
    > Accept: */*
    
    < HTTP/1.1 200 OK
    < Transfer-Encoding: chunked
    < Date: Sat, 01 Oct 2022 12:15:25 GMT
    < Server: kupo
    < X-Block-Header-Hash: 6c735d4883068bb7d3c5b22b1b8166490317480bab3454e2fe227a351448bdb8
    < X-Most-Recent-Checkpoint: 70257131
    < Content-Type: application/json;charset=utf-8
    <
    [
    {
      "hash": "04e6464091d5b8b5555fc4c3e2240c8676ece51ebd7c090115013f8fe74f07dd",
      "raw": "a11902a2a1636d736781687465737420313233",
      "schema": {
        "674": {
          "map": [
            {
              "k": {
                "string": "msg"
              },
              "v": {
                "list": [
                  {
                    "string": "test 123"
                  }
                ]
              }
            }
          ]
        }
      }
    }
    ]
    0.00s user 0.01s system 57% cpu 0.022 total
  • 📍 Write more elaborate Metadata generator, going through all primitives.

  • 📍 Implement JSON parser for metadata detailed JSON
    And roundtrip test the heck out of it. This is THE missing piece for the Ogmios' implementation. Now that this is done, rest should be a piece of cake.

  • 📍 Implement FetchBlockClient for Ogmios.
    Effectively enabling metadata retrieving through Ogmios as well.

  • 📍 Add optional filter by transaction id on metadata retrieving.

… point.

  This is pretty rudimentary but _works_. This assumes that the caller
  has ways to retrieve any checkpoint necessary for them to find blocks.

  In the case of kupo, this is now trivially done since every on-chain
  point is stored in the database, with proper indexing allowing to
  navigate backward from any point.

  The Ogmios client is still to be written but will be done in a similar
  fashion.
  Going for something simple, allowing to retrieve metadata behind

  GET /metadata/{slot-no}

  Possibly, authorizing a filter by transaction-id. Note that I am
  tempting to change the response already to move the 'created_at' point
  to the top-level of the response -- since it'll be identical for all
  items in the array in case where there are multiple metadata object.

  Plus, it'll make it easier for clients to verify that the points
  they're getting is indeed matching whatever they expect (and we should
  encourage them to do in the documentation!) because fetching by slot
  is "unsafe" for blocks that aren't at least k=2160 blocks deep in the
  chain.
  Retrieving these from Ogmios isn't implemented yet. It isn't trivial as it requires to parse the detailed JSON metadata description. It't not essential to move on with testing and the rest of the implementation so I'll leave it for later.
…h-block client.

   This was a bit more involved than I originally envisionned, because of the state-machine testing mainly which obliges the application logic to somewhat abstract the fetch-block client too. This is fortunately relatively easy to do, and has the nice benefits of making it testable too through the state-machine machinery. I am starting off a simple generator for metadata (though I'll make it more stuffed as soon as I begin looking into testing serialization roundtrips when implementing the Ogmios' part).

   So far, everything compiles (without +production) and tests pass; so this isn't breaking any existing functionalities. I can now extend the test-suite to demonstrate that the new functionality work properly. Eventually, an end-to-end scenario will confirm that the integration with actual chain producers also work (since the state-machine tests mocks them away).
@KtorZ KtorZ self-assigned this Sep 30, 2022
KtorZ added 6 commits October 1, 2022 13:24
  And fix various little things accordingly, including the missing chain-sync exception handler around the fetch-block client.
  ```console
  $ time curl -v http://localhost:1442/metadata/70146054 | jq
  > GET /metadata/70146054 HTTP/1.1
  > Host: localhost:1442
  > User-Agent: curl/7.77.0
  > Accept: */*

  < HTTP/1.1 200 OK
  < Transfer-Encoding: chunked
  < Date: Sat, 01 Oct 2022 12:15:25 GMT
  < Server: kupo
  < X-Block-Header-Hash: 6c735d4883068bb7d3c5b22b1b8166490317480bab3454e2fe227a351448bdb8
  < X-Most-Recent-Checkpoint: 70257131
  < Content-Type: application/json;charset=utf-8
  <
  [
    {
      "hash": "04e6464091d5b8b5555fc4c3e2240c8676ece51ebd7c090115013f8fe74f07dd",
      "raw": "a11902a2a1636d736781687465737420313233",
      "schema": {
        "674": {
          "map": [
            {
              "k": {
                "string": "msg"
              },
              "v": {
                "list": [
                  {
                    "string": "test 123"
                  }
                ]
              }
            }
          ]
        }
      }
    }
  ]
  0.00s user 0.01s system 57% cpu 0.022 total
  ```
  And roundtrip test the heck out of it. This is THE missing piece for the Ogmios' implementation. Now that this is done, rest should be a piece of cake.
  Effectively enabling metadata retrieving through Ogmios as well.
@KtorZ KtorZ merged commit b98b4ce into master Oct 1, 2022
@KtorZ KtorZ deleted the search-metadata-by-slot branch October 15, 2022 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Metadata with output
1 participant