RFC: Hand-rolling our web server <> GraphQL API <> Data layer integration

[ra0x3]

Context

• This is the RFC discussion task for Integrate web server, GraphQL layer, and data layer #1145

The Idea

• We are hand-rolling this web server <> GraphQL query <> SQL integration because no 3rd party crate exists to do this
• We are effectively taking an raw GraphQL query String and converting it into a SQL query String, executing the query, and returning the results
• The flow would be:
  • HTTP Request -> {parse} -> GraphQL Query Document -> {translate} -> SQL query tokens -> {execute} -> SQL Rows -> {translate} -> GraphQL Response -> {convert} -> HTTP Response
  • The idea that the psuedo-code below is based on (which may very well be incorrect as I'm not a GraphQL expert), is that a GraphQL Query document is just a tree with a start (root node) and end nodes (leaf nodes), and since GraphQL query documents are parsed top-down, left-to-right (inside-to-outside), we can build a Node and Edge based representation of this document, then iteratively convert each Node/Edge into a set of SQL tokens

Considerations

• Account for FK and M2M relationships
  • If I give you a m2m field that doesn't actually exist on the typdef table, look up the value in the m2m field
    • E.g., Given query { block { hash height transaction { id } } } you know that 'Transaction' is a m2m child FK of 'Block', so for this value you look into the m2m table for the id, then the actual typdef table for the field value(s)
    • Should work for FK and m2m FK
• Queries can specify particular fields of a JSON item
• JSON/virtual fields are NodeKind::Virtual
• Multiple types of pagination (or no pagination depending on user request)
• Introspection
  • On a per-resolver basis?
    • This is what TheGraph uses
• Should account for queries as functions (aliases)
  • E.g., query below:

query {
  feed(filter: "QL") {
    id
    description
    url
    postedBy {
      id
      name
    }
  }
}

• Will add more considerations as we think of them

Pseudo-Code

• This is just rough pseudo-code to show what I'm thinking
• We can integrate this with our existing GraphQL code
• This code shouldn't be overly generic - it's only translating GraphQL --> SQL
• The pseudo-code does not include code for the resolver(s)

enum Sort {
  Asc,  // "ASC"
  Desc,  // "DESC"
}

impl Display for Sort;

struct GraphQLOptions {
  max_edge_depth: usize,
  bail_on_empty_queries: bool,
  max_join_complexity: usize,
}


enum PageDirection {
  #[default]
  Forward,
  Backward,
}

struct Cursor {
  direction: PageDirection
  offset: usize,
  size: usize,
}

impl Display for Cursor;

impl From<Cursor> for String;

struct Offset {
  offset: usize
  limit: usize
}

enum PaginationKind {
 	Cursor(Cursor),
    Offset(Offset),
    None,
}


enum EdgeKind {
  Field,
  Entities, // m2m
  Entity, // foreign key
}

enum NodeKind {
  Field,
  VirtualField, // a key in a JSON blob
  Entity,
  Other
}
  
  
// arbitrary object where the keys are fields on the typedef and values are `Sort`
type OrderBy = serde::Value;
  
  
// arbitrary object where the keys are fields on the typedef and values are `Sort`
type Filter = serde::Value; 
  
    
struct Take(pub usize);
  

struct Skip(pub usize);

enum ParamKind {
  Filter(Filter),  // "filter"
  Sort(Sort),      // "sort"
  Skip(Skip),      // "skip"
  Take(Take),      // "take"
  OrderBy(OrderBy) // "orderBy"
}


struct Param {
  kind: ParamKind
  value: serde::Value,
}

struct Node {
  typdef: TypeDefinition,
  kind: NodeKind,
  schema: Rc<ParsedGrapQLSchema>,
  edge: Rc<Edge>,
  params: Vec<Param>,
}


struct Edge {
  parent: Rc<Node>,
  node: Option<Rc<Node>>,
  kind: EdgeKind,
}


impl From<Node> for JoinMetadata;


struct GraphQLQuery {
    text: String,
    head: Rc<Node>,
    pagination: PaginationKind,
    schema: Rc<ParsedGraphQLSchema>,
}


struct SQLQueryResult {
    data: serde::Value,
    size: usize,
}

impl SQLQueryResult {
   fn has_next_page(&self) -> usize {
   		self.data.len() - self.size > 0  
   }
}

struct SQLQuery {
 	ast: sqlparser::AST,
    cursor: Cursor,
    result: SQLQueryResult,
}


struct GraphQLRequest {
  cursor: Cursor,
  query: GraphQLQuery,
}

impl GraphQLRequest {
 
    // Needs to be able to:
    //     - detect cycles in query
    //.    - ensure num of joins (whethere single or m2m) isn't too large
    //.    - ensure query is not too deeply nested
    //.    - maybe run query cost plan first and bail if query is too costly
 	pub fn validate(&self, options: GraphQLOptions) -> Result<()>; 
}

struct GraphQLResponse {
  cursor: Cursor,
  data: Option<SQLQueryResult>,
  has_next_page: bool
}

impl GraphQLResponse {
    fn result_set_size(&self) -> usize {
        self.data.size
    }
}

trait Translate
  type Output;

  fn translate(&self, schema: &ParsedGraphQLSchema) -> Self::Output;
}

trait Query {
  fn parse(q: String) -> Self;
}


impl Query for GraphQLQuery {
  fn parse(text: String, cursor: Cursor, pagination: PaginationKind) -> Self {
    let nodes = do_the_stuff_to_parse_nodes_from_query_string();
     Self { 
       text, 
	   cursor, 
       nodes,
       pagination,
    }
  }
}


impl Translate for GraphQLQuery  {
    type Output = SQLQuery;
  
	fn translate(&self, schema: &ParsedGraphQLSchema) -> Self::Output {
      SQLQuery::from_graphql_query(self, schema)
	}
}

impl SQLQuery {
   pub async fn from_graphql_query(query: GraphQLQuery, schema: &ParsedGraphQLSchema, conn: Connection) -> Self; 
        // traverse this query's nodes and edges
        // validate these nodes and edges
        // iterate over nodes and edges, converting them to SQL fragments
        // execute SQL according to GraphQLOptions
        // get actual data/payload from results
        // return a graphl-spec response in GraphQLResponse using GraphQLOptions
        self.ast = build_the_ast()?;
        self.execute(conn).await
  
  
    async fn execute(&self, conn: Connection) -> Result<()> {
        let result = conn.execute(self.ast.to_string()).await?;
        self.result = result;
        Ok(())
    }
  
    pub fn into_inner() -> serde::Value {
       self.result  
    }
}

struct GraphQLResponse {
   cursor: Option<Cursor>,
   data: serde::Value,
   has_next_page: bool,
}

impl From<SQLQueryResult> for GraphQLResponse {
  fn from_sql(result: SQLQueryResult, cursor: Cursor) -> Self {
    GraphQLResponse {
        cursor,
        has_next_page: result.has_next_page(),
        data: result.into_inner(),
    }
  }
}


impl From<GraphQLResponse> for axum::Json;

[ra0x3]

CC @deekerno, @AlicanC and @lostman for thoughts first. Then I'll bring in the fuel-core team to see what they think

[AlicanC]

We should adapt structures and methods described here: https://relay.dev/graphql/connections.htm

[AlicanC]

FYI, I already have them implemented in my codegen.

[ra0x3]

@AlicanC Feel free to copy the pseudo-code or make any changes to it, or paste your own 👌🏽 just riffing and throwing out ideas

[ra0x3]

I believe this https://spec.graphql.org/June2018/ is the actual GraphQL spec? So might paste a 2nd iteration of pseudo-code that's a bit more inline with that ☝🏽

I think the specific parts of the spec that related to us are:

• https://spec.graphql.org/June2018/#sec-Language.Fields
• https://spec.graphql.org/June2018/#sec-Language.Arguments
• https://spec.graphql.org/June2018/#sec-Field-Alias
• https://spec.graphql.org/June2018/#sec-Selection-Sets
  • I see the value of fragments 🤔 I also see the complexity
• https://spec.graphql.org/June2018/#sec-Input-Values
• https://spec.graphql.org/June2018/#sec-Introspection
• https://spec.graphql.org/June2018/#sec-Executing-Requests
  • I'm not sure how the concept of Subscriptions would work manually 🤔

[AlicanC]

I've attached below the generated types for some dummy Block and Transaction types from the code I've extracted from the codegen.

While the code was initially written for generating types from contract ABIs, it defines (or helps you define) basic GQL concepts like nodes, connections and such so it's usable anywhere. Everything should be up to spec besides ordering and filtering (which aren't specced and are up to us to define.)

The main concept here is the Node interface. This interface is implemented on all GraphQL objects we intend to address by an id. Connections and their params like first and after are used for cursor pagination.

Note: the schema below doesn't implement the whole spec yet, and filtering/ordering is WIP.

type Block implements Node {
  id: ID!
  number: Int!
  hash: String!
}
type BlockConnection {
  totalCount: Int!
  nodes: [Block!]!
  edges: [BlockEdge!]!
  pageInfo: PageInfo!
}
type BlockEdge {
  node: Block!
  cursor: String!
}
input BlockFilterInput {
  and: [BlockFilterInput!]
  or: [BlockFilterInput!]
  not: [BlockFilterInput!]
  number: IntFilterInput
  hash: StringFilterInput
}
input BlockOrderInput {
  number: OrderDirection
  hash: OrderDirection
}
input IDFilterInput {
  and: [IDFilterInput!]
  or: [IDFilterInput!]
  not: [IDFilterInput!]
  eq: IDFilterInput
  in: [IDFilterInput!]
}
input IntFilterInput {
  and: [IntFilterInput!]
  or: [IntFilterInput!]
  not: [IntFilterInput!]
  eq: IntFilterInput
  in: [IntFilterInput!]
  gt: IntFilterInput
  gte: IntFilterInput
  lt: IntFilterInput
  lte: IntFilterInput
}
interface Node {
  id: ID!
}
input NodeFilterInput {
  and: [NodeFilterInput!]
  or: [NodeFilterInput!]
  not: [NodeFilterInput!]
  id: IDFilterInput
}
input NodeOrderInput {
  id: OrderDirection
}
enum OrderDirection {
  asc
  desc
}
type PageInfo {
  hasNextPage: Boolean!
  hasPreviousPage: Boolean!
  startCursor: String
  endCursor: String
}
type Query {
  block(id: ID!): Block
  blocks(filter: BlockFilterInput, order: BlockOrderInput, first: Int, after: String, last: Int, before: String): BlockConnection!
  transaction(id: ID!): Transaction
  transactions(filter: TransactionFilterInput, order: TransactionOrderInput, first: Int, after: String, last: Int, before: String): TransactionConnection!
}
input StringFilterInput {
  and: [StringFilterInput!]
  or: [StringFilterInput!]
  not: [StringFilterInput!]
  eq: StringFilterInput
  in: [StringFilterInput!]
  gt: StringFilterInput
  gte: StringFilterInput
  lt: StringFilterInput
  lte: StringFilterInput
}
type Transaction implements Node {
  id: ID!
  index: Int!
  hash: String!
}
type TransactionConnection {
  totalCount: Int!
  nodes: [Transaction!]!
  edges: [TransactionEdge!]!
  pageInfo: PageInfo!
}
type TransactionEdge {
  node: Transaction!
  cursor: String!
}
input TransactionFilterInput {
  and: [TransactionFilterInput!]
  or: [TransactionFilterInput!]
  not: [TransactionFilterInput!]
  index: IntFilterInput
  hash: StringFilterInput
}
input TransactionOrderInput {
  index: OrderDirection
  hash: OrderDirection
}
schema {
  query: Query
}

[ra0x3]

@AlicanC +1 from me

• I imagine this would be compiled during forc index build ? (also we can talk about this over in Auto-generate GraphQL schema using contract ABI #846 as well)

[ra0x3]

Update:

After what @AlicanC mentioned earlier, I think we can break the execution part issue into 2 parts.

• Part 1: pre-canned queries

• Part 2: arbitrary queries

• Pre-canned queries

  • We require the user to include a type Query TypeDefinition in their schema
    • This auto-generated type Query TypeDefinition could be autogenerated during forc index build
      • As a follow up to Auto-generate GraphQL schema using contract ABI #846
    • Once the user deploys their indexer, it's pretty easy to convert the pre-canned query (see below) into SQL
      • This makes getting a v1 of this issue (using proper, idiomatic GraphQL) much easier to get done

• Arbitrary queries

  • This would be much harder than pre-canned queries, and was my original idea
  • This is the case where a user submits a query {} that is not defined in type Query
  • We already support a bit of arbitrary queries

• I'm thinking we could always get a quick win with pre-canned queries, and then circle back for arbitrary queries

  • Would also give us more time to think of the best way to do arbitrary queries

Example using pseudo-code

" schema file"
type FooEntity @entity {
  id: ID!
  name: Charfield!
}
  
type AnotherEntity @entity {
   id: ID!
   foo: FooEntity!
   name: Charfield!
}

" << WITH ALL THE METADATA FROM JC's WORK >> "

type Query {
    singleFoo(id: ID, name: Charfield) FooEntity
    allFoos() [FooEntity]!
    singleFooInList(id: [ID!]!) FooEntity
    " so on and so forth.... "
    
    manyFooInList(id: [ID!]!) [FooEntity]!
    singleAnotherEntity(id: ID, name: Charfield) AnotherEntity
    allAnotherEntities() [AnotherEntity]!
    " so on and so forth.."
   
}

Thoughts? (cc @deekerno @AlicanC @lostman )

[deekerno]

re: pre-canned queries -- I think this is a good idea. Just so I understand correctly: the suggestion is just to auto-generate a set of queries for each type marked with @entity, is that right? If so, we should decide on a sensible (and hopefully small) set of pre-canned queries to support; taking inspiration from the above example, we could do the following queries:

1. query for a paginated list of records with no arguments: [ExampleEntity]!
   -- Sort argument could be optional but should have a default sort for deterministic pagination
   -- We should probably enforce a sensible max limit for elements in this list, similar to how fuel-core doesn't allow users to request the entirety of all blocks
2. query for a single record using an ID or any @unique field (but not both): ExampleEntity
3. query for a set of records using a group of IDs or any @unique fields (all elements should be the same type): [ExampleEntity]!

Since we would have a strict criteria for pre-canned queries, we should be able to create methods that essentially concatenate format!()-ed string fragments into a final query string without all the parsing machinery that we currently utilize for arbitrary queries. Also, it should be relatively easy to enforce correct argument usage since we would only support at max one element (for a single record query) or one element type (for a query returning a set of records).

---

re: arbitrary queries -- I think we have fairly solid functionality here already; it just needs to be tightened up and tested within an inch of its life. I think it would be helpful to have the client team (or even grantees) share what their queries they've seen or will typically use.

[Voxelot]

Here are all the precanned queries the fuel-core-client supports currently: https://github.com/FuelLabs/fuel-core/tree/master/crates/client/src/client/schema/snapshots

However, this set of precanned queries in the rust client may not be representative of what swayswap or the old block explorer needed to use, since they could setup arbitrary queries against fuel-core.