Best folder structure for large schemas

[gijswobben]

This isn't really an issue but more of a question: What would be the best way to split up large schemas into separate files? What folders do I need? Do I split the resolvers?

[oharlem]

@gijswobben
Working in Django, for example, through some trial and error I found it useful to split as simple as:

foo_graphql/
  errors.py
  middlewares.py
  dataloaders.py
  schema.py
  types.py
  util.py
  ...etc

The largest file is obviously types.py.
With a number of types growing, will just move into its own dir with types as separate files.

[ahopkins]

I have mine broken out into sub "apps", and then I have a utility to "auto load" my queries and migrations.

So, it looks like this:

data
    player
        mutations.py
        queries.py
        types.py
    team
        mutations.py
        queries.py
        types.py
    user
        mutations.py
        queries.py
        types.py
    mutation.py
    query.py

The contents of data/query.py is something like this:

class QueriesAbstract(graphene.ObjectType):
    pass

queries_base_classes = [QueriesAbstract]
current_directory = os.path.dirname(os.path.abspath(__file__))
current_module = current_directory.split('/')[-1]
subdirectories = [
    x
    for x in os.listdir(current_directory)
    if os.path.isdir(os.path.join(current_directory, x)) and
    x != '__pycache__'
]
for directory in subdirectories:
    try:
        module = importlib.import_module(f'{current_module}.{directory}.queries')
        if module:
            classes = [x for x in getmembers(module, isclass)]
            queries = [x[1] for x in classes if 'Query' in x[0]]
            queries_base_classes += queries
    except ModuleNotFoundError:
        pass

queries_base_classes = queries_base_classes[::-1]
properties = {}
for base_class in queries_base_classes:
    properties.update(base_class.__dict__['_meta'].fields)

Queries = type(
    'Queries',
    tuple(queries_base_classes),
    properties
)

And, a similar file for mutation.py

Probably could be improved some. But, what this allows.. me to do:

schema = Schema(query=query.Queries, mutation=mutation.Mutations)

And, all I need to do is create a file in the right directory and everything gets loaded automatically.

[ProjectCheshire]

@ahopkins you rock my socks. my folder structure is almost the exact same as yours and I'm totally going to steal your code. Err, 'make use of open source ecosystem'

• input
• type
• model // this is my OGM layer since I'm working with neo4j
• mutations
• queries
• resolvers // I've been doing from . import resolvers as resolve so I can do resolver=resolve.create_ or resolve.users and keep the functional bits all together and the other files crisp.

FWIW, I've been loving using neomodel with graphene if anyone else goes down the neo4j-python-graphql rabbit hole :)

[nishant-jain-94]

@ahopkins I like your folder structure. But I have something which is bothering me. How do you manage field resolvers, for instance your team may have a list of players, where do you exactly resolve that? In the data/team/queries.py? or probably somewhere else?

[ahopkins]

@nishant-jain-94

It is hooked up to a DB. In this case, it is using Neo4j. So, inside data/player/types.py is the following:

import graphene

from data.base_abstracts import BaseAbstract
from lib.utils import import_module
from lib.db import fields
from lib.db.resolvers import RelatedNodeResolver

class SchoolAbstract(BaseAbstract):  # BaseAbstract just defines some fields that should be on ALL types
    name = graphene.String()
    slug = graphene.String()
    abbreviation = graphene.String()
    ...

class School(SchoolAbstract, graphene.ObjectType):
    players = fields.NodeList('data.player.types.Player')
    ...

    ###########################
    #        RESOLVERS        #
    ###########################
    def resolve_players(self, args, context, info):
        Player = import_module('data.player.types.Player')
        return RelatedNodeResolver(self, 'Player', args, context, info).execute(Player)

Basically, my types.py file has two things (perhaps I could break them out into separate files, I have not found the need yet).

1. Abstract classes
2. Object classes

The Abstract classes define the fields. The Object classes define the resolvers, and inherit from the Abstract. This separation makes it so that everything is easy (for me) to find.

As for RelatedNodeResolver, I have a set of custom resolvers that I have abstracted away so that each of my resolvers inside the types.py files are very short. They put together my cypher queries for Neo4j and execute to return the results. If you are also using Neo4j and curious what that looks like, let me know.

[ahopkins]

@ProjectCheshire Glad to hear you are using it. Or something like it. I agree that there is room for improvement, and I may borrow some of your adjustments too!

As for neomodel, I thought about using it but ended up writing some custom resolvers that build the queries and execute them, mostly because I was interested to play around with cypher directly.

[cmmartti]

The examples on this page were a little confusing, so once I figured it out, I thought I'd put together a small working example of the fractal-style schema approach, which you can find at https://github.com/cmmartti/fractal-style-schema.

[oppianmatt]

auto configuring from python files in a folder? That's not very python. Better to be explicit than implicit. Is it really hard to just import the modules you use? And then the ide/static analysis tools work better.

That said, it looks like it just combine it into one class. How does it account for conflicts? if 2 resolvers have the same name in different modules, who wins?

[dbertella]

Thanks @cmmartti for the working example it's super super useful, just wondering, where did you find this syntax:

games = List(
        lambda: Game,
        description="A list of video games.",
        resolver=resolve.all_games
)

I love it but I can't find any documentation around, usually query contains resolver too.
Also it will be awesome if you can add mutations to the example, I'm trying to figure out how to add them in the same elegant way as you add the query but I'm struggling a bit to achieve the same result.

[cmmartti]

@dbertella Resolvers can be located outside of the class, as documented.

I don't have the time or motivation to update the example with mutations (I've only dealt with read-only GraphQL APIs so far), but if you do manage to get it working, do feel free to put together a PR.

[dbertella]

Oh cool I missed that, I will update as soon as I can figure it out then! Thank you very much!
My problem is that I can easily create a mutation this way:

class CreatePerson(graphene.Mutation):
    class Arguments:
        name = graphene.String()

    Output = Person

    def mutate(self, info, name):
        return Person(name=name)

But what I can't figure out is how to split at least mutate from this class using the same syntax as used above. Ideally I'd like to have CreatePerson under types and then the mutation in a mutations.py file where I import the resolver, but I can't figure it out unfortunately, I'll try again and update this in case.

[lie-nielsen]

Here's a large implemetation: https://github.com/mirumee/saleor

[dbertella]

wow that's super nice, thanks for sharing!

[atlasloewenherz]

hi @ahopkins

thanks for sharing your code!

I have a similar approach derived from yours and I am trying to resolve queries via inheritance as follows, but I do not progress since I hit the described error and would be more than thankful if you or someone else can point me to the right direction!!

thanks

class QueriesAbstract(graphene.ObjectType):
    pass

Loading/Resolving the Queries

def loadQueries():
    queries = [QueriesAbstract]
    base_dir = Path('modules').resolve()
    subdirectories = [
        x
        for x in os.listdir(base_dir)
        if os.path.isdir(os.path.join(base_dir, x)) and x not in ['main', 'configs', '__pycache__' ] and
        x != '__pycache__' and x!= 'main' 
    ]
    for directory in subdirectories:
        try:
            #logging.debug(importlib.import_module(f'modules.{directory}.schema'))
            module = importlib.import_module(f'modules.{directory}.schema')
            #logging.debug(module)
            if module:
                classes = [x for x in getmembers(module, isclass)]
                query_classes = [cls for cls in classes if issubclass(cls[1], SQLAlchemyObjectType)]
                queries += query_classes
        except ModuleNotFoundError:
            pass
    return queries[::-1]

loading the properties

def loadProperties(queries):
    properties = {}
    for base_class in queries:
        #logging.debug(dir(base_class))
        logging.debug(type(base_class[1])) # this leads to the error described in the following        
        properties.update(base_class[1].__dict__['_meta'].fields)
    return properties

main execution:

queries_ = loadQueries()
# the above works just fine
properties = {}
properties = loadProperties(queries_)

Queries = type(
    'Queries',
    tuple(queries_),
    properties
)

the error:

2020-08-31 10:44:48,103 - root - MainThread - 10 - DEBUG - <module 'logging' from '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/__init__.py'>
2020-08-31 10:44:48,137 - root - MainThread - 10 - DEBUG - <class 'graphene.utils.subclass_with_meta.SubclassWithMeta_Meta'>
2020-08-31 10:44:48,138 - root - MainThread - 10 - DEBUG - <class 'graphene.utils.subclass_with_meta.SubclassWithMeta_Meta'>
2020-08-31 10:44:48,138 - root - MainThread - 10 - DEBUG - <class 'graphene.utils.subclass_with_meta.SubclassWithMeta_Meta'>
2020-08-31 10:44:48,138 - root - MainThread - 10 - DEBUG - <class 'graphene.utils.subclass_with_meta.SubclassWithMeta_Meta'>
2020-08-31 10:44:48,138 - root - MainThread - 10 - DEBUG - <class 'graphene.utils.subclass_with_meta.SubclassWithMeta_Meta'>
Traceback (most recent call last):
  File "manage.py", line 24, in <module>
    app = create_app(os.getenv('FLASK_CONFIG') or default_config)
  File "/Users/ely/projects/grapene.poc/backend/meem/modules/main/__init__.py", line 101, in create_app
    from modules.main.schema import schema
  File "/Users/ely/projects/grapene.poc/backend/meem/modules/main/schema.py", line 107, in <module>
    properties = loadProperties(queries_)
  File "/Users/ely/projects/grapene.poc/backend/meem/modules/main/schema.py", line 98, in loadProperties
    logging.debug(type(base_class[1]))
TypeError: 'SubclassWithMeta_Meta' object is not subscriptable
    ~/p/grapene.poc/backend/meem                                                                                                                            1 ✘  backend   system   kubernetes-admin@kubernetes ⎈  10:44:48  
```

[ahopkins]

Why are you trying to get from base_class[1]?

queries looks to be a list of classes. And, there for base_class is also a class. Therefore, it makes sense that you cannot [1], and the error is not subscriptable makes sense. What are you trying to achieve there?

[atlasloewenherz]

hi @ahopkins,

thanks for the hint, it was exactly the problem i was facing.

I am trying to load queries, mutations and types per convention / dynamically so when the flask app starts its processing any of the predefined locations and modules for all subclasses of BaseQuery, BaseMutation and BaseSuscription and process accordingly.

the main idea is to have an flask application that is sort taking control of loading schemas when ever they are available. i hope you could get a rough idea of what I am trying to achieve!

cheers!

[ahopkins]

Just try getting rid of [1].

From your code, it looks like you are looping through a list of classes. base_class looks to be enough.

[atlasloewenherz]

Hi @ahopkins

thanks to your inspiration, I managed to make some progress.

I created a SchemaBuilder class that is supposed to load all Queries, Mutations that inherits from my BaseQuery or BaseMutation and later subscription... etc. and creates a Schema containing all Queries and Mutations found. the schema is then being passed to my flask app to work with it.

Currently, I am struggling with the execution of my mutation. So I wrote a unit test to demonstrate the problem

when I execute the test I get the following result and output

The interesting part is line 33 as it claims:

'message': 'Unknown argument "user_data" on field "user" of type ' '"Mutation".

The CreateUser Mutation does have an argument user_data and is loaded according to my logging output and also visible in my Graphql web endpoint.

I'm desperately lost here and I really need some help! if you or anyone else could help to identify the problem, I will be more than glad about that!!

thanks
@atlasloewenherz