[Issue 1135] Setup lookup value logic within the API

[chouinar]

Summary

Fixes #1135

Time to review: 15 mins

Changes proposed

Added the ability to configure lookup values across the API and database as Python enums - this includes:

• Various utility container classes for connecting the components of the approach
• A global LookupRegistry where we validate and store the configurations
• A SQLAlchemy column type which handles converting the DB lookup IDs into enums automatically
• Updated the DB migrations to automatically update the lookup tables
• Automatically handle converting enums to integers in the DB and back using SQLAlchemy type decorators.

Context for reviewers

This implementation is largely borrowed from a prior project that used the template repo as well, and worked well there.

While this is quite a bit, it's a lot of boilerplatey-setup, testing, and validation to make usage of it pretty foolproof (ie. hard to setup wrong). Actual usage of it should be pretty simple, just create/update your enum, update the config, and attach it to a table. Adding a new value just requires a 2-line change and the API, DB table, and all validation "just works". See the new .md file added for examples.

I'm not 100% on the exact file naming/structure, but there's a lot of risk for circular dependencies with how it is currently setup, and this is what I could get to work, happy to adjust if you've got any suggestions.

A few pre-emptive answers to questions:

• Why use foreign keys in the DB instead of enums or check constraints?
  • Alembic struggles to detect changes to allowed values, and even a custom approach would be very difficult. Modifying an existing enum/check constraints is either complex or requires dropping and remaking the constraint. Also, this approach lets the ID in the DB be separate from the value which is nice for renaming them.
• Why make a separate LookupConfig class instead of putting the value in the Enum/why not make a class that does both?
  • I looked into this before. The very short version is that adding more info to an existing Python enum is difficult, you can make the values something like a tuple, but it causes a lot of its convenience to break (can't do str to enum). Also looked into building my own equivalent to Python enums and that gets into complex metaclass logic, which even if you make a proper implementation, python typing libraries like MyPy struggle to interpret it like a Python enum (which under the hood is really really strange as each enum member is also the class itself?).

Additional information

Running migrations locally creates the lk_opportunity_category table and populates it with the values we've configured:

The opportunity table has has a foreign key column with this field:

Note that the factory we use to generate data locally just knows about the enum, the LookupColumn that we have configured automatically handles converting that to the integer without us needing to be aware of it.

[jamesbursa]

Looks good, thanks.

[jamesbursa]

An idea for making this check stronger (currently it could allow the wrong enum type to be set I think).

• Add an enum_type argument to __init__() and store it as a member variable.
• Change this line to isinstance(value, self.enum_type)
• Wherever LookupColumn is created, add the enum type. For example:

   category: Mapped[OpportunityCategory | None] = mapped_column(
        "opportunity_category_id",
        LookupColumn(LkOpportunityCategory, OpportunityCategory),  # 2nd argument added
        ForeignKey(LkOpportunityCategory.opportunity_category_id),
        index=True,
    )

[chouinar]

I could also just make it so each Lk___ class has the enum specifically registered with it and use that? It registers the mapping of enum values, but if I just add the enum itself that would accomplish the same thing

[chouinar]

In the LookupConfig we already were grabbing all of the enums that were associated with a config (note that while not used now - this is setup to allow multiple python enums for a given Lk table), just piped that through via the LookupRegistry