Is there a way to add a "catch-all" fallback hook?

[yukw777]

• cattrs version: 22.2.0
• Python version: 3.9.12
• Operating System: Ubuntu

Description

First of all, thanks for the awesome library!

My use-case is that I want to be able to serialize as much as I can by falling back to something like pickle for non-attrs classes. I understand that pickle can't serialize everything, but I'm just trying to cover as many classes as I can. Unstructuring is not a problem by using things like jsonpickle or the default option of orjson, but structuring is giving me a lot of problems. I've tried to use cattrs.register_structure_hook_func, but I haven't found a way to make it work. I guess one solution would be to find a way to check if a type is supported (either implicitly or explicitly) by the current converter or not, but I haven't found a good way to do so.

What I Did

# infinite recursion b/c of attrs class that have not been explicitly registered.
def fallback(d, t):
    if attrs.has(t):
        return cattrs.structure(d, t)
    return pickles.loads(d)
cattrs.register_structure_hook_func(lambda cls: True, lambda d, t: t.deserialize(d))

# fails on collections of attrs classes such as list[attrs]
def fallback(d, t):
    if attrs.has(t):
        return cattrs.structure_attrs_fromdict(d)
    return pickles.loads(d)
cattrs.register_structure_hook_func(lambda cls: True, lambda d, t: t.deserialize(d))

Any help would be greatly appreciated!

[Tinche]

Hello, good question.

To give a little context, cattrs uses two mechanisms to get your hook - the first one is a functools.singledispatch, and the second one is a list of predicate functions. When you use register_structure_hook_func, you're basically registering this hook at the start of the list, so it'll be checked first. The problem is, the fallback handler is the last item in that list, not the first.

We don't have a very good way of handling this currently, but I think we should.

Here's what you can do right now to solve your problem:

c = Converter()

c._structure_func._function_dispatch._handler_pairs[-1] = (
    lambda _: True,
    lambda v, _: pickle.loads(v),
    False,
)

You basically overwrite the fallback handler with your own. It's kind of filthy though, as evidenced by all the underscores.

I think I'll do a small refactor to make setting this on the MultiStrategyDispatch easier. Ultimately, maybe this could be an argument to the Converter?

I'm a little wary of adding arguments to the Converter since there are so many already, it's getting to be intimidating for users.

[yukw777]

I see, I was kind of going that way too and thinking about monkeypatching (yuk) BaseConverter._structure_error, which I believe is used as the fallback function for MultiStrategyDispatch, but I'll use your code snippet! Thank you so much!

I don't think the number of arguments for Converter is too large now, but maybe another way to handle is to give a way for people to register a fallback function?

[Tinche]

Hm, you could maybe subclass Converter and just override _structure_error?

Let's keep this issue open to track it as we think about it.

[yukw777]

That was my initial plan, but the problem is MultiStrategyDispatch is initialized using BaseConverter._structure_error instead of self._structure_error. Hence, my modified plan of monkeypatching. Interestingly, MultiStrategyDispatch for unstructure is initialized with self._unstructure_identity.

[Tinche]

Bleh, that's by accident probably.

[yukw777]

haha I see. If that's fixed, I could totally subclass Converter to add my fallback. Maybe we should rename _structure_error() and _unstructure_identity() to mark them public? I'm kind of partial to registering a fallback though. That being said, I don't have any strong preference on the interface. I'm sure you have more context and knowledge to come up with a good one.

[Tinche]

I just applied a refactor to the main branch that improves the architecture a little.

Now you can do this:

import pickle

from cattrs import Converter
from cattrs.dispatch import MultiStrategyDispatch

c = Converter()

dispatch = MultiStrategyDispatch(lambda v, _: pickle.loads(v))
c._structure_func.copy_to(dispatch)
c._structure_func = dispatch


class Test:
    def __init__(self, a: int) -> None:
        self.a = a

    def __repr__(self) -> str:
        return f"Test(a={self.a})"


print(c.structure([pickle.dumps(Test(2))], list[Test]))

[yukw777]

Thanks! I'll watch out for the next release!

[Goldziher]

Hiya,

So I am one of the maintainers of starlite, and I am integrating attrs/cattrs as an optional parsing/validation framework (users will be able to choose between it an pydantic).

The issue I am now faced with is exactly tha ability to handle catch-all structuring / unstructuring hooks, to deal with generic union types and all sorts of other userland types.

It would be great if there was some way to configure a catch-all hook on the converter itself. Not necessarily as an init parameter - i can easily see it as a method that you can override with subclasses or something along these lines- since we are already knee deep into creating our own Converter class.

[Goldziher]

Ok, so after playing with this for a while, this is what I am now doing to be able to handle union properly:

def _create_default_structuring_hooks(
    converter: cattrs.Converter,
) -> tuple[Callable[[Any, type[Any]], Any], Callable[[Any], Any]]:
    """Create scoped default hooks for a given converter.
    
    Notes:
        - We are forced to use this pattern because some types cannot be hanlded by cattrs out of the box. For example,
            union types, optionals, complex union types etc. 
    
    Args:
        converter: A conveter instance 

    Returns:
        A tuple of hook handlers
    """
    def _default_unstructuring_hook(value: Any) -> Any:
        return converter.unstructure(value)

    def _default_structuring_hook(value: Any, annotation: Any) -> Any:
        for arg in unwrap_union(annotation) or get_args(annotation):
            try:
                return converter.structure(arg, value)
            except ValueError:
                continue
        return converter.structure(annotation, value)

    return (
        _default_unstructuring_hook,
        _default_structuring_hook,
    )


class Converter(cattrs.Converter):
    def __init__(self) -> None:
        super().__init__()

        # this is a hack to create a catch-all hook, see: https://github.com/python-attrs/cattrs/issues/311
        self._structure_func._function_dispatch._handler_pairs[-1] = (
            *_create_default_structuring_hooks(self),
            False,
        )


_converter: Converter = Converter()

I would suggest that the converter exposes simple setters for this-

converter_instance.set_default_structuring_hook(...)
converter_instance.set_default_unstructuring_hook(...)

Additionally, it would be very good to be able to cache these. I will try throwing in an lru_cache decorator on the top, but I am not sure this is wise in this context.

[Tinche]

Howdy,

I've just merged #441 bringing support for fallback hook factories (and fallback hooks). The docs are available here: https://catt.rs/en/latest/converters.html#fallback-hook-factories

I've included a simple example with pickle. Hope this meets your needs!

[yukw777]

Wow thank you! @Tinche