pydantic>=2.5 classes can't be serialized

[isidentical]

import dill
from pydantic import BaseModel, Field


dill.settings["recurse"] = True


class Input(BaseModel):
    prompt: str = Field(
        ..., title="Prompt", description="The prompt to use for the completion."
    )
    num_inference_steps: int = Field(
        default=25,
        ge=20,
        le=100,
        title="Number of Inference Steps",
        description="The number of inference steps to take for each prompt.",
    )


Input2 = dill.loads(dill.dumps(Input))
print(Input2(prompt="test", num_inference_steps=25))

same example works with cloudpickle

import cloudpickle
from pydantic import BaseModel, Field


class Input(BaseModel):
    prompt: str = Field(
        ..., title="Prompt", description="The prompt to use for the completion."
    )
    num_inference_steps: int = Field(
        default=25,
        ge=20,
        le=100,
        title="Number of Inference Steps",
        description="The number of inference steps to take for each prompt.",
    )


Input2 = cloudpickle.loads(cloudpickle.dumps(Input))
print(Input2(prompt="test", num_inference_steps=25))

[gabrielmbmb]

I also encountered this issue. Not sure if the issue is on dill or pydantic side (even if pydantic.BaseModels can be serialized/deserialized with pickle and cloudpickle).

This the minimum code required to reproduce the error with dill==0.3.8, pydantic==2.7.0 and pydantic_core==2.18.1:

import dill
from pydantic import BaseModel


class MyModel(BaseModel):
    pass


dill.loads(dill.dumps(MyModel()))

and the error:

/Users/gabrielmbmb/Source/Argilla/distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py:414: PicklingWarning: Cannot locate reference to <class '__main__.MyModel'>.
  StockPickler.save(self, obj, save_persistent_id)
/Users/gabrielmbmb/Source/Argilla/distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py:414: PicklingWarning: Cannot pickle <class '__main__.MyModel'>: __main__.MyModel has recursive self-references that trigger a RecursionError.
  StockPickler.save(self, obj, save_persistent_id)
Traceback (most recent call last):
  File "/Users/gabrielmbmb/Source/Argilla/distilabel/test_step_decorator.py", line 13, in <module>
    dill.loads(dill.dumps(MyModel()))
  File "/Users/gabrielmbmb/Source/Argilla/distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 303, in loads
    return load(file, ignore, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gabrielmbmb/Source/Argilla/distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 289, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gabrielmbmb/Source/Argilla/distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 444, in load
    obj = StockUnpickler.load(self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gabrielmbmb/Source/Argilla/distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 593, in _create_type
    return typeobj(*args)
           ^^^^^^^^^^^^^^
  File "/Users/gabrielmbmb/Source/Argilla/distilabel/.venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 93, in __new__
    private_attributes = inspect_namespace(
                         ^^^^^^^^^^^^^^^^^^
  File "/Users/gabrielmbmb/Source/Argilla/distilabel/.venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 406, in inspect_namespace
    raise PydanticUserError(
pydantic.errors.PydanticUserError: A non-annotated attribute was detected: `model_fields = {}`. All model fields require a type annotation; if `model_fields` is not meant to be a field, you may be able to resolve this error by annotating it as a `ClassVar` or updating `model_config['ignored_types']`.

For further information visit https://errors.pydantic.dev/2.7/u/model-field-missing-annotation

This only occurs if the pydantic.BaseModel have been declared in __main__. If it's declared in another module, then everything works.

[mmckerns]

dill doesn't explicitly support pickling of pydantic classes, but I can help figure out if there's a patch to be applied in dill (due to something in the standard library), or in pydantic, or elsewhere.

If earlier versions of dill, pydantic, etc had serialized a BaseModel instance, then one easy thing to do is to walk back over commits and see which commit corresponds to the change in behavior. Also, dill provides a serialization traceback, that traces the recursive pickling process... so it's helpful to debug a failure to serialize with dill.detect.trace(True). I can help decipher what the trace is telling you.

[gabrielmbmb]

Thanks for the info @mmckerns. I'll do some more tests and try to figure out what's happening.

[Rocamonde]

Also facing this issue — we use pydantic for our run configs in ML experiments, and pickling is essential since we do distributed training. Any updates would be super helpful! Thank you

[zhc7]

Any updates here? Also facing the issue. Does anyone know how to solve this? Thanks!

[mmckerns]

my suggestion is to set dill.detect.trace(True), then post the traceback.

[zhc7]

ofcourse, here is a minimal reproduce.

from pydantic import BaseModel
import dill

class A(BaseModel):
    foo: str

dill.detect.trace(True)
dill.dumps(A)

output:

┬ T2: <class '__main__.A'>
├┬ ABC: <class '__main__.A'>
│└ # ABC
├┬ F2: <function _create_type at 0x101ef7f60>
│└ # F2 [30 B]
├┬ T4: <class 'pydantic._internal._model_construction.ModelMetaclass'>
│└ # T4 [60 B]
├┬ T4: <class 'pydantic.main.BaseModel'>
│└ # T4 [30 B]
├┬ D2: <dict object at 0x0102058f80>
│├┬ D2: <dict object at 0x0101392cc0>
││├┬ T1: <class 'str'>
│││├┬ F2: <function _load_type at 0x101ef7ec0>
││││└ # F2 [17 B]
│││└ # T1 [27 B]
││└ # D2 [36 B]
│├┬ D2: <dict object at 0x010128cfc0>
││└ # D2 [2 B]
│├┬ D2: <dict object at 0x0102058f40>
││└ # D2 [2 B]
│├┬ T4: <class 'pydantic._internal._decorators.DecoratorInfos'>
││└ # T4 [52 B]
│├┬ D2: <dict object at 0x01020ffd40>
││├┬ D2: <dict object at 0x01017d8d40>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x010128f500>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x0102075540>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01020fc980>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01020fc940>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01020fc900>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01020fc8c0>
│││└ # D2 [2 B]
││└ # D2 [145 B]
│├┬ D2: <dict object at 0x01020fce00>
││└ # D2 [36 B]
│├┬ D2: <dict object at 0x01020fcac0>
││├┬ T4: <class 'pydantic.fields.FieldInfo'>
│││└ # T4 [32 B]
││├┬ D2: <dict object at 0x01018e8c40>
│││├┬ D2: <dict object at 0x01020ffe80>
││││└ # D2 [11 B]
│││└ # D2 [419 B]
││└ # D2 [463 B]
│├┬ D2: <dict object at 0x0102176ec0>
││├┬ T4: <class '__main__.A'>
/opt/miniconda3/lib/python3.12/site-packages/dill/_dill.py:422: PicklingWarning: Cannot locate reference to <class '__main__.A'>.
  StockPickler.save(self, obj, save_persistent_id)
/opt/miniconda3/lib/python3.12/site-packages/dill/_dill.py:422: PicklingWarning: Cannot pickle <class '__main__.A'>: __main__.A has recursive self-references that trigger a RecursionError.
  StockPickler.save(self, obj, save_persistent_id)
│││└ # T4 [6 B]
││├┬ D2: <dict object at 0x0102176e40>
│││├┬ D2: <dict object at 0x0102176e00>
││││├┬ D2: <dict object at 0x0102176dc0>
│││││├┬ D2: <dict object at 0x0102176d40>
││││││└ # D2 [7 B]
│││││├┬ D2: <dict object at 0x0102176d00>
││││││└ # D2 [2 B]
│││││└ # D2 [33 B]
││││└ # D2 [38 B]
│││└ # D2 [87 B]
││├┬ D2: <dict object at 0x0102176cc0>
│││└ # D2 [7 B]
││├┬ D2: <dict object at 0x0102176c80>
│││├┬ Me1: <bound method BaseModel.__get_pydantic_json_schema__ of <class '__main__.A'>>
││││├┬ T1: <class 'method'>
│││││└ # T1 [19 B]
││││├┬ F1: <function BaseModel.__get_pydantic_json_schema__ at 0x10195c9a0>
│││││├┬ F2: <function _create_function at 0x101f00040>
││││││└ # F2 [23 B]
│││││├┬ Co: <code object __get_pydantic_json_schema__ at 0x10136a5d0, file "/opt/miniconda3/lib/python3.12/site-packages/pydantic/main.py", line 704>
││││││├┬ F2: <function _create_code at 0x101f00540>
│││││││└ # F2 [19 B]
││││││└ # Co [1 MiB]
│││││├┬ D4: <dict object at 0x010122be80>
││││││└ # D4 [24 B]
│││││├┬ D2: <dict object at 0x01020fcd80>
││││││└ # D2 [2 B]
│││││├┬ D2: <dict object at 0x01021085c0>
││││││├┬ D2: <dict object at 0x010194f580>
│││││││└ # D2 [71 B]
││││││└ # D2 [98 B]
│││││└ # F1 [1 MiB]
││││└ # Me1 [1 MiB]
│││└ # D2 [1 MiB]
││└ # D2 [1 MiB]
│├┬ T4: <class 'pydantic_core._pydantic_core.SchemaValidator'>
││└ # T4 [51 B]
│├┬ D2: <dict object at 0x01020fe380>
││└ # D2 [7 B]
│├┬ T4: <class 'pydantic_core._pydantic_core.SchemaSerializer'>
││└ # T4 [52 B]
│├┬ T4: <class 'pydantic._internal._utils.LazyClassAttribute'>
││└ # T4 [51 B]
│├┬ D2: <dict object at 0x01020ffec0>
││├┬ T1: <class 'functools.partial'>
│││└ # T1 [20 B]
││├┬ F2: <function generate_pydantic_signature at 0x10195aca0>
│││└ # F2 [64 B]
││├┬ D2: <dict object at 0x01022f3000>
│││├┬ F2: <function BaseModel.__init__ at 0x10195be20>
││││└ # F2 [25 B]
│││└ # D2 [64 B]
││└ # D2 [185 B]
│├┬ D2: <dict object at 0x01020fdac0>
││└ # D2 [2 B]
│└ # D2 [3 MiB]
└ # T2 [3 MiB]

environment:

dill                                              0.3.9
pydantic                                          2.10.3
pydantic_core                                     2.27.1

[mmckerns]

@zhc7: ... and I assume it throws the same error noted earlier upon load? What if you dump with recurse=True (or use dill.settings['recurse'] = True)?

[zhc7]

@zhc7: ... and I assume it throws the same error noted earlier upon load? What if you dump with recurse=True (or use dill.settings['recurse'] = True)?

from pydantic import BaseModel
import dill

dill.settings['recurse'] = True

class A(BaseModel):
    foo: str

dill.detect.trace(True)
dill.dumps(A)

┬ T2: <class '__main__.A'>
├┬ ABC: <class '__main__.A'>
│└ # ABC
├┬ F2: <function _create_type at 0x1016cbf60>
│└ # F2 [30 B]
├┬ T4: <class 'pydantic._internal._model_construction.ModelMetaclass'>
│└ # T4 [60 B]
├┬ T4: <class 'pydantic.main.BaseModel'>
│└ # T4 [30 B]
├┬ D2: <dict object at 0x010182d0c0>
│├┬ D2: <dict object at 0x0101121d40>
││├┬ T1: <class 'str'>
│││├┬ F2: <function _load_type at 0x1016cbec0>
││││└ # F2 [17 B]
│││└ # T1 [27 B]
││└ # D2 [36 B]
│├┬ D2: <dict object at 0x0101710540>
││└ # D2 [2 B]
│├┬ D2: <dict object at 0x010182d080>
││└ # D2 [2 B]
│├┬ T4: <class 'pydantic._internal._decorators.DecoratorInfos'>
││└ # T4 [52 B]
│├┬ D2: <dict object at 0x01018d3d40>
││├┬ D2: <dict object at 0x0100ab1040>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x0101849640>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01018d26c0>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01018d2700>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01018d2740>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01018d27c0>
│││└ # D2 [2 B]
││├┬ D2: <dict object at 0x01018d2800>
│││└ # D2 [2 B]
││└ # D2 [145 B]
│├┬ D2: <dict object at 0x01018d3840>
││└ # D2 [36 B]
│├┬ D2: <dict object at 0x01018d3980>
││├┬ T4: <class 'pydantic.fields.FieldInfo'>
│││└ # T4 [32 B]
││├┬ D2: <dict object at 0x01018ae900>
│││├┬ D2: <dict object at 0x01018d3e80>
││││└ # D2 [11 B]
│││└ # D2 [419 B]
││└ # D2 [463 B]
│├┬ D2: <dict object at 0x010194b300>
││├┬ T4: <class '__main__.A'>
/opt/miniconda3/lib/python3.12/site-packages/dill/_dill.py:422: PicklingWarning: Cannot locate reference to <class '__main__.A'>.
  StockPickler.save(self, obj, save_persistent_id)
/opt/miniconda3/lib/python3.12/site-packages/dill/_dill.py:422: PicklingWarning: Cannot pickle <class '__main__.A'>: __main__.A has recursive self-references that trigger a RecursionError.
  StockPickler.save(self, obj, save_persistent_id)
│││└ # T4 [6 B]
││├┬ D2: <dict object at 0x010194b280>
│││├┬ D2: <dict object at 0x010194b240>
││││├┬ D2: <dict object at 0x010194b200>
│││││├┬ D2: <dict object at 0x010194b180>
││││││└ # D2 [7 B]
│││││├┬ D2: <dict object at 0x010194b140>
││││││└ # D2 [2 B]
│││││└ # D2 [33 B]
││││└ # D2 [38 B]
│││└ # D2 [87 B]
││├┬ D2: <dict object at 0x010194b100>
│││└ # D2 [7 B]
││├┬ D2: <dict object at 0x010194b0c0>
│││├┬ Me1: <bound method BaseModel.__get_pydantic_json_schema__ of <class '__main__.A'>>
││││├┬ T1: <class 'method'>
│││││└ # T1 [19 B]
││││├┬ F1: <function BaseModel.__get_pydantic_json_schema__ at 0x1011809a0>
│││││├┬ F2: <function _create_function at 0x1016d4040>
││││││└ # F2 [23 B]
│││││├┬ Co: <code object __get_pydantic_json_schema__ at 0x100b8e5d0, file "/opt/miniconda3/lib/python3.12/site-packages/pydantic/main.py", line 704>
││││││├┬ F2: <function _create_code at 0x1016d4540>
│││││││└ # F2 [19 B]
││││││└ # Co [1 MiB]
│││││├┬ D2: <dict object at 0x01018dc740>
││││││└ # D2 [16 B]
│││││├┬ D2: <dict object at 0x01018d0140>
││││││└ # D2 [2 B]
│││││├┬ D2: <dict object at 0x0101ad4700>
││││││├┬ D2: <dict object at 0x0101173880>
│││││││└ # D2 [71 B]
││││││└ # D2 [98 B]
│││││└ # F1 [1 MiB]
││││└ # Me1 [1 MiB]
│││└ # D2 [1 MiB]
││└ # D2 [1 MiB]
│├┬ T4: <class 'pydantic_core._pydantic_core.SchemaValidator'>
││└ # T4 [51 B]
│├┬ D2: <dict object at 0x0101679280>
││└ # D2 [7 B]
│├┬ T4: <class 'pydantic_core._pydantic_core.SchemaSerializer'>
││└ # T4 [52 B]
│├┬ T4: <class 'pydantic._internal._utils.LazyClassAttribute'>
││└ # T4 [51 B]
│├┬ D2: <dict object at 0x01018dd3c0>
││├┬ T1: <class 'functools.partial'>
│││└ # T1 [20 B]
││├┬ F2: <function generate_pydantic_signature at 0x10117eca0>
│││└ # F2 [64 B]
││├┬ D2: <dict object at 0x0101ac6f00>
│││├┬ F2: <function BaseModel.__init__ at 0x10117fe20>
││││└ # F2 [25 B]
│││└ # D2 [64 B]
││└ # D2 [185 B]
│├┬ D2: <dict object at 0x01018d3a80>
││└ # D2 [2 B]
│└ # D2 [3 MiB]
└ # T2 [3 MiB]

The warning still exists.

[mmckerns]

@zhc7: Are you seeing an error upon load, or just the warning on dump?