int and set subclass __sizeof__ under-reports the instance dictionary pointer

[ionite34]

Bug report

Most built-in types, when subclassed, will have a __sizeof__ that is 8 (pointer size) larger than the parent object, due to having an instance dictionary.

class UserFloat(float): pass
assert UserFloat().__sizeof__() - (0.0).__sizeof__() == 8

class UserTuple(tuple): pass
assert UserTuple().__sizeof__() - ().__sizeof__() == 8

class UserList(list): pass
assert UserList().__sizeof__() - [].__sizeof__() == 8

class UserDict(dict): pass
assert UserDict().__sizeof__() - {}.__sizeof__() == 8

This is unexpectedly, not the case for int and set, which exactly match the original __sizeof__

class UserSet(set): pass
print(UserSet().__sizeof__(), set().__sizeof__())  # 200 200

class UserInt(int): pass
print(UserInt().__sizeof__(), (0).__sizeof__())  # 24 24

As a result this makes __dictoffset__ usages incorrect as well

from ctypes import py_object

class UserTuple(tuple): pass

x = UserTuple()
x.a = 5
print(x.__dict__)
>> {'a': 5}
py_object.from_address(id(x) + x.__sizeof__() + UserTuple.__dictoffset__)
>> py_object({'a': 5})

from ctypes import py_object

class UserInt(int): pass

x = UserInt()
x.a = 5
print(x.__dict__)
>> {'a': 5}
py_object.from_address(id(x) + x.__sizeof__() + UserInt.__dictoffset__)
>> py_object(<NULL>)

Your environment

• CPython versions tested on: 3.11, 3.12.0a3

Linked PRs

• gh-101266: Fix __sizeof__ for subclasses of int #101394
• [3.11] gh-101266: Fix __sizeof__ for subclasses of int (GH-101394) #101579
• [3.11] gh-101266: Revert fix __sizeof__ for subclasses of int #101638
• Finding a test_importlib refleak on aarch64 RHEL8 #101394 #101847

[mdickinson]

I'll leave the set part of this to someone more qualified. (@rhettinger?)

For int, I think this is an easy fix; I'll make a PR shortly.

Though oddly enough, I believe the current behaviour in main is correct (though for the wrong reasons), since UserInt.__basicsize__ now matches int.__basicsize__.

[mdickinson]

Actually, I'm not sure there's any bug here for set: the __dict__ pointer for a UserSet instance is stored in the preheader (the flags for UserSet include Py_TPFLAGS_MANAGED_DICT), and is accounted for there when computing the size with sys.getsizeof.

Python 3.12.0a4 (main, Jan 20 2023, 16:04:05) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class UserSet(set): pass
... 
>>> s = {1, 2, 3}
>>> us = UserSet(s)
>>> import sys
>>> sys.getsizeof(us) - sys.getsizeof(s)
16

[mdickinson]

The int side of this is fixed in main, with the fix backported to 3.11 (but not to 3.10, since the "correct" behaviour is a bit less clear there, and since we didn't backport the previous fix to 3.10).

I believe there's no actual issue on the set side of things (@ionite34 please comment if you disagree), so I'm going to close here.