diff --git a/pep-0670.rst b/pep-0670.rst index fefe73d9647..27902be96d0 100644 --- a/pep-0670.rst +++ b/pep-0670.rst @@ -12,7 +12,13 @@ Python-Version: 3.11 Abstract ======== -Convert macros to static inline functions or regular functions. +Convert macros to static inline functions or regular functions to avoid +macro pitfalls. + +Convert macros and static inline functions to regular functions to make +them usable by Python extensions which cannot use macros or static +inline functions, like extensions written in a programming languages +other than C or C++. Remove the return value of macros having a return value, whereas they should not, to aid detecting bugs in C extensions when the C API is @@ -21,6 +27,9 @@ misused. Some function arguments are still cast to ``PyObject*`` to prevent emitting new compiler warnings. +Macros which can be used as l-value in an assignment are not converted +to functions to avoid introducing incompatible changes. + Rationale ========= @@ -163,9 +172,11 @@ The following macros should not be converted: * Compatibility layer for different C compilers, C language extensions, or recent C features. Example: ``#define Py_ALWAYS_INLINE __attribute__((always_inline))``. -* Macros that need the stringification or concatenation feature of the C preprocessor. +* Macros that need C preprocessor features, like stringification and + concatenation. Example: ``Py_STRINGIFY()``. * Macros which can be used as l-value in an assignment. This change is - an incompatible change and it is out of the scope of this PEP. + an incompatible change and is out of the scope of this PEP. + Example: ``PyBytes_AS_STRING()``. Convert static inline functions to regular functions @@ -214,11 +225,35 @@ Remove the return value ----------------------- When a macro is implemented as an expression, it has an implicit return -value. This macro pitfall can be misused in third party C extensions. See -`bpo-30459 `_ regarding the misuse of the -``PyList_SET_ITEM()`` and ``PyCell_SET()`` macros. Such pitfalls are hard to -catch while reviewing macro code. Removing the return value aids detecting -bugs in C extensions when the C API is misused. +value. This return value can be misused in third party C extensions. +See `bpo-30459 `__ regarding the +misuse of the ``PyList_SET_ITEM()`` and ``PyCell_SET()`` macros. + +Such issue is hard to catch while reviewing macro code. Removing the +return value aids detecting bugs in C extensions when the C API is +misused. + +The issue has already been fixed in public C API macros by the +`bpo-30459 `__ in Python 3.10: add a +``(void)`` cast to the affected macros. Example of the +``PyTuple_SET_ITEM()`` macro:: + + #define PyTuple_SET_ITEM(op, i, v) ((void)(_PyTuple_CAST(op)->ob_item[i] = v)) + +Example of macros currently using a ``(void)`` cast to have no return +value: + +* ``PyCell_SET()`` +* ``PyList_SET_ITEM()`` +* ``PyTuple_SET_ITEM()`` +* ``Py_BUILD_ASSERT()`` +* ``_PyGCHead_SET_FINALIZED()`` +* ``_PyGCHead_SET_NEXT()`` +* ``_PyObject_ASSERT_FROM()`` +* ``_Py_atomic_signal_fence()`` +* ``_Py_atomic_store_64bit()`` +* ``asdl_seq_SET()`` +* ``asdl_seq_SET_UNTYPED()`` Backwards Compatibility @@ -227,6 +262,9 @@ Backwards Compatibility Removing the return value of macros is an incompatible API change made on purpose: see the `Remove the return value`_ section. +Some function arguments are still cast to ``PyObject*`` to prevent +emitting new compiler warnings. + Macros which can be used as l-value in an assignment are not modified by this PEP to avoid incompatible changes. @@ -238,9 +276,8 @@ Keep macros, but fix some macro issues -------------------------------------- Converting macros to functions is not needed to `remove the return -value`_: casting a macro return value to ``void`` also fix the issue. -For example, the ``PyList_SET_ITEM()`` macro was already fixed like -that. +value`_: adding a ``(void)`` cast is enough. For example, the +``PyList_SET_ITEM()`` macro was already fixed like that. Macros are always "inlined" with any C compiler. @@ -255,8 +292,11 @@ to miss a macro pitfall when writing and reviewing macro code. Moreover, macros are harder to read and maintain than functions. -Examples of duplication of side effects -======================================= +Examples of Macro Pitfalls +========================== + +Duplication of side effects +--------------------------- Macros:: @@ -269,6 +309,49 @@ Macros:: If the *op* or the *X* argument has a side effect, the side effect is duplicated: it executed twice by ``PySet_Check()`` and ``Py_IS_NAN()``. +For example, the ``pos++`` argument in the +``PyUnicode_WRITE(kind, data, pos++, ch)`` code has a side effect. +This code is safe because the ``PyUnicode_WRITE()`` macro only uses its +3rd argument once and so does not duplicate ``pos++`` side effect. + +Misnesting +---------- + +Example of the `bpo-43181: Python macros don't shield arguments +`_. The ``PyObject_TypeCheck()`` +macro before it has been fixed:: + + #define PyObject_TypeCheck(ob, tp) \ + (Py_IS_TYPE(ob, tp) || PyType_IsSubtype(Py_TYPE(ob), (tp))) + +C++ usage example:: + + PyObject_TypeCheck(ob, U(f(c))) + +The preprocessor first expands it:: + + (Py_IS_TYPE(ob, f(c)) || ...) + +C++ ``"<"`` and ``">"`` characters are not treated as brackets by the +preprocessor, so the ``Py_IS_TYPE()`` macro is invoked with 3 arguments: + +* ``ob`` +* ``f(c)`` + +The compilation fails with an error on ``Py_IS_TYPE()`` which only takes +2 arguments. + +The bug is that the *op* and *tp* arguments of ``PyObject_TypeCheck()`` +must be put between parentheses: replace ``Py_IS_TYPE(ob, tp)`` with +``Py_IS_TYPE((ob), (tp))``. In regular C code, these parentheses are +redundant, can be seen as a bug, and so are often forgotten when writing +macros. + +To avoid Macro Pitfalls, the ``PyObject_TypeCheck()`` macro has been +converted to a static inline function: +`commit `__. + Examples of hard to read macros =============================== @@ -303,7 +386,7 @@ Python 3.8 function (simplified code):: as a single long line. * Inside the function, the *op* argument has the well defined type ``PyObject*`` and so doesn't need casts like ``(PyObject *)(op)``. -* Arguments don't need to be put inside parenthesis: use ``typeobj``, +* Arguments don't need to be put inside parentheses: use ``typeobj``, rather than ``(typeobj)``. _Py_NewReference() @@ -375,6 +458,12 @@ Possible implementation as a static inlined function:: Macros converted to functions since Python 3.8 ============================================== +List of macros already converted to functions between Python 3.8 and +Python 3.11 showing that these conversions didn't not impact the Python +performance and didn't break the backward compatibility, even if some +converted macros are very commonly used by C extensions like +``Py_INCREF()``. + Macros converted to static inline functions ------------------------------------------- @@ -389,15 +478,6 @@ Python 3.8: * ``_PyObject_GC_UNTRACK()`` * ``_Py_Dealloc()`` -Python 3.10: - -* ``Py_REFCNT()`` - -Python 3.11: - -* ``Py_TYPE()`` -* ``Py_SIZE()`` - Macros converted to regular functions ------------------------------------- @@ -418,6 +498,7 @@ private static inline functions have been added to the internal C API: * ``_PyType_HasFeature()`` * ``_PyType_IS_GC()`` + Static inline functions converted to regular functions ------------------------------------------------------- @@ -434,6 +515,22 @@ private static inline function has been added to the internal C API: * ``_PyVectorcall_FunctionInline()`` +Incompatible changes +-------------------- + +While other converted macros didn't break the backward compatibility, +there are is an exception. + +The 3 macros ``Py_REFCNT()``, ``Py_TYPE()`` and ``Py_SIZE()`` have been +converted to static inline functions in Python 3.10 and 3.11 to disallow +using them as l-value in assignment. It is an incompatible change made +on purpose: see `bpo-39573 `_ for +the rationale. + +This PEP does not convert macros which can be used as l-value to avoid +introducing incompatible changes. + + Benchmark comparing macros and static inline functions ======================================================