-
-
Notifications
You must be signed in to change notification settings - Fork 30.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-98724: Fix Py_CLEAR() macro side effects #99100
Conversation
I looked at the x86-64 machine code of test_py_clear() built by gcc -O3. I don't see any
|
I completed test_py_clear() to test calling Py_CLEAR() with a side effect: test based on #98724 (comment) idea. |
I have some suggestion for the test: It should crash with the old |
It does crash with the old Py_CLEAR() macro:
|
cc @serhiy-storchaka: there is now a unit test which shows that the old macro had a bug. |
Py_SETREF? |
I don't understand your comment. This PR fix a bug in Py_CLEAR(). Can you please elaborate? |
Py_SETREF and Py_XSETREF are similar to Py_CLEAR. If change the latter, why not change the formers? |
I think all macros should be changed so they only reference their arguments once. This avoids the whole issue with side effects applied multiple times. |
That's one of the purpose of PEP 670. It's just that @erlend-aasland and me missed these macros (Py_CLEAR, Py_SETREF, Py_XSETREF). |
You're right. I completed my PR to fix also Py_SETREF() and Py_XSETREF() macros. Py_SETREF() and Py_XSETREF() macros require an additional
I added unit tests. |
I would document it as a change in 3.12. And older versions need documenting that the first argument can be evaluated more than once. |
Ok, I documented the Py_CLEAR() change with a "versionchanged". But Py_SETREF() and Py_XSETREF() are not documented. Maybe it's no purpose, so I didn't modify their (non existing) documentation. |
I would consider adding a What's New item. |
Ok, I added a What's New in Python 3.12 entry. I copied the same text everywhere. Would you mind to review the update documentation text? |
The Py_CLEAR(), Py_SETREF() and Py_XSETREF() macros now only evaluate their argument once. If the argument has side effects, these side effects are no longer duplicated. Add test_py_clear() and test_py_setref() unit tests to _testcapi.
I renamed Py_SETREF() and Py_XSETREF() arguments, I documented Py_SETREF() and Py_XSETREF(). I fixed a typo in the documentation. @erlend-aasland: Would you mind to review the updated PR? Sorry, I abused |
.. c:macro:: Py_SETREF(dst, src) | ||
|
||
Macro safely decrementing the `dst` reference count and setting `dst` to | ||
`src`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to make it clear that Py_SETREF() and Py_XSETREF() are implemented as macros.
Py_SETREF() and Py_XSETREF() are intentionally not documented because they are not in the public API. |
If I include Multiple projects use these macros. Code search in PyPI top 5000 projects using 16 projects (177 matching lines in total):
numpy and the pythoncapi-compat project (code) define these macros on Python older than 3.5.2. IMO it's too late, people are now consuming the macros, so it's good to document them. Not documenting them doesn't prevent people from using them. |
Oh, on these 16 projects, 4 projects only define Py_SETREF() and Py_XSETREF() in a copy of the pythoncapi_capi.h header file:
They don't use these macros. |
Doc/c-api/refcounting.rst contains two times an error: If the argument has side effects, there are no longer duplicated. Further down it is correct with these are no longer duplicated. |
Ah, I hesitated between both :-) I changed for "these are". |
Unfortunately, anything that's included by Python.h is de facto part of the public C API, documented or not. |
Adding these functions to the Limited API should be discussed in a separate issue. It is out of scope for gh-98724, and this PR is definitely not the correct place to discuss it. Regarding the backports, which should be discussed here (or on the linked issue): if we are not to update the docs in the backports, I don't think the backports should be made. It is unfortunate if the docs are inaccurate because of the backports. |
What if I put a "versionchanged" with Python 3.11.x and Python 3.10.x bugfix versions, rather than "3.11" and "3.10"? |
I'm not sure how to handle that correctly in Sphinx. In order to be accurate, the |
I just feel uncomfortable that these macros were documented without discussion. What is they official status? |
I don't understand your point about the fact that functions which are excluded from the limited C API should not be documented. Most of the C API is excluded from the limited C API and it is documented. For example, https://docs.python.org/dev/c-api/frame.html documents many PyFrame functions, PyFrame_GetBack() is documented and excluded from the limited C API. Only some functions have the "Part of the Stable ABI" label (see this Frame C API doc). By the way, many C API macros are documented.
Py_CLEAR(), Py_SETREF(), Py_XSETREF() are part of the public C API, are tested and documented in Python 3.12. How is it an issue? Sorry, I still don't get your point. |
That's problematic in practice if 3.10 or 3.11 get a special release just for a single security fix, not based on the current 3.10 and 3.11 branch. Well, I don't recall that it ever happened. But I would prefer to not attempt to which today what will be the 3.x.y release in 3.10, 3.11 and main branches which will include the change. I prefer to only put 3.10.x in 3.10 doc, 3.11.x in the 3.11 doc, and 3.12 in the main branch. We did that multiple times in the past for security fixes. See also:
I'm not suggesting to document Py_SETREF() change as a "notable change". I would prefer to not even document it in 3.10 and 3.11 branches. I don't get why you guys consider that Py_SETREF() is special enough to require that the bugfix is documented. But I'm fine with document it if you prefer. |
Ok, that makes sense.
+1
The easiest thing to do would of course be to just keep this bugfix/feature in |
Since there were objections and concerns, you should ask at least for opinions of @rhettinger, @pitrou and @ncoghlan. |
What's the question exactly? |
What is the status of |
I think they can be part of the public API, as they are useful helpers. |
Did you even read my messages? These macros are public and are used in the wild. Removing these macros would break projects. |
They were added as non-public with intention to make them public in the next feature release. But the latter thing did never happen. They are not officially public. |
Only in docs. But they are part of whats included in Python.h, so they are de facto public. Might as well document them. As Victor points out, removing them from Python.h is a breaking change, which proves the fact that they are public. |
Py_CLEAR() was added to Python 2.4 in 2004 with commit 8c5aeaa. Py_SETREF() was added to Python 3.6 (and 3.5.2) in 2016 with commit 57a01d3. The issue #98724 was only reported in 2022. So people were fine with duplicated side effects in Py_CLEAR() for 18 years and in Py_SETREF() for 6 years. Well, ok, I revert my Python 3.11 change: #99573 In Python 3.11 and older, there is simple workaround: avoid side effects in these macros. I also suspect that the issue #98724 doesn't come from a real application, but was just spotted as a theorical issue. So yeah, there is no urgency to fix it. And again, there are trivial ways to workaround the issue. |
I created issue #99574 "Add Py_SETREF() and Py_XSETREF() macros to the limited C API 3.12". |
GH-99573 is a backport of this pull request to the 3.11 branch. |
Ok, I reverted my 3.11 change to keep the duplication of side effects in Python 3.11. |
I reverted the PR with commit: 3a803bc Rationale: #99701 (comment) |
The Py_CLEAR(), Py_SETREF() and Py_XSETREF() macros now only evaluate their arguments once. If an argument has side effects, these side effects are no longer duplicated. Use temporary variables to avoid duplicating side effects of macro arguments. If available, use _Py_TYPEOF() to avoid type punning. Otherwise, use memcpy() for the assignment to prevent a miscompilation with strict aliasing caused by type punning. Add _Py_TYPEOF() macro: __typeof__() on GCC and clang. Add test_py_clear() and test_py_setref() unit tests to _testcapi.
|
The Py_CLEAR(), Py_SETREF() and Py_XSETREF() macros now only evaluate
their argument once. If the argument has side effects, these side
effects are no longer duplicated.
Add test_py_clear() and test_py_setref() unit tests to _testcapi.