PyUnicode_AsUnicode is deprecated, slated for removal in 3.12

[mattip]

A call to PyUnicode_AsUnicode exposes the PyUnicodeObject.wstr field. This is removed in 3.12. The documentation says it is deprecated since 3.3, and it was removed in python/cpython#92537. This will require a workaround for this code

cython/Cython/Utility/TypeConversion.c

Line 101 in 849d147

#define __Pyx_PyUnicode_AsUnicode PyUnicode_AsUnicode

which is used here

cython/Cython/Compiler/PyrexTypes.py

Line 2563 in 0e1d4bc

self.from_py_function = "__Pyx_PyUnicode_AsUnicode"

I ran up against this in the py_unicode_strings tests when it uses a unicode object in a call to a cdef function with this signature cdef assert_Py_UNICODE_equal(const Py_UNICODE* u1, const Py_UNICODE* u2)

cython/tests/run/py_unicode_strings.pyx

Line 98 in e9ffbb8

assert_Py_UNICODE_equal(uwide_literal, <Py_UNICODE*>c_pu_wide_literal)

.

The code might be hard to replace 1:1 since there is no longer a safe way to access the underlying PyUnicodeObject internal buffer, the remaining interfaces all need to deal with freeing the buffer when done with it.

[da-woods]

I'm not convinced that this is a huge issue. Essentially this conversion function only applies for pointers defined as Py_UNICODE*. If any user code uses those then it'll probably become obsolete anyway and it's not our fault.

So really the question is "should we remove the test, change the test, or only run it on supported versions"?

[da-woods]

In terms of the type conversion, I think we should just add a warning to Cython when people try to use it saying "type conversion from Python objects to Py_UNICODE* will only work on versions of Python that provide Py_UNICODE APIs. These are scheduled for removal in Python 3.12" or something similar. I don't think Cython is obliged to find workarounds beyond that.

[scoder]

only run it on supported versions

This.

add a warning to Cython

Sounds good, too, although not visible by default, as we usually do. Now that there is an actual removal, it's good to warn about it, but people who use it probably know what they're doing at some point and don't need the warning any more, even if they keep their code around for supported Python versions.

[mattip]

Closing, although I never found a good place to add a warning. Please reopen if that should be part of this issue.