Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python 3.13 support #442

Merged
merged 47 commits into from
Oct 10, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
a9144ea
chore(deps): upgrade numpy to support Python 3.13
Xmader Aug 21, 2024
d80c162
fix: `_PyDictView` removed from Python's public API
Xmader Aug 21, 2024
dac79df
fix: Python 3.13 removed `_PyThreadState_GetDict(PyThreadState *)` fr…
Xmader Aug 21, 2024
9b7c598
chore(CI): enable Python 3.13 build in the CI
Xmader Aug 21, 2024
fc734e2
Merge branch 'main' into Xmader/feat/python-3.13-support
Xmader Aug 21, 2024
dc4442d
WIP: remove the use of the `_PyArg_CheckPositional` private API
Xmader Aug 21, 2024
3feb5b7
fix: Python 3.13 moved the undocumented `_PyArg_UnpackKeywords` funct…
Xmader Aug 22, 2024
e4a4fda
fix: Python 3.13 moved the undocumented `_PyErr_SetKeyError` function…
Xmader Aug 22, 2024
428353c
fix: Python 3.13 added the `PyLong_AsNativeBytes` API, but changed th…
Xmader Aug 22, 2024
a96d52b
Merge branch 'main' into Xmader/feat/python-3.13-support
Xmader Sep 12, 2024
2f30f99
cleanup unused code
philippedistributive Sep 17, 2024
db5ba71
fix: `_Py_IsFinalizing` is renamed to `Py_IsFinalizing` in Python 3.13
Xmader Sep 20, 2024
07da1db
refactor: move the `_PyDictViewObject` shim to the `pyshim.hh` header…
Xmader Sep 20, 2024
a8ac1f6
feat: add shim for `_PyDictView_New`
Xmader Sep 20, 2024
9fd5412
fix: `_PyErr_SetKeyError` is removed in Python 3.13
Xmader Sep 20, 2024
8d2f1a1
WIP: `_PyThreadState_GetDict(tstate)` API gets removed in Python 3.13
Xmader Sep 20, 2024
9826939
WIP: `_PyArg_UnpackKeywords` is removed
Xmader Sep 20, 2024
f6de9c3
fix: `_PyDictViewObject` typedef redefinition with different types on…
Xmader Sep 20, 2024
7d07ee1
fix: `Py_NewRef` API is not available in Python < 3.10
Xmader Sep 20, 2024
3368d3b
Revert "WIP: remove the use of the `_PyArg_CheckPositional` private API"
Xmader Sep 20, 2024
943a431
fix: `_PyArg_CheckPositional` function became an internal API since P…
Xmader Sep 20, 2024
7c451e5
refactor: move the `_PyDictView_New` shim to pyshim.hh
Xmader Sep 20, 2024
d6e4cff
refactor: shim for `_PyArg_CheckPositional`
Xmader Sep 20, 2024
7c03098
Revert "refactor: shim for `_PyArg_CheckPositional`"
Xmader Sep 20, 2024
d3c0f85
fix: `_PyArg_UnpackKeywords` API is removed, use `PyArg_ParseTupleAnd…
Xmader Sep 20, 2024
3706a47
fix: `_PyArg_UnpackKeywords` API is removed, use `PyArg_ParseTupleAnd…
Xmader Sep 20, 2024
92798a8
fix: `_PyThreadState_GetDict(tstate)` API gets removed in Python 3.13
Xmader Sep 20, 2024
ec20bdf
fix: replacement for the `_PyThreadState_GetDict(tstate)` API in Pyth…
Xmader Sep 20, 2024
aea53bd
get rid of the build warnings
Xmader Sep 20, 2024
b2fb169
refactor: `_getLoopOnThread` getting thread dict
Xmader Sep 20, 2024
a5fbef8
feat: to get the running event-loop in Python 3.13 is as simple as `t…
Xmader Sep 20, 2024
29b6bae
feat: keep track of the local variables inside Python `eval`/`exec` t…
Xmader Sep 20, 2024
7f02431
fix: `_PyArg_CheckPositional` may also be when unpacking a tuple
Xmader Sep 23, 2024
a2d9ce8
refactor: use the `_PyDictView_New` function in Python < 3.13
Xmader Sep 23, 2024
623d83e
refactor: keep using `_PyErr_SetKeyError` in Python < 3.13
Xmader Sep 23, 2024
a1cdfb3
refactor: move `_PyErr_SetKeyError` shim to pyshim.hh
Xmader Sep 23, 2024
56b6409
refactor: rename the `PyDictViewObject_new` shim to `PyDictView_New`
Xmader Sep 25, 2024
3f7f722
refactor: add shim for `Py_SET_SIZE`
Xmader Sep 25, 2024
bcfcbf3
refactor: add shim for `PyObject_CallOneArg`
Xmader Oct 1, 2024
68c288e
refactor: `_PyObject_CallOneArg` is already an alias for `PyObject_Ca…
Xmader Oct 1, 2024
23d0576
revert commit 68c288eadaf6fe37f62b09b2107e2363eedc46b1.
Xmader Oct 1, 2024
6bf3d70
refactor: replace `PyObject_CallNoArgs(func)` calls with `PyObject_Ca…
Xmader Oct 1, 2024
c4d9405
refactor: add shim for `PyLong_AsByteArray` to work with different fu…
Xmader Oct 1, 2024
3e2aad2
feat: add back the ability to mutate parent scope variables with `pm.…
Xmader Oct 1, 2024
7ebe149
fix: add back `python.getenv` that was accidentally deleted in `3e2aad2`
Xmader Oct 1, 2024
bb78bbd
CI: use the release version of Python 3.13
Xmader Oct 8, 2024
1a98043
fix the reference count for dicts `test_get_default_not_found`
Xmader Oct 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/test-and-publish.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ on:
- '3.10'
- '3.11'
- '3.12'
- '3.13-dev'
build_type:
type: choice
description: 'Choose the build type to use'
Expand Down Expand Up @@ -132,7 +133,7 @@ jobs:
fail-fast: false
matrix:
os: [ 'ubuntu-20.04', 'macos-12', 'macos-14', 'windows-2022', 'pi' ]
python_version: [ '3.8', '3.9', '3.10', '3.11', '3.12' ]
python_version: [ '3.8', '3.9', '3.10', '3.11', '3.12', '3.13-dev' ]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
Expand Down
1 change: 1 addition & 0 deletions include/JSObjectItemsProxy.hh
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include <jsapi.h>

#include <Python.h>
#include "include/dictviewShim.hh"


/**
Expand Down
1 change: 1 addition & 0 deletions include/JSObjectKeysProxy.hh
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include <jsapi.h>

#include <Python.h>
#include "include/dictviewShim.hh"


/**
Expand Down
1 change: 1 addition & 0 deletions include/JSObjectValuesProxy.hh
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include <jsapi.h>

#include <Python.h>
#include "include/dictviewShim.hh"


/**
Expand Down
4 changes: 4 additions & 0 deletions include/PyEventLoop.hh
Original file line number Diff line number Diff line change
Expand Up @@ -331,4 +331,8 @@ private:
static inline std::vector<AsyncHandle> _timeoutIdMap;
};

// See https://github.com/python/cpython/blob/v3.13.0rc1/Python/pystate.c#L1940-L1951
// Python 3.13 removed it from the public API. Re-exporting here.
extern PyObject *_PyThreadState_GetDict(PyThreadState *tstate);

#endif
28 changes: 28 additions & 0 deletions include/dictviewShim.hh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
/**
Xmader marked this conversation as resolved.
Show resolved Hide resolved
* @file dictviewShim.hh
* @author Tom Tang ([email protected])
* @brief Since Python 3.13, `_PyDictView` moved from Python's public API to the **internal** header file `internal/pycore_dict.h`.
* This file behaves as a shim to bring back its definitions.
* The code here is copied from https://github.com/python/cpython/blob/v3.13.0rc1/Include/internal/pycore_dict.h#L64-L72.
* @date 2024-08-21
*
* @copyright Copyright (c) 2024 Distributive Corp.
*
*/

#ifndef PythonMonkey_dictview_shim_
#define PythonMonkey_dictview_shim_

#include <Python.h>

/* _PyDictView */

typedef struct {
PyObject_HEAD
PyDictObject *dv_dict;
} _PyDictViewObject;

extern PyObject *_PyDictView_New(PyObject *, PyTypeObject *);
extern PyObject *_PyDictView_Intersect(PyObject *self, PyObject *other);

#endif
512 changes: 312 additions & 200 deletions poetry.lock

Large diffs are not rendered by default.

5 changes: 3 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,9 @@ pmjs = "pythonmonkey.cli.pmjs:main"
pytest = "^7.3.1"
pip = "^23.1.2"
numpy = [
{version = "^1.26.1", python = ">=3.9,<3.13"},
{version = "^1.24.3", python = "3.8.*"},
{version = "^2.1.0", python = ">=3.10"},
{version = "^2.0.1", python = ">=3.9,<3.10"}, # NumPy 2.1.0 drops support for Python 3.9
{version = "^1.24.3", python = ">=3.8,<3.9"}, # NumPy 1.25.0 drops support for Python 3.8
]
pminit = { path = "./python/pminit", develop = true }

Expand Down
4 changes: 4 additions & 0 deletions src/IntType.cc
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,11 @@ JS::BigInt *IntType::toJsBigInt(JSContext *cx, PyObject *pyObject) {
// Convert to bytes of 8-bit "digits" in **big-endian** order
size_t byteCount = (size_t)JS_DIGIT_BYTE * jsDigitCount;
uint8_t *bytes = (uint8_t *)PyMem_Malloc(byteCount);
#if PY_VERSION_HEX >= 0x030d0000 // Python version is greater than 3.13
_PyLong_AsByteArray((PyLongObject *)pyObject, bytes, byteCount, /*is_little_endian*/ false, false, false);
#else
_PyLong_AsByteArray((PyLongObject *)pyObject, bytes, byteCount, /*is_little_endian*/ false, false);
#endif
Xmader marked this conversation as resolved.
Show resolved Hide resolved

// Convert pm.bigint to JS::BigInt through hex strings (no public API to convert directly through bytes)
// TODO (Tom Tang): We could manually allocate the memory, https://hg.mozilla.org/releases/mozilla-esr102/file/tip/js/src/vm/BigIntType.cpp#l162, but still no public API
Expand Down
20 changes: 9 additions & 11 deletions src/JSArrayProxy.cc
Original file line number Diff line number Diff line change
Expand Up @@ -789,10 +789,6 @@ PyObject *JSArrayProxyMethodDefinitions::JSArrayProxy_insert(JSArrayProxy *self,
Py_ssize_t index;
PyObject *value;

if (!_PyArg_CheckPositional("insert", nargs, 2, 2)) {
return NULL;
}

{
Py_ssize_t ival = -1;
PyObject *iobj = PyNumber_Index(args[0]);
Expand Down Expand Up @@ -899,10 +895,6 @@ PyObject *JSArrayProxyMethodDefinitions::JSArrayProxy_extend(JSArrayProxy *self,
PyObject *JSArrayProxyMethodDefinitions::JSArrayProxy_pop(JSArrayProxy *self, PyObject *const *args, Py_ssize_t nargs) {
Py_ssize_t index = -1;

if (!_PyArg_CheckPositional("pop", nargs, 0, 1)) {
return NULL;
}

if (nargs >= 1) {
Py_ssize_t ival = -1;
PyObject *iobj = PyNumber_Index(args[0]);
Expand Down Expand Up @@ -987,9 +979,6 @@ PyObject *JSArrayProxyMethodDefinitions::JSArrayProxy_index(JSArrayProxy *self,
Py_ssize_t start = 0;
Py_ssize_t stop = PY_SSIZE_T_MAX;

if (!_PyArg_CheckPositional("index", nargs, 1, 3)) {
return NULL;
}
value = args[0];
if (nargs < 2) {
goto skip_optional;
Expand Down Expand Up @@ -1179,6 +1168,15 @@ static bool sort_compare_default(JSContext *cx, unsigned argc, JS::Value *vp) {
return true;
}

// Python 3.13 moved this function to private API. Re-exporting it.
extern PyObject *const *_PyArg_UnpackKeywords(
PyObject *const *args, Py_ssize_t nargs,
PyObject *kwargs, PyObject *kwnames,
struct _PyArg_Parser *parser,
int minpos, int maxpos, int minkw,
PyObject **buf
);

PyObject *JSArrayProxyMethodDefinitions::JSArrayProxy_sort(JSArrayProxy *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) {
#if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)

Expand Down
1 change: 1 addition & 0 deletions src/JSObjectKeysProxy.cc
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <jsfriendapi.h>

#include <Python.h>
#include "include/dictviewShim.hh"



Expand Down
13 changes: 4 additions & 9 deletions src/JSObjectProxy.cc
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
#include <jsfriendapi.h>

#include <Python.h>
#include "include/dictviewShim.hh"

#include <object.h>

Expand Down Expand Up @@ -622,9 +623,6 @@ PyObject *JSObjectProxyMethodDefinitions::JSObjectProxy_get_method(JSObjectProxy
PyObject *key;
PyObject *default_value = Py_None;

if (!_PyArg_CheckPositional("get", nargs, 1, 2)) {
return NULL;
}
key = args[0];
if (nargs < 2) {
goto skip_optional;
Expand All @@ -646,9 +644,6 @@ PyObject *JSObjectProxyMethodDefinitions::JSObjectProxy_setdefault_method(JSObje
PyObject *key;
PyObject *default_value = Py_None;

if (!_PyArg_CheckPositional("setdefault", nargs, 1, 2)) {
return NULL;
}
key = args[0];
if (nargs < 2) {
goto skip_optional;
Expand All @@ -674,13 +669,13 @@ PyObject *JSObjectProxyMethodDefinitions::JSObjectProxy_setdefault_method(JSObje
return value;
}

// Python 3.13 moved this undocumented function to private API. Re-exporting it here.
extern void _PyErr_SetKeyError(PyObject *);

PyObject *JSObjectProxyMethodDefinitions::JSObjectProxy_pop_method(JSObjectProxy *self, PyObject *const *args, Py_ssize_t nargs) {
PyObject *key;
PyObject *default_value = NULL;

if (!_PyArg_CheckPositional("pop", nargs, 1, 2)) {
return NULL;
}
key = args[0];
if (nargs < 2) {
goto skip_optional;
Expand Down
1 change: 1 addition & 0 deletions src/JSObjectValuesProxy.cc
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <jsfriendapi.h>

#include <Python.h>
#include "include/dictviewShim.hh"



Expand Down
Loading