From 3d08f89c7686a5d576091a5e197084b1c10f9829 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Mon, 26 Feb 2024 23:17:39 +0000 Subject: [PATCH 01/18] wip bound docs --- guide/src/SUMMARY.md | 27 +- guide/src/conversions/tables.md | 71 +-- guide/src/function-calls.md | 1 + guide/src/index.md | 16 + guide/src/python-from-rust.md | 506 ------------------ .../python-from-rust/calling-existing-code.md | 390 ++++++++++++++ guide/src/python-from-rust/function-calls.md | 114 ++++ guide/src/rust-from-python.md | 1 + guide/src/rust_from_python.md | 13 + guide/src/types.md | 55 +- 10 files changed, 630 insertions(+), 564 deletions(-) create mode 100644 guide/src/function-calls.md create mode 100644 guide/src/python-from-rust/calling-existing-code.md create mode 100644 guide/src/python-from-rust/function-calls.md create mode 100644 guide/src/rust-from-python.md create mode 100644 guide/src/rust_from_python.md diff --git a/guide/src/SUMMARY.md b/guide/src/SUMMARY.md index 6aea71b1722..4c22c26f587 100644 --- a/guide/src/SUMMARY.md +++ b/guide/src/SUMMARY.md @@ -5,22 +5,25 @@ --- - [Getting started](getting-started.md) -- [Python modules](module.md) -- [Python functions](function.md) - - [Function signatures](function/signature.md) - - [Error handling](function/error-handling.md) -- [Python classes](class.md) - - [Class customizations](class/protocols.md) - - [Basic object customization](class/object.md) - - [Emulating numeric types](class/numeric.md) - - [Emulating callable objects](class/call.md) +- [Using Rust from Python](rust-from-python.md) + - [Python modules](module.md) + - [Python functions](function.md) + - [Function signatures](function/signature.md) + - [Error handling](function/error-handling.md) + - [Python classes](class.md) + - [Class customizations](class/protocols.md) + - [Basic object customization](class/object.md) + - [Emulating numeric types](class/numeric.md) + - [Emulating callable objects](class/call.md) +- [Calling Python from Rust](python-from-rust.md) + - [Python object types](types.md) + - [Python exceptions](exception.md) + - [Calling Python functions](python-from-rust/function-calls.md) + - [Executing existing Python code](python-from-rust/calling-existing-code.md) - [Type conversions](conversions.md) - [Mapping of Rust types to Python types](conversions/tables.md) - [Conversion traits](conversions/traits.md) -- [Python exceptions](exception.md) -- [Calling Python from Rust](python-from-rust.md) - [Using `async` and `await`](async-await.md) -- [GIL, mutability and object types](types.md) - [Parallelism](parallelism.md) - [Debugging](debugging.md) - [Features reference](features.md) diff --git a/guide/src/conversions/tables.md b/guide/src/conversions/tables.md index b6a2d30e55a..e61932f0206 100644 --- a/guide/src/conversions/tables.md +++ b/guide/src/conversions/tables.md @@ -12,49 +12,49 @@ The table below contains the Python type and the corresponding function argument | Python | Rust | Rust (Python-native) | | ------------- |:-------------------------------:|:--------------------:| -| `object` | - | `&PyAny` | -| `str` | `String`, `Cow`, `&str`, `char`, `OsString`, `PathBuf`, `Path` | `&PyString`, `&PyUnicode` | -| `bytes` | `Vec`, `&[u8]`, `Cow<[u8]>` | `&PyBytes` | -| `bool` | `bool` | `&PyBool` | -| `int` | `i8`, `u8`, `i16`, `u16`, `i32`, `u32`, `i64`, `u64`, `i128`, `u128`, `isize`, `usize`, `num_bigint::BigInt`[^1], `num_bigint::BigUint`[^1] | `&PyLong` | -| `float` | `f32`, `f64` | `&PyFloat` | -| `complex` | `num_complex::Complex`[^2] | `&PyComplex` | -| `list[T]` | `Vec` | `&PyList` | -| `dict[K, V]` | `HashMap`, `BTreeMap`, `hashbrown::HashMap`[^3], `indexmap::IndexMap`[^4] | `&PyDict` | -| `tuple[T, U]` | `(T, U)`, `Vec` | `&PyTuple` | -| `set[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^3] | `&PySet` | -| `frozenset[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^3] | `&PyFrozenSet` | -| `bytearray` | `Vec`, `Cow<[u8]>` | `&PyByteArray` | -| `slice` | - | `&PySlice` | -| `type` | - | `&PyType` | -| `module` | - | `&PyModule` | +| `object` | - | `PyAny` | +| `str` | `String`, `Cow`, `&str`, `char`, `OsString`, `PathBuf`, `Path` | `PyString`, `PyUnicode` | +| `bytes` | `Vec`, `&[u8]`, `Cow<[u8]>` | `PyBytes` | +| `bool` | `bool` | `PyBool` | +| `int` | `i8`, `u8`, `i16`, `u16`, `i32`, `u32`, `i64`, `u64`, `i128`, `u128`, `isize`, `usize`, `num_bigint::BigInt`[^1], `num_bigint::BigUint`[^1] | `PyLong` | +| `float` | `f32`, `f64` | `PyFloat` | +| `complex` | `num_complex::Complex`[^2] | `PyComplex` | +| `list[T]` | `Vec` | `PyList` | +| `dict[K, V]` | `HashMap`, `BTreeMap`, `hashbrown::HashMap`[^3], `indexmap::IndexMap`[^4] | `PyDict` | +| `tuple[T, U]` | `(T, U)`, `Vec` | `PyTuple` | +| `set[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^3] | `PySet` | +| `frozenset[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^3] | `PyFrozenSet` | +| `bytearray` | `Vec`, `Cow<[u8]>` | `PyByteArray` | +| `slice` | - | `PySlice` | +| `type` | - | `PyType` | +| `module` | - | `PyModule` | | `collections.abc.Buffer` | - | `PyBuffer` | -| `datetime.datetime` | `SystemTime`, `chrono::DateTime`[^5], `chrono::NaiveDateTime`[^5] | `&PyDateTime` | -| `datetime.date` | `chrono::NaiveDate`[^5] | `&PyDate` | -| `datetime.time` | `chrono::NaiveTime`[^5] | `&PyTime` | -| `datetime.tzinfo` | `chrono::FixedOffset`[^5], `chrono::Utc`[^5], `chrono_tz::TimeZone`[^6] | `&PyTzInfo` | -| `datetime.timedelta` | `Duration`, `chrono::Duration`[^5] | `&PyDelta` | +| `datetime.datetime` | `SystemTime`, `chrono::DateTime`[^5], `chrono::NaiveDateTime`[^5] | `PyDateTime` | +| `datetime.date` | `chrono::NaiveDate`[^5] | `PyDate` | +| `datetime.time` | `chrono::NaiveTime`[^5] | `PyTime` | +| `datetime.tzinfo` | `chrono::FixedOffset`[^5], `chrono::Utc`[^5], `chrono_tz::TimeZone`[^6] | `PyTzInfo` | +| `datetime.timedelta` | `Duration`, `chrono::Duration`[^5] | `PyDelta` | | `decimal.Decimal` | `rust_decimal::Decimal`[^7] | - | | `ipaddress.IPv4Address` | `std::net::IpAddr`, `std::net::IpV4Addr` | - | | `ipaddress.IPv6Address` | `std::net::IpAddr`, `std::net::IpV6Addr` | - | -| `os.PathLike ` | `PathBuf`, `Path` | `&PyString`, `&PyUnicode` | -| `pathlib.Path` | `PathBuf`, `Path` | `&PyString`, `&PyUnicode` | +| `os.PathLike ` | `PathBuf`, `Path` | `PyString`, `PyUnicode` | +| `pathlib.Path` | `PathBuf`, `Path` | `PyString`, `PyUnicode` | | `typing.Optional[T]` | `Option` | - | -| `typing.Sequence[T]` | `Vec` | `&PySequence` | +| `typing.Sequence[T]` | `Vec` | `PySequence` | | `typing.Mapping[K, V]` | `HashMap`, `BTreeMap`, `hashbrown::HashMap`[^3], `indexmap::IndexMap`[^4] | `&PyMapping` | -| `typing.Iterator[Any]` | - | `&PyIterator` | +| `typing.Iterator[Any]` | - | `PyIterator` | | `typing.Union[...]` | See [`#[derive(FromPyObject)]`](traits.md#deriving-frompyobject-for-enums) | - | -There are also a few special types related to the GIL and Rust-defined `#[pyclass]`es which may come in useful: +It is also worth remembering the following special types: -| What | Description | -| ------------- | ------------------------------------- | -| `Python` | A GIL token, used to pass to PyO3 constructors to prove ownership of the GIL | -| `Py` | A Python object isolated from the GIL lifetime. This can be sent to other threads. | -| `PyObject` | An alias for `Py` | -| `&PyCell` | A `#[pyclass]` value owned by Python. | -| `PyRef` | A `#[pyclass]` borrowed immutably. | -| `PyRefMut` | A `#[pyclass]` borrowed mutably. | +| What | Description | +| ---------------- | ------------------------------------- | +| `Python<'py>` | A GIL token, used to pass to PyO3 constructors to prove ownership of the GIL. | +| `Bound<'py, T>` | A Python object connected to the GIL lifetime. This provides access to most of PyO3's APIs. | +| `Py` | A Python object isolated from the GIL lifetime. This can be sent to other threads. | +| `PyObject` | An alias for `Py` | +| `PyRef` | A `#[pyclass]` borrowed immutably. | +| `PyRefMut` | A `#[pyclass]` borrowed mutably. | For more detail on accepting `#[pyclass]` values as function arguments, see [the section of this guide on Python Classes](../class.md). @@ -95,7 +95,8 @@ Finally, the following Rust types are also able to convert to Python as return v | `BTreeMap` | `Dict[K, V]` | | `HashSet` | `Set[T]` | | `BTreeSet` | `Set[T]` | -| `&PyCell` | `T` | +| `Py` | `T` | +| `Bound` | `T` | | `PyRef` | `T` | | `PyRefMut` | `T` | diff --git a/guide/src/function-calls.md b/guide/src/function-calls.md new file mode 100644 index 00000000000..e5c9c1ed9b3 --- /dev/null +++ b/guide/src/function-calls.md @@ -0,0 +1 @@ +# Calling Python functions diff --git a/guide/src/index.md b/guide/src/index.md index 80534caea5f..f47334e1d28 100644 --- a/guide/src/index.md +++ b/guide/src/index.md @@ -2,6 +2,22 @@ Welcome to the PyO3 user guide! This book is a companion to [PyO3's API docs](https://docs.rs/pyo3). It contains examples and documentation to explain all of PyO3's use cases in detail. +The rough order of material in this user guide is as follows: + 1. Getting started + 2. Wrapping Rust code for use from Python + 3. How to use Python code from Rust + 4. Remaining topics which go into advanced concepts in detail + Please choose from the chapters on the left to jump to individual topics, or continue below to start with PyO3's README. +
+⚠️ Warning: API update in progress 🛠️ + +PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer `Bound`. + +While much of the guide has been updated to the new API, it is possible some stray references to the older "GIL Refs" API such as `&PyAny` may remain. +
+ +
+ {{#include ../../README.md}} diff --git a/guide/src/python-from-rust.md b/guide/src/python-from-rust.md index 4a81d9a668f..9cc647ffa7c 100644 --- a/guide/src/python-from-rust.md +++ b/guide/src/python-from-rust.md @@ -3,509 +3,3 @@ This chapter of the guide documents some ways to interact with Python code from Rust: - How to call Python functions - How to execute existing Python code - -## Calling Python functions - -Any Python-native object reference (such as `&PyAny`, `&PyList`, or `&PyCell`) can be used to call Python functions. - -PyO3 offers two APIs to make function calls: - -* [`call`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call) - call any callable Python object. -* [`call_method`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call_method) - call a method on the Python object. - -Both of these APIs take `args` and `kwargs` arguments (for positional and keyword arguments respectively). There are variants for less complex calls: - -* [`call1`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call1) and [`call_method1`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call_method1) to call only with positional `args`. -* [`call0`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call0) and [`call_method0`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call_method0) to call with no arguments. - -For convenience the [`Py`](types.md#pyt-and-pyobject) smart pointer also exposes these same six API methods, but needs a `Python` token as an additional first argument to prove the GIL is held. - -The example below calls a Python function behind a `PyObject` (aka `Py`) reference: - -```rust -use pyo3::prelude::*; -use pyo3::types::PyTuple; - -fn main() -> PyResult<()> { - let arg1 = "arg1"; - let arg2 = "arg2"; - let arg3 = "arg3"; - - Python::with_gil(|py| { - let fun: Py = PyModule::from_code_bound( - py, - "def example(*args, **kwargs): - if args != (): - print('called with args', args) - if kwargs != {}: - print('called with kwargs', kwargs) - if args == () and kwargs == {}: - print('called with no arguments')", - "", - "", - )? - .getattr("example")? - .into(); - - // call object without any arguments - fun.call0(py)?; - - // call object with PyTuple - let args = PyTuple::new_bound(py, &[arg1, arg2, arg3]); - fun.call1(py, args)?; - - // pass arguments as rust tuple - let args = (arg1, arg2, arg3); - fun.call1(py, args)?; - Ok(()) - }) -} -``` - -### Creating keyword arguments - -For the `call` and `call_method` APIs, `kwargs` can be `None` or `Some(&PyDict)`. You can use the [`IntoPyDict`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.IntoPyDict.html) trait to convert other dict-like containers, e.g. `HashMap` or `BTreeMap`, as well as tuples with up to 10 elements and `Vec`s where each element is a two-element tuple. - -```rust -use pyo3::prelude::*; -use pyo3::types::IntoPyDict; -use std::collections::HashMap; - -fn main() -> PyResult<()> { - let key1 = "key1"; - let val1 = 1; - let key2 = "key2"; - let val2 = 2; - - Python::with_gil(|py| { - let fun: Py = PyModule::from_code_bound( - py, - "def example(*args, **kwargs): - if args != (): - print('called with args', args) - if kwargs != {}: - print('called with kwargs', kwargs) - if args == () and kwargs == {}: - print('called with no arguments')", - "", - "", - )? - .getattr("example")? - .into(); - - // call object with PyDict - let kwargs = [(key1, val1)].into_py_dict_bound(py); - fun.call_bound(py, (), Some(&kwargs))?; - - // pass arguments as Vec - let kwargs = vec![(key1, val1), (key2, val2)]; - fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?; - - // pass arguments as HashMap - let mut kwargs = HashMap::<&str, i32>::new(); - kwargs.insert(key1, 1); - fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?; - - Ok(()) - }) -} -``` - -
- -During PyO3's [migration from "GIL Refs" to the `Bound` smart pointer](./migration.md#migrating-from-the-gil-refs-api-to-boundt), [`Py::call`]({{#PYO3_DOCS_URL}}/pyo3/struct.Py.html#method.call) is temporarily named `call_bound` (and `call_method` is temporarily `call_method_bound`). - -(This temporary naming is only the case for the `Py` smart pointer. The methods on the `&PyAny` GIL Ref such as `call` have not been given replacements, and the methods on the `Bound` smart pointer such as [`Bound::call`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call) already use follow the newest API conventions.) - -
- -## Executing existing Python code - -If you already have some existing Python code that you need to execute from Rust, the following FAQs can help you select the right PyO3 functionality for your situation: - -### Want to access Python APIs? Then use `PyModule::import`. - -[`Pymodule::import`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.import) can -be used to get handle to a Python module from Rust. You can use this to import and use any Python -module available in your environment. - -```rust -use pyo3::prelude::*; - -fn main() -> PyResult<()> { - Python::with_gil(|py| { - let builtins = PyModule::import_bound(py, "builtins")?; - let total: i32 = builtins - .getattr("sum")? - .call1((vec![1, 2, 3],))? - .extract()?; - assert_eq!(total, 6); - Ok(()) - }) -} -``` - -### Want to run just an expression? Then use `eval`. - -[`Python::eval`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.eval) is -a method to execute a [Python expression](https://docs.python.org/3.7/reference/expressions.html) -and return the evaluated value as a `&PyAny` object. - -```rust -use pyo3::prelude::*; - -# fn main() -> Result<(), ()> { -Python::with_gil(|py| { - let result = py - .eval_bound("[i * 10 for i in range(5)]", None, None) - .map_err(|e| { - e.print_and_set_sys_last_vars(py); - })?; - let res: Vec = result.extract().unwrap(); - assert_eq!(res, vec![0, 10, 20, 30, 40]); - Ok(()) -}) -# } -``` - -### Want to run statements? Then use `run`. - -[`Python::run`] is a method to execute one or more -[Python statements](https://docs.python.org/3.7/reference/simple_stmts.html). -This method returns nothing (like any Python statement), but you can get -access to manipulated objects via the `locals` dict. - -You can also use the [`py_run!`] macro, which is a shorthand for [`Python::run`]. -Since [`py_run!`] panics on exceptions, we recommend you use this macro only for -quickly testing your Python extensions. - -```rust -use pyo3::prelude::*; -use pyo3::py_run; - -# fn main() { -#[pyclass] -struct UserData { - id: u32, - name: String, -} - -#[pymethods] -impl UserData { - fn as_tuple(&self) -> (u32, String) { - (self.id, self.name.clone()) - } - - fn __repr__(&self) -> PyResult { - Ok(format!("User {}(id: {})", self.name, self.id)) - } -} - -Python::with_gil(|py| { - let userdata = UserData { - id: 34, - name: "Yu".to_string(), - }; - let userdata = Py::new(py, userdata).unwrap(); - let userdata_as_tuple = (34, "Yu"); - py_run!(py, userdata userdata_as_tuple, r#" -assert repr(userdata) == "User Yu(id: 34)" -assert userdata.as_tuple() == userdata_as_tuple - "#); -}) -# } -``` - -## You have a Python file or code snippet? Then use `PyModule::from_code`. - -[`PyModule::from_code`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.from_code) -can be used to generate a Python module which can then be used just as if it was imported with -`PyModule::import`. - -**Warning**: This will compile and execute code. **Never** pass untrusted code -to this function! - -```rust -use pyo3::{ - prelude::*, - types::IntoPyDict, -}; - -# fn main() -> PyResult<()> { -Python::with_gil(|py| { - let activators = PyModule::from_code_bound( - py, - r#" -def relu(x): - """see https://en.wikipedia.org/wiki/Rectifier_(neural_networks)""" - return max(0.0, x) - -def leaky_relu(x, slope=0.01): - return x if x >= 0 else x * slope - "#, - "activators.py", - "activators", - )?; - - let relu_result: f64 = activators.getattr("relu")?.call1((-1.0,))?.extract()?; - assert_eq!(relu_result, 0.0); - - let kwargs = [("slope", 0.2)].into_py_dict_bound(py); - let lrelu_result: f64 = activators - .getattr("leaky_relu")? - .call((-1.0,), Some(&kwargs))? - .extract()?; - assert_eq!(lrelu_result, -0.2); -# Ok(()) -}) -# } -``` - -### Want to embed Python in Rust with additional modules? - -Python maintains the `sys.modules` dict as a cache of all imported modules. -An import in Python will first attempt to lookup the module from this dict, -and if not present will use various strategies to attempt to locate and load -the module. - -The [`append_to_inittab`]({{#PYO3_DOCS_URL}}/pyo3/macro.append_to_inittab.html) -macro can be used to add additional `#[pymodule]` modules to an embedded -Python interpreter. The macro **must** be invoked _before_ initializing Python. - -As an example, the below adds the module `foo` to the embedded interpreter: - -```rust -use pyo3::prelude::*; - -#[pyfunction] -fn add_one(x: i64) -> i64 { - x + 1 -} - -#[pymodule] -fn foo(foo_module: &Bound<'_, PyModule>) -> PyResult<()> { - foo_module.add_function(wrap_pyfunction!(add_one, foo_module)?)?; - Ok(()) -} - -fn main() -> PyResult<()> { - pyo3::append_to_inittab!(foo); - Python::with_gil(|py| Python::run_bound(py, "import foo; foo.add_one(6)", None, None)) -} -``` - -If `append_to_inittab` cannot be used due to constraints in the program, -an alternative is to create a module using [`PyModule::new`] -and insert it manually into `sys.modules`: - -```rust -use pyo3::prelude::*; -use pyo3::types::PyDict; - -#[pyfunction] -pub fn add_one(x: i64) -> i64 { - x + 1 -} - -fn main() -> PyResult<()> { - Python::with_gil(|py| { - // Create new module - let foo_module = PyModule::new_bound(py, "foo")?; - foo_module.add_function(wrap_pyfunction!(add_one, &foo_module)?)?; - - // Import and get sys.modules - let sys = PyModule::import_bound(py, "sys")?; - let py_modules: Bound<'_, PyDict> = sys.getattr("modules")?.downcast_into()?; - - // Insert foo into sys.modules - py_modules.set_item("foo", foo_module)?; - - // Now we can import + run our python code - Python::run_bound(py, "import foo; foo.add_one(6)", None, None) - }) -} -``` - -### Include multiple Python files - -You can include a file at compile time by using -[`std::include_str`](https://doc.rust-lang.org/std/macro.include_str.html) macro. - -Or you can load a file at runtime by using -[`std::fs::read_to_string`](https://doc.rust-lang.org/std/fs/fn.read_to_string.html) function. - -Many Python files can be included and loaded as modules. If one file depends on -another you must preserve correct order while declaring `PyModule`. - -Example directory structure: -```text -. -├── Cargo.lock -├── Cargo.toml -├── python_app -│ ├── app.py -│ └── utils -│ └── foo.py -└── src - └── main.rs -``` - -`python_app/app.py`: -```python -from utils.foo import bar - - -def run(): - return bar() -``` - -`python_app/utils/foo.py`: -```python -def bar(): - return "baz" -``` - -The example below shows: -* how to include content of `app.py` and `utils/foo.py` into your rust binary -* how to call function `run()` (declared in `app.py`) that needs function - imported from `utils/foo.py` - -`src/main.rs`: -```rust,ignore -use pyo3::prelude::*; - -fn main() -> PyResult<()> { - let py_foo = include_str!(concat!( - env!("CARGO_MANIFEST_DIR"), - "/python_app/utils/foo.py" - )); - let py_app = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/python_app/app.py")); - let from_python = Python::with_gil(|py| -> PyResult> { - PyModule::from_code_bound(py, py_foo, "utils.foo", "utils.foo")?; - let app: Py = PyModule::from_code_bound(py, py_app, "", "")? - .getattr("run")? - .into(); - app.call0(py) - }); - - println!("py: {}", from_python?); - Ok(()) -} -``` - -The example below shows: -* how to load content of `app.py` at runtime so that it sees its dependencies - automatically -* how to call function `run()` (declared in `app.py`) that needs function - imported from `utils/foo.py` - -It is recommended to use absolute paths because then your binary can be run -from anywhere as long as your `app.py` is in the expected directory (in this example -that directory is `/usr/share/python_app`). - -`src/main.rs`: -```rust,no_run -use pyo3::prelude::*; -use pyo3::types::PyList; -use std::fs; -use std::path::Path; - -fn main() -> PyResult<()> { - let path = Path::new("/usr/share/python_app"); - let py_app = fs::read_to_string(path.join("app.py"))?; - let from_python = Python::with_gil(|py| -> PyResult> { - let syspath = py.import_bound("sys")?.getattr("path")?.downcast_into::()?; - syspath.insert(0, &path)?; - let app: Py = PyModule::from_code_bound(py, &py_app, "", "")? - .getattr("run")? - .into(); - app.call0(py) - }); - - println!("py: {}", from_python?); - Ok(()) -} -``` - - -[`Python::run`]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.run -[`py_run!`]: {{#PYO3_DOCS_URL}}/pyo3/macro.py_run.html - -## Need to use a context manager from Rust? - -Use context managers by directly invoking `__enter__` and `__exit__`. - -```rust -use pyo3::prelude::*; - -fn main() { - Python::with_gil(|py| { - let custom_manager = PyModule::from_code_bound( - py, - r#" -class House(object): - def __init__(self, address): - self.address = address - def __enter__(self): - print(f"Welcome to {self.address}!") - def __exit__(self, type, value, traceback): - if type: - print(f"Sorry you had {type} trouble at {self.address}") - else: - print(f"Thank you for visiting {self.address}, come again soon!") - - "#, - "house.py", - "house", - ) - .unwrap(); - - let house_class = custom_manager.getattr("House").unwrap(); - let house = house_class.call1(("123 Main Street",)).unwrap(); - - house.call_method0("__enter__").unwrap(); - - let result = py.eval_bound("undefined_variable + 1", None, None); - - // If the eval threw an exception we'll pass it through to the context manager. - // Otherwise, __exit__ is called with empty arguments (Python "None"). - match result { - Ok(_) => { - let none = py.None(); - house - .call_method1("__exit__", (&none, &none, &none)) - .unwrap(); - } - Err(e) => { - house - .call_method1("__exit__", (e.get_type_bound(py), e.value_bound(py), e.traceback_bound(py))) - .unwrap(); - } - } - }) -} -``` - -## Handling system signals/interrupts (Ctrl-C) - -The best way to handle system signals when running Rust code is to periodically call `Python::check_signals` to handle any signals captured by Python's signal handler. See also [the FAQ entry](./faq.md#ctrl-c-doesnt-do-anything-while-my-rust-code-is-executing). - -Alternatively, set Python's `signal` module to take the default action for a signal: - -```rust -use pyo3::prelude::*; - -# fn main() -> PyResult<()> { -Python::with_gil(|py| -> PyResult<()> { - let signal = py.import_bound("signal")?; - // Set SIGINT to have the default action - signal - .getattr("signal")? - .call1((signal.getattr("SIGINT")?, signal.getattr("SIG_DFL")?))?; - Ok(()) -}) -# } -``` - - -[`PyModule::new`]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.new diff --git a/guide/src/python-from-rust/calling-existing-code.md b/guide/src/python-from-rust/calling-existing-code.md new file mode 100644 index 00000000000..0fef7f2dc17 --- /dev/null +++ b/guide/src/python-from-rust/calling-existing-code.md @@ -0,0 +1,390 @@ +# Executing existing Python code + +If you already have some existing Python code that you need to execute from Rust, the following FAQs can help you select the right PyO3 functionality for your situation: + +## Want to access Python APIs? Then use `PyModule::import`. + +[`Pymodule::import`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.import) can +be used to get handle to a Python module from Rust. You can use this to import and use any Python +module available in your environment. + +```rust +use pyo3::prelude::*; + +fn main() -> PyResult<()> { + Python::with_gil(|py| { + let builtins = PyModule::import_bound(py, "builtins")?; + let total: i32 = builtins + .getattr("sum")? + .call1((vec![1, 2, 3],))? + .extract()?; + assert_eq!(total, 6); + Ok(()) + }) +} +``` + +## Want to run just an expression? Then use `eval`. + +[`Python::eval`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.eval) is +a method to execute a [Python expression](https://docs.python.org/3.7/reference/expressions.html) +and return the evaluated value as a `&PyAny` object. + +```rust +use pyo3::prelude::*; + +# fn main() -> Result<(), ()> { +Python::with_gil(|py| { + let result = py + .eval_bound("[i * 10 for i in range(5)]", None, None) + .map_err(|e| { + e.print_and_set_sys_last_vars(py); + })?; + let res: Vec = result.extract().unwrap(); + assert_eq!(res, vec![0, 10, 20, 30, 40]); + Ok(()) +}) +# } +``` + +## Want to run statements? Then use `run`. + +[`Python::run`] is a method to execute one or more +[Python statements](https://docs.python.org/3.7/reference/simple_stmts.html). +This method returns nothing (like any Python statement), but you can get +access to manipulated objects via the `locals` dict. + +You can also use the [`py_run!`] macro, which is a shorthand for [`Python::run`]. +Since [`py_run!`] panics on exceptions, we recommend you use this macro only for +quickly testing your Python extensions. + +```rust +use pyo3::prelude::*; +use pyo3::py_run; + +# fn main() { +#[pyclass] +struct UserData { + id: u32, + name: String, +} + +#[pymethods] +impl UserData { + fn as_tuple(&self) -> (u32, String) { + (self.id, self.name.clone()) + } + + fn __repr__(&self) -> PyResult { + Ok(format!("User {}(id: {})", self.name, self.id)) + } +} + +Python::with_gil(|py| { + let userdata = UserData { + id: 34, + name: "Yu".to_string(), + }; + let userdata = Py::new(py, userdata).unwrap(); + let userdata_as_tuple = (34, "Yu"); + py_run!(py, userdata userdata_as_tuple, r#" +assert repr(userdata) == "User Yu(id: 34)" +assert userdata.as_tuple() == userdata_as_tuple + "#); +}) +# } +``` + +## You have a Python file or code snippet? Then use `PyModule::from_code`. + +[`PyModule::from_code`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.from_code) +can be used to generate a Python module which can then be used just as if it was imported with +`PyModule::import`. + +**Warning**: This will compile and execute code. **Never** pass untrusted code +to this function! + +```rust +use pyo3::{ + prelude::*, + types::IntoPyDict, +}; + +# fn main() -> PyResult<()> { +Python::with_gil(|py| { + let activators = PyModule::from_code_bound( + py, + r#" +def relu(x): + """see https://en.wikipedia.org/wiki/Rectifier_(neural_networks)""" + return max(0.0, x) + +def leaky_relu(x, slope=0.01): + return x if x >= 0 else x * slope + "#, + "activators.py", + "activators", + )?; + + let relu_result: f64 = activators.getattr("relu")?.call1((-1.0,))?.extract()?; + assert_eq!(relu_result, 0.0); + + let kwargs = [("slope", 0.2)].into_py_dict_bound(py); + let lrelu_result: f64 = activators + .getattr("leaky_relu")? + .call((-1.0,), Some(&kwargs))? + .extract()?; + assert_eq!(lrelu_result, -0.2); +# Ok(()) +}) +# } +``` + +## Want to embed Python in Rust with additional modules? + +Python maintains the `sys.modules` dict as a cache of all imported modules. +An import in Python will first attempt to lookup the module from this dict, +and if not present will use various strategies to attempt to locate and load +the module. + +The [`append_to_inittab`]({{#PYO3_DOCS_URL}}/pyo3/macro.append_to_inittab.html) +macro can be used to add additional `#[pymodule]` modules to an embedded +Python interpreter. The macro **must** be invoked _before_ initializing Python. + +As an example, the below adds the module `foo` to the embedded interpreter: + +```rust +use pyo3::prelude::*; + +#[pyfunction] +fn add_one(x: i64) -> i64 { + x + 1 +} + +#[pymodule] +fn foo(foo_module: &Bound<'_, PyModule>) -> PyResult<()> { + foo_module.add_function(wrap_pyfunction!(add_one, foo_module)?)?; + Ok(()) +} + +fn main() -> PyResult<()> { + pyo3::append_to_inittab!(foo); + Python::with_gil(|py| Python::run_bound(py, "import foo; foo.add_one(6)", None, None)) +} +``` + +If `append_to_inittab` cannot be used due to constraints in the program, +an alternative is to create a module using [`PyModule::new`] +and insert it manually into `sys.modules`: + +```rust +use pyo3::prelude::*; +use pyo3::types::PyDict; + +#[pyfunction] +pub fn add_one(x: i64) -> i64 { + x + 1 +} + +fn main() -> PyResult<()> { + Python::with_gil(|py| { + // Create new module + let foo_module = PyModule::new_bound(py, "foo")?; + foo_module.add_function(wrap_pyfunction!(add_one, &foo_module)?)?; + + // Import and get sys.modules + let sys = PyModule::import_bound(py, "sys")?; + let py_modules: Bound<'_, PyDict> = sys.getattr("modules")?.downcast_into()?; + + // Insert foo into sys.modules + py_modules.set_item("foo", foo_module)?; + + // Now we can import + run our python code + Python::run_bound(py, "import foo; foo.add_one(6)", None, None) + }) +} +``` + +## Include multiple Python files + +You can include a file at compile time by using +[`std::include_str`](https://doc.rust-lang.org/std/macro.include_str.html) macro. + +Or you can load a file at runtime by using +[`std::fs::read_to_string`](https://doc.rust-lang.org/std/fs/fn.read_to_string.html) function. + +Many Python files can be included and loaded as modules. If one file depends on +another you must preserve correct order while declaring `PyModule`. + +Example directory structure: +```text +. +├── Cargo.lock +├── Cargo.toml +├── python_app +│ ├── app.py +│ └── utils +│ └── foo.py +└── src + └── main.rs +``` + +`python_app/app.py`: +```python +from utils.foo import bar + + +def run(): + return bar() +``` + +`python_app/utils/foo.py`: +```python +def bar(): + return "baz" +``` + +The example below shows: +* how to include content of `app.py` and `utils/foo.py` into your rust binary +* how to call function `run()` (declared in `app.py`) that needs function + imported from `utils/foo.py` + +`src/main.rs`: +```rust,ignore +use pyo3::prelude::*; + +fn main() -> PyResult<()> { + let py_foo = include_str!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/python_app/utils/foo.py" + )); + let py_app = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/python_app/app.py")); + let from_python = Python::with_gil(|py| -> PyResult> { + PyModule::from_code_bound(py, py_foo, "utils.foo", "utils.foo")?; + let app: Py = PyModule::from_code_bound(py, py_app, "", "")? + .getattr("run")? + .into(); + app.call0(py) + }); + + println!("py: {}", from_python?); + Ok(()) +} +``` + +The example below shows: +* how to load content of `app.py` at runtime so that it sees its dependencies + automatically +* how to call function `run()` (declared in `app.py`) that needs function + imported from `utils/foo.py` + +It is recommended to use absolute paths because then your binary can be run +from anywhere as long as your `app.py` is in the expected directory (in this example +that directory is `/usr/share/python_app`). + +`src/main.rs`: +```rust,no_run +use pyo3::prelude::*; +use pyo3::types::PyList; +use std::fs; +use std::path::Path; + +fn main() -> PyResult<()> { + let path = Path::new("/usr/share/python_app"); + let py_app = fs::read_to_string(path.join("app.py"))?; + let from_python = Python::with_gil(|py| -> PyResult> { + let syspath = py.import_bound("sys")?.getattr("path")?.downcast_into::()?; + syspath.insert(0, &path)?; + let app: Py = PyModule::from_code_bound(py, &py_app, "", "")? + .getattr("run")? + .into(); + app.call0(py) + }); + + println!("py: {}", from_python?); + Ok(()) +} +``` + + +[`Python::run`]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.run +[`py_run!`]: {{#PYO3_DOCS_URL}}/pyo3/macro.py_run.html + +#£ Need to use a context manager from Rust? + +Use context managers by directly invoking `__enter__` and `__exit__`. + +```rust +use pyo3::prelude::*; + +fn main() { + Python::with_gil(|py| { + let custom_manager = PyModule::from_code_bound( + py, + r#" +class House(object): + def __init__(self, address): + self.address = address + def __enter__(self): + print(f"Welcome to {self.address}!") + def __exit__(self, type, value, traceback): + if type: + print(f"Sorry you had {type} trouble at {self.address}") + else: + print(f"Thank you for visiting {self.address}, come again soon!") + + "#, + "house.py", + "house", + ) + .unwrap(); + + let house_class = custom_manager.getattr("House").unwrap(); + let house = house_class.call1(("123 Main Street",)).unwrap(); + + house.call_method0("__enter__").unwrap(); + + let result = py.eval_bound("undefined_variable + 1", None, None); + + // If the eval threw an exception we'll pass it through to the context manager. + // Otherwise, __exit__ is called with empty arguments (Python "None"). + match result { + Ok(_) => { + let none = py.None(); + house + .call_method1("__exit__", (&none, &none, &none)) + .unwrap(); + } + Err(e) => { + house + .call_method1("__exit__", (e.get_type_bound(py), e.value_bound(py), e.traceback_bound(py))) + .unwrap(); + } + } + }) +} +``` + +## Handling system signals/interrupts (Ctrl-C) + +The best way to handle system signals when running Rust code is to periodically call `Python::check_signals` to handle any signals captured by Python's signal handler. See also [the FAQ entry](../faq.md#ctrl-c-doesnt-do-anything-while-my-rust-code-is-executing). + +Alternatively, set Python's `signal` module to take the default action for a signal: + +```rust +use pyo3::prelude::*; + +# fn main() -> PyResult<()> { +Python::with_gil(|py| -> PyResult<()> { + let signal = py.import_bound("signal")?; + // Set SIGINT to have the default action + signal + .getattr("signal")? + .call1((signal.getattr("SIGINT")?, signal.getattr("SIG_DFL")?))?; + Ok(()) +}) +# } +``` + + +[`PyModule::new`]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.new diff --git a/guide/src/python-from-rust/function-calls.md b/guide/src/python-from-rust/function-calls.md new file mode 100644 index 00000000000..f97de1f24ce --- /dev/null +++ b/guide/src/python-from-rust/function-calls.md @@ -0,0 +1,114 @@ +# Calling Python functions + +The `Bound<'py, T>` smart pointer (such as `Bound<'py, PyAny>`, `Bound<'py, PyList>`, or `Bound<'py, MyClass>`) can be used to call Python functions. + +PyO3 offers two APIs to make function calls: + +* [`call`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call) - call any callable Python object. +* [`call_method`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call_method) - call a method on the Python object. + +Both of these APIs take `args` and `kwargs` arguments (for positional and keyword arguments respectively). There are variants for less complex calls: + +* [`call1`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call1) and [`call_method1`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call_method1) to call only with positional `args`. +* [`call0`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call0) and [`call_method0`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call_method0) to call with no arguments. + +For convenience the [`Py`](../types.md#pyt-and-pyobject) smart pointer also exposes these same six API methods, but needs a `Python` token as an additional first argument to prove the GIL is held. + +The example below calls a Python function behind a `PyObject` (aka `Py`) reference: + +```rust +use pyo3::prelude::*; +use pyo3::types::PyTuple; + +fn main() -> PyResult<()> { + let arg1 = "arg1"; + let arg2 = "arg2"; + let arg3 = "arg3"; + + Python::with_gil(|py| { + let fun: Py = PyModule::from_code_bound( + py, + "def example(*args, **kwargs): + if args != (): + print('called with args', args) + if kwargs != {}: + print('called with kwargs', kwargs) + if args == () and kwargs == {}: + print('called with no arguments')", + "", + "", + )? + .getattr("example")? + .into(); + + // call object without any arguments + fun.call0(py)?; + + // pass object with Rust tuple of positional arguments + let args = (arg1, arg2, arg3); + fun.call1(py, args)?; + + // call object with Python tuple of positional arguments + let args = PyTuple::new_bound(py, &[arg1, arg2, arg3]); + fun.call1(py, args)?; + Ok(()) + }) +} +``` + +## Creating keyword arguments + +For the `call` and `call_method` APIs, `kwargs` are `Option<&Bound<'py, PyDict>>`, so can either be `None` or `Some(&dict)`. You can use the [`IntoPyDict`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.IntoPyDict.html) trait to convert other dict-like containers, e.g. `HashMap` or `BTreeMap`, as well as tuples with up to 10 elements and `Vec`s where each element is a two-element tuple. + +```rust +use pyo3::prelude::*; +use pyo3::types::IntoPyDict; +use std::collections::HashMap; + +fn main() -> PyResult<()> { + let key1 = "key1"; + let val1 = 1; + let key2 = "key2"; + let val2 = 2; + + Python::with_gil(|py| { + let fun: Py = PyModule::from_code_bound( + py, + "def example(*args, **kwargs): + if args != (): + print('called with args', args) + if kwargs != {}: + print('called with kwargs', kwargs) + if args == () and kwargs == {}: + print('called with no arguments')", + "", + "", + )? + .getattr("example")? + .into(); + + // call object with PyDict + let kwargs = [(key1, val1)].into_py_dict_bound(py); + fun.call_bound(py, (), Some(&kwargs))?; + + // pass arguments as Vec + let kwargs = vec![(key1, val1), (key2, val2)]; + fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?; + + // pass arguments as HashMap + let mut kwargs = HashMap::<&str, i32>::new(); + kwargs.insert(key1, 1); + fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?; + + Ok(()) + }) +} +``` + +
+ +During PyO3's [migration from "GIL Refs" to the `Bound` smart pointer](../migration.md#migrating-from-the-gil-refs-api-to-boundt), [`Py::call`]({{#PYO3_DOCS_URL}}/pyo3/struct.Py.html#method.call) is temporarily named `call_bound` (and `call_method` is temporarily `call_method_bound`). + +(This temporary naming is only the case for the `Py` smart pointer. The methods on the `&PyAny` GIL Ref such as `call` have not been given replacements, and the methods on the `Bound` smart pointer such as [`Bound::call`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.call) already use follow the newest API conventions.) + +
diff --git a/guide/src/rust-from-python.md b/guide/src/rust-from-python.md new file mode 100644 index 00000000000..8e74bcae219 --- /dev/null +++ b/guide/src/rust-from-python.md @@ -0,0 +1 @@ +# Using Rust from Python diff --git a/guide/src/rust_from_python.md b/guide/src/rust_from_python.md new file mode 100644 index 00000000000..470d5719098 --- /dev/null +++ b/guide/src/rust_from_python.md @@ -0,0 +1,13 @@ +# Using Rust from Python + +This chapter of the guide is dedicated to explaining how to wrap Rust code into Python objects. + +PyO3 uses Rust's "procedural macros" to provide a powerful yet simple API to denote what Rust code should map into Python objects. + +The three types of Python objects which PyO3 can produce are: + +- Python modules, via the `#[pymodule]` macro +- Python functions, via the `#[pyfunction]` macro +- Python classes, via the `#[pyclass]` macro (plus `#[pymethods]` to define methods for those clases) + +The following subchapters go through each of these in turn. diff --git a/guide/src/types.md b/guide/src/types.md index 372c6c8632f..c2d5988409d 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -1,22 +1,55 @@ -# GIL lifetimes, mutability and Python object types +# Python object types -On first glance, PyO3 provides a huge number of different types that can be used +PyO3 offers two main sets of types to interact with Python objects. This section of the guide expands into detail about these types and how to choose which to use. + +The first set of types is are the "smart pointers" which all Python objects are wrapped in. These are `Py`, `Bound<'py, T>`, and `Borrowed<'a, 'py, T>`. The [first section below](#pyo3s-smart-pointers) expands on each of these in detail and why there are three of them. + +The second set of types are types which fill in the generic parameter `T` of the smart pointers. The most common is `PyAny`, which represents any Python object (similar to Python's `typing.Any`). There are also concrete types for many Python built-in types, such as `PyList`, `PyDict`, and `PyTuple`. User defined `#[pyclass]` types also fit this category. The [second section below](#concrete-python-types) expands on how to use these types. + +Before PyO3 0.21, PyO3's main API to interact with Python objects was a deprecated API known as the "GIL Refs" API, containing reference types such as `&PyAny`, `&PyList`, and `&PyCell` for user-defined `#[pyclass]` types. The [third section below](#the-gil-refs-api) details this deprecated API. + +## PyO3's smart pointers + +PyO3's API offers three generic smart pointers: `Py`, `Bound<'py, T>` and `Borrowed<'a, 'py, T>`. For each of these the type parameter `T` will be filled by a [concrete Python type](#concrete-python-types). For example, a Python list object can be represented by `Py`, `Bound<'py, PyList>`, and `Borrowed<'a, 'py, PyList>`. + +These smart pointers have different numbers of lifetime parameters, which defines how they behave. `Py` has no lifetime parameters, `Bound<'py, T>` has a lifetime parameter `'py`, and `Borrowed<'a, 'py, T>` has two lifetime parameters `'a` and `'py`. + +Python objects are reference counted, like [`std::sync::Arc`](https://doc.rust-lang.org/stable/std/sync/struct.Arc.html). A major reason for these smart pointers is to bring Python's reference counting to a Rust API. + +The recommendation of when to use each of these smart pointers is as follows: + +- Use `Bound<'py, T>` for as much as possible, as it offers the most efficient and complete API. +- Use `Py` mostly just for storage inside Rust `struct`s which do not want to add a lifetime parameter for `Bound<'py, T>`. +- `Borrowed<'a, 'py, T>` is almost never used. It is occasionally present at the boundary between Rust and the Python interpreter, for example when borrowing data from Python tuples (which is safe because they are immutable). + +The sections below also explain these smart pointers in a little more detail. + +### `Py` (and `PyObject`) + +### `Bound<'py, T>` + +### `Borrowed<'a, 'py, T>` + +GIL lifetimes, mutability and Python object types + +At first glance, PyO3 provides a huge number of different types that can be used to wrap or refer to Python objects. This page delves into the details and gives an overview of their intended meaning, with examples when each type is best used. +## Concrete Python types + + ## The Python GIL, mutability, and Rust types -Since Python has no concept of ownership, and works solely with boxed objects, -any Python object can be referenced any number of times, and mutation is allowed -from any reference. +Python code differs from Rust in two key ways: +- There is no concept of ownership; all Python objects are reference counted +- There is no concept of exclusive (`&mut`) references; any reference can mutate a Python object + +PyO3's API for interacting with Python objects is built with this in mind. There are two generic "smart pointers" to Python objects, `Py` and `Bound<'py, T>`, which both use Python reference counting as their memory management. Almost all methods on these smart pointers use `&self` receivers, that is, a `&mut Py` or `&mut Bound<'py, T>` are almost never used in PyO3 code. -The situation is helped a little by the Global Interpreter Lock (GIL), which -ensures that only one thread can use the Python interpreter and its API at the -same time, while non-Python operations (system calls and extension code) can -unlock the GIL. (See [the section on parallelism](parallelism.md) for how to do -that in PyO3.) +The situation is helped by the Global Interpreter Lock (GIL), which ensures that only one thread can use the Python interpreter and its API at the same time, while non-Python operations (system calls and extension code) can unlock the GIL. (See [the section on parallelism](parallelism.md) for how to do that in PyO3.) In PyO3, holding the GIL is modeled by acquiring a token of the type `Python<'py>`, which serves three purposes: @@ -47,7 +80,7 @@ references is done at runtime using `PyCell`, a scheme very similar to To get hold of a `Python<'py>` token to prove the GIL is held, consult [PyO3's documentation][obtaining-py]. -## Object types +## The GIL Refs API ### [`PyAny`][PyAny] From 11de7f081b43d019d79b7fda0bc46f744ca6fcc8 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Thu, 7 Mar 2024 09:45:07 +0000 Subject: [PATCH 02/18] Update guide/src/python_from_rust/calling-existing-code.md Co-authored-by: Lily Foote --- guide/src/python-from-rust/calling-existing-code.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guide/src/python-from-rust/calling-existing-code.md b/guide/src/python-from-rust/calling-existing-code.md index 0fef7f2dc17..46a6b6ebbed 100644 --- a/guide/src/python-from-rust/calling-existing-code.md +++ b/guide/src/python-from-rust/calling-existing-code.md @@ -310,7 +310,7 @@ fn main() -> PyResult<()> { [`Python::run`]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.run [`py_run!`]: {{#PYO3_DOCS_URL}}/pyo3/macro.py_run.html -#£ Need to use a context manager from Rust? +## Need to use a context manager from Rust? Use context managers by directly invoking `__enter__` and `__exit__`. From 6d4093d85b40c97b2b29e469c13ad91c8f8ae797 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Thu, 7 Mar 2024 23:34:52 +0000 Subject: [PATCH 03/18] continue to move and tidy up --- guide/src/index.md | 2 +- guide/src/python-from-rust.md | 46 +++++++++++++++++++++++++++++++++++ guide/src/types.md | 40 ------------------------------ 3 files changed, 47 insertions(+), 41 deletions(-) diff --git a/guide/src/index.md b/guide/src/index.md index f47334e1d28..87914975afe 100644 --- a/guide/src/index.md +++ b/guide/src/index.md @@ -15,7 +15,7 @@ Please choose from the chapters on the left to jump to individual topics, or con PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer `Bound`. -While much of the guide has been updated to the new API, it is possible some stray references to the older "GIL Refs" API such as `&PyAny` may remain. +While most of this guide has been updated to the new API, it is possible some stray references to the older "GIL Refs" API such as `&PyAny` remain.
diff --git a/guide/src/python-from-rust.md b/guide/src/python-from-rust.md index 9cc647ffa7c..f903a6531c3 100644 --- a/guide/src/python-from-rust.md +++ b/guide/src/python-from-rust.md @@ -3,3 +3,49 @@ This chapter of the guide documents some ways to interact with Python code from Rust: - How to call Python functions - How to execute existing Python code + +## Interacting with the Python interpreter + +To safely interact with the Python interpreter a Rust thread must have a corresponding Python thread state and hold the Global Interpreter Lock (GIL). PyO3 has a `Python<'py>` token which is used to prove that these conditions +are met. It's lifetime `'py` is a central part of PyO3's API. + +The `Python<'py>` token serves three purposes: + +* It provides global API for the Python interpreter, such as [`py.eval()`][eval] and [`py.import()`][import]. +* It can be passed to functions that require a proof of holding the GIL, such as [`Py::clone_ref`][clone_ref]. +* Its lifetime `'py` is used to bind many of PyO3's types to the Python interpreter, such as [`Bound<'py, T>`][Bound]. + +PyO3's types which are bound to the `'py` lifetime, for example `Bound<'py, T>`, all contain a `Python<'py>` token. This means they have full access to the Python interpreter and offer a complete API for interacting with Python objects. + +Consult [PyO3's API documentation][obtaining-py] to learn how to acquire one of these tokens. + +## Python's memory model + +Python memory model differs from Rust's memory model in two key ways: +- There is no concept of ownership; all Python objects are reference counted +- There is no concept of exclusive (`&mut`) references; any reference can mutate a Python object + +PyO3's API reflects this by providing smart pointer types, `Py`, `Bound<'py, T>`, and (the very rarely used) `Borrowed<'a, 'py, T>`. These smart pointers all use Python reference counting. See the [subchapter on types](./types.md) for more detail on these types. + +### Mutable access to `#[pyclass]` types + +The situation is helped by the Global Interpreter Lock (GIL), which ensures that only one thread can use the Python interpreter and its API at the same time, while non-Python operations (system calls and extension code) can unlock the GIL. (See [the section on parallelism](parallelism.md) for how to do that in PyO3.) + + + +The latter two points are the reason why some APIs in PyO3 require the `py: +Python` argument, while others don't. + +The PyO3 API for Python objects is written such that instead of requiring a +mutable Rust reference for mutating operations such as +[`PyList::append`][PyList_append], a shared reference (which, in turn, can only +be created through `Python<'_>` with a GIL lifetime) is sufficient. + +However, Rust structs wrapped as Python objects (called `pyclass` types) usually +*do* need `&mut` access. Due to the GIL, PyO3 *can* guarantee thread-safe access +to them, but it cannot statically guarantee uniqueness of `&mut` references once +an object's ownership has been passed to the Python interpreter, ensuring +references is done at runtime using `PyCell`, a scheme very similar to +`std::cell::RefCell`. + +[obtaining-py]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#obtaining-a-python-token diff --git a/guide/src/types.md b/guide/src/types.md index c2d5988409d..95ebb41dd3a 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -41,45 +41,6 @@ used. -## The Python GIL, mutability, and Rust types - -Python code differs from Rust in two key ways: -- There is no concept of ownership; all Python objects are reference counted -- There is no concept of exclusive (`&mut`) references; any reference can mutate a Python object - -PyO3's API for interacting with Python objects is built with this in mind. There are two generic "smart pointers" to Python objects, `Py` and `Bound<'py, T>`, which both use Python reference counting as their memory management. Almost all methods on these smart pointers use `&self` receivers, that is, a `&mut Py` or `&mut Bound<'py, T>` are almost never used in PyO3 code. - -The situation is helped by the Global Interpreter Lock (GIL), which ensures that only one thread can use the Python interpreter and its API at the same time, while non-Python operations (system calls and extension code) can unlock the GIL. (See [the section on parallelism](parallelism.md) for how to do that in PyO3.) - -In PyO3, holding the GIL is modeled by acquiring a token of the type -`Python<'py>`, which serves three purposes: - -* It provides some global API for the Python interpreter, such as - [`eval`][eval]. -* It can be passed to functions that require a proof of holding the GIL, - such as [`Py::clone_ref`][clone_ref]. -* Its lifetime can be used to create Rust references that implicitly guarantee - holding the GIL, such as [`&'py PyAny`][PyAny]. - -The latter two points are the reason why some APIs in PyO3 require the `py: -Python` argument, while others don't. - -The PyO3 API for Python objects is written such that instead of requiring a -mutable Rust reference for mutating operations such as -[`PyList::append`][PyList_append], a shared reference (which, in turn, can only -be created through `Python<'_>` with a GIL lifetime) is sufficient. - -However, Rust structs wrapped as Python objects (called `pyclass` types) usually -*do* need `&mut` access. Due to the GIL, PyO3 *can* guarantee thread-safe access -to them, but it cannot statically guarantee uniqueness of `&mut` references once -an object's ownership has been passed to the Python interpreter, ensuring -references is done at runtime using `PyCell`, a scheme very similar to -`std::cell::RefCell`. - -### Accessing the Python GIL - -To get hold of a `Python<'py>` token to prove the GIL is held, consult [PyO3's documentation][obtaining-py]. - ## The GIL Refs API ### [`PyAny`][PyAny] @@ -346,4 +307,3 @@ This trait marks structs that mirror native Python types, such as `PyList`. [PyAny]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyAny.html [PyList_append]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyList.html#method.append [RefCell]: https://doc.rust-lang.org/std/cell/struct.RefCell.html -[obtaining-py]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#obtaining-a-python-token From ea6101e5b0cd7b5744b6b06e603d42f454b5a52a Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Fri, 8 Mar 2024 12:07:11 +0000 Subject: [PATCH 04/18] Apply suggestions from code review Co-authored-by: Lily Foote --- guide/src/python-from-rust.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/guide/src/python-from-rust.md b/guide/src/python-from-rust.md index f903a6531c3..073efb95623 100644 --- a/guide/src/python-from-rust.md +++ b/guide/src/python-from-rust.md @@ -7,11 +7,11 @@ This chapter of the guide documents some ways to interact with Python code from ## Interacting with the Python interpreter To safely interact with the Python interpreter a Rust thread must have a corresponding Python thread state and hold the Global Interpreter Lock (GIL). PyO3 has a `Python<'py>` token which is used to prove that these conditions -are met. It's lifetime `'py` is a central part of PyO3's API. +are met. Its lifetime `'py` is a central part of PyO3's API. The `Python<'py>` token serves three purposes: -* It provides global API for the Python interpreter, such as [`py.eval()`][eval] and [`py.import()`][import]. +* It provides global APIs for the Python interpreter, such as [`py.eval()`][eval] and [`py.import()`][import]. * It can be passed to functions that require a proof of holding the GIL, such as [`Py::clone_ref`][clone_ref]. * Its lifetime `'py` is used to bind many of PyO3's types to the Python interpreter, such as [`Bound<'py, T>`][Bound]. From b009dafdda5552c8b24f7dda45f7536f79e6c82f Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Fri, 8 Mar 2024 13:13:23 +0000 Subject: [PATCH 05/18] update URL --- guide/src/migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guide/src/migration.md b/guide/src/migration.md index 3ed3e015b3b..f2c478da4a8 100644 --- a/guide/src/migration.md +++ b/guide/src/migration.md @@ -495,7 +495,7 @@ drop(first); drop(second); ``` -The replacement is [`Python::with_gil`]() which is more cumbersome but enforces the proper nesting by design, e.g. +The replacement is [`Python::with_gil`](https://docs.rs/pyo3/0.18.3/pyo3/marker/struct.Python.html#method.with_gil) which is more cumbersome but enforces the proper nesting by design, e.g. ```rust # #![allow(dead_code)] From 718faac8f4300245db7a6e55c0ca0e678872094d Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Fri, 8 Mar 2024 14:44:44 +0000 Subject: [PATCH 06/18] complete python-from-rust.md --- guide/src/class.md | 30 +++++++++++++----------------- guide/src/python-from-rust.md | 32 ++++++++++---------------------- noxfile.py | 1 + src/sync.rs | 5 +++++ 4 files changed, 29 insertions(+), 39 deletions(-) diff --git a/guide/src/class.md b/guide/src/class.md index 93920a40d88..20a78a7f40c 100644 --- a/guide/src/class.md +++ b/guide/src/class.md @@ -187,23 +187,21 @@ fn my_module(m: &Bound<'_, PyModule>) -> PyResult<()> { } ``` -## Bound and interior mutability +## Bound and interior mutability -You sometimes need to convert your `#[pyclass]` into a Python object and access it -from Rust code (e.g., for testing it). -[`Bound`] is the primary interface for that. +Often is is useful to turn a `#[pyclass]` type `T` into a Python object and access it from Rust code. The [`Py`] and [`Bound<'py, T>`] smart pointers are the ways to represent a Python object in PyO3's API. More detail can be found about them [in the Python objects](./types.md#pyo3s-smart-pointers) section of the guide. -To mutate data behind a `Bound<'_, T>` safely, PyO3 employs the -[Interior Mutability Pattern](https://doc.rust-lang.org/book/ch15-05-interior-mutability.html) -like [`RefCell`]. +Most Python objects do not offer exclusive (`&mut`) access (see the [section on Python's memory model](./python-from-rust.md#pythons-memory-model)). However, Rust structs wrapped as Python objects (called `pyclass` types) often *do* need `&mut` access. Due to the GIL, PyO3 *can* guarantee exclusive access to them. -Users who are familiar with `RefCell` can use `Bound` just like `RefCell`. +The Rust borrow checker cannot reason about `&mut` references once an object's ownership has been passed to the Python interpreter. This means that borrow checking is done at runtime using with a scheme very similar to `std::cell::RefCell`. This is known as [interior mutability](https://doc.rust-lang.org/book/ch15-05-interior-mutability.html). -For users who are not very familiar with `RefCell`, here is a reminder of Rust's rules of borrowing: +Users who are familiar with `RefCell` can use `Py` and `Bound<'py, T>` just like `RefCell`. + +For users who are not very familiar with `RefCell`, here is a reminder of Rust's rules of borrowing: - At any given time, you can have either (but not both of) one mutable reference or any number of immutable references. -- References must always be valid. +- References can never outlast the data they refer to. -`Bound`, like `RefCell`, ensures these borrowing rules by tracking references at runtime. +`Py` and `Bound<'py, T>`, like `RefCell`, ensure these borrowing rules by tracking references at runtime. ```rust # use pyo3::prelude::*; @@ -233,10 +231,8 @@ Python::with_gil(|py| { }); ``` -A `Bound<'py, T>` is restricted to the GIL lifetime `'py`. -To make the object longer lived (for example, to store it in a struct on the -Rust side), you can use `Py`, which stores an object longer than the GIL -lifetime, and therefore needs a `Python<'_>` token to access. +A `Bound<'py, T>` is restricted to the GIL lifetime `'py`. To make the object longer lived (for example, to store it in a struct on the +Rust side), use `Py`. `Py` needs a `Python<'_>` token to allow access: ```rust # use pyo3::prelude::*; @@ -252,7 +248,7 @@ fn return_myclass() -> Py { let obj = return_myclass(); Python::with_gil(|py| { - let bound = obj.bind(py); // Py::bind returns &Bound<'_, MyClass> + let bound = obj.bind(py); // Py::bind returns &Bound<'py, MyClass> let obj_ref = bound.borrow(); // Get PyRef assert_eq!(obj_ref.num, 1); }); @@ -1319,7 +1315,7 @@ impl pyo3::impl_::pyclass::PyClassImpl for MyClass { [`PyTypeInfo`]: {{#PYO3_DOCS_URL}}/pyo3/type_object/trait.PyTypeInfo.html [`Py`]: {{#PYO3_DOCS_URL}}/pyo3/struct.Py.html -[`Bound`]: {{#PYO3_DOCS_URL}}/pyo3/struct.Bound.html +[`Bound<'_, T>`]: {{#PYO3_DOCS_URL}}/pyo3/struct.Bound.html [`PyClass`]: {{#PYO3_DOCS_URL}}/pyo3/pyclass/trait.PyClass.html [`PyRef`]: {{#PYO3_DOCS_URL}}/pyo3/pycell/struct.PyRef.html [`PyRefMut`]: {{#PYO3_DOCS_URL}}/pyo3/pycell/struct.PyRefMut.html diff --git a/guide/src/python-from-rust.md b/guide/src/python-from-rust.md index 073efb95623..449a5fd7772 100644 --- a/guide/src/python-from-rust.md +++ b/guide/src/python-from-rust.md @@ -6,7 +6,7 @@ This chapter of the guide documents some ways to interact with Python code from ## Interacting with the Python interpreter -To safely interact with the Python interpreter a Rust thread must have a corresponding Python thread state and hold the Global Interpreter Lock (GIL). PyO3 has a `Python<'py>` token which is used to prove that these conditions +To safely interact with the Python interpreter a Rust thread must have a corresponding Python thread state and hold the [Global Interpreter Lock (GIL)](#the-global-interpreter-lock). PyO3 has a `Python<'py>` token that is used to prove that these conditions are met. Its lifetime `'py` is a central part of PyO3's API. The `Python<'py>` token serves three purposes: @@ -15,10 +15,16 @@ The `Python<'py>` token serves three purposes: * It can be passed to functions that require a proof of holding the GIL, such as [`Py::clone_ref`][clone_ref]. * Its lifetime `'py` is used to bind many of PyO3's types to the Python interpreter, such as [`Bound<'py, T>`][Bound]. -PyO3's types which are bound to the `'py` lifetime, for example `Bound<'py, T>`, all contain a `Python<'py>` token. This means they have full access to the Python interpreter and offer a complete API for interacting with Python objects. +PyO3's types that are bound to the `'py` lifetime, for example `Bound<'py, T>`, all contain a `Python<'py>` token. This means they have full access to the Python interpreter and offer a complete API for interacting with Python objects. Consult [PyO3's API documentation][obtaining-py] to learn how to acquire one of these tokens. +### The Global Interpreter Lock + +Concurrent programming in Python is aided by the Global Interpreter Lock (GIL), which ensures that only one Python thread can use the Python interpreter and its API at the same time. This allows it to be used to synchronize code. See the [`pyo3::sync`] module for synchronization tools PyO3 offers that are based on the GIL's guarantees. + +Non-Python operations (system calls and native Rust code) can unlock the GIL. See [the section on parallelism](parallelism.md) for how to do that using PyO3's API. + ## Python's memory model Python memory model differs from Rust's memory model in two key ways: @@ -27,25 +33,7 @@ Python memory model differs from Rust's memory model in two key ways: PyO3's API reflects this by providing smart pointer types, `Py`, `Bound<'py, T>`, and (the very rarely used) `Borrowed<'a, 'py, T>`. These smart pointers all use Python reference counting. See the [subchapter on types](./types.md) for more detail on these types. -### Mutable access to `#[pyclass]` types - -The situation is helped by the Global Interpreter Lock (GIL), which ensures that only one thread can use the Python interpreter and its API at the same time, while non-Python operations (system calls and extension code) can unlock the GIL. (See [the section on parallelism](parallelism.md) for how to do that in PyO3.) - - - -The latter two points are the reason why some APIs in PyO3 require the `py: -Python` argument, while others don't. - -The PyO3 API for Python objects is written such that instead of requiring a -mutable Rust reference for mutating operations such as -[`PyList::append`][PyList_append], a shared reference (which, in turn, can only -be created through `Python<'_>` with a GIL lifetime) is sufficient. - -However, Rust structs wrapped as Python objects (called `pyclass` types) usually -*do* need `&mut` access. Due to the GIL, PyO3 *can* guarantee thread-safe access -to them, but it cannot statically guarantee uniqueness of `&mut` references once -an object's ownership has been passed to the Python interpreter, ensuring -references is done at runtime using `PyCell`, a scheme very similar to -`std::cell::RefCell`. +Because of the lack of exclusive `&mut` references, PyO3's APIs for Python objects, for example [`PyListMethods::append`], use shared references. This is safe because Python objects have internal mechanisms to prevent data races (as of time of writing, the Python GIL). [obtaining-py]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#obtaining-a-python-token +[`pyo3::sync`]: {{#PYO3_DOCS_URL}}/pyo3/sync/index.html diff --git a/noxfile.py b/noxfile.py index f70fced76d6..5f7ca8c04f8 100644 --- a/noxfile.py +++ b/noxfile.py @@ -372,6 +372,7 @@ def docs(session: nox.Session) -> None: "--no-deps", "--workspace", *cargo_flags, + *session.posargs, ) diff --git a/src/sync.rs b/src/sync.rs index f4ffe0409a7..e89a8edd853 100644 --- a/src/sync.rs +++ b/src/sync.rs @@ -1,4 +1,9 @@ //! Synchronization mechanisms based on the Python GIL. +//! +//! With the acceptance of [PEP 703] (aka a "freethreaded Python") for Python 3.13, these +//! are likely to undergo significant developments in the future. +//! +//! [PEP 703]: https://peps.python.org/pep-703/ use crate::{ types::{any::PyAnyMethods, PyString, PyType}, Bound, Py, PyResult, PyVisit, Python, From 5a6944578df4b04decdc6df99b2ad4d311052432 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sat, 9 Mar 2024 08:31:51 +0000 Subject: [PATCH 07/18] progress on types.md; probably more to go --- guide/src/python-from-rust.md | 10 +- guide/src/types.md | 337 +++++++++++++++++++++++++++++----- 2 files changed, 301 insertions(+), 46 deletions(-) diff --git a/guide/src/python-from-rust.md b/guide/src/python-from-rust.md index 449a5fd7772..5c6e089abd3 100644 --- a/guide/src/python-from-rust.md +++ b/guide/src/python-from-rust.md @@ -1,10 +1,16 @@ # Calling Python in Rust code -This chapter of the guide documents some ways to interact with Python code from Rust: +This chapter of the guide documents some ways to interact with Python code from Rust. + +Below is an introduction to the `'py` lifetime and some general remarks about how PyO3's API reasons about Python code. + +The subchapters also cover the following topics: + - Python object types available in PyO3's API + - How to work with Python exceptions - How to call Python functions - How to execute existing Python code -## Interacting with the Python interpreter +## The `'py` lifetime To safely interact with the Python interpreter a Rust thread must have a corresponding Python thread state and hold the [Global Interpreter Lock (GIL)](#the-global-interpreter-lock). PyO3 has a `Python<'py>` token that is used to prove that these conditions are met. Its lifetime `'py` is a central part of PyO3's API. diff --git a/guide/src/types.md b/guide/src/types.md index 95ebb41dd3a..c8528681e97 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -26,78 +26,164 @@ The sections below also explain these smart pointers in a little more detail. ### `Py` (and `PyObject`) +[`Py`][Py] is the foundational smart pointer in PyO3's API. The type parameter `T` denotes the type of the Python object. Very frequently this is `PyAny`, meaning any Python object. This is so common that `Py` has a type alias `PyObject`. + +Because `Py` is not bound to [the `'py` lifetime](./python-from-rust.md#the-py-lifetime), it is the type to use when storing a Python object inside a Rust `struct` or `enum` which do not want to have a lifetime parameter. In particular, [`#[pyclass]`][pyclass] types are not permitted to have a lifetime, so `Py` is the correct type to store Python objects inside them. + +The lack of binding to the `'py` lifetime also carries drawbacks: + - Almost all methods on `Py` require a `Python<'py>` token as the first argument + - Other functionality, such as [`Drop`][Drop], needs to check at runtime for attachment to the Python GIL, at a small performance cost + +Because of the drawbacks `Bound<'py, T>` is preferred for many of PyO3's APIs. + +To convert a `Py` into a `Bound<'py, T>`, the `Py::bind` and `Py::into_bound` methods are available. `Bound<'py, T>` can be converted back into `Py` using `Bound::unbind`. + ### `Bound<'py, T>` +[`Bound<'py, T>`][Bound] is the counterpart to `Py` which is also bound to the `'py` lifetime. It can be thought of as equivalent to the Rust tuple `(Python<'py>, Py)`. + +By having the binding to the `'py` lifetime, `Bound<'py, T>` can offer the complete PyO3 API at maximum efficiency. This means that in almost all cases where `Py` is not necessary for lifetime reasons, `Bound<'py, T>` should be used. + +`Bound<'py, T>` engages in Python reference counting. This means that `Bound<'py, T>` owns a Python object. Rust code which just wants to borrow a Python object should use a shared reference `&Bound<'py, T>`. Just like `std::sync::Arg`, using `.clone()` and `drop()` will cheaply implement and decrement the reference count of the object (just in this case, the reference counting is implemented by the Python interpreter itself). + +To give an example of how `Bound<'py, T>` is PyO3's primary API type, consider the following Python code: + +```python +def example(): + x = list() # create a Python list + x.append(1) # append the integer 1 to it + y = x # create a second reference to the list + del x # delete the original reference +``` + +Using PyO3's API, and in particular `Bound<'py, PyList>`, this code translates into the following Rust code: + +```rust +use pyo3::prelude::*; +use pyo3::types::PyList; + +# fn main() -> PyResult<()> { +// the 'py lifetime doesn't need to be named explicitly in this code, +// so '_ is used. +Python::with_gil(|py: Python<'_>| -> PyResult<()> { + let x: Bound<'_, PyList> = PyList::empty_bound(py); + x.append(1)?; + let y: Bound<'_, PyList> = x.clone(); // y is a new reference to the same list + drop(x); // release the original reference x + Ok(()) +})?; // y dropped implicitly here at the end of the closure +# Ok(()) +# } +``` + +Or, without the type annotations: + +```rust +use pyo3::prelude::*; +use pyo3::types::PyList; + +# fn main() -> PyResult<()> { +Python::with_gil(|py| { + let x = PyList::empty_bound(py); + x.append(1)?; + let y = x.clone(); + drop(x); + Ok(()) +})?; +# Ok(()) +# } +``` + ### `Borrowed<'a, 'py, T>` -GIL lifetimes, mutability and Python object types +[`Borrowed<'a, 'py, T>`][Borrowed] is an advanced type used just occasionally at the edge of interaction with the Python interpreter. It can be thought of as analogous to the shared reference `&'a Bound<'py, T>`. The difference is that `Borrowed<'a, 'py, T>` is just a smart pointer rather than a reference-to-a-smart-pointer, which is a helpful reduction in indirection in specific interactions with the Python interpreter. + +`Borrowed<'a, 'py, T>` dereferences to `Bound<'py, T>`, so all methods on `Bound<'py, T>` are available on `Borrowed<'a, 'py, T>`. -At first glance, PyO3 provides a huge number of different types that can be used -to wrap or refer to Python objects. This page delves into the details and gives -an overview of their intended meaning, with examples when each type is best -used. +An example where `Borrowed<'a, 'py, T>` is used is in [`PyTupleMethods::get_borrowed_item`](#{{PYO3_DOCS_URL}}/pyo3/types/trait.PyTupleMethods.html#tymethod.get_item): + +```rust +use pyo3::prelude::*; +use pyo3::types::PyTuple; + +# fn main() -> PyResult<()> { +Python::with_gil(|py| { + // Create a new tuple with the elements (0, 1, 2) + let t = PyTuple::new_bound(py, 0..=2); + for i in 0..=2 { + let entry: Borrowed<'_, '_, PyAny> = t.get_borrowed_item(i)?; + // `PyAnyMethods::extract` is available on `Borrowed` + // via the dereference to `Bound` + let value: usize = entry.extract()?; + assert_eq!(i, value); + } +})?; +# Ok(()) +# } +``` ## Concrete Python types +In all of `Py`, `Bound<'py, T>`, and `Borrowed<'a, 'py, T>`, the type parameter `T` denotes the type of the Python object referred to by the smart pointer. +This parameter `T` can be filled by: + - [`PyAny`][PyAny], which represents any Python object, + - Native Python types such as `PyList`, `PyTuple`, and `PyDict`, and + - [`#[pyclass]`][pyclass] types defined from Rust -## The GIL Refs API +The following subsections elaborate on working with these types in a little more detail. ### [`PyAny`][PyAny] -**Represents:** a Python object of unspecified type, restricted to a GIL -lifetime. Currently, `PyAny` can only ever occur as a reference, `&PyAny`. +A Python object of unspecified type. -**Used:** Whenever you want to refer to some Python object and will have the -GIL for the whole duration you need to access that object. For example, -intermediate values and arguments to `pyfunction`s or `pymethod`s implemented -in Rust where any type is allowed. +**Uses:** Whenever you want to refer to some Python object without knowing its type. -Many general methods for interacting with Python objects are on the `PyAny` struct, -such as `getattr`, `setattr`, and `.call`. +Many general methods for interacting with Python objects are on the `Bound<'py, PyAny>` smart pointer, such as `.getattr`, `.setattr`, and `.call`. **Conversions:** -For a `&PyAny` object reference `any` where the underlying object is a Python-native type such as -a list: +For a `Bound<'py, PyAny>` where the underlying object is a Python-native type such as a list: ```rust # use pyo3::prelude::*; # use pyo3::types::PyList; -# Python::with_gil(|py| -> PyResult<()> { -#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API. -let obj: &PyAny = PyList::empty(py); +# fn example<'py>(py: Python<'py>) -> PyResult<()> { +let obj: Bound<'py, PyAny> = PyList::empty_bound(py).into_any(); -// To &PyList with PyAny::downcast -let _: &PyList = obj.downcast()?; +// To &Bound<'py, PyList> with PyAnyMethods::downcast +let _: &Bound<'py, PyList> = obj.downcast()?; -// To Py (aka PyObject) with .into() -let _: Py = obj.into(); +# let obj2 = obj.clone(); +// To Py (aka PyObject) with .unbind() +let _: Py = obj.unbind(); +# let obj = obj2; -// To Py with PyAny::extract -let _: Py = obj.extract()?; +// To Bound<'py, PyList> with PyAnyMethods::downcast_into +let list: Bound<'py, PyList> = obj.downcast_into()?; # Ok(()) -# }).unwrap(); +# } +# Python::with_gil(example).unwrap() ``` -For a `&PyAny` object reference `any` where the underlying object is a `#[pyclass]`: +For a `Bound<'_, PyAny>` where the underlying object is a `#[pyclass]`: ```rust # use pyo3::prelude::*; # #[pyclass] #[derive(Clone)] struct MyClass { } -# Python::with_gil(|py| -> PyResult<()> { -# #[allow(deprecated)] -let obj: &PyAny = Py::new(py, MyClass {})?.into_ref(py); +# fn example<'py>(py: Python<'py>) -> PyResult<()> { +let obj: Bound<'_, PyAny> = Bound::new(py, MyClass {})?.into_any(); -// To &PyCell with PyAny::downcast -# #[allow(deprecated)] -let _: &PyCell = obj.downcast()?; +// To &Bound<'py, MyClass> with PyAnyMethods::downcast +let _: &Bound<'_, MyClass> = obj.downcast()?; -// To Py (aka PyObject) with .into() -let _: Py = obj.into(); +# let obj2 = obj.clone(); +// To Py (aka PyObject) with .unbind() +let _: Py = obj.unbind(); +# let obj = obj2; -// To Py with PyAny::extract -let _: Py = obj.extract()?; +// To Bound<'_, MyClass> with PyAnyMethods::downcast_into +let _: Bound<'_, MyClass> = obj.downcast_into()?; // To MyClass with PyAny::extract, if MyClass: Clone let _: MyClass = obj.extract()?; @@ -106,7 +192,8 @@ let _: MyClass = obj.extract()?; let _: PyRef<'_, MyClass> = obj.extract()?; let _: PyRefMut<'_, MyClass> = obj.extract()?; # Ok(()) -# }).unwrap(); +# } +# Python::with_gil(example).unwrap() ``` ### `PyTuple`, `PyDict`, and many more @@ -288,19 +375,181 @@ borrows, analog to `Ref` and `RefMut` used by `RefCell`. **Used:** while borrowing a `PyCell`. They can also be used with `.extract()` on types like `Py` and `PyAny` to get a reference quickly. +## The GIL Refs API -## Related traits and types +The GIL Refs API was PyO3's primary API prior to PyO3 0.21. The main difference was that instead of the `Bound<'py, PyAny>` smart pointer, the "GIL Reference" `&'py PyAny` was used. (This was similar for other Python types.) -### `PyClass` +As of PyO3 0.21, the GIL Refs API is deprecated. See the [migration guide](./migration.md#from-020-to-021) for details on how to upgrade. -This trait marks structs defined in Rust that are also usable as Python classes, -usually defined using the `#[pyclass]` macro. +The following sections note some historical detail about the GIL Refs API. -### `PyNativeType` +### [`PyAny`][PyAny] -This trait marks structs that mirror native Python types, such as `PyList`. +**Represented:** a Python object of unspecified type. In the GIL Refs API, this was only accessed as the GIL Ref `&'py PyAny`. + +**Used:** `&'py PyAny` was used to refer to some Python object when the GIL lifetime was available for the whole duration access was needed. For example, intermediate values and arguments to `pyfunction`s or `pymethod`s implemented in Rust where any type is allowed. + +**Conversions:** + +For a `&PyAny` object reference `any` where the underlying object is a Python-native type such as +a list: + +```rust +# use pyo3::prelude::*; +# use pyo3::types::PyList; +# Python::with_gil(|py| -> PyResult<()> { +#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API. +let obj: &PyAny = PyList::empty(py); + +// To &PyList with PyAny::downcast +let _: &PyList = obj.downcast()?; + +// To Py (aka PyObject) with .into() +let _: Py = obj.into(); + +// To Py with PyAny::extract +let _: Py = obj.extract()?; +# Ok(()) +# }).unwrap(); +``` + +For a `&PyAny` object reference `any` where the underlying object is a `#[pyclass]`: + +```rust +# use pyo3::prelude::*; +# #[pyclass] #[derive(Clone)] struct MyClass { } +# Python::with_gil(|py| -> PyResult<()> { +#[allow(deprecated)] // into_ref is part of the deprecated GIL Refs API +let obj: &PyAny = Py::new(py, MyClass {})?.into_ref(py); + +// To &PyCell with PyAny::downcast +#[allow(deprecated)] // &PyCell is part of the deprecated GIL Refs API +let _: &PyCell = obj.downcast()?; + +// To Py (aka PyObject) with .into() +let _: Py = obj.into(); + +// To Py with PyAny::extract +let _: Py = obj.extract()?; + +// To MyClass with PyAny::extract, if MyClass: Clone +let _: MyClass = obj.extract()?; +// To PyRef<'_, MyClass> or PyRefMut<'_, MyClass> with PyAny::extract +let _: PyRef<'_, MyClass> = obj.extract()?; +let _: PyRefMut<'_, MyClass> = obj.extract()?; +# Ok(()) +# }).unwrap(); +``` + +### `PyTuple`, `PyDict`, and many more + +**Represented:** a native Python object of known type. In the GIL Refs API, they were only accessed as the GIL Refs `&'py PyTuple`, `&'py PyDict`. + +**Used:** `&'py PyTuple` and similar were used to operate with native Python types while holding the GIL. Like `PyAny`, this is the most convenient form to use for function arguments and intermediate values. + +These GIL Refs implement `Deref`, so they all expose the same methods which can be found on `PyAny`. + +To see all Python types exposed by `PyO3` consult the [`pyo3::types`][pyo3::types] module. + +**Conversions:** + +```rust +# use pyo3::prelude::*; +# use pyo3::types::PyList; +# Python::with_gil(|py| -> PyResult<()> { +#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API. +let list = PyList::empty(py); + +// Use methods from PyAny on all Python types with Deref implementation +let _ = list.repr()?; + +// To &PyAny automatically with Deref implementation +let _: &PyAny = list; + +// To &PyAny explicitly with .as_ref() +#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. +let _: &PyAny = list.as_ref(); + +// To Py with .into() or Py::from() +let _: Py = list.into(); + +// To PyObject with .into() or .to_object(py) +let _: PyObject = list.into(); +# Ok(()) +# }).unwrap(); +``` + +### `Py` and `PyObject` + +**Represented:** a GIL-independent reference to a Python object. This can be a Python native type +(like `PyTuple`), or a `pyclass` type implemented in Rust. The most commonly-used variant, +`Py`, is also known as `PyObject`. + +**Used:** Whenever you want to carry around references to a Python object without caring about a +GIL lifetime. For example, storing Python object references in a Rust struct that outlives the +Python-Rust FFI boundary, or returning objects from functions implemented in Rust back to Python. + +Can be cloned using Python reference counts with `.clone()`. + +### `PyCell` + +**Represented:** a reference to a Rust object (instance of `PyClass`) wrapped in a Python object. The cell part is an analog to stdlib's [`RefCell`][RefCell] to allow access to `&mut` references. + +**Used:** for accessing pure-Rust API of the instance (members and functions taking `&SomeType` or `&mut SomeType`) while maintaining the aliasing rules of Rust references. + +Like PyO3's Python native types, the GIL Ref `&PyCell` implements `Deref`, so it also exposed all of the methods on `PyAny`. + +**Conversions:** + +`PyCell` was used to access `&T` and `&mut T` via `PyRef` and `PyRefMut` respectively. + +```rust +# use pyo3::prelude::*; +# #[pyclass] struct MyClass { } +# Python::with_gil(|py| -> PyResult<()> { +#[allow(deprecated)] // &PyCell is part of the deprecated GIL Refs API +let cell: &PyCell = PyCell::new(py, MyClass {})?; + +// To PyRef with .borrow() or .try_borrow() +let py_ref: PyRef<'_, MyClass> = cell.try_borrow()?; +let _: &MyClass = &*py_ref; +# drop(py_ref); + +// To PyRefMut with .borrow_mut() or .try_borrow_mut() +let mut py_ref_mut: PyRefMut<'_, MyClass> = cell.try_borrow_mut()?; +let _: &mut MyClass = &mut *py_ref_mut; +# Ok(()) +# }).unwrap(); +``` + +`PyCell` was also accessed like a Python-native type. + +```rust +# use pyo3::prelude::*; +# #[pyclass] struct MyClass { } +# Python::with_gil(|py| -> PyResult<()> { +#[allow(deprecated)] // &PyCell is part of the deprecate GIL Refs API +let cell: &PyCell = PyCell::new(py, MyClass {})?; + +// Use methods from PyAny on PyCell with Deref implementation +let _ = cell.repr()?; + +// To &PyAny automatically with Deref implementation +let _: &PyAny = cell; + +// To &PyAny explicitly with .as_ref() +#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. +let _: &PyAny = cell.as_ref(); +# Ok(()) +# }).unwrap(); +``` +[pyclass]: class.md +[Py]: {{#PYO3_DOCS_URL}}/pyo3/struct.Py.html +[Bound]: {{#PYO3_DOCS_URL}}/pyo3/struct.Bound.html +[Borrowed]: {{#PYO3_DOCS_URL}}/pyo3/struct.Borrowed.html +[Drop]: https://doc.rust-lang.org/std/drop/trait.Drop.html [eval]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.eval [clone_ref]: {{#PYO3_DOCS_URL}}/pyo3/struct.Py.html#method.clone_ref [pyo3::types]: {{#PYO3_DOCS_URL}}/pyo3/types/index.html From 03ff845b61c0566877f44d82cf9c0ab2cafdbd71 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sat, 9 Mar 2024 08:42:14 +0000 Subject: [PATCH 08/18] update doctest paths --- guide/src/rust-from-python.md | 12 ++++++++++++ guide/src/rust_from_python.md | 13 ------------- src/lib.rs | 3 +++ 3 files changed, 15 insertions(+), 13 deletions(-) delete mode 100644 guide/src/rust_from_python.md diff --git a/guide/src/rust-from-python.md b/guide/src/rust-from-python.md index 8e74bcae219..470d5719098 100644 --- a/guide/src/rust-from-python.md +++ b/guide/src/rust-from-python.md @@ -1 +1,13 @@ # Using Rust from Python + +This chapter of the guide is dedicated to explaining how to wrap Rust code into Python objects. + +PyO3 uses Rust's "procedural macros" to provide a powerful yet simple API to denote what Rust code should map into Python objects. + +The three types of Python objects which PyO3 can produce are: + +- Python modules, via the `#[pymodule]` macro +- Python functions, via the `#[pyfunction]` macro +- Python classes, via the `#[pyclass]` macro (plus `#[pymethods]` to define methods for those clases) + +The following subchapters go through each of these in turn. diff --git a/guide/src/rust_from_python.md b/guide/src/rust_from_python.md deleted file mode 100644 index 470d5719098..00000000000 --- a/guide/src/rust_from_python.md +++ /dev/null @@ -1,13 +0,0 @@ -# Using Rust from Python - -This chapter of the guide is dedicated to explaining how to wrap Rust code into Python objects. - -PyO3 uses Rust's "procedural macros" to provide a powerful yet simple API to denote what Rust code should map into Python objects. - -The three types of Python objects which PyO3 can produce are: - -- Python modules, via the `#[pymodule]` macro -- Python functions, via the `#[pyfunction]` macro -- Python classes, via the `#[pyclass]` macro (plus `#[pymethods]` to define methods for those clases) - -The following subchapters go through each of these in turn. diff --git a/src/lib.rs b/src/lib.rs index bc88838d4b1..a6482015de3 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -512,7 +512,10 @@ pub mod doc_test { "guide/src/parallelism.md" => guide_parallelism_md, "guide/src/performance.md" => guide_performance_md, "guide/src/python-from-rust.md" => guide_python_from_rust_md, + "guide/src/python-from-rust/calling-existing-code.md" => guide_pfr_calling_existing_code_md, + "guide/src/python-from-rust/function-calls.md" => guide_pfr_function_calls_md, "guide/src/python-typing-hints.md" => guide_python_typing_hints_md, + "guide/src/rust-from-python.md" => guide_rust_from_python_md, "guide/src/trait-bounds.md" => guide_trait_bounds_md, "guide/src/types.md" => guide_types_md, } From 4682449b428ef48b32241483b3fc1c204c130d43 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sat, 9 Mar 2024 08:45:27 +0000 Subject: [PATCH 09/18] review: Icxolu --- guide/src/python-from-rust.md | 4 ++-- guide/src/types.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/guide/src/python-from-rust.md b/guide/src/python-from-rust.md index 5c6e089abd3..6bc15e2e82c 100644 --- a/guide/src/python-from-rust.md +++ b/guide/src/python-from-rust.md @@ -17,7 +17,7 @@ are met. Its lifetime `'py` is a central part of PyO3's API. The `Python<'py>` token serves three purposes: -* It provides global APIs for the Python interpreter, such as [`py.eval()`][eval] and [`py.import()`][import]. +* It provides global APIs for the Python interpreter, such as [`py.eval_bound()`][eval] and [`py.import_bound()`][import]. * It can be passed to functions that require a proof of holding the GIL, such as [`Py::clone_ref`][clone_ref]. * Its lifetime `'py` is used to bind many of PyO3's types to the Python interpreter, such as [`Bound<'py, T>`][Bound]. @@ -33,7 +33,7 @@ Non-Python operations (system calls and native Rust code) can unlock the GIL. Se ## Python's memory model -Python memory model differs from Rust's memory model in two key ways: +Python's memory model differs from Rust's memory model in two key ways: - There is no concept of ownership; all Python objects are reference counted - There is no concept of exclusive (`&mut`) references; any reference can mutate a Python object diff --git a/guide/src/types.md b/guide/src/types.md index c8528681e97..f3231418837 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -19,7 +19,7 @@ Python objects are reference counted, like [`std::sync::Arc`](https://doc.rust-l The recommendation of when to use each of these smart pointers is as follows: - Use `Bound<'py, T>` for as much as possible, as it offers the most efficient and complete API. -- Use `Py` mostly just for storage inside Rust `struct`s which do not want to add a lifetime parameter for `Bound<'py, T>`. +- Use `Py` mostly just for storage inside Rust `struct`s which do not want to or can't add a lifetime parameter for `Bound<'py, T>`. - `Borrowed<'a, 'py, T>` is almost never used. It is occasionally present at the boundary between Rust and the Python interpreter, for example when borrowing data from Python tuples (which is safe because they are immutable). The sections below also explain these smart pointers in a little more detail. From a6da5cf7fc4dbb127d1b9d1f723416ac24eda48f Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 12:14:42 +0100 Subject: [PATCH 10/18] finish updating `types.md` to Bound API --- Architecture.md | 29 +--- guide/src/types.md | 374 +++++++++++++++++---------------------------- src/lib.rs | 49 +++--- 3 files changed, 171 insertions(+), 281 deletions(-) diff --git a/Architecture.md b/Architecture.md index 60083d71c98..a4218a7f71f 100644 --- a/Architecture.md +++ b/Architecture.md @@ -59,6 +59,7 @@ Those flags are set in [`build.rs`](#6-buildrs-and-pyo3-build-config). [`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html) of Python, such as `dict` and `list`. For historical reasons, Python's `object` is called `PyAny` in PyO3 and located in [`src/types/any.rs`]. + Currently, `PyAny` is a straightforward wrapper of `ffi::PyObject`, defined as: ```rust @@ -66,38 +67,16 @@ Currently, `PyAny` is a straightforward wrapper of `ffi::PyObject`, defined as: pub struct PyAny(UnsafeCell); ``` -All built-in types are defined as a C struct. -For example, `dict` is defined as: - -```c -typedef struct { - /* Base object */ - PyObject ob_base; - /* Number of items in the dictionary */ - Py_ssize_t ma_used; - /* Dictionary version */ - uint64_t ma_version_tag; - PyDictKeysObject *ma_keys; - PyObject **ma_values; -} PyDictObject; -``` - -However, we cannot access such a specific data structure with `#[cfg(Py_LIMITED_API)]` set. -Thus, all builtin objects are implemented as opaque types by wrapping `PyAny`, e.g.,: +Concrete Python objects are implemented by wrapping `PyAny`, e.g.,: ```rust #[repr(transparent)] pub struct PyDict(PyAny); ``` -Note that `PyAny` is not a pointer, and it is usually used as a pointer to the object in the -Python heap, as `&PyAny`. -This design choice can be changed -(see the discussion in [#1056](https://github.com/PyO3/pyo3/issues/1056)). +These types are not intended to be accessed directly, and instead are used through the `Py` and `Bound` smart pointers. -Since we need lots of boilerplate for implementing common traits for these types -(e.g., `AsPyPointer`, `AsRef`, and `Debug`), we have some macros in -[`src/types/mod.rs`]. +We have some macros in [`src/types/mod.rs`] which make it easier to implement APIs for concrete Python types. ## 3. `PyClass` and related functionalities diff --git a/guide/src/types.md b/guide/src/types.md index f3231418837..9c4ed4dc849 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -12,7 +12,7 @@ Before PyO3 0.21, PyO3's main API to interact with Python objects was a deprecat PyO3's API offers three generic smart pointers: `Py`, `Bound<'py, T>` and `Borrowed<'a, 'py, T>`. For each of these the type parameter `T` will be filled by a [concrete Python type](#concrete-python-types). For example, a Python list object can be represented by `Py`, `Bound<'py, PyList>`, and `Borrowed<'a, 'py, PyList>`. -These smart pointers have different numbers of lifetime parameters, which defines how they behave. `Py` has no lifetime parameters, `Bound<'py, T>` has a lifetime parameter `'py`, and `Borrowed<'a, 'py, T>` has two lifetime parameters `'a` and `'py`. +These smart pointers behave differently due to their lifetime parameters. `Py` has no lifetime parameters, `Bound<'py, T>` has [the `'py` lifetime](./python-from-rust.md#the-py-lifetime) as a parameter, and `Borrowed<'a, 'py, T>` has the `'py` lifetime plus an additional lifetime `'a` to denote the lifetime it is borrowing data for. (You can read more about these lifetimes in the subsections below). Python objects are reference counted, like [`std::sync::Arc`](https://doc.rust-lang.org/stable/std/sync/struct.Arc.html). A major reason for these smart pointers is to bring Python's reference counting to a Rust API. @@ -34,9 +34,9 @@ The lack of binding to the `'py` lifetime also carries drawbacks: - Almost all methods on `Py` require a `Python<'py>` token as the first argument - Other functionality, such as [`Drop`][Drop], needs to check at runtime for attachment to the Python GIL, at a small performance cost -Because of the drawbacks `Bound<'py, T>` is preferred for many of PyO3's APIs. +Because of the drawbacks `Bound<'py, T>` is preferred for many of PyO3's APIs. In particular, `Bound<'py, T>` is the better for function arguments. -To convert a `Py` into a `Bound<'py, T>`, the `Py::bind` and `Py::into_bound` methods are available. `Bound<'py, T>` can be converted back into `Py` using `Bound::unbind`. +To convert a `Py` into a `Bound<'py, T>`, the `Py::bind` and `Py::into_bound` methods are available. `Bound<'py, T>` can be converted back into `Py` using [`Bound::unbind`]. ### `Bound<'py, T>` @@ -44,7 +44,7 @@ To convert a `Py` into a `Bound<'py, T>`, the `Py::bind` and `Py::into_bound` By having the binding to the `'py` lifetime, `Bound<'py, T>` can offer the complete PyO3 API at maximum efficiency. This means that in almost all cases where `Py` is not necessary for lifetime reasons, `Bound<'py, T>` should be used. -`Bound<'py, T>` engages in Python reference counting. This means that `Bound<'py, T>` owns a Python object. Rust code which just wants to borrow a Python object should use a shared reference `&Bound<'py, T>`. Just like `std::sync::Arg`, using `.clone()` and `drop()` will cheaply implement and decrement the reference count of the object (just in this case, the reference counting is implemented by the Python interpreter itself). +`Bound<'py, T>` engages in Python reference counting. This means that `Bound<'py, T>` owns a Python object. Rust code which just wants to borrow a Python object should use a shared reference `&Bound<'py, T>`. Just like `std::sync::Arc`, using `.clone()` and `drop()` will cheaply implement and decrement the reference count of the object (just in this case, the reference counting is implemented by the Python interpreter itself). To give an example of how `Bound<'py, T>` is PyO3's primary API type, consider the following Python code: @@ -62,18 +62,14 @@ Using PyO3's API, and in particular `Bound<'py, PyList>`, this code translates i use pyo3::prelude::*; use pyo3::types::PyList; -# fn main() -> PyResult<()> { -// the 'py lifetime doesn't need to be named explicitly in this code, -// so '_ is used. -Python::with_gil(|py: Python<'_>| -> PyResult<()> { - let x: Bound<'_, PyList> = PyList::empty_bound(py); +fn example<'py>(py: Python<'py>) -> PyResult<()> { + let x: Bound<'py, PyList> = PyList::empty_bound(py); x.append(1)?; - let y: Bound<'_, PyList> = x.clone(); // y is a new reference to the same list + let y: Bound<'py, PyList> = x.clone(); // y is a new reference to the same list drop(x); // release the original reference x Ok(()) -})?; // y dropped implicitly here at the end of the closure -# Ok(()) -# } +} +# Python::with_gil(example).unwrap(); ``` Or, without the type annotations: @@ -82,16 +78,56 @@ Or, without the type annotations: use pyo3::prelude::*; use pyo3::types::PyList; -# fn main() -> PyResult<()> { -Python::with_gil(|py| { +# fn example(py: Python<'_>) -> PyResult<()> { let x = PyList::empty_bound(py); x.append(1)?; let y = x.clone(); drop(x); Ok(()) -})?; -# Ok(()) -# } +} +# Python::with_gil(example).unwrap(); +``` + +#### Function argument lifetimes + +Because the `'py` lifetime often appears in many function arguments as part of the `Bound<'py, T>` smart pointer, the Rust compiler will often require annotations of input and output lifetimes. This occurs when the function output has at least one lifetime, and there is more than one lifetime present on the inputs. + +To demonstrate, consider this function which takes accepts Python objects and applies the [Python `+` operation][PyAnyMethods::add] to them: + +```rust,compile_fail +# use pyo3::prelude::*; +fn add(left: &'_ Bound<'_, PyAny>, right: &'_ Bound<'_, PyAny>) -> PyResult> { + left.add(right) +} +``` + +Because the Python `+` operation might raise an exception, this function returns `PyResult>`. It doesn't need ownership of the inputs, so it takes `&Bound<'_, PyAny>` shared references. To demonstrate the point, all lifetimes have used the wildcard `'_` to allow the Rust compiler to attempt to infer them. Because there are four input lifetimes (two lifetimes of the shared references, and two `'py` lifetimes unnamed inside the `Bound<'_, PyAny>` pointers), the compiler cannot reason about which must be connected to the output. + +The correct way to solve this is to add the `'py` lifetime as a parameter for the function, and name all the `'py` lifetimes inside the `Bound<'py, PyAny>` smart pointers. For the shared references, it's also fine to reduce `&'_` to just `&`. The working end result is below: + +```rust +# use pyo3::prelude::*; +fn add<'py>(left: &Bound<'py, PyAny>, right: &Bound<'py, PyAny>) -> PyResult> { + left.add(right) +} +# Python::with_gil(|py| { +# let s = pyo3::types::PyString::new_bound(py, "s"); +# assert!(add(&s, &s).unwrap().eq("ss").unwrap()); +# }) +``` + +If naming the `'py` lifetime adds unwanted complexity to the function signature, it is also acceptable to return `PyObject` (aka `Py`), which has no lifetime. The cost is instead paid by a slight increase in implementation complexity, as seen by the introduction of a call to [`Bound::unbind`]: + +```rust +# use pyo3::prelude::*; +fn add(left: &Bound<'_, PyAny>, right: &Bound<'_, PyAny>) -> PyResult { + let output: Bound<'_, PyAny> = left.add(right)?; + Ok(output.unbind()) +} +# Python::with_gil(|py| { +# let s = pyo3::types::PyString::new_bound(py, "s"); +# assert!(add(&s, &s).unwrap().bind(py).eq("ss").unwrap()); +# }) ``` ### `Borrowed<'a, 'py, T>` @@ -100,26 +136,25 @@ Python::with_gil(|py| { `Borrowed<'a, 'py, T>` dereferences to `Bound<'py, T>`, so all methods on `Bound<'py, T>` are available on `Borrowed<'a, 'py, T>`. -An example where `Borrowed<'a, 'py, T>` is used is in [`PyTupleMethods::get_borrowed_item`](#{{PYO3_DOCS_URL}}/pyo3/types/trait.PyTupleMethods.html#tymethod.get_item): +An example where `Borrowed<'a, 'py, T>` is used is in [`PyTupleMethods::get_borrowed_item`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.PyTupleMethods.html#tymethod.get_item): ```rust use pyo3::prelude::*; use pyo3::types::PyTuple; -# fn main() -> PyResult<()> { -Python::with_gil(|py| { - // Create a new tuple with the elements (0, 1, 2) - let t = PyTuple::new_bound(py, 0..=2); - for i in 0..=2 { - let entry: Borrowed<'_, '_, PyAny> = t.get_borrowed_item(i)?; - // `PyAnyMethods::extract` is available on `Borrowed` - // via the dereference to `Bound` - let value: usize = entry.extract()?; - assert_eq!(i, value); - } -})?; +# fn example<'py>(py: Python<'py>) -> PyResult<()> { +// Create a new tuple with the elements (0, 1, 2) +let t = PyTuple::new_bound(py, [0, 1, 2]); +for i in 0..=2 { + let entry: Borrowed<'_, 'py, PyAny> = t.get_borrowed_item(i)?; + // `PyAnyMethods::extract` is available on `Borrowed` + // via the dereference to `Bound` + let value: usize = entry.extract()?; + assert_eq!(i, value); +} # Ok(()) # } +# Python::with_gil(example).unwrap(); ``` ## Concrete Python types @@ -131,249 +166,107 @@ This parameter `T` can be filled by: - Native Python types such as `PyList`, `PyTuple`, and `PyDict`, and - [`#[pyclass]`][pyclass] types defined from Rust -The following subsections elaborate on working with these types in a little more detail. +The following subsections covers some further detail about how to work with these types: +- the APIs that are available for these concrete types, +- how to cast `Bound<'py, T>` to a specific concrete type, and +- how to get Rust data out of a `Bound<'py, T>`. -### [`PyAny`][PyAny] +### Using APIs for concrete Python types -A Python object of unspecified type. +Each concrete Python type such as `PyAny`, `PyTuple` and `PyDict` exposes its API on the corresponding bound smart pointer `Bound<'py, PyAny>`, `Bound<'py, PyTuple>` and `Bound<'py, PyDict>`. -**Uses:** Whenever you want to refer to some Python object without knowing its type. +Each type's API is exposed as a trait: [`PyAnyMethods`], [`PyTupleMethods`], [`PyDictMethods`], and so on for all concrete types. Using traits rather than associated methods on the `Bound` smart pointer is done for a couple of reasons: +- Clarity of documentation: each trait gets its own documentation page in the PyO3 API docs. If all methods were on the `Bound` smart pointer directly, the vast majority of PyO3's API would be on a single, extremely long, documentation page. +- Consistency: downstream code implementing [Rust APIs for existing Python types](#creating-a-rust-api-for-an-existing-python-type) can also follow this pattern of using a trait. Downstream code would not be allowed to add new associated methods directly on the `Bound` type. +- Future design: it is hoped that a future Rust with [arbitrary self types](TODO) will remove the need for these traits in favour of placing the methods directly on `PyAny`, `PyTuple`, `PyDict`, and so on. -Many general methods for interacting with Python objects are on the `Bound<'py, PyAny>` smart pointer, such as `.getattr`, `.setattr`, and `.call`. +These traits are all included in the `pyo3::prelude` module, so with the glob import `use pyo3::prelude::*` the full PyO3 API is made available to downstream code. -**Conversions:** - -For a `Bound<'py, PyAny>` where the underlying object is a Python-native type such as a list: +The following function accesses the first item in the input Python list, using the `.get_item()` method from the `PyListMethods` trait: ```rust -# use pyo3::prelude::*; -# use pyo3::types::PyList; -# fn example<'py>(py: Python<'py>) -> PyResult<()> { -let obj: Bound<'py, PyAny> = PyList::empty_bound(py).into_any(); +use pyo3::prelude::*; +use pyo3::types::PyList; + +fn get_first_item<'py>(list: &Bound<'py, PyList>) -> PyResult> { + list.get_item(0) +} +# Python::with_gil(|py| { +# let l = PyList::new_bound(py, ["hello world"]); +# assert!(get_first_item(&l).unwrap().eq("hello world").unwrap()); +# }) +``` -// To &Bound<'py, PyList> with PyAnyMethods::downcast -let _: &Bound<'py, PyList> = obj.downcast()?; +### Casting between Python object types -# let obj2 = obj.clone(); -// To Py (aka PyObject) with .unbind() -let _: Py = obj.unbind(); -# let obj = obj2; +To cast `Bound<'py, T>` smart pointers to some other type, use the [`.downcast()`][PyAnyMethods::downcast] family of functions. This converts `&Bound<'py, T>` to a different `&Bound<'py, U>`, without transferring ownership. There is also [`.downcast_into()`][PyAnyMethods::downcast_into] to convert `Bound<'py, T>` to `Bound<'py, U>` with transfer of ownership. These methods are available for all types `T` which implement the [`PyTypeCheck`] trait. -// To Bound<'py, PyList> with PyAnyMethods::downcast_into -let list: Bound<'py, PyList> = obj.downcast_into()?; -# Ok(()) -# } -# Python::with_gil(example).unwrap() -``` +Casting to `Bound<'py, PyAny>` can be done with `.as_any()` or `.into_any()`. -For a `Bound<'_, PyAny>` where the underlying object is a `#[pyclass]`: +For example, the following snippet shows how to cast `Bound<'py, PyAny>` to `Bound<'py, PyTuple>`: ```rust # use pyo3::prelude::*; -# #[pyclass] #[derive(Clone)] struct MyClass { } +# use pyo3::types::PyTuple; # fn example<'py>(py: Python<'py>) -> PyResult<()> { -let obj: Bound<'_, PyAny> = Bound::new(py, MyClass {})?.into_any(); - -// To &Bound<'py, MyClass> with PyAnyMethods::downcast -let _: &Bound<'_, MyClass> = obj.downcast()?; - -# let obj2 = obj.clone(); -// To Py (aka PyObject) with .unbind() -let _: Py = obj.unbind(); -# let obj = obj2; +// create a new Python `tuple`, and use `.into_any()` to erase the type +let obj: Bound<'py, PyAny> = PyTuple::empty_bound(py).into_any(); -// To Bound<'_, MyClass> with PyAnyMethods::downcast_into -let _: Bound<'_, MyClass> = obj.downcast_into()?; - -// To MyClass with PyAny::extract, if MyClass: Clone -let _: MyClass = obj.extract()?; +// use `.downcast()` to cast to `PyTuple` without transferring ownership +let _: &Bound<'py, PyTuple> = obj.downcast()?; -// To PyRef<'_, MyClass> or PyRefMut<'_, MyClass> with PyAny::extract -let _: PyRef<'_, MyClass> = obj.extract()?; -let _: PyRefMut<'_, MyClass> = obj.extract()?; +// use `.downcast_into()` to cast to `PyTuple` with transfer of ownership +let _: Bound<'py, PyTuple> = obj.downcast_into()?; # Ok(()) # } # Python::with_gil(example).unwrap() ``` -### `PyTuple`, `PyDict`, and many more - -**Represents:** a native Python object of known type, restricted to a GIL -lifetime just like `PyAny`. - -**Used:** Whenever you want to operate with native Python types while holding -the GIL. Like `PyAny`, this is the most convenient form to use for function -arguments and intermediate values. - -These types all implement `Deref`, so they all expose the same -methods which can be found on `PyAny`. - -To see all Python types exposed by `PyO3` you should consult the -[`pyo3::types`][pyo3::types] module. - -**Conversions:** - -```rust -# use pyo3::prelude::*; -# use pyo3::types::PyList; -# Python::with_gil(|py| -> PyResult<()> { -#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API. -let list = PyList::empty(py); - -// Use methods from PyAny on all Python types with Deref implementation -let _ = list.repr()?; - -// To &PyAny automatically with Deref implementation -let _: &PyAny = list; - -// To &PyAny explicitly with .as_ref() -#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. -let _: &PyAny = list.as_ref(); - -// To Py with .into() or Py::from() -let _: Py = list.into(); - -// To PyObject with .into() or .to_object(py) -let _: PyObject = list.into(); -# Ok(()) -# }).unwrap(); -``` - -### `Py` and `PyObject` - -**Represents:** a GIL-independent reference to a Python object. This can be a Python native type -(like `PyTuple`), or a `pyclass` type implemented in Rust. The most commonly-used variant, -`Py`, is also known as `PyObject`. - -**Used:** Whenever you want to carry around references to a Python object without caring about a -GIL lifetime. For example, storing Python object references in a Rust struct that outlives the -Python-Rust FFI boundary, or returning objects from functions implemented in Rust back to Python. - -Can be cloned using Python reference counts with `.clone()`. - -**Conversions:** - -For a `Py`, the conversions are as below: - -```rust -# use pyo3::prelude::*; -# use pyo3::types::PyList; -# Python::with_gil(|py| { -let list: Py = PyList::empty_bound(py).unbind(); - -// To &PyList with Py::as_ref() (borrows from the Py) -#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. -let _: &PyList = list.as_ref(py); - -# let list_clone = list.clone(); // Because `.into_ref()` will consume `list`. -// To &PyList with Py::into_ref() (moves the pointer into PyO3's object storage) -# #[allow(deprecated)] -let _: &PyList = list.into_ref(py); - -# let list = list_clone; -// To Py (aka PyObject) with .into() -let _: Py = list.into(); -# }) -``` - -For a `#[pyclass] struct MyClass`, the conversions for `Py` are below: +Custom [`#[pyclass]`][pyclass] types implement [`PyTypeCheck`], so `.downcast()` also works for these types. The snippet below is the same as the snippet above casting instead to a custom type `MyClass`: ```rust -# use pyo3::prelude::*; -# Python::with_gil(|py| { -# #[pyclass] struct MyClass { } -# Python::with_gil(|py| -> PyResult<()> { -let my_class: Py = Py::new(py, MyClass { })?; - -// To &PyCell with Py::as_ref() (borrows from the Py) -#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. -let _: &PyCell = my_class.as_ref(py); +use pyo3::prelude::*; -# let my_class_clone = my_class.clone(); // Because `.into_ref()` will consume `my_class`. -// To &PyCell with Py::into_ref() (moves the pointer into PyO3's object storage) -# #[allow(deprecated)] -let _: &PyCell = my_class.into_ref(py); +#[pyclass] +struct MyClass { } -# let my_class = my_class_clone.clone(); -// To Py (aka PyObject) with .into_py(py) -let _: Py = my_class.into_py(py); +# fn example<'py>(py: Python<'py>) -> PyResult<()> { +// create a new Python `tuple`, and use `.into_any()` to erase the type +let obj: Bound<'py, PyAny> = Bound::new(py, MyClass { })?.into_any(); -# let my_class = my_class_clone; -// To PyRef<'_, MyClass> with Py::borrow or Py::try_borrow -let _: PyRef<'_, MyClass> = my_class.try_borrow(py)?; +// use `.downcast()` to cast to `MyClass` without transferring ownership +let _: &Bound<'py, MyClass> = obj.downcast()?; -// To PyRefMut<'_, MyClass> with Py::borrow_mut or Py::try_borrow_mut -let _: PyRefMut<'_, MyClass> = my_class.try_borrow_mut(py)?; +// use `.downcast_into()` to cast to `MyClass` with transfer of ownership +let _: Bound<'py, MyClass> = obj.downcast_into()?; # Ok(()) -# }).unwrap(); -# }); +# } +# Python::with_gil(example).unwrap() ``` -### `PyCell` - -**Represents:** a reference to a Rust object (instance of `PyClass`) which is -wrapped in a Python object. The cell part is an analog to stdlib's -[`RefCell`][RefCell] to allow access to `&mut` references. - -**Used:** for accessing pure-Rust API of the instance (members and functions -taking `&SomeType` or `&mut SomeType`) while maintaining the aliasing rules of -Rust references. - -Like PyO3's Python native types, `PyCell` implements `Deref`, -so it also exposes all of the methods on `PyAny`. +### Extracting Rust data from Python objects -**Conversions:** +To extract Rust data from Python objects, use [`.extract()`][PyAnyMethods::extract] instead of `.downcast()`. This method is available for all types which implement the [`FromPyObject`] trait. -`PyCell` can be used to access `&T` and `&mut T` via `PyRef` and `PyRefMut` respectively. +For example, the following snippet extracts a Rust tuple of integers from a Python tuple: ```rust # use pyo3::prelude::*; -# #[pyclass] struct MyClass { } -# Python::with_gil(|py| -> PyResult<()> { -# #[allow(deprecated)] -let cell: &PyCell = PyCell::new(py, MyClass {})?; - -// To PyRef with .borrow() or .try_borrow() -let py_ref: PyRef<'_, MyClass> = cell.try_borrow()?; -let _: &MyClass = &*py_ref; -# drop(py_ref); - -// To PyRefMut with .borrow_mut() or .try_borrow_mut() -let mut py_ref_mut: PyRefMut<'_, MyClass> = cell.try_borrow_mut()?; -let _: &mut MyClass = &mut *py_ref_mut; -# Ok(()) -# }).unwrap(); -``` - -`PyCell` can also be accessed like a Python-native type. - -```rust -# use pyo3::prelude::*; -# #[pyclass] struct MyClass { } -# Python::with_gil(|py| -> PyResult<()> { -# #[allow(deprecated)] -let cell: &PyCell = PyCell::new(py, MyClass {})?; - -// Use methods from PyAny on PyCell with Deref implementation -let _ = cell.repr()?; - -// To &PyAny automatically with Deref implementation -let _: &PyAny = cell; +# use pyo3::types::PyTuple; +# fn example<'py>(py: Python<'py>) -> PyResult<()> { +// create a new Python `tuple`, and use `.into_any()` to erase the type +let obj: Bound<'py, PyAny> = PyTuple::new_bound(py, [1, 2, 3]).into_any(); -// To &PyAny explicitly with .as_ref() -#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. -let _: &PyAny = cell.as_ref(); +// extracting the Python `tuple` to a rust `(i32, i32, i32)` tuple +let (x, y, z) = obj.extract::<(i32, i32, i32)>()?; +assert_eq!((x, y, z), (1, 2, 3)); # Ok(()) -# }).unwrap(); +# } +# Python::with_gil(example).unwrap() ``` -### `PyRef` and `PyRefMut` - -**Represents:** reference wrapper types employed by `PyCell` to keep track of -borrows, analog to `Ref` and `RefMut` used by `RefCell`. - -**Used:** while borrowing a `PyCell`. They can also be used with `.extract()` -on types like `Py` and `PyAny` to get a reference quickly. +To avoid copying data, [`#[pyclass]`][pyclass] types can directly reference Rust data stored within the Python objects without needing to `.extract()`. See the [corresponding documentation in the class section of the guide](./class. +md#bound-and-interior-mutability) for more detail. ## The GIL Refs API @@ -545,9 +438,18 @@ let _: &PyAny = cell.as_ref(); # }).unwrap(); ``` -[pyclass]: class.md -[Py]: {{#PYO3_DOCS_URL}}/pyo3/struct.Py.html [Bound]: {{#PYO3_DOCS_URL}}/pyo3/struct.Bound.html +[`Bound::unbind`]: {{#PYO3_DOCS_URL}}/pyo3/struct.Bound.html#method.unbind +[Py]: {{#PYO3_DOCS_URL}}/pyo3/struct.Py.html +[PyAnyMethods::add]: {{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.add +[PyAnyMethods::extract]: {{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.extract +[PyAnyMethods::downcast]: {{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.downcast +[PyAnyMethods::downcast_into]: {{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html#tymethod.downcast_into +[`PyTypeCheck`]: {{#PYO3_DOCS_URL}}/pyo3/type_object/trait.PyTypeCheck.html +[`PyAnyMethods`]: {{#PYO3_DOCS_URL}}/pyo3/types/trait.PyAnyMethods.html +[`PyDictMethods`]: {{#PYO3_DOCS_URL}}/pyo3/types/trait.PyDictMethods.html +[`PyTupleMethods`]: {{#PYO3_DOCS_URL}}/pyo3/types/trait.PyTupleMethods.html +[pyclass]: class.md [Borrowed]: {{#PYO3_DOCS_URL}}/pyo3/struct.Borrowed.html [Drop]: https://doc.rust-lang.org/std/drop/trait.Drop.html [eval]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.eval diff --git a/src/lib.rs b/src/lib.rs index a6482015de3..95c758569a8 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -30,29 +30,38 @@ //! //! PyO3 has several core types that you should familiarize yourself with: //! -//! ## The Python<'py> object -//! -//! Holding the [global interpreter lock] (GIL) is modeled with the [`Python<'py>`](crate::Python) -//! token. All APIs that require that the GIL is held require this token as proof that you really -//! are holding the GIL. It can be explicitly acquired and is also implicitly acquired by PyO3 as -//! it wraps Rust functions and structs into Python functions and objects. -//! -//! ## The GIL-dependent types -//! -//! For example `&`[`PyAny`]. These are only ever seen as references, with a lifetime that is only -//! valid for as long as the GIL is held, which is why using them doesn't require a -//! [`Python<'py>`](crate::Python) token. The underlying Python object, if mutable, can be mutated -//! through any reference. +//! ## The `Python<'py>` object, and the `'py` lifetime +//! +//! Holding the [global interpreter lock] (GIL) is modeled with the [`Python<'py>`](Python) token. Many +//! Python APIs require that the GIL is held, and PyO3 uses this token as proof that these APIs +//! can be called safely. It can be explicitly acquired and is also implicitly acquired by PyO3 +//! as it wraps Rust functions and structs into Python functions and objects. +//! +//! The [`Python<'py>`](Python) token's lifetime `'py` is common to many PyO3 APIs: +//! - Types that also have the `'py` lifetime, such as the [`Bound<'py, T>`](Bound) smart pointer, are +//! bound to the Python GIL and rely on this to offer their functionality. These types often +//! have a [`.py()`](Bound::py) method to get the associated [`Python<'py>`](Python) token. +//! - Functions which depend on the `'py` lifetime, such as [`PyList::new_bound`](types::PyList::new_bound), +//! require a [`Python<'py>`](Python) token as an input. Sometimes the token is passed implicitly by +//! taking a [`Bound<'py, T>`](Bound) or other type which is bound to the `'py` lifetime. +//! - Traits which depend on the `'py` lifetime, such as [`FromPyObject<'py>`](FromPyObject), usually have +//! inputs or outputs which depend on the lifetime. Adding the lifetime to the trait allows +//! these inputs and outputs to express their binding to the GIL in the Rust type system. +//! +//! ## Python object smart pointers +//! +//! PyO3 has two core smart pointers to refer to Python objects, [`Py`](Py) and its GIL-bound +//! form [`Bound<'py, T>`](Bound) which carries the `'py` lifetime. (There is also +//! [`Borrowed<'a, 'py, T>`](instance::Borrowed), but it is used much more rarely. +//! +//! The type parameter `T` in these smart pointers can be filled by: +//! - [`PyAny`], e.g. `Py` or `Bound<'py, PyAny>`, where the Python object type is not +//! known. `Py` is so common it has a type alias [`PyObject`]. +//! - Concrete Python types like [`PyList`](types::PyList) or [`PyTuple`](types::PyTuple). +//! - Rust types which are exposed to Python using the [`#[pyclass]`](macro@pyclass) macro. //! //! See the [guide][types] for an explanation of the different Python object types. //! -//! ## The GIL-independent types -//! -//! When wrapped in [`Py`]`<...>`, like with [`Py`]`<`[`PyAny`]`>` or [`Py`]``, Python -//! objects no longer have a limited lifetime which makes them easier to store in structs and pass -//! between functions. However, you cannot do much with them without a -//! [`Python<'py>`](crate::Python) token, for which you’d need to reacquire the GIL. -//! //! ## PyErr //! //! The vast majority of operations in this library will return [`PyResult<...>`](PyResult). From 92ba7f11421032b25f1b98268e5a079513882c37 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 12:15:24 +0000 Subject: [PATCH 11/18] update remainder of the guide to Bound API --- guide/src/advanced.md | 7 -- guide/src/async-await.md | 12 ++- guide/src/class.md | 10 ++- guide/src/conversions/tables.md | 4 +- guide/src/conversions/traits.md | 2 +- guide/src/ecosystem/async-await.md | 12 +-- guide/src/exception.md | 2 +- guide/src/function.md | 4 +- guide/src/memory.md | 32 ++++++-- guide/src/migration.md | 14 +++- guide/src/performance.md | 42 +++++----- .../python-from-rust/calling-existing-code.md | 21 +++-- guide/src/trait-bounds.md | 80 +++++++------------ guide/src/types.md | 35 ++++---- noxfile.py | 1 - 15 files changed, 144 insertions(+), 134 deletions(-) diff --git a/guide/src/advanced.md b/guide/src/advanced.md index 8264c14dd17..61dc66382f4 100644 --- a/guide/src/advanced.md +++ b/guide/src/advanced.md @@ -5,10 +5,3 @@ PyO3 exposes much of Python's C API through the `ffi` module. The C API is naturally unsafe and requires you to manage reference counts, errors and specific invariants yourself. Please refer to the [C API Reference Manual](https://docs.python.org/3/c-api/) and [The Rustonomicon](https://doc.rust-lang.org/nightly/nomicon/ffi.html) before using any function from that API. - -## Memory management - -PyO3's `&PyAny` "owned references" and `Py` smart pointers are used to -access memory stored in Python's heap. This memory sometimes lives for longer -than expected because of differences in Rust and Python's memory models. See -the chapter on [memory management](./memory.md) for more information. diff --git a/guide/src/async-await.md b/guide/src/async-await.md index c354be7f1b7..0db1720340d 100644 --- a/guide/src/async-await.md +++ b/guide/src/async-await.md @@ -30,13 +30,13 @@ async fn sleep(seconds: f64, result: Option) -> Option { Resulting future of an `async fn` decorated by `#[pyfunction]` must be `Send + 'static` to be embedded in a Python object. -As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a signature like `async fn does_not_compile(arg: &PyAny, py: Python<'_>) -> &PyAny`. +As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a signature like `async fn does_not_compile<'py>(arg: Bound<'py, PyAny>) -> Bound<'py, PyAny>`. -However, there is an exception for method receiver, so async methods can accept `&self`/`&mut self`. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows `&self` to exclusive ones `&mut self` to avoid racy borrow check failures at runtime. +However, there is an exception for method receivers, so async methods can accept `&self`/`&mut self`. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows `&self` to exclusive ones `&mut self` to avoid racy borrow check failures at runtime. ## Implicit GIL holding -Even if it is not possible to pass a `py: Python<'_>` parameter to `async fn`, the GIL is still held during the execution of the future – it's also the case for regular `fn` without `Python<'_>`/`&PyAny` parameter, yet the GIL is held. +Even if it is not possible to pass a `py: Python<'py>` parameter to `async fn`, the GIL is still held during the execution of the future – it's also the case for regular `fn` without `Python<'py>`/`Bound<'py, PyAny>` parameter, yet the GIL is held. It is still possible to get a `Python` marker using [`Python::with_gil`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.with_gil); because `with_gil` is reentrant and optimized, the cost will be negligible. @@ -47,7 +47,11 @@ There is currently no simple way to release the GIL when awaiting a future, *but Here is the advised workaround for now: ```rust,ignore -use std::{future::Future, pin::{Pin, pin}, task::{Context, Poll}}; +use std::{ + future::Future, + pin::{Pin, pin}, + task::{Context, Poll}, +}; use pyo3::prelude::*; struct AllowThreads(F); diff --git a/guide/src/class.md b/guide/src/class.md index 20a78a7f40c..2c6d854ff08 100644 --- a/guide/src/class.md +++ b/guide/src/class.md @@ -60,7 +60,7 @@ enum Shape { Circle { radius: f64 }, Rectangle { width: f64, height: f64 }, RegularPolygon { side_count: u32, radius: f64 }, - Nothing { }, + Nothing {}, } ``` @@ -89,7 +89,7 @@ Currently, the best alternative is to write a macro which expands to a new `#[py use pyo3::prelude::*; struct GenericClass { - data: T + data: T, } macro_rules! create_interface { @@ -102,7 +102,9 @@ macro_rules! create_interface { impl $name { #[new] pub fn new(data: $type) -> Self { - Self { inner: GenericClass { data: data } } + Self { + inner: GenericClass { data: data }, + } } } }; @@ -427,7 +429,7 @@ impl DictWithCounter { Self::default() } - fn set(slf: &Bound<'_, Self>, key: String, value: &PyAny) -> PyResult<()> { + fn set(slf: &Bound<'_, Self>, key: String, value: Bound<'_, PyAny>) -> PyResult<()> { slf.borrow_mut().counter.entry(key.clone()).or_insert(0); let dict = slf.downcast::()?; dict.set_item(key, value) diff --git a/guide/src/conversions/tables.md b/guide/src/conversions/tables.md index e61932f0206..eb33b17acf7 100644 --- a/guide/src/conversions/tables.md +++ b/guide/src/conversions/tables.md @@ -72,9 +72,9 @@ For most PyO3 usage the conversion cost is worth paying to get these benefits. A ### Returning Rust values to Python -When returning values from functions callable from Python, Python-native types (`&PyAny`, `&PyDict` etc.) can be used with zero cost. +When returning values from functions callable from Python, [PyO3's smart pointers](../types.md#pyo3s-smart-pointers) (`Py`, `Bound<'py, T>`, and `Borrowed<'a, 'py, T>`) can be used with zero cost. -Because these types are references, in some situations the Rust compiler may ask for lifetime annotations. If this is the case, you should use `Py`, `Py` etc. instead - which are also zero-cost. For all of these Python-native types `T`, `Py` can be created from `T` with an `.into()` conversion. +Because `Bound<'py, T>` and `Borrowed<'a, 'py, T>` have lifetime parameters, the Rust compiler may ask for lifetime annotations to be added to your function. See the [section of the guide dedicated to this](../types.md#function-argument-lifetimes). If your function is fallible, it should return `PyResult` or `Result` where `E` implements `From for PyErr`. This will raise a `Python` exception if the `Err` variant is returned. diff --git a/guide/src/conversions/traits.md b/guide/src/conversions/traits.md index 5cdf2c590b9..3d1d5e21d85 100644 --- a/guide/src/conversions/traits.md +++ b/guide/src/conversions/traits.md @@ -484,7 +484,7 @@ If the input is neither a string nor an integer, the error message will be: - `pyo3(from_py_with = "...")` - apply a custom function to convert the field from Python the desired Rust type. - the argument must be the name of the function as a string. - - the function signature must be `fn(&PyAny) -> PyResult` where `T` is the Rust type of the argument. + - the function signature must be `fn(Bound) -> PyResult` where `T` is the Rust type of the argument. ### `IntoPy` diff --git a/guide/src/ecosystem/async-await.md b/guide/src/ecosystem/async-await.md index 1e4ea4ab4b9..0319fa05063 100644 --- a/guide/src/ecosystem/async-await.md +++ b/guide/src/ecosystem/async-await.md @@ -120,7 +120,7 @@ Export an async function that makes use of `async-std`: use pyo3::{prelude::*, wrap_pyfunction}; #[pyfunction] -fn rust_sleep(py: Python<'_>) -> PyResult<&PyAny> { +fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> { pyo3_asyncio::async_std::future_into_py(py, async { async_std::task::sleep(std::time::Duration::from_secs(1)).await; Ok(Python::with_gil(|py| py.None())) @@ -143,7 +143,7 @@ If you want to use `tokio` instead, here's what your module should look like: use pyo3::{prelude::*, wrap_pyfunction}; #[pyfunction] -fn rust_sleep(py: Python<'_>) -> PyResult<&PyAny> { +fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> { pyo3_asyncio::tokio::future_into_py(py, async { tokio::time::sleep(std::time::Duration::from_secs(1)).await; Ok(Python::with_gil(|py| py.None())) @@ -233,7 +233,7 @@ a coroutine argument: ```rust #[pyfunction] -fn await_coro(coro: &PyAny) -> PyResult<()> { +fn await_coro(coro: &Bound<'_, PyAny>>) -> PyResult<()> { // convert the coroutine into a Rust future using the // async_std runtime let f = pyo3_asyncio::async_std::into_future(coro)?; @@ -261,7 +261,7 @@ If for you wanted to pass a callable function to the `#[pyfunction]` instead, (i ```rust #[pyfunction] -fn await_coro(callable: &PyAny) -> PyResult<()> { +fn await_coro(callable: &Bound<'_, PyAny>>) -> PyResult<()> { // get the coroutine by calling the callable let coro = callable.call0()?; @@ -317,7 +317,7 @@ async fn rust_sleep() { } #[pyfunction] -fn call_rust_sleep(py: Python<'_>) -> PyResult<&PyAny> { +fn call_rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> { pyo3_asyncio::async_std::future_into_py(py, async move { rust_sleep().await; Ok(Python::with_gil(|py| py.None())) @@ -467,7 +467,7 @@ tokio = "1.4" use pyo3::{prelude::*, wrap_pyfunction}; #[pyfunction] -fn rust_sleep(py: Python<'_>) -> PyResult<&PyAny> { +fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> { pyo3_asyncio::tokio::future_into_py(py, async { tokio::time::sleep(std::time::Duration::from_secs(1)).await; Ok(Python::with_gil(|py| py.None())) diff --git a/guide/src/exception.md b/guide/src/exception.md index ee36f544d74..3e2f5034897 100644 --- a/guide/src/exception.md +++ b/guide/src/exception.md @@ -111,7 +111,7 @@ mod io { pyo3::import_exception!(io, UnsupportedOperation); } -fn tell(file: &PyAny) -> PyResult { +fn tell(file: &Bound<'_, PyAny>) -> PyResult { match file.call_method0("tell") { Err(_) => Err(io::UnsupportedOperation::new_err("not supported: tell")), Ok(x) => x.extract::(), diff --git a/guide/src/function.md b/guide/src/function.md index cfb1e5ef81e..86ac4c89b46 100644 --- a/guide/src/function.md +++ b/guide/src/function.md @@ -87,7 +87,9 @@ The `#[pyo3]` attribute can be used to modify properties of the generated Python #[pyfunction] #[pyo3(pass_module)] - fn pyfunction_with_module<'py>(module: &Bound<'py, PyModule>) -> PyResult> { + fn pyfunction_with_module<'py>( + module: &Bound<'py, PyModule>, + ) -> PyResult> { module.name() } diff --git a/guide/src/memory.md b/guide/src/memory.md index b14ce4496ad..f40c4c6fd25 100644 --- a/guide/src/memory.md +++ b/guide/src/memory.md @@ -1,5 +1,15 @@ # Memory management +
+⚠️ Warning: API update in progress 🛠️ + +PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer `Bound`. + +This section on memory management is heavily weighted towards the now-deprecated "GIL Refs" API, which suffered from the drawbacks detailed here as well as CPU overheads. + +See [the smart pointer types](./types.md#pyo3s-smart-pointers) for description on the new, simplified, memory model of the Bound API, which is built as a thin wrapper on Python reference counting. +
+ Rust and Python have very different notions of memory management. Rust has a strict memory model with concepts of ownership, borrowing, and lifetimes, where memory is freed at predictable points in program execution. Python has @@ -10,12 +20,12 @@ Memory in Python is freed eventually by the garbage collector, but not usually in a predictable way. PyO3 bridges the Rust and Python memory models with two different strategies for -accessing memory allocated on Python's heap from inside Rust. These are -GIL-bound, or "owned" references, and GIL-independent `Py` smart pointers. +accessing memory allocated on Python's heap from inside Rust. These are +GIL Refs such as `&'py PyAny`, and GIL-independent `Py` smart pointers. ## GIL-bound memory -PyO3's GIL-bound, "owned references" (`&PyAny` etc.) make PyO3 more ergonomic to +PyO3's GIL Refs such as `&'py PyAny` make PyO3 more ergonomic to use by ensuring that their lifetime can never be longer than the duration the Python GIL is held. This means that most of PyO3's API can assume the GIL is held. (If PyO3 could not assume this, every PyO3 API would need to take a @@ -27,7 +37,9 @@ very simple and easy-to-understand programs like this: # use pyo3::types::PyString; # fn main() -> PyResult<()> { Python::with_gil(|py| -> PyResult<()> { - let hello = py.eval_bound("\"Hello World!\"", None, None)?.downcast_into::()?; + let hello = py + .eval_bound("\"Hello World!\"", None, None)? + .downcast_into::()?; println!("Python says: {}", hello); Ok(()) })?; @@ -48,7 +60,9 @@ of the time we don't have to think about this, but consider the following: # fn main() -> PyResult<()> { Python::with_gil(|py| -> PyResult<()> { for _ in 0..10 { - let hello = py.eval_bound("\"Hello World!\"", None, None)?.downcast_into::()?; + let hello = py + .eval_bound("\"Hello World!\"", None, None)? + .downcast_into::()?; println!("Python says: {}", hello); } // There are 10 copies of `hello` on Python's heap here. @@ -76,7 +90,9 @@ is to acquire and release the GIL with each iteration of the loop. # fn main() -> PyResult<()> { for _ in 0..10 { Python::with_gil(|py| -> PyResult<()> { - let hello = py.eval_bound("\"Hello World!\"", None, None)?.downcast_into::()?; + let hello = py + .eval_bound("\"Hello World!\"", None, None)? + .downcast_into::()?; println!("Python says: {}", hello); Ok(()) })?; // only one copy of `hello` at a time @@ -97,7 +113,9 @@ Python::with_gil(|py| -> PyResult<()> { for _ in 0..10 { let pool = unsafe { py.new_pool() }; let py = pool.python(); - let hello = py.eval_bound("\"Hello World!\"", None, None)?.downcast_into::()?; + let hello = py + .eval_bound("\"Hello World!\"", None, None)? + .downcast_into::()?; println!("Python says: {}", hello); } Ok(()) diff --git a/guide/src/migration.md b/guide/src/migration.md index f2c478da4a8..5af6eb2336c 100644 --- a/guide/src/migration.md +++ b/guide/src/migration.md @@ -190,7 +190,9 @@ struct PyClassAsyncIter { impl PyClassAsyncIter { fn __anext__(&mut self) -> PyClassAwaitable { self.number += 1; - PyClassAwaitable { number: self.number } + PyClassAwaitable { + number: self.number, + } } fn __aiter__(slf: Py) -> Py { @@ -312,7 +314,10 @@ Python::with_gil(|py| { // `b` is not in the dictionary assert!(dict.get_item("b").is_none()); // `dict` is not hashable, so this fails with a `TypeError` - assert!(dict.get_item_with_error(dict).unwrap_err().is_instance_of::(py)); + assert!(dict + .get_item_with_error(dict) + .unwrap_err() + .is_instance_of::(py)); }); # } ``` @@ -333,7 +338,10 @@ Python::with_gil(|py| -> PyResult<()> { // `b` is not in the dictionary assert!(dict.get_item("b")?.is_none()); // `dict` is not hashable, so this fails with a `TypeError` - assert!(dict.get_item(dict).unwrap_err().is_instance_of::(py)); + assert!(dict + .get_item(dict) + .unwrap_err() + .is_instance_of::(py)); Ok(()) }); diff --git a/guide/src/performance.md b/guide/src/performance.md index fe362bed953..3db23ddf01e 100644 --- a/guide/src/performance.md +++ b/guide/src/performance.md @@ -4,26 +4,26 @@ To achieve the best possible performance, it is useful to be aware of several tr ## `extract` versus `downcast` -Pythonic API implemented using PyO3 are often polymorphic, i.e. they will accept `&PyAny` and try to turn this into multiple more concrete types to which the requested operation is applied. This often leads to chains of calls to `extract`, e.g. +Pythonic API implemented using PyO3 are often polymorphic, i.e. they will accept `&Bound<'_, PyAny>` and try to turn this into multiple more concrete types to which the requested operation is applied. This often leads to chains of calls to `extract`, e.g. ```rust # #![allow(dead_code)] # use pyo3::prelude::*; # use pyo3::{exceptions::PyTypeError, types::PyList}; -fn frobnicate_list(list: &PyList) -> PyResult<&PyAny> { +fn frobnicate_list<'py>(list: &Bound<'_, PyList>) -> PyResult> { todo!() } -fn frobnicate_vec(vec: Vec<&PyAny>) -> PyResult<&PyAny> { +fn frobnicate_vec<'py>(vec: Vec>) -> PyResult> { todo!() } #[pyfunction] -fn frobnicate(value: &PyAny) -> PyResult<&PyAny> { - if let Ok(list) = value.extract::<&PyList>() { - frobnicate_list(list) - } else if let Ok(vec) = value.extract::>() { +fn frobnicate<'py>(value: &Bound<'py, PyAny>) -> PyResult> { + if let Ok(list) = value.extract::>() { + frobnicate_list(&list) + } else if let Ok(vec) = value.extract::>>() { frobnicate_vec(vec) } else { Err(PyTypeError::new_err("Cannot frobnicate that type.")) @@ -37,15 +37,15 @@ This suboptimal as the `FromPyObject` trait requires `extract` to have a `Res # #![allow(dead_code)] # use pyo3::prelude::*; # use pyo3::{exceptions::PyTypeError, types::PyList}; -# fn frobnicate_list(list: &PyList) -> PyResult<&PyAny> { todo!() } -# fn frobnicate_vec(vec: Vec<&PyAny>) -> PyResult<&PyAny> { todo!() } +# fn frobnicate_list<'py>(list: &Bound<'_, PyList>) -> PyResult> { todo!() } +# fn frobnicate_vec<'py>(vec: Vec>) -> PyResult> { todo!() } # #[pyfunction] -fn frobnicate(value: &PyAny) -> PyResult<&PyAny> { +fn frobnicate<'py>(value: &Bound<'py, PyAny>) -> PyResult> { // Use `downcast` instead of `extract` as turning `PyDowncastError` into `PyErr` is quite costly. if let Ok(list) = value.downcast::() { frobnicate_list(list) - } else if let Ok(vec) = value.extract::>() { + } else if let Ok(vec) = value.extract::>>() { frobnicate_vec(vec) } else { Err(PyTypeError::new_err("Cannot frobnicate that type.")) @@ -53,9 +53,9 @@ fn frobnicate(value: &PyAny) -> PyResult<&PyAny> { } ``` -## Access to GIL-bound reference implies access to GIL token +## Access to Bound implies access to GIL token -Calling `Python::with_gil` is effectively a no-op when the GIL is already held, but checking that this is the case still has a cost. If an existing GIL token can not be accessed, for example when implementing a pre-existing trait, but a GIL-bound reference is available, this cost can be avoided by exploiting that access to GIL-bound reference gives zero-cost access to a GIL token via `PyAny::py`. +Calling `Python::with_gil` is effectively a no-op when the GIL is already held, but checking that this is the case still has a cost. If an existing GIL token can not be accessed, for example when implementing a pre-existing trait, but a GIL-bound reference is available, this cost can be avoided by exploiting that access to GIL-bound reference gives zero-cost access to a GIL token via `Bound::py`. For example, instead of writing @@ -66,34 +66,32 @@ For example, instead of writing struct Foo(Py); -struct FooRef<'a>(&'a PyList); +struct FooBound<'py>(Bound<'py, PyList>); -impl PartialEq for FooRef<'_> { +impl PartialEq for FooBound<'_> { fn eq(&self, other: &Foo) -> bool { Python::with_gil(|py| { - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. - let len = other.0.as_ref(py).len(); + let len = other.0.bind(py).len(); self.0.len() == len }) } } ``` -use more efficient +use the more efficient ```rust # #![allow(dead_code)] # use pyo3::prelude::*; # use pyo3::types::PyList; # struct Foo(Py); -# struct FooRef<'a>(&'a PyList); +# struct FooBound<'py>(Bound<'py, PyList>); # -impl PartialEq for FooRef<'_> { +impl PartialEq for FooBound<'_> { fn eq(&self, other: &Foo) -> bool { // Access to `&'a PyAny` implies access to `Python<'a>`. let py = self.0.py(); - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. - let len = other.0.as_ref(py).len(); + let len = other.0.bind(py).len(); self.0.len() == len } } diff --git a/guide/src/python-from-rust/calling-existing-code.md b/guide/src/python-from-rust/calling-existing-code.md index 46a6b6ebbed..53051d4ce51 100644 --- a/guide/src/python-from-rust/calling-existing-code.md +++ b/guide/src/python-from-rust/calling-existing-code.md @@ -28,7 +28,7 @@ fn main() -> PyResult<()> { [`Python::eval`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.eval) is a method to execute a [Python expression](https://docs.python.org/3.7/reference/expressions.html) -and return the evaluated value as a `&PyAny` object. +and return the evaluated value as a `Bound<'py, PyAny>` object. ```rust use pyo3::prelude::*; @@ -105,10 +105,7 @@ can be used to generate a Python module which can then be used just as if it was to this function! ```rust -use pyo3::{ - prelude::*, - types::IntoPyDict, -}; +use pyo3::{prelude::*, types::IntoPyDict}; # fn main() -> PyResult<()> { Python::with_gil(|py| { @@ -293,7 +290,10 @@ fn main() -> PyResult<()> { let path = Path::new("/usr/share/python_app"); let py_app = fs::read_to_string(path.join("app.py"))?; let from_python = Python::with_gil(|py| -> PyResult> { - let syspath = py.import_bound("sys")?.getattr("path")?.downcast_into::()?; + let syspath = py + .import_bound("sys")? + .getattr("path")? + .downcast_into::()?; syspath.insert(0, &path)?; let app: Py = PyModule::from_code_bound(py, &py_app, "", "")? .getattr("run")? @@ -357,7 +357,14 @@ class House(object): } Err(e) => { house - .call_method1("__exit__", (e.get_type_bound(py), e.value_bound(py), e.traceback_bound(py))) + .call_method1( + "__exit__", + ( + e.get_type_bound(py), + e.value_bound(py), + e.traceback_bound(py), + ), + ) .unwrap(); } } diff --git a/guide/src/trait-bounds.md b/guide/src/trait-bounds.md index b0eee80c80a..65842dee71f 100644 --- a/guide/src/trait-bounds.md +++ b/guide/src/trait-bounds.md @@ -66,6 +66,7 @@ The following wrapper will call the Python model from Rust, using a struct to ho ```rust # #![allow(dead_code)] use pyo3::prelude::*; +use pyo3::types::PyList; # pub trait Model { # fn set_variables(&mut self, inputs: &Vec); @@ -81,12 +82,8 @@ impl Model for UserModel { fn set_variables(&mut self, var: &Vec) { println!("Rust calling Python to set the variables"); Python::with_gil(|py| { - let values: Vec = var.clone(); - let list: PyObject = values.into_py(py); - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. - let py_model = self.model.as_ref(py); - py_model - .call_method("set_variables", (list,), None) + self.model.bind(py) + .call_method("set_variables", (PyList::new_bound(py, var),), None) .unwrap(); }) } @@ -94,9 +91,8 @@ impl Model for UserModel { fn get_results(&self) -> Vec { println!("Rust calling Python to get the results"); Python::with_gil(|py| { - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. self.model - .as_ref(py) + .bind(py) .call_method("get_results", (), None) .unwrap() .extract() @@ -107,9 +103,8 @@ impl Model for UserModel { fn compute(&mut self) { println!("Rust calling Python to perform the computation"); Python::with_gil(|py| { - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. self.model - .as_ref(py) + .bind(py) .call_method("compute", (), None) .unwrap(); }) @@ -168,6 +163,7 @@ This wrapper will also perform the type conversions between Python and Rust. ```rust # #![allow(dead_code)] # use pyo3::prelude::*; +# use pyo3::types::PyList; # # pub trait Model { # fn set_variables(&mut self, inputs: &Vec); @@ -184,12 +180,8 @@ This wrapper will also perform the type conversions between Python and Rust. # fn set_variables(&mut self, var: &Vec) { # println!("Rust calling Python to set the variables"); # Python::with_gil(|py| { -# let values: Vec = var.clone(); -# let list: PyObject = values.into_py(py); -# #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. -# let py_model = self.model.as_ref(py); -# py_model -# .call_method("set_variables", (list,), None) +# self.model.bind(py) +# .call_method("set_variables", (PyList::new_bound(py, var),), None) # .unwrap(); # }) # } @@ -197,9 +189,8 @@ This wrapper will also perform the type conversions between Python and Rust. # fn get_results(&self) -> Vec { # println!("Rust calling Python to get the results"); # Python::with_gil(|py| { -# #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. # self.model -# .as_ref(py) +# .bind(py) # .call_method("get_results", (), None) # .unwrap() # .extract() @@ -210,9 +201,8 @@ This wrapper will also perform the type conversions between Python and Rust. # fn compute(&mut self) { # println!("Rust calling Python to perform the computation"); # Python::with_gil(|py| { -# #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. # self.model -# .as_ref(py) +# .bind(py) # .call_method("compute", (), None) # .unwrap(); # }) @@ -340,6 +330,7 @@ We used in our `get_results` method the following call that performs the type co ```rust # #![allow(dead_code)] # use pyo3::prelude::*; +# use pyo3::types::PyList; # # pub trait Model { # fn set_variables(&mut self, inputs: &Vec); @@ -355,10 +346,9 @@ We used in our `get_results` method the following call that performs the type co impl Model for UserModel { fn get_results(&self) -> Vec { println!("Rust calling Python to get the results"); - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. Python::with_gil(|py| { self.model - .as_ref(py) + .bind(py) .call_method("get_results", (), None) .unwrap() .extract() @@ -368,12 +358,8 @@ impl Model for UserModel { # fn set_variables(&mut self, var: &Vec) { # println!("Rust calling Python to set the variables"); # Python::with_gil(|py| { -# let values: Vec = var.clone(); -# let list: PyObject = values.into_py(py); -# #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. -# let py_model = self.model.as_ref(py); -# py_model -# .call_method("set_variables", (list,), None) +# self.model.bind(py) +# .call_method("set_variables", (PyList::new_bound(py, var),), None) # .unwrap(); # }) # } @@ -381,9 +367,8 @@ impl Model for UserModel { # fn compute(&mut self) { # println!("Rust calling Python to perform the computation"); # Python::with_gil(|py| { -# #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. # self.model -# .as_ref(py) +# .bind(py) # .call_method("compute", (), None) # .unwrap(); # }) @@ -396,6 +381,7 @@ Let's break it down in order to perform better error handling: ```rust # #![allow(dead_code)] # use pyo3::prelude::*; +# use pyo3::types::PyList; # # pub trait Model { # fn set_variables(&mut self, inputs: &Vec); @@ -412,10 +398,9 @@ impl Model for UserModel { fn get_results(&self) -> Vec { println!("Get results from Rust calling Python"); Python::with_gil(|py| { - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. - let py_result: &PyAny = self + let py_result: Bound<'_, PyAny> = self .model - .as_ref(py) + .bind(py) .call_method("get_results", (), None) .unwrap(); @@ -432,12 +417,8 @@ impl Model for UserModel { # fn set_variables(&mut self, var: &Vec) { # println!("Rust calling Python to set the variables"); # Python::with_gil(|py| { -# let values: Vec = var.clone(); -# let list: PyObject = values.into_py(py); -# #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. -# let py_model = self.model.as_ref(py); -# py_model -# .call_method("set_variables", (list,), None) +# let py_model = self.model.bind(py) +# .call_method("set_variables", (PyList::new_bound(py, var),), None) # .unwrap(); # }) # } @@ -445,9 +426,8 @@ impl Model for UserModel { # fn compute(&mut self) { # println!("Rust calling Python to perform the computation"); # Python::with_gil(|py| { -# #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. # self.model -# .as_ref(py) +# .bind(py) # .call_method("compute", (), None) # .unwrap(); # }) @@ -478,6 +458,7 @@ It is also required to make the struct public. ```rust # #![allow(dead_code)] use pyo3::prelude::*; +use pyo3::types::PyList; pub trait Model { fn set_variables(&mut self, var: &Vec); @@ -533,12 +514,9 @@ impl Model for UserModel { fn set_variables(&mut self, var: &Vec) { println!("Rust calling Python to set the variables"); Python::with_gil(|py| { - let values: Vec = var.clone(); - let list: PyObject = values.into_py(py); - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. - let py_model = self.model.as_ref(py); - py_model - .call_method("set_variables", (list,), None) + self.model + .bind(py) + .call_method("set_variables", (PyList::new_bound(py, var),), None) .unwrap(); }) } @@ -546,10 +524,9 @@ impl Model for UserModel { fn get_results(&self) -> Vec { println!("Get results from Rust calling Python"); Python::with_gil(|py| { - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. - let py_result: &PyAny = self + let py_result: Bound<'_, PyAny> = self .model - .as_ref(py) + .bind(py) .call_method("get_results", (), None) .unwrap(); @@ -567,9 +544,8 @@ impl Model for UserModel { fn compute(&mut self) { println!("Rust calling Python to perform the computation"); Python::with_gil(|py| { - #[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. self.model - .as_ref(py) + .bind(py) .call_method("compute", (), None) .unwrap(); }) diff --git a/guide/src/types.md b/guide/src/types.md index 9c4ed4dc849..2ba69f72669 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -65,8 +65,8 @@ use pyo3::types::PyList; fn example<'py>(py: Python<'py>) -> PyResult<()> { let x: Bound<'py, PyList> = PyList::empty_bound(py); x.append(1)?; - let y: Bound<'py, PyList> = x.clone(); // y is a new reference to the same list - drop(x); // release the original reference x + let y: Bound<'py, PyList> = x.clone(); // y is a new reference to the same list + drop(x); // release the original reference x Ok(()) } # Python::with_gil(example).unwrap(); @@ -107,7 +107,10 @@ The correct way to solve this is to add the `'py` lifetime as a parameter for th ```rust # use pyo3::prelude::*; -fn add<'py>(left: &Bound<'py, PyAny>, right: &Bound<'py, PyAny>) -> PyResult> { +fn add<'py>( + left: &Bound<'py, PyAny>, + right: &Bound<'py, PyAny>, +) -> PyResult> { left.add(right) } # Python::with_gil(|py| { @@ -177,8 +180,8 @@ Each concrete Python type such as `PyAny`, `PyTuple` and `PyDict` exposes its AP Each type's API is exposed as a trait: [`PyAnyMethods`], [`PyTupleMethods`], [`PyDictMethods`], and so on for all concrete types. Using traits rather than associated methods on the `Bound` smart pointer is done for a couple of reasons: - Clarity of documentation: each trait gets its own documentation page in the PyO3 API docs. If all methods were on the `Bound` smart pointer directly, the vast majority of PyO3's API would be on a single, extremely long, documentation page. -- Consistency: downstream code implementing [Rust APIs for existing Python types](#creating-a-rust-api-for-an-existing-python-type) can also follow this pattern of using a trait. Downstream code would not be allowed to add new associated methods directly on the `Bound` type. -- Future design: it is hoped that a future Rust with [arbitrary self types](TODO) will remove the need for these traits in favour of placing the methods directly on `PyAny`, `PyTuple`, `PyDict`, and so on. +- Consistency: downstream code implementing Rust APIs for existing Python types can also follow this pattern of using a trait. Downstream code would not be allowed to add new associated methods directly on the `Bound` type. +- Future design: it is hoped that a future Rust with [arbitrary self types](https://github.com/rust-lang/rust/issues/44874) will remove the need for these traits in favour of placing the methods directly on `PyAny`, `PyTuple`, `PyDict`, and so on. These traits are all included in the `pyo3::prelude` module, so with the glob import `use pyo3::prelude::*` the full PyO3 API is made available to downstream code. @@ -228,11 +231,11 @@ Custom [`#[pyclass]`][pyclass] types implement [`PyTypeCheck`], so `.downcast()` use pyo3::prelude::*; #[pyclass] -struct MyClass { } +struct MyClass {} # fn example<'py>(py: Python<'py>) -> PyResult<()> { // create a new Python `tuple`, and use `.into_any()` to erase the type -let obj: Bound<'py, PyAny> = Bound::new(py, MyClass { })?.into_any(); +let obj: Bound<'py, PyAny> = Bound::new(py, MyClass {})?.into_any(); // use `.downcast()` to cast to `MyClass` without transferring ownership let _: &Bound<'py, MyClass> = obj.downcast()?; @@ -291,7 +294,7 @@ a list: # use pyo3::prelude::*; # use pyo3::types::PyList; # Python::with_gil(|py| -> PyResult<()> { -#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API. +#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API. let obj: &PyAny = PyList::empty(py); // To &PyList with PyAny::downcast @@ -312,11 +315,11 @@ For a `&PyAny` object reference `any` where the underlying object is a `#[pyclas # use pyo3::prelude::*; # #[pyclass] #[derive(Clone)] struct MyClass { } # Python::with_gil(|py| -> PyResult<()> { -#[allow(deprecated)] // into_ref is part of the deprecated GIL Refs API +#[allow(deprecated)] // into_ref is part of the deprecated GIL Refs API let obj: &PyAny = Py::new(py, MyClass {})?.into_ref(py); // To &PyCell with PyAny::downcast -#[allow(deprecated)] // &PyCell is part of the deprecated GIL Refs API +#[allow(deprecated)] // &PyCell is part of the deprecated GIL Refs API let _: &PyCell = obj.downcast()?; // To Py (aka PyObject) with .into() @@ -351,7 +354,7 @@ To see all Python types exposed by `PyO3` consult the [`pyo3::types`][pyo3::type # use pyo3::prelude::*; # use pyo3::types::PyList; # Python::with_gil(|py| -> PyResult<()> { -#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API. +#[allow(deprecated)] // PyList::empty is part of the deprecated "GIL Refs" API. let list = PyList::empty(py); // Use methods from PyAny on all Python types with Deref implementation @@ -361,7 +364,7 @@ let _ = list.repr()?; let _: &PyAny = list; // To &PyAny explicitly with .as_ref() -#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. +#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. let _: &PyAny = list.as_ref(); // To Py with .into() or Py::from() @@ -401,7 +404,7 @@ Like PyO3's Python native types, the GIL Ref `&PyCell` implements `Deref PyResult<()> { -#[allow(deprecated)] // &PyCell is part of the deprecated GIL Refs API +#[allow(deprecated)] // &PyCell is part of the deprecated GIL Refs API let cell: &PyCell = PyCell::new(py, MyClass {})?; // To PyRef with .borrow() or .try_borrow() @@ -422,7 +425,7 @@ let _: &mut MyClass = &mut *py_ref_mut; # use pyo3::prelude::*; # #[pyclass] struct MyClass { } # Python::with_gil(|py| -> PyResult<()> { -#[allow(deprecated)] // &PyCell is part of the deprecate GIL Refs API +#[allow(deprecated)] // &PyCell is part of the deprecate GIL Refs API let cell: &PyCell = PyCell::new(py, MyClass {})?; // Use methods from PyAny on PyCell with Deref implementation @@ -432,7 +435,7 @@ let _ = cell.repr()?; let _: &PyAny = cell; // To &PyAny explicitly with .as_ref() -#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. +#[allow(deprecated)] // as_ref is part of the deprecated "GIL Refs" API. let _: &PyAny = cell.as_ref(); # Ok(()) # }).unwrap(); @@ -451,7 +454,7 @@ let _: &PyAny = cell.as_ref(); [`PyTupleMethods`]: {{#PYO3_DOCS_URL}}/pyo3/types/trait.PyTupleMethods.html [pyclass]: class.md [Borrowed]: {{#PYO3_DOCS_URL}}/pyo3/struct.Borrowed.html -[Drop]: https://doc.rust-lang.org/std/drop/trait.Drop.html +[Drop]: https://doc.rust-lang.org/std/ops/trait.Drop.html [eval]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.eval [clone_ref]: {{#PYO3_DOCS_URL}}/pyo3/struct.Py.html#method.clone_ref [pyo3::types]: {{#PYO3_DOCS_URL}}/pyo3/types/index.html diff --git a/noxfile.py b/noxfile.py index 5f7ca8c04f8..f70fced76d6 100644 --- a/noxfile.py +++ b/noxfile.py @@ -372,7 +372,6 @@ def docs(session: nox.Session) -> None: "--no-deps", "--workspace", *cargo_flags, - *session.posargs, ) From 4b8d6cc80f6b09c21fafbdb18aaa65bf032c93ff Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 14:24:39 +0000 Subject: [PATCH 12/18] Update guide/src/performance.md Co-authored-by: Icxolu <10486322+Icxolu@users.noreply.github.com> --- guide/src/performance.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guide/src/performance.md b/guide/src/performance.md index 3db23ddf01e..c47a91deee5 100644 --- a/guide/src/performance.md +++ b/guide/src/performance.md @@ -89,7 +89,7 @@ use the more efficient # impl PartialEq for FooBound<'_> { fn eq(&self, other: &Foo) -> bool { - // Access to `&'a PyAny` implies access to `Python<'a>`. + // Access to `&Bound<'py, PyAny>` implies access to `Python<'py>`. let py = self.0.py(); let len = other.0.bind(py).len(); self.0.len() == len From 16925a1bea7dea61a835b66029c44853d0797009 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 14:24:50 +0000 Subject: [PATCH 13/18] Update guide/src/types.md Co-authored-by: Icxolu <10486322+Icxolu@users.noreply.github.com> --- guide/src/types.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guide/src/types.md b/guide/src/types.md index 2ba69f72669..6b586719cc3 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -44,7 +44,7 @@ To convert a `Py` into a `Bound<'py, T>`, the `Py::bind` and `Py::into_bound` By having the binding to the `'py` lifetime, `Bound<'py, T>` can offer the complete PyO3 API at maximum efficiency. This means that in almost all cases where `Py` is not necessary for lifetime reasons, `Bound<'py, T>` should be used. -`Bound<'py, T>` engages in Python reference counting. This means that `Bound<'py, T>` owns a Python object. Rust code which just wants to borrow a Python object should use a shared reference `&Bound<'py, T>`. Just like `std::sync::Arc`, using `.clone()` and `drop()` will cheaply implement and decrement the reference count of the object (just in this case, the reference counting is implemented by the Python interpreter itself). +`Bound<'py, T>` engages in Python reference counting. This means that `Bound<'py, T>` owns a Python object. Rust code which just wants to borrow a Python object should use a shared reference `&Bound<'py, T>`. Just like `std::sync::Arc`, using `.clone()` and `drop()` will cheaply increment and decrement the reference count of the object (just in this case, the reference counting is implemented by the Python interpreter itself). To give an example of how `Bound<'py, T>` is PyO3's primary API type, consider the following Python code: From df1ff750f93b96e51a94ebd7295b0981c06d6308 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 14:25:12 +0000 Subject: [PATCH 14/18] Update src/lib.rs --- src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib.rs b/src/lib.rs index 95c758569a8..a7c03d1b6cb 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -52,7 +52,7 @@ //! //! PyO3 has two core smart pointers to refer to Python objects, [`Py`](Py) and its GIL-bound //! form [`Bound<'py, T>`](Bound) which carries the `'py` lifetime. (There is also -//! [`Borrowed<'a, 'py, T>`](instance::Borrowed), but it is used much more rarely. +//! [`Borrowed<'a, 'py, T>`](instance::Borrowed), but it is used much more rarely). //! //! The type parameter `T` in these smart pointers can be filled by: //! - [`PyAny`], e.g. `Py` or `Bound<'py, PyAny>`, where the Python object type is not From 403dda4c268411affcf3bec488945ae02af55439 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 14:39:20 +0000 Subject: [PATCH 15/18] review: Icxolu --- guide/src/conversions/traits.md | 2 +- guide/src/memory.md | 47 ++++++++++++++++++++++----------- guide/src/trait-bounds.md | 3 ++- 3 files changed, 35 insertions(+), 17 deletions(-) diff --git a/guide/src/conversions/traits.md b/guide/src/conversions/traits.md index 3d1d5e21d85..3a00a160c65 100644 --- a/guide/src/conversions/traits.md +++ b/guide/src/conversions/traits.md @@ -484,7 +484,7 @@ If the input is neither a string nor an integer, the error message will be: - `pyo3(from_py_with = "...")` - apply a custom function to convert the field from Python the desired Rust type. - the argument must be the name of the function as a string. - - the function signature must be `fn(Bound) -> PyResult` where `T` is the Rust type of the argument. + - the function signature must be `fn(&Bound) -> PyResult` where `T` is the Rust type of the argument. ### `IntoPy` diff --git a/guide/src/memory.md b/guide/src/memory.md index f40c4c6fd25..fe98184e3e1 100644 --- a/guide/src/memory.md +++ b/guide/src/memory.md @@ -37,9 +37,10 @@ very simple and easy-to-understand programs like this: # use pyo3::types::PyString; # fn main() -> PyResult<()> { Python::with_gil(|py| -> PyResult<()> { + #[allow(deprecated)] // py.eval() is part of the GIL Refs API let hello = py - .eval_bound("\"Hello World!\"", None, None)? - .downcast_into::()?; + .eval("\"Hello World!\"", None, None)? + .downcast::()?; println!("Python says: {}", hello); Ok(()) })?; @@ -60,9 +61,10 @@ of the time we don't have to think about this, but consider the following: # fn main() -> PyResult<()> { Python::with_gil(|py| -> PyResult<()> { for _ in 0..10 { + #[allow(deprecated)] // py.eval() is part of the GIL Refs API let hello = py - .eval_bound("\"Hello World!\"", None, None)? - .downcast_into::()?; + .eval("\"Hello World!\"", None, None)? + .downcast::()?; println!("Python says: {}", hello); } // There are 10 copies of `hello` on Python's heap here. @@ -90,9 +92,10 @@ is to acquire and release the GIL with each iteration of the loop. # fn main() -> PyResult<()> { for _ in 0..10 { Python::with_gil(|py| -> PyResult<()> { + #[allow(deprecated)] // py.eval() is part of the GIL Refs API let hello = py - .eval_bound("\"Hello World!\"", None, None)? - .downcast_into::()?; + .eval("\"Hello World!\"", None, None)? + .downcast::()?; println!("Python says: {}", hello); Ok(()) })?; // only one copy of `hello` at a time @@ -113,9 +116,10 @@ Python::with_gil(|py| -> PyResult<()> { for _ in 0..10 { let pool = unsafe { py.new_pool() }; let py = pool.python(); + #[allow(deprecated)] // py.eval() is part of the GIL Refs API let hello = py - .eval_bound("\"Hello World!\"", None, None)? - .downcast_into::()?; + .eval("\"Hello World!\"", None, None)? + .downcast::()?; println!("Python says: {}", hello); } Ok(()) @@ -162,8 +166,12 @@ reference count reaches zero? It depends whether or not we are holding the GIL. # use pyo3::types::PyString; # fn main() -> PyResult<()> { Python::with_gil(|py| -> PyResult<()> { - let hello: Py = py.eval_bound("\"Hello World!\"", None, None)?.extract()?; - println!("Python says: {}", hello.bind(py)); + #[allow(deprecated)] // py.eval() is part of the GIL Refs API + let hello: Py = py.eval("\"Hello World!\"", None, None)?.extract()?; + #[allow(deprecated)] // as_ref is part of the GIL Refs API + { + println!("Python says: {}", hello.as_ref(py)); + } Ok(()) })?; # Ok(()) @@ -184,7 +192,8 @@ we are *not* holding the GIL? # use pyo3::types::PyString; # fn main() -> PyResult<()> { let hello: Py = Python::with_gil(|py| { - py.eval_bound("\"Hello World!\"", None, None)?.extract() + #[allow(deprecated)] // py.eval() is part of the GIL Refs API + py.eval("\"Hello World!\"", None, None)?.extract() })?; // Do some stuff... // Now sometime later in the program we want to access `hello`. @@ -216,12 +225,16 @@ We can avoid the delay in releasing memory if we are careful to drop the # use pyo3::prelude::*; # use pyo3::types::PyString; # fn main() -> PyResult<()> { +#[allow(deprecated)] // py.eval() is part of the GIL Refs API let hello: Py = - Python::with_gil(|py| py.eval_bound("\"Hello World!\"", None, None)?.extract())?; + Python::with_gil(|py| py.eval("\"Hello World!\"", None, None)?.extract())?; // Do some stuff... // Now sometime later in the program: Python::with_gil(|py| { - println!("Python says: {}", hello.bind(py)); + #[allow(deprecated)] // as_ref is part of the GIL Refs API + { + println!("Python says: {}", hello.as_ref(py)); + } drop(hello); // Memory released here. }); # Ok(()) @@ -238,12 +251,16 @@ until the GIL is dropped. # use pyo3::prelude::*; # use pyo3::types::PyString; # fn main() -> PyResult<()> { +#[allow(deprecated)] // py.eval() is part of the GIL Refs API let hello: Py = - Python::with_gil(|py| py.eval_bound("\"Hello World!\"", None, None)?.extract())?; + Python::with_gil(|py| py.eval("\"Hello World!\"", None, None)?.extract())?; // Do some stuff... // Now sometime later in the program: Python::with_gil(|py| { - println!("Python says: {}", hello.into_bound(py)); + #[allow(deprecated)] // into_ref is part of the GIL Refs API + { + println!("Python says: {}", hello.into_ref(py)); + } // Memory not released yet. // Do more stuff... // Memory released here at end of `with_gil()` closure. diff --git a/guide/src/trait-bounds.md b/guide/src/trait-bounds.md index 65842dee71f..0644e679190 100644 --- a/guide/src/trait-bounds.md +++ b/guide/src/trait-bounds.md @@ -82,7 +82,8 @@ impl Model for UserModel { fn set_variables(&mut self, var: &Vec) { println!("Rust calling Python to set the variables"); Python::with_gil(|py| { - self.model.bind(py) + self.model + .bind(py) .call_method("set_variables", (PyList::new_bound(py, var),), None) .unwrap(); }) From b0159826edfd526a8990022937f4c7ae25fe56d6 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 14:53:58 +0000 Subject: [PATCH 16/18] Update guide/src/python-from-rust.md Co-authored-by: Adam Reichold --- guide/src/python-from-rust.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guide/src/python-from-rust.md b/guide/src/python-from-rust.md index 6bc15e2e82c..085b3b03280 100644 --- a/guide/src/python-from-rust.md +++ b/guide/src/python-from-rust.md @@ -34,7 +34,7 @@ Non-Python operations (system calls and native Rust code) can unlock the GIL. Se ## Python's memory model Python's memory model differs from Rust's memory model in two key ways: -- There is no concept of ownership; all Python objects are reference counted +- There is no concept of ownership; all Python objects are shared and usually implemented via reference counting - There is no concept of exclusive (`&mut`) references; any reference can mutate a Python object PyO3's API reflects this by providing smart pointer types, `Py`, `Bound<'py, T>`, and (the very rarely used) `Borrowed<'a, 'py, T>`. These smart pointers all use Python reference counting. See the [subchapter on types](./types.md) for more detail on these types. From 5a41b61901ad143370ba386060b20e5457f384c2 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 15:16:07 +0000 Subject: [PATCH 17/18] Update guide/src/async-await.md Co-authored-by: Adam Reichold --- guide/src/async-await.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guide/src/async-await.md b/guide/src/async-await.md index 0db1720340d..06fa1580ad7 100644 --- a/guide/src/async-await.md +++ b/guide/src/async-await.md @@ -32,7 +32,7 @@ Resulting future of an `async fn` decorated by `#[pyfunction]` must be `Send + ' As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a signature like `async fn does_not_compile<'py>(arg: Bound<'py, PyAny>) -> Bound<'py, PyAny>`. -However, there is an exception for method receivers, so async methods can accept `&self`/`&mut self`. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows `&self` to exclusive ones `&mut self` to avoid racy borrow check failures at runtime. +However, there is an exception for method receivers, so async methods can accept `&self`/`&mut self`. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows `&self` over exclusive ones `&mut self` to avoid racy borrow check failures at runtime. ## Implicit GIL holding From 676524cac23c6fb042822753cf2904843bb8bca2 Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Sun, 10 Mar 2024 15:16:26 +0000 Subject: [PATCH 18/18] review: adamreichold --- guide/src/python-from-rust.md | 3 ++- guide/src/types.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/guide/src/python-from-rust.md b/guide/src/python-from-rust.md index 085b3b03280..ee618f3fa47 100644 --- a/guide/src/python-from-rust.md +++ b/guide/src/python-from-rust.md @@ -37,9 +37,10 @@ Python's memory model differs from Rust's memory model in two key ways: - There is no concept of ownership; all Python objects are shared and usually implemented via reference counting - There is no concept of exclusive (`&mut`) references; any reference can mutate a Python object -PyO3's API reflects this by providing smart pointer types, `Py`, `Bound<'py, T>`, and (the very rarely used) `Borrowed<'a, 'py, T>`. These smart pointers all use Python reference counting. See the [subchapter on types](./types.md) for more detail on these types. +PyO3's API reflects this by providing [smart pointer][smart-pointers] types, `Py`, `Bound<'py, T>`, and (the very rarely used) `Borrowed<'a, 'py, T>`. These smart pointers all use Python reference counting. See the [subchapter on types](./types.md) for more detail on these types. Because of the lack of exclusive `&mut` references, PyO3's APIs for Python objects, for example [`PyListMethods::append`], use shared references. This is safe because Python objects have internal mechanisms to prevent data races (as of time of writing, the Python GIL). +[smart-pointers]: https://doc.rust-lang.org/book/ch15-00-smart-pointers.html [obtaining-py]: {{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#obtaining-a-python-token [`pyo3::sync`]: {{#PYO3_DOCS_URL}}/pyo3/sync/index.html diff --git a/guide/src/types.md b/guide/src/types.md index 6b586719cc3..0f1fa3d0af1 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -2,7 +2,7 @@ PyO3 offers two main sets of types to interact with Python objects. This section of the guide expands into detail about these types and how to choose which to use. -The first set of types is are the "smart pointers" which all Python objects are wrapped in. These are `Py`, `Bound<'py, T>`, and `Borrowed<'a, 'py, T>`. The [first section below](#pyo3s-smart-pointers) expands on each of these in detail and why there are three of them. +The first set of types is are the [smart pointers][smart-pointers] which all Python objects are wrapped in. These are `Py`, `Bound<'py, T>`, and `Borrowed<'a, 'py, T>`. The [first section below](#pyo3s-smart-pointers) expands on each of these in detail and why there are three of them. The second set of types are types which fill in the generic parameter `T` of the smart pointers. The most common is `PyAny`, which represents any Python object (similar to Python's `typing.Any`). There are also concrete types for many Python built-in types, such as `PyList`, `PyDict`, and `PyTuple`. User defined `#[pyclass]` types also fit this category. The [second section below](#concrete-python-types) expands on how to use these types. @@ -461,3 +461,4 @@ let _: &PyAny = cell.as_ref(); [PyAny]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyAny.html [PyList_append]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyList.html#method.append [RefCell]: https://doc.rust-lang.org/std/cell/struct.RefCell.html +[smart-pointers]: https://doc.rust-lang.org/book/ch15-00-smart-pointers.html