update remainder of the guide to Bound API

guide/src/advanced.md

@@ -5,10 +5,3 @@
 PyO3 exposes much of Python's C API through the `ffi` module.
 
 The C API is naturally unsafe and requires you to manage reference counts, errors and specific invariants yourself. Please refer to the [C API Reference Manual](https://docs.python.org/3/c-api/) and [The Rustonomicon](https://doc.rust-lang.org/nightly/nomicon/ffi.html) before using any function from that API.
-
-## Memory management
-
-PyO3's `&PyAny` "owned references" and `Py<PyAny>` smart pointers are used to
-access memory stored in Python's heap.  This memory sometimes lives for longer
-than expected because of differences in Rust and Python's memory models.  See
-the chapter on [memory management](./memory.md) for more information.

guide/src/async-await.md

@@ -30,13 +30,13 @@ async fn sleep(seconds: f64, result: Option<PyObject>) -> Option<PyObject> {
 
 Resulting future of an `async fn` decorated by `#[pyfunction]` must be `Send + 'static` to be embedded in a Python object.
 
-As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a signature like `async fn does_not_compile(arg: &PyAny, py: Python<'_>) -> &PyAny`.
+As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a signature like `async fn does_not_compile<'py>(arg: Bound<'py, PyAny>) -> Bound<'py, PyAny>`.
 
-However, there is an exception for method receiver, so async methods can accept `&self`/`&mut self`. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows `&self` to exclusive ones `&mut self` to avoid racy borrow check failures at runtime.
+However, there is an exception for method receivers, so async methods can accept `&self`/`&mut self`. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows `&self` to exclusive ones `&mut self` to avoid racy borrow check failures at runtime.
 
 ## Implicit GIL holding
 
-Even if it is not possible to pass a `py: Python<'_>` parameter to `async fn`, the GIL is still held during the execution of the future – it's also the case for regular `fn` without `Python<'_>`/`&PyAny` parameter, yet the GIL is held.
+Even if it is not possible to pass a `py: Python<'py>` parameter to `async fn`, the GIL is still held during the execution of the future – it's also the case for regular `fn` without `Python<'py>`/`Bound<'py, PyAny>` parameter, yet the GIL is held.
 
 It is still possible to get a `Python` marker using [`Python::with_gil`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.with_gil); because `with_gil` is reentrant and optimized, the cost will be negligible.
 
@@ -47,7 +47,11 @@ There is currently no simple way to release the GIL when awaiting a future, *but
 Here is the advised workaround for now:
 
 ```rust,ignore
-use std::{future::Future, pin::{Pin, pin}, task::{Context, Poll}};
+use std::{
+    future::Future,
+    pin::{Pin, pin},
+    task::{Context, Poll},
+};
 use pyo3::prelude::*;
 
 struct AllowThreads<F>(F);

guide/src/class.md

@@ -60,7 +60,7 @@ enum Shape {
     Circle { radius: f64 },
     Rectangle { width: f64, height: f64 },
     RegularPolygon { side_count: u32, radius: f64 },
-    Nothing { },
+    Nothing {},
 }
 ```
 
@@ -89,7 +89,7 @@ Currently, the best alternative is to write a macro which expands to a new `#[py
 use pyo3::prelude::*;
 
 struct GenericClass<T> {
-   data: T
+    data: T,
 }
 
 macro_rules! create_interface {
@@ -102,7 +102,9 @@ macro_rules! create_interface {
         impl $name {
             #[new]
             pub fn new(data: $type) -> Self {
-                Self { inner: GenericClass { data: data } }
+                Self {
+                    inner: GenericClass { data: data },
+                }
             }
         }
     };
@@ -427,7 +429,7 @@ impl DictWithCounter {
         Self::default()
     }
 
-    fn set(slf: &Bound<'_, Self>, key: String, value: &PyAny) -> PyResult<()> {
+    fn set(slf: &Bound<'_, Self>, key: String, value: Bound<'_, PyAny>) -> PyResult<()> {
         slf.borrow_mut().counter.entry(key.clone()).or_insert(0);
         let dict = slf.downcast::<PyDict>()?;
         dict.set_item(key, value)

guide/src/conversions/tables.md

@@ -72,9 +72,9 @@ For most PyO3 usage the conversion cost is worth paying to get these benefits. A
 
 ### Returning Rust values to Python
 
-When returning values from functions callable from Python, Python-native types (`&PyAny`, `&PyDict` etc.) can be used with zero cost.
+When returning values from functions callable from Python, [PyO3's smart pointers](../types.md#pyo3s-smart-pointers) (`Py<T>`, `Bound<'py, T>`, and `Borrowed<'a, 'py, T>`) can be used with zero cost.
 
-Because these types are references, in some situations the Rust compiler may ask for lifetime annotations. If this is the case, you should use `Py<PyAny>`, `Py<PyDict>` etc. instead - which are also zero-cost. For all of these Python-native types `T`, `Py<T>` can be created from `T` with an `.into()` conversion.
+Because `Bound<'py, T>` and `Borrowed<'a, 'py, T>` have lifetime parameters, the Rust compiler may ask for lifetime annotations to be added to your function. See the [section of the guide dedicated to this](../types.md#function-argument-lifetimes).
 
 If your function is fallible, it should return `PyResult<T>` or `Result<T, E>` where `E` implements `From<E> for PyErr`. This will raise a `Python` exception if the `Err` variant is returned.
 

guide/src/conversions/traits.md

@@ -484,7 +484,7 @@ If the input is neither a string nor an integer, the error message will be:
 - `pyo3(from_py_with = "...")`
     - apply a custom function to convert the field from Python the desired Rust type.
     - the argument must be the name of the function as a string.
-    - the function signature must be `fn(&PyAny) -> PyResult<T>` where `T` is the Rust type of the argument.
+    - the function signature must be `fn(Bound<PyAny>) -> PyResult<T>` where `T` is the Rust type of the argument.
 
 ### `IntoPy<T>`
 

guide/src/ecosystem/async-await.md

@@ -120,7 +120,7 @@ Export an async function that makes use of `async-std`:
 use pyo3::{prelude::*, wrap_pyfunction};
 
 #[pyfunction]
-fn rust_sleep(py: Python<'_>) -> PyResult<&PyAny> {
+fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
     pyo3_asyncio::async_std::future_into_py(py, async {
         async_std::task::sleep(std::time::Duration::from_secs(1)).await;
         Ok(Python::with_gil(|py| py.None()))
@@ -143,7 +143,7 @@ If you want to use `tokio` instead, here's what your module should look like:
 use pyo3::{prelude::*, wrap_pyfunction};
 
 #[pyfunction]
-fn rust_sleep(py: Python<'_>) -> PyResult<&PyAny> {
+fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
     pyo3_asyncio::tokio::future_into_py(py, async {
         tokio::time::sleep(std::time::Duration::from_secs(1)).await;
         Ok(Python::with_gil(|py| py.None()))
@@ -233,7 +233,7 @@ a coroutine argument:
 
 ```rust
 #[pyfunction]
-fn await_coro(coro: &PyAny) -> PyResult<()> {
+fn await_coro(coro: &Bound<'_, PyAny>>) -> PyResult<()> {
     // convert the coroutine into a Rust future using the
     // async_std runtime
     let f = pyo3_asyncio::async_std::into_future(coro)?;
@@ -261,7 +261,7 @@ If for you wanted to pass a callable function to the `#[pyfunction]` instead, (i
 
 ```rust
 #[pyfunction]
-fn await_coro(callable: &PyAny) -> PyResult<()> {
+fn await_coro(callable: &Bound<'_, PyAny>>) -> PyResult<()> {
     // get the coroutine by calling the callable
     let coro = callable.call0()?;
 
@@ -317,7 +317,7 @@ async fn rust_sleep() {
 }
 
 #[pyfunction]
-fn call_rust_sleep(py: Python<'_>) -> PyResult<&PyAny> {
+fn call_rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
     pyo3_asyncio::async_std::future_into_py(py, async move {
         rust_sleep().await;
         Ok(Python::with_gil(|py| py.None()))
@@ -467,7 +467,7 @@ tokio = "1.4"
 use pyo3::{prelude::*, wrap_pyfunction};
 
 #[pyfunction]
-fn rust_sleep(py: Python<'_>) -> PyResult<&PyAny> {
+fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
     pyo3_asyncio::tokio::future_into_py(py, async {
         tokio::time::sleep(std::time::Duration::from_secs(1)).await;
         Ok(Python::with_gil(|py| py.None()))

guide/src/exception.md

@@ -111,7 +111,7 @@ mod io {
     pyo3::import_exception!(io, UnsupportedOperation);
 }
 
-fn tell(file: &PyAny) -> PyResult<u64> {
+fn tell(file: &Bound<'_, PyAny>) -> PyResult<u64> {
     match file.call_method0("tell") {
         Err(_) => Err(io::UnsupportedOperation::new_err("not supported: tell")),
         Ok(x) => x.extract::<u64>(),

guide/src/function.md

@@ -87,7 +87,9 @@ The `#[pyo3]` attribute can be used to modify properties of the generated Python
 
     #[pyfunction]
     #[pyo3(pass_module)]
-    fn pyfunction_with_module<'py>(module: &Bound<'py, PyModule>) -> PyResult<Bound<'py, PyString>> {
+    fn pyfunction_with_module<'py>(
+        module: &Bound<'py, PyModule>,
+    ) -> PyResult<Bound<'py, PyString>> {
         module.name()
     }
 

guide/src/memory.md

@@ -1,5 +1,15 @@
 # Memory management
 
+<div class="warning">
+⚠️ Warning: API update in progress 🛠️
+
+PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer `Bound<T>`.
+
+This section on memory management is heavily weighted towards the now-deprecated "GIL Refs" API, which suffered from the drawbacks detailed here as well as CPU overheads.
+
+See [the smart pointer types](./types.md#pyo3s-smart-pointers) for description on the new, simplified, memory model of the Bound API, which is built as a thin wrapper on Python reference counting.
+</div>
+
 Rust and Python have very different notions of memory management.  Rust has
 a strict memory model with concepts of ownership, borrowing, and lifetimes,
 where memory is freed at predictable points in program execution.  Python has
@@ -10,12 +20,12 @@ Memory in Python is freed eventually by the garbage collector, but not usually
 in a predictable way.
 
 PyO3 bridges the Rust and Python memory models with two different strategies for
-accessing memory allocated on Python's heap from inside Rust.  These are
-GIL-bound, or "owned" references, and GIL-independent `Py<Any>` smart pointers.
+accessing memory allocated on Python's heap from inside Rust. These are
+GIL Refs such as `&'py PyAny`, and GIL-independent `Py<Any>` smart pointers.
 
 ## GIL-bound memory
 
-PyO3's GIL-bound, "owned references" (`&PyAny` etc.) make PyO3 more ergonomic to
+PyO3's GIL Refs such as `&'py PyAny` make PyO3 more ergonomic to
 use by ensuring that their lifetime can never be longer than the duration the
 Python GIL is held.  This means that most of PyO3's API can assume the GIL is
 held. (If PyO3 could not assume this, every PyO3 API would need to take a
@@ -27,7 +37,9 @@ very simple and easy-to-understand programs like this:
 # use pyo3::types::PyString;
 # fn main() -> PyResult<()> {
 Python::with_gil(|py| -> PyResult<()> {
-    let hello = py.eval_bound("\"Hello World!\"", None, None)?.downcast_into::<PyString>()?;
+    let hello = py
+        .eval_bound("\"Hello World!\"", None, None)?
+        .downcast_into::<PyString>()?;
     println!("Python says: {}", hello);
     Ok(())
 })?;
@@ -48,7 +60,9 @@ of the time we don't have to think about this, but consider the following:
 # fn main() -> PyResult<()> {
 Python::with_gil(|py| -> PyResult<()> {
     for _ in 0..10 {
-        let hello = py.eval_bound("\"Hello World!\"", None, None)?.downcast_into::<PyString>()?;
+        let hello = py
+            .eval_bound("\"Hello World!\"", None, None)?
+            .downcast_into::<PyString>()?;
         println!("Python says: {}", hello);
     }
     // There are 10 copies of `hello` on Python's heap here.
@@ -76,7 +90,9 @@ is to acquire and release the GIL with each iteration of the loop.
 # fn main() -> PyResult<()> {
 for _ in 0..10 {
     Python::with_gil(|py| -> PyResult<()> {
-        let hello = py.eval_bound("\"Hello World!\"", None, None)?.downcast_into::<PyString>()?;
+        let hello = py
+            .eval_bound("\"Hello World!\"", None, None)?
+            .downcast_into::<PyString>()?;
         println!("Python says: {}", hello);
         Ok(())
     })?; // only one copy of `hello` at a time
@@ -97,7 +113,9 @@ Python::with_gil(|py| -> PyResult<()> {
     for _ in 0..10 {
         let pool = unsafe { py.new_pool() };
         let py = pool.python();
-        let hello = py.eval_bound("\"Hello World!\"", None, None)?.downcast_into::<PyString>()?;
+        let hello = py
+            .eval_bound("\"Hello World!\"", None, None)?
+            .downcast_into::<PyString>()?;
         println!("Python says: {}", hello);
     }
     Ok(())

guide/src/migration.md

@@ -190,7 +190,9 @@ struct PyClassAsyncIter {
 impl PyClassAsyncIter {
     fn __anext__(&mut self) -> PyClassAwaitable {
         self.number += 1;
-        PyClassAwaitable { number: self.number }
+        PyClassAwaitable {
+            number: self.number,
+        }
     }
 
     fn __aiter__(slf: Py<Self>) -> Py<Self> {
@@ -312,7 +314,10 @@ Python::with_gil(|py| {
     // `b` is not in the dictionary
     assert!(dict.get_item("b").is_none());
     // `dict` is not hashable, so this fails with a `TypeError`
-    assert!(dict.get_item_with_error(dict).unwrap_err().is_instance_of::<PyTypeError>(py));
+    assert!(dict
+        .get_item_with_error(dict)
+        .unwrap_err()
+        .is_instance_of::<PyTypeError>(py));
 });
 # }
 ```
@@ -333,7 +338,10 @@ Python::with_gil(|py| -> PyResult<()> {
     // `b` is not in the dictionary
     assert!(dict.get_item("b")?.is_none());
     // `dict` is not hashable, so this fails with a `TypeError`
-    assert!(dict.get_item(dict).unwrap_err().is_instance_of::<PyTypeError>(py));
+    assert!(dict
+        .get_item(dict)
+        .unwrap_err()
+        .is_instance_of::<PyTypeError>(py));
 
     Ok(())
 });

guide/src/performance.md

@@ -4,26 +4,26 @@ To achieve the best possible performance, it is useful to be aware of several tr
 
 ## `extract` versus `downcast`
 
-Pythonic API implemented using PyO3 are often polymorphic, i.e. they will accept `&PyAny` and try to turn this into multiple more concrete types to which the requested operation is applied. This often leads to chains of calls to `extract`, e.g.
+Pythonic API implemented using PyO3 are often polymorphic, i.e. they will accept `&Bound<'_, PyAny>` and try to turn this into multiple more concrete types to which the requested operation is applied. This often leads to chains of calls to `extract`, e.g.
 
 ```rust
 # #![allow(dead_code)]
 # use pyo3::prelude::*;
 # use pyo3::{exceptions::PyTypeError, types::PyList};
 
-fn frobnicate_list(list: &PyList) -> PyResult<&PyAny> {
+fn frobnicate_list<'py>(list: &Bound<'_, PyList>) -> PyResult<Bound<'py, PyAny>> {
     todo!()
 }
 
-fn frobnicate_vec(vec: Vec<&PyAny>) -> PyResult<&PyAny> {
+fn frobnicate_vec<'py>(vec: Vec<Bound<'py, PyAny>>) -> PyResult<Bound<'py, PyAny>> {
     todo!()
 }
 
 #[pyfunction]
-fn frobnicate(value: &PyAny) -> PyResult<&PyAny> {
-    if let Ok(list) = value.extract::<&PyList>() {
+fn frobnicate<'py>(value: &Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
+    if let Ok(list) = value.extract::<Bound<'_, PyList>>() {
         frobnicate_list(list)
-    } else if let Ok(vec) = value.extract::<Vec<&PyAny>>() {
+    } else if let Ok(vec) = value.extract::<Vec<Bound<'_, PyAny>>>() {
         frobnicate_vec(vec)
     } else {
         Err(PyTypeError::new_err("Cannot frobnicate that type."))
@@ -37,25 +37,25 @@ This suboptimal as the `FromPyObject<T>` trait requires `extract` to have a `Res
 # #![allow(dead_code)]
 # use pyo3::prelude::*;
 # use pyo3::{exceptions::PyTypeError, types::PyList};
-# fn frobnicate_list(list: &PyList) -> PyResult<&PyAny> { todo!() }
-# fn frobnicate_vec(vec: Vec<&PyAny>) -> PyResult<&PyAny> { todo!() }
+# fn frobnicate_list<'py>(list: &Bound<'_, PyList>) -> PyResult<Bound<'py, PyAny>> { todo!() }
+# fn frobnicate_vec<'py>(vec: Vec<Bound<'py, PyAny>>) -> PyResult<Bound<'py, PyAny>> { todo!() }
 #
 #[pyfunction]
-fn frobnicate(value: &PyAny) -> PyResult<&PyAny> {
+fn frobnicate<'py>(value: &Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
     // Use `downcast` instead of `extract` as turning `PyDowncastError` into `PyErr` is quite costly.
     if let Ok(list) = value.downcast::<PyList>() {
         frobnicate_list(list)
-    } else if let Ok(vec) = value.extract::<Vec<&PyAny>>() {
+    } else if let Ok(vec) = value.extract::<Vec<Bound<'_, PyAny>>>() {
         frobnicate_vec(vec)
     } else {
         Err(PyTypeError::new_err("Cannot frobnicate that type."))
     }
 }
 ```
 
-## Access to GIL-bound reference implies access to GIL token
+## Access to Bound implies access to GIL token
 
-Calling `Python::with_gil` is effectively a no-op when the GIL is already held, but checking that this is the case still has a cost. If an existing GIL token can not be accessed, for example when implementing a pre-existing trait, but a GIL-bound reference is available, this cost can be avoided by exploiting that access to GIL-bound reference gives zero-cost access to a GIL token via `PyAny::py`.
+Calling `Python::with_gil` is effectively a no-op when the GIL is already held, but checking that this is the case still has a cost. If an existing GIL token can not be accessed, for example when implementing a pre-existing trait, but a GIL-bound reference is available, this cost can be avoided by exploiting that access to GIL-bound reference gives zero-cost access to a GIL token via `Bound::py`.
 
 For example, instead of writing
 
@@ -66,34 +66,32 @@ For example, instead of writing
 
 struct Foo(Py<PyList>);
 
-struct FooRef<'a>(&'a PyList);
+struct FooBound<'py>(Bound<'py, PyList>);
 
-impl PartialEq<Foo> for FooRef<'_> {
+impl PartialEq<Foo> for FooBound<'py> {
     fn eq(&self, other: &Foo) -> bool {
         Python::with_gil(|py| {
-            #[allow(deprecated)]  // as_ref is part of the deprecated "GIL Refs" API.
-            let len = other.0.as_ref(py).len();
+            let len = other.0.bind(py).len();
             self.0.len() == len
         })
     }
 }
 ```
 
-use more efficient
+use the more efficient
 
 ```rust
 # #![allow(dead_code)]
 # use pyo3::prelude::*;
 # use pyo3::types::PyList;
 # struct Foo(Py<PyList>);
-# struct FooRef<'a>(&'a PyList);
+# struct FooBound<'py>(Bound<'py, PyList>);
 #
-impl PartialEq<Foo> for FooRef<'_> {
+impl PartialEq<Foo> for FooBound<'_> {
     fn eq(&self, other: &Foo) -> bool {
         // Access to `&'a PyAny` implies access to `Python<'a>`.
         let py = self.0.py();
-        #[allow(deprecated)]  // as_ref is part of the deprecated "GIL Refs" API.
-        let len = other.0.as_ref(py).len();
+        let len = other.0.bind(py).len();
         self.0.len() == len
     }
 }