RLG fork of pybind11 incompatible with Python 3.13

[jwnimmer-tri]

Error message from a CPython 3.13 wheel build:

pybind11/pytypes.h: In member function 'void pybind11::wrapper<Base>::resurrect_python3()':
pybind11/pytypes.h:2526:13: error: there are no arguments to '_PyGC_FINALIZED' that depend on a template parameter, so a declaration of '_PyGC_FINALIZED' must be available [-fpermissive]
2526 |         if (_PyGC_FINALIZED(patient_.ptr())) {
     |             ^~~~~~~~~~~~~~~

See https://bugs.python.org/issue40240 and https://bugs.python.org/issue32377. The private macros _PyGC_FINALIZED and _PyGC_SET_FINALIZED are no longer available. The former has a replacement function available, but the latter does not. Those two macros are used by the bespoke RLG patch in our custom class wrapper:

https://github.com/RobotLocomotion/pybind11/blob/51d715e037386fcdbeda75ffab15f02f8e4388d8/include/pybind11/pytypes.h#L2455-L2461

[rpoyner-tri]

Three pieces of interest in the fork:

• unique_ptr args
• Wrapper
• eigen/numpy dtype magic for autodiff py objects

[rpoyner-tri]

Above list is priority order for porting away. Jeremy has a pybind11 branch with partial reverts of the custom features.

[jwnimmer-tri]

Here's my WIP branch to explore the problem: https://github.com/jwnimmer-tri/drake/commits/pybind11-tmp/

• It switches pybind11 to my fork which is exactly an upstream release except for our eigen/matrix.h patch for dtypes.
• It cherry-picks some Drake PRs in flight.
• It puts #if 0 around the remaining bindings that don't compile.

As such, everything builds but some tests fail (due to the missing bindings).

The inventory if #if 0 locations forms the worklist of problems that need solving.

Probably the next problem worth tackling is getting the Simulator working, since that halts a bunch of other tests so hides any deeper problems.

I have some small fixes for a couple of the other #if 0 problems I'll also try to resurrect soon (e.g., LCM serializer bindings IIRC). Edit: Pushed to the branch now.

[jwnimmer-tri]

See also #12811 for simulator-context discussion.

[jwnimmer-tri]

FYI I've rebased my -tmp branch to cherry-pick all of the recent PRs (both open and merged PRs).

I'm going to start looking at the ValueProducer::AllocateCallback problems next (framework_py_semantics, framework_py_systems).

=> #22470

[jwnimmer-tri]

Next I'm going to tackle the system scalar converter.
=> #22491

[jwnimmer-tri]

I've rebased my pybind11-tmp branch atop everything in flight. What still remains as #if 0 are:

• AddMultibodyPlantSceneGraph
• CollisionChecker
• RandomSimulation / MonteCarloSimulation

[jwnimmer-tri]

The monte carlo looks to easy, so I'll take that one (#22528) and leave the other two for you.

[jwnimmer-tri]

Also the final capstone -- we need to summarize the status quo / playbook for Drake Developers who maintain the bindings on an ongoing basis, probably at https://github.com/RobotLocomotion/drake/blob/master/bindings/pydrake/pydrake_doxygen.h.

[rpoyner-tri]

#22595 is in review.
Need to patch CollisionChecker, and that should be the end of unique pointer patching.

[jwnimmer-tri]

@rpoyner-tri I rebased my https://github.com/jwnimmer-tri/drake/commits/pybind11-tmp/ WIP branch. I see only the 7 test failures.

[rpoyner-tri]

#22668 is merged, which takes care of one of the resistant failures.

I believe that the remaining problems are variations of pybind/pybind11#1333 . It's not clear that there is a clean minimal patch available.