fix: allow the ABI string to be forced

[henryiii]

Description

In version 2.4, commit c9f5a added extra protections to the ABI string, which affect some power users (see commits there) as well as PyTorch. This is a small patch that simply allows the various components to be pre-defined by users who know what they are doing (or a user on an odd compiler could set the string for the new compiler!). This is not a "fix", per se, which would take carful study to check ABI compatibility, but would allow users stuck on 2.3 to upgrade or work in special cases.

To restore 2.3 like behavior:

#define PYBIND11_COMPILER_TYPE ""
#define PYBIND11_STDLIB ""
#define PYBIND11_BUILD_ABI ""

To force a specfic compiler:

#define PYBIND11_COMPILER_TYPE "_gcc"

Suggested changelog entry:

Should it be only for power users, or have a small changelog entry, and maybe even be listed in the docs?

[rwgk]

Very minor & optional: I'd undo the whitespace change here, to reduce the potential for confusion.

[henryiii]

There was a mix of 2-space and 3-space indentation in these, I normalized to two space. You can use -w when calling git blame to ignore whitespace, and there's a temporary ignore whitespace setting for the git diffs.

[henryiii]

https://github.com/pybind/pybind11/pull/2602/files?diff=unified&w=1

[henryiii]

(I can still undo if you think it should be undone - obviously I didn't search out and fix all uneven whitespace ;) )

[rwgk]

I'm with the PR fine as-is. Thanks!

[wjakob]

(Already merged, but for the record this look all good to me.)

[YannickJadoul]

What's the immediate motivation behind this, actually? Is there any way 2.3 can work with 2.6?

This does feel like it could cause weird segfaults that we'll have to debug for users? :-(

EDIT: Sorry, I have found the thread on commit c9f5a46, now: c9f5a46#commitcomment-35812965

[henryiii]

The immediate need is for PyTorch. They have been stuck trying to upgrade to 2.4 since the beginning of the year in order to fix bugs, but haven't been able to. The problem is that they compile a bunch of modules, then at run time they also have a JIT that compiles and needs to use the existing module. If the JIT compiler ID doesn't exactly match, 2.4 and later fail due to this addition. They have been happily using it without this for now, and this would allow the upgrade to move forward separately of this change. We could not even debug the issue without the ability to force this.

Also other users who were using pybind11 to mix clang and gcc could continue to do so now without being stuck on an old version.

we'll have to debug for users

So if a user is intentionally forcing the ABI string with these defs, then coming to us and complaining that the ABI is not actually compatible?

And, it would be nice to be able to eventually either support a compiler-independent ABI (might already at least support gcc vs. clang), and have only truly incompatible modules separated by different ABI strings. To test that, we have to be able to force the ABI strings to match.

See pytorch/pytorch#46415 and pytorch/pytorch#46367

[YannickJadoul]

Right. Thanks, @henryiii!

So if a user is intentionally forcing the ABI string with these defs, then coming to us and complaining that the ABI is not actually compatible?

Well, the danger is not knowing it when the user forgets to report it. Guess we'll just have to be extra careful?