Arbitrary self types v2: stabilize

[adetaylor]

This PR stabilizes the arbitrary self types v2 feature, tracked in #44874.

r? @wesleywiser

Stabilization report

I'd like to stabilize Arbitrary Self Types in some upcoming Rust version. A stabilization PR is here. What follows is the list of questions from Niko's new template plus various sections I've seen in other stabilization reports.

Summary

This feature allows custom smart pointer types to be used as method receivers:

#![feature(arbitrary_self_types)]

struct MySmartPtr<T>(T);

impl<T> core::ops::Receiver for MySmartPtr<T> {
  type Target = T;
}

struct Content;

impl Content {
  fn method(self: MySmartPtr<Self>) { // note self type
  }
}

fn main() {
  let p = MySmartPtr(Content);
  p.method();
}

What is the RFC for this feature and what changes have occurred to the user-facing design since the RFC was finalized?

Changes since the RFC:

• The RFC specified additional diagnostics where people forgot to add a T: ?Sized bound - this is in progress here - @cramertj has kindly taken this on, and has determined that this is a pre-existing more general problem which she'll work on. We do not need to block stabilization to wait for this. (The other diagnostics noted in the RFC have been added).

In two cases the RFC wasn't very specific and the implementation has required some choices:

• The RFC specified that we should ban "generic" arbitrary self types, but we failed to define what that actually meant. Subsequent discussion yielded a fairly narrow definition of "generic".
• The RFC specifies that we should attempt to detect methods on smart pointers which might "shadow" pre-existing methods on the pointee. We are going ahead and doing this, but some simplifying assumptions were made, as noted in this comment. The likelihood of shadowing occurring in these cases seems low, relative to the likelihood of false positives in these cases.

What behavior are we committing to that has been controversial? Summarize the major arguments pro/con.

In general, Arbitrary Self Types is the opposite of controversial. It has always been anomalous that some specific stdlib smart pointer types have been hard-coded; this work removes that hard-coding.

The specific points which have involved discussion during the process:

• A prior version of Arbitrary Self Types relied on the Deref trait. This version introduces a new Receiver trait, such that types can be used as self types even if they can't safely devolve to a reference (e.g. because they're a wrapper for a pointer that can't safely comply with Rust reference semantics). This has been largely uncontroversial, but I'd note there's a blanket implementation of Receiver for T: Deref. This ensures all existing smart pointer types continue to work, and reduces complexity in users' brains. I'm 100% sure that this is the right thing to do, but highlighting it here because blanket impls are fairly unusual in the Rust standard library.
• Most of the discussion and complexity has been around the deshadowing algorithm. We can't add new methods to Rust smart pointer types - e.g. Rc, Box - because we may generate errors if they match the name of methods in pointees. It's already good practice to avoid these because of the risk of shadowing pointee methods, so these types tend to have associated functions instead. Now, any such shadowing will generate an error instead of silently shadowing.
• For a while we were considering a much more complex version which would have allowed us to add new methods to Rc, Box etc. and allow smart pointer types such as NonNull and Weak. We decided not to do that.
• There is a theoretical possibility of breaking existing code noted in this comment - which we don't think can happen in practice because any such code would be pointless.

Are there extensions to this feature that remain unstable? How do we know that we are not accidentally committing to those.

There is an arbitrary_self_types_pointers feature gate which allows raw pointers to be used as self types. Arguments against using this are summarized here.

This was a pre-existing aspect of the former arbitrary_self_types feature gate which has been split out into its own new feature gate because we don't want to stabilize it at this time, but we don't feel a strong need to remove this option from nightly Rust users.

Summarize the major parts of the implementation and provide links into the code (or to PRs)

The three main PRs are:

• Block generic arbitrary self types
• Library changes - adds the Receiver trait
• Compiler changes - switches the compiler to using Receiver instead of Deref; adds the deshadowing algorithm which is responsible for most of the complexity

Summarize existing test coverage of this feature

• The main PR for this feature added 17 tests: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17. These are a combination of basic functional tests, and tests relating to questions or corner cases which arose in development. These tests should comprehensively cover the basic method dispatch, the deshadowing algorithm (the only complex bit), and additional diagnostics.
• An additional test is in this PR
• Fairly extensive playing around with this has happened in autocxx
• Rust for Linux uses arbitrary self types for its Arc type and has partially adapted to the changes here
• A previous simulated stabilization PR was here and has been through two crater runs. It had perf results which look broadly neutral to me but I'm not an expert at interpreting them. (We expect some small performance penalty by checking for shadowed methods.)

Has a call-for-testing period been conducted? If so, what feedback was received?

No; though an earlier version of arbitrary self types has been in nightly for years. This has been experimented with by multiple communities - Rust/C++ interop, Rust/Python interop, and Rust for Linux. As the RFC notes, v2 was proposed based on experiences with v1.

What outstanding bugs in the issue tracker involve this feature? Are they stabilization-blocking?

• We need to land this fix for it to work well with the unstable DispatchFromDyn and the being-stabilized CoercePointee. This should block stabilization of either this feature, or CoercePointee.

Summarize contributors to the feature by name for recognition and assuredness that people involved in the feature agree with stabilization

• Me: @adetaylor
• Folks mostly involved in motivating the requirements: @davidhewitt @Darksonn @PoignardAzur
• Contributors to the RFC process: @Nadrieril @Manishearth @RalfJung @Veykril @madsmtm @Urhengulas @mikeyhew @joshtriplett @nikomatsakis @scottmcm @programmerjake @nbdd0121 @tmandry (this is not an exhaustive list; thanks to everyone else as well)
• Reviewers of the implementation: @wesleywiser @compiler-errors

Very sorry to those who I've missed; it's been a long road :)

What FIXMEs are still in the code for that feature and why is it ok to leave them there?

None.

What static checks are done that are needed to prevent undefined behavior?

None.

In what way does this feature interact with the reference/specification, and are those edits prepared

• rustc-dev-guide update (draft, needs rebase)
• Rust reference updates (draft)
• Book stuff (per comments from @chriskrycho we might not even want this)

Does this feature introduce new expressions and can they produce temporaries? What are the lifetimes of those temporaries?

No.

What other unstable features may be exposed by this feature?

None, though the separate derive(CoercePointee) becomes more useful when this is also stabilized.

What is tooling support like for this feature, rustdoc/clippy/rust-analzyer/rustfmt/etc

I don't anticipate any work here being needed other than for rust-analyzer, tracked here.

[adetaylor]

@rustbot label -S-waiting-on-review

[nikomatsakis]

Thanks for this! I already see an edit to the template that could be useful, clarifying what static checks are required and linking to tests that demonstrate them, but it kind of duplicates the reference/spec work

[PoignardAzur]

So the report mentions that testing already happened in a few forms.

How interested would you be in other libraries using this feature? We could implement it in Masonry, but since we don't want our crate to be nightly-only, we'd have to either use a feature flag or keep it to a separate branch.

If we do end up writing an implementation, is there any kind of feedback you'd be specifically interested in?

[adetaylor]

How interested would you be in other libraries using this feature? We could implement it in Masonry, but since we don't want our crate to be nightly-only, we'd have to either use a feature flag or keep it to a separate branch.

Good question and thanks for the offer! I personally have run out of runway to work on this feature - so I'm keeping my fingers crossed that no major semantic changes are needed. Or if they are, it will need someone else to pick up the work. So, the most interesting and useful testing to me would be looking for corner cases or oddities in the current implementation which cause ICEs or other oddities. (An example is the interaction with CoercePointee which caused a crash). It's hard for me to guess how likely it is that you'd find any such problem. Obviously I hope you wouldn't, but real world testing always has a habit of finding fun surprises.

Overall: up to you! If you think it would be useful for Masonry long-term then perhaps it's worth having a play in the hopes and expectations that this will be stabilized soon and you can therefore merge it into your main branch before too long.

[obi1kenobi]

I'd like to ask that the cargo SemVer reference be updated to describe any new SemVer hazards that are introduced here. This doesn't have to be a stabilization blocker, but I'd appreciate it if it can be done at least shortly thereafter so I can make sure cargo-semver-checks quickly offers the best possible support for linting this excellent new functionality.

[adetaylor]

@obi1kenobi good idea, will do.

@rustbot label +S-waiting-on-review -S-waiting-on-author

[rustbot]

The Miri subtree was changed

cc @rust-lang/miri

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

Some changes occurred in tests/codegen/sanitizer

cc @rust-lang/project-exploit-mitigations, @rcvalle

[compiler-errors]

Some more thoughts. I only scanned briefly but want to ensure these are addressed:

• Could you please also document the interaction with *const T/*mut T receivers? i.e. discuss the feature gate that gates those methods and why we're not stabilizing that right now.
• Have you documented any changes to method shadowing/deshadowing lint? I see references to it, but I'd want to make sure it lives somewhere more persistent like the dev guide (or at least as a dev comment in the method probing code). I'd want to make sure
• Can you make sure the Receiver trait is documented, and its interaction (i.e. blanket) with Deref, and also perhaps we should have something somewhere that advises smart pointer users when it's wise to implement Receiver on their own types?
• The What outstanding bugs in the issue tracker involve this feature? section currently doesn't mention all of the issues tagged F-arbitrary_self_types. Probably best if it does, even if they're not actionable.

T-types should also definitely be on this FCP. I'll adjust the labels accordingly:
@rustbot label: -T-compiler +T-types

[Kixunil]

• note there's a blanket implementation of Receiver for T: Deref

Just in case, I'm calling out that this enables ManuallyDrop, which is not a pointer, to be used in receiver position. I don't think there's anything broken about it but it's a bit weird. Also IIUC this allows to write things like:

impl Path {
    fn foo(self: PathBuf) { ... }
}

Which again I don't think is wrong but looks quite weird.

[traviscross]

cc @rust-lang/lang
cc @rust-lang/types