Class extension members

[eernstg]

This issue is a proposal for a variant of extension members that (1) are declared in the target class, and (2) are "sticky". The main motivation for this mechanism is that it allows for adding a new member to an existing class, and providing an implementation of that member that will be available for all subtypes, such that it does not break existing subtypes that they don't implement the new member.

For example, the update problem arises if we change a class A which is implemented by a class B as follows:

// ** Example using the instance members that we have today **

// --- Library 'lib.dart', old version.
class A {}

// --- Library 'lib.dart', new version.
class A { void newMember() {}}

// --- Library 'some_client.dart'.
import 'lib.dart';

// This class breaks when `A` is updated: "No implementation of `newMember`".
class B implements A {}

With class extension members, we can add the newMember to A as a "sticky" extension method. This means that it works just like an extension method (in particular, it is always resolved statically), and it is possible to execute that method implementation with a receiver of type A or any subtype of A. But, in contrast to regular extension methods, it is not necessary to directly import the library that declares the class extension member in order to have access to it (so it's "sticky" in the sense that it automatically goes with the type).

The trade-off is that newMember is resolved statically, which means that subtypes like B do not get the opportunity to run a different implementation when newMember is invoked on an instance of type B unless the static type of the receiver is B (and B declares a different implementation of newMember). For example:

// ** Example using class extension members **

// --- Library 'lib.dart', old version.
class A {}

// --- Library 'lib.dart', new version.
class A { extension void newMember() {}}

// --- Library 'some_client.dart'.
import 'lib.dart';

// This class continues to work when `A` is updated.
class B implements A {}

// A similar class may choose to provide its own implementation.
class B2 implements A {
  extension newMember() {} // Could also omit `extension`.
}

void main() {
  B2 b2 = B2();
  A a = b2;
  b2.newMember(); // Calls `B2.newMember`.
  a.newMember(); // Calls `A.newMember` even though `a is B2`.
}

Syntax

The only syntactic change is that a class member declaration may have the modifier extension.

<classMemberDefinition> ::=
  'extension'? <methodSignature> <functionBody> |
  'extension'? <declaration> ';'

A compile-time error occurs if a class member definition D has the modifier extension, and D declares an instance variable, a constructor, a static member, an abstract member, or an external member.

Static Analysis

Static analysis of a class extension member declaration of the form extension classMember in a class or mixin declaration named C with type parameters X1 extends B1 .. Xk extends Bk is performed as if it had occurred as follows, in the same library as the declaration C:

extension Name<X1 extends B1 .. Xk extends Bk> on C<X1 .. Xk> {
  classMember
}

In this declaration, Name is a globally fresh public name.

In particular, this means that a super invocation of an instance member (like super.foo() or super + 3) is a compile-time error, and the body of the class extension method has access to the type variables of the class, but the value of those type variables at run time is determined by the static information available at the call site.

A compile-time error occurs if classMember is named n and it is declared in a class or mixin whose interface has a member named n.

In particular, a class extension member declaration cannot "override" an instance member declaration. In contrast, it is indeed allowed for an instance member declaration in a subtype to "override" a class extension member. There is no requirement that the instance member declaration has a signature which is a correct override of the member signature of the class extension member. This is similar to the treatment of a static method: An instance method is allowed to "override" a static method, but it's not an object-oriented override relation because invocations of the static method will never call the instance method at run time, because those invocations are statically resolved.

We use the word redeclare to describe the relationship between two class extension member declarations named m in D1 respectively D2, or one class extension member declaration in D1 and one instance member declaration in D2, or vice versa, both again named m, when D2 is a direct superinterface of D1.

As usual, with a declaration like class B implements A {}, B has a declaration of m if B declares m, or A declares m, or A has a superinterface that has m. So the phrase 'direct superinterface' will give us exactly the same relationship as the normal 'override' relation.

A compile-time error occurs if multiple superinterfaces of a class C have a class extension method with the same name, and they do not all refer to the same declaration.

Hence, a class C that implements or extends A and B is a compile-time error if both A and B has a class extension method named n, and they do not both resolve to the same declaration, but C can eliminate the error by declaring a member (of any kind) named n.

Assume that a class declaration D named C in a library L contains a class extension member m and the desugaring to an extension declaration containing m has the name CExtension.

Any library L2 that contains an expression whose type resolves to D or a subtype thereof will then have an implicitly induced import as follows:

import 'L' as FreshName show CExtension;

where FreshName is a name which is fresh in L2.

During static analysis of a member access (e.g., a method call) e.m..., if multiple extensions are accessible and available, then all extensions obtained by desugaring of class extension members are considered more specific than
all other extensions.

This implies that class extension methods have higher priority than other extension methods. Class extension methods cannot be conflicted among each other, because it's required that every class that inherits multiple class extension methods with the same name must have a redeclaration to resolve the conflict. It is possible to specify any of those other extension methods by means of an explicit extension method invocation.

Dynamic Semantics

The dynamic semantics of class extension methods is fully determined by the desugaring.

Discussion

Note that the dynamic semantics of some other proposals is more powerful. For instance, interface default methods are proper instance methods, so they allow for overriding, dynamic invocations, and access to the run-time value of the type parameters of the enclosing class/mixin.

[lrhn]

I'd avoid using the word "override" for relations which are not virtual overrides. I don't have a better word, "shadow" is not good because it assumes a point of view. Maybe no term is needed: "Can declare a member with the same name".

The rules against conflict means that it's possible for two independent interfaces to add an extension member with the same name, and that causing a third class, in a separate package, to become a compile-time error.
I'm a little worried about that. I think I'd prefer to allow the conflict, but disallow invocations on that type (as for any other extension conflict where neither declared on type is more specific than the other). That way, you only get an error in code that is aware of at least one of the conflicting extension members.

It's also a problem if two superinterfaces both add extension members with the same name and no reasonable way to combine them (say a getter/setter and a function taking one required argument).
In order for a subclass to placate the compiler, it would have to add some member with the same public name to its interface. It might just be extension get banana => throw UnsupportedError("banana!");, but it still pollutes your public API documentation, even though you have nothing to do with it.
You can obviously choose to do extension get banana => (this as OneSuperInterface).banana; and ignore the other extension, after all you are implementing the interface, so you might want to support it. But I don't think you should have to.

What extension methods can do, which interface methods cannot, is to only apply to some instantiations of a generic class.
Take:

extension ComparableSorter<T extends Comparable<T>> on List<T> {
  List<T> sorted([int Function(T, T) compare]);
}
extension Sorter<T> on List<T> {
  List<T> sorted(int Function(T, T) compare);
}

(A pattern we have used in package:collection)

Here the extension method in ComparableSorter only applies to some instantiations of List.
It cannot be added as a class extension member on List, because those must apply to all instances.

Declaring a method as en extension method instead of an instance method, when you have access to the library declaring the target type, suggests that you either want the restricted receiver set, or you use extensions instead of instance methods to avoid breaking existing implementations of the interface.

I don't find extension methods to be the best way to add methods to an existing interface in a non-breaking way. I'd still prefer interface default methods for that.

Which mans that class extensions members is not actually the best solution for either of the problems that you use extension members for today.

I think this design should either include a way to apply extension members to a subset of instances of the class, or we should just use the implementation effort on interface default methods instead.

[eernstg]

I'd avoid using the word "override"

Agreed! I'll think some more about what else we can use.

[Edit: Now using 'redeclare'.]

The rules against conflict means that it's possible for two independent interfaces
to add an extension member with the same name, and that causing a third class,
in a separate package, to become a compile-time error.

Let's say it is class C implements A, B {}, and someone adds a class extension method m to A and B.

We could of course delay the error to the call site, just like we do with imports. However, I tend to prefer an early detection and handling of the conflict (that is, make C an error).

Otherwise, if we allow C and make myC.m() an error, the error may be reported at a location which is far away from the declaration of C. In contrast, clashing names in imports can be resolved when they are detected, because it occurs in the same library where the imports are specified.

So if the two declarations of m are compatible (such that it makes sense to implement C.m and let it be called both in order to serve the purpose of A.m and the purpose of B.m), then I think it's a useful service to the community for C to implement m and eliminate all those errors that might otherwise arise for clients of C.

In contrast, I don't think it's a good idea to silently build up technical debt whereby lots of extension methods can't be called because of this kind of conflict (unless you do something like (myC as A).m() or (myC as B).m()). After all, it is the maintainer of C that knows best how the given behavior of A.m and B.m can be handled properly when the receiver is a C.

It is true that in the case where A.m and B.m are semantically incompatible (conceptually: they have different purposes, technically: there is no implementation that does the job of both), reporting an error for myC.m() is more meaningful than running any implementation. We must insist that the caller informs us about the purpose of calling myC.m() — should it do the A.m thing or the B.m thing?

However, that's the same dilemma that we would have if we're declaring class F implements D, E {} where D and E both have a regular instance method n, but D.n and E.n have a completely different purpose. It would be highly error-prone to introduce any implementation of F.n. Still, it's not a situation that comes up very frequently...

So I still tend to think that it's better to detect and handle the conflict early (which is also what we usually want). If there is an insurmountable incompatibility between A.m and B.m then we can at least indicate that there is a conflict by declaring a C.m that basically serves as an error message:

class A { extension int get m => 1; }
class B { extension String m(bool b) {}

/// Serves to indicate an error: [A.m] and [B.m] are incompatible.
///
/// This class is used to inform the reader of error messages mentioning
/// this class that the extension methods [A.m] and [B.m] are incompatible,
/// and the caller of [C.m] must choose explicitly which one to call (say,
/// using `(myC as A).m`).
abstract class _mOfAorBPlease {}

class C {
   extension _mOfAorBPlease m({required _mOfAorBPlease mOfAOrBPlease}) => throw 0;
}

Of course, the community would then put some pressure on the maintainers of A and B such that they very quickly switch to use distinct names for distinct purposes, and then we can remove the "error member" m from C.

What extension methods can do, which interface methods cannot, is to
only apply to some instantiations of a generic class.

Cf. #2313, we could easily add that capability.

Also, today's extension methods are not "sticky", which means that if we add a new member by means of a regular extension method then all clients need to add some imports in order to get access to it. Class extension methods are "sticky" in the sense that they are available on any expression whose type is a subtype of the type that declares that class extension method, and you don't need to import anything directly in order to enable that.

the best way to add methods to an existing interface in a non-breaking way

Interface default methods indeed have a more powerful dynamic semantics (they allow for access to the actual values of the type arguments of the enclosing class/mixin, and they can be overridden).

However, class extension members can do a couple of things as well: They can allow redeclaring declarations with incompatible signatures (which means that they can adapt to needs that a set of overriding instance members cannot), and they are more lightweight than instance members (for instance, they can be inlined at call sites).

So the comparison is not totally one-sided.

[lrhn]

The problem with early errors is that it makes adding an extension method to a class potentially breaking, much more than adding the extension method on the side.
If the goal is to add a new method in a non-breaking way, then in some ways you're better off putting it on the side. (And worse off, in that users then need to import the extension).

If the incentives of a feature is such that it's better to not use the feature, I think that suggests a flaw in the design.

We already have extension methods. Those only clash at the invocation point.

Adding class extension methods which do the same as normal extension methods, with better import behavior (well, different, you also cannot not import them), but worse conflict resolution, it feels like a real tradeoff whether I should use one or the other.
If the class extensions used the same conflict resolution as other extensions methods, then the tradeoff balance would flip towards the new feature. That's good. Why add a feature which isn't more desirable than what we already have?

Extensions can do things interface default methods can't, but nothing a static helper function can't - except more ergonomic invocation. The power of extensions lies in the syntax used to call them, nothing more.

Adding new member to an interface that other people are implementing (which is where you need to be non-breaking) is a place where allowing subclasses to add more efficient overrides is important. That's why I think interface default methods are better than extension methods for adding new methods to existing interfaces - if can't just add the instance method to begin with, then you'd likely also get an advantage from being virtual.

[eernstg]

Arguments against class extension members seem to imply a repetition of this one:

we should just use the implementation effort on interface default methods instead

So how exactly are you going to make it a non-error to have the following update of A and B?:

// Old version.
abstract class A {}
abstract class B {}

// New version.
abstract class A { default int get m => 1; }
abstract class B { default void m(String s) {}}

// Class that implements both is not an error after that update?
class C implements A, B {}

Note that we can resolve the error by adding any declaration of m that may seem useful if we add class extension methods to A and B rather than interface default methods. But with interface default methods we will need to specify an instance member that overrides both A.m and B.m, and that is a plain impossibility.

The motivation for class extension members is almost exclusively to deliver a sticky version of the extension members that we have today. They do that. Conflicts may still arise, but that's true for every alternative, just with different trade-offs.

[lrhn]

True, interface default methods are more breaking than extension methods (at least if extension method conflict is not an early error), and harder to fix if they occur.

I believe the tradeoff is worth it.

I guess interfaces can often be separated into, effectively, API-interfaces and "mixin"/functionality-interfaces.

The former is what you mostly think of when you say "an interface". It's abstract data types like Iterable, List, and Stream.
You generally do not mix those interface with other similar interfaces, instead you have multiple different implementations. These interfaces can be large. (We have exceptions, mostly in dart:io, where something is a Stream instead of having a Stream. Those were a mistake.)
Some of these interfaces may be smaller, and form a common base for other, larger, interfaces, but you rarely combine two such API interfaces. These interfaces describe how to manage internal data of the class.

The latter are small interfaces intended to be combined into other classes. Think Observable, Disposable, ... really "able"-names in general. They say that an object supports some specific general functionality in their own way. Those interfaces rarely grow, and they can be as small as a single method. It's common to implement multiple of these, so they are already prepared to avoid naming conflicts. The interfaces are hooks for other code to work with the class in a specific way.

I want interface default methods for adding extra functionality to the API interfaces. Since you only implement one of those, conflicts are unlikely, having different implementations and wanting to specialize the operation is likely..

I'd want extension methods for adding utility to the mixin-interfaces, like adding a registerInDisposer(Disposer) to every Disposable.

I guess that means that I don't worry as much about conflicts for interface default methods, because I don't expect multiple different source of them. I do worry more about it for extension methods.

But maybe the world is not so easily divided in only two groups. I'd expect a Widget to be a little of both, and I'm mainly focusing on my needs in the platform libraries, and similar packages. I'm not a framework writer :)

[eernstg]

interface default methods are more breaking .. at least if extension method
conflict is not an early error

Also if class extension method conflict is an early error: With interface default methods you simply cannot resolve the conflict among two incompatible signatures (unless you can stop implementing one of those interfaces, but that's hugely breaking for your clients as well). With class extension methods it is always sufficient to write a declaration of the conflicted name. By the way, with interface default methods you don't even have the option to report the error late (at call sites).

but you rarely combine two such API interfaces

Just checking a bunch of source code with simple regular expressions, I can see 42583 occurrences of implements followed by no commas on that same line, and 16479 occurrences of implements followed by at least one comma on the same line. (Surely there are more occurrences of implements with multiple superinterfaces, because I made no attempt to find occurrences that span multiple lines).

So you're saying that those approximately 39% of the implements clauses that have multiple superinterfaces do not matter? If we want to say "that doesn't happen in practice" I would expect something more like 0.39%, at most...

I don't worry as much about conflicts for interface default methods, because I don't
expect multiple different source of them. I do worry more about it for extension methods.

Right, interface default methods may have fewer conflicts because they are provided by the maintainers of the target classes (and nobody else can write them), but regular extension methods can be written by anyone.

But that argument works equally well in favor of class extension methods: They are also written in the target class itself, and they have higher priority than other extension methods. It might then serve as a tiebreaker that conflicts between incompatible signatures simply cannot be resolved with interface default methods. ;-)

[ds84182]

I'm not sure this should use extension as the prefix, seems like default is much better suited and in-line with Java.

[eernstg]

We could use default, but that's a pretty strong hint in the direction of a different mechanism, interface default methods.

They differ from class extension members in that they are actual instance members which are added implicitly to any concrete class that must have an implementation, but doesn't have it. In other words, 'default' describes the semantics in a very direct manner.

With class extension members (this proposal), invocations are determined by the static knowledge about the receiver, just like extension members (that is, members declared in an extension declaration). It would probably be more confusing than helpful to use default for class extension members.