PEP 677: Runtime Behavior Specification

pep-0677.rst

@@ -635,24 +635,108 @@ callable types and ``=>`` for lambdas.
 Runtime Behavior
 ----------------
 
-Our tentative plan is that:
+The new AST nodes need to evaluate to runtime types, and we have two goals for the
+behavior of these runtime types:
 
-- The ``__repr__`` will show an arrow syntax literal.
-- We will provide a new API where the runtime data structure can be
-  accessed in the same manner as the AST data structure.
-- We will ensure that we provide an API that is backward-compatible
-  with ``typing.Callable`` and ``typing.Concatenate``, specifically
-  the behavior of ``__args__`` and ``__parameters__``.
+- They should expose a structured API that is descriptive and powerful
+  enough to be compatible with extending the type to include new features
+  like named and variadic arguments.
+- They should also expose an API that is fully backward-compatible with
+  ``typing.Callable``.
 
-Because these details are still under debate we are currently
-maintaining `a separate doc
-<https://docs.google.com/document/d/15nmTDA_39Lo-EULQQwdwYx_Q1IYX4dD5WPnHbFG71Lk/edit>`_
-with details about the new builtins, the evaluation model, how to
-provide both a backward-compatible and more structured API, and
-possible alternatives to the current plan.
+Evaluation and Structured API
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We intend to create new builtin types to which the new AST nodes will
+evaluate, exposing them in the ``types`` module.
+
+Our plan is to expose a structured API as if they were defined as follows::
+
+    class CallableType:
+        is_async: bool
+        arguments: builtins.Ellipsis | CallableArgumentsList | CallableConcatenation
+        return_type:: Typing.type
+
+    class CallableArgumentsList:
+        positional_only: tuple[typing.Type]
+
+    class CallableConcatenation:
+        positional_only: tuple[typing.Type]
+        param_spec: typing.ParamSpec
+
+The evaluation model is that:
+
+  - ``AnyArguments`` evaluates to ``builtins.Ellipsis``.
+  - ``ArgumentsList`` evaluates to ``CallableArgumentsList``.
+  - ``Concatenation`` evaluates to ``CallableConcatenation``.
+  - To evaluate a ``CallableType`` or ``AsyncCallableType`` to a
+    ``types.CallableType`` value ``t``:
+
+    - Define ``t.is_async`` in the obvious way
+    - Evaluate ``arguments`` to define ``t.arguments``
+    - Evaluate ``returns`` to define ``t.return_type``
+
+Backward-Compatible API
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To get backward compatibility with the existing ``types.Callable`` API,
+which relies on fields ``__args__`` and ``__parameters__``, we can define
+them as if they were written in terms of the following::
+
+    import itertools
+    import typing
+
+    def get_args(
+        t: CallableType
+    ):
+        return_type_arg = (
+            typing.Awaitable[t.return_type]
+            if t.is_async
+            else t.return_type
+        )
+        arguments = t.arguments
+        if isinstance(arguments, Ellipses):
+            argument_args = (Ellipses,)
+        elif isinstance(arguments, CallableArgumentsList):
+            argument_args = arguments.positional_only
+        else:
+            argument_args = (*arguments.positional_only, arguments.param_spec)
+        return (*arguments_args, return_type_arg)
+
+    def _positional_only_parameters(
+        positional_only: tuple[typing.Type]
+    ):
+        return tuple(itertools.chain(
+            arg.__parameters__ for arg in positional_only
+        ))
+
+    def get_parameters(t: CallableType):
+        arguments = t.arguments
+        if isinstance(arguments, Ellipses):
+            argument_parameters = ()
+        elif isinstance(arguments, Typ):
+            argument_parameters = _positional_only_parameters(
+                arguments.positional_only
+            )
+        else:
+            argument_parameters = (
+                *_positional_only_parameters(arguments.positional_only)
+                arguments.param_spec
+            )
+        return (
+            *argument_parameters,
+            *t.return_type.__parameters__
+        )
+
+Additional Behaviors of ``types.CallableType``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- The ``__eq__`` method should treat equivalent ``typing.Callable``
+  values as equal to values constructed using the builtin syntax, and
+  otherwise should behave like the ``__eq__`` of ``typing.Callable``.
+- The ``__repr__`` method should produce an arrow syntax representation that,
+  when evaluated, gives us back an equal ``types.CallableType`` instance.
 
-Once the plan is finalized we will include a full specification of
-runtime behavior in this section of the PEP.
 
 Rejected Alternatives
 =====================
@@ -991,6 +1075,56 @@ Moreover, none of these ideas help as much with reducing verbosity
 as the current proposal, nor do they introduce as strong a visual cue
 as the ``->`` between the parameter types and the return type.
 
+Alternative Runtime Behaviors
+-----------------------------
+
+The hard requirements on our runtime API are that:
+
+- It must preserve backward compatibility with ``typing.Callable`` via
+  ``__args__`` and ``__params__``.
+- It must provide a structured API, which should be extensible if
+  in the future we try to support named and variadic arguments.
+
+Using a flat API
+~~~~~~~~~~~~~~~~
+
+Our runtime API mirrors the AST in that we distinguish the case
+where the arguments is ``Ellipsis`` versus ``CallableArgumentsList``
+versus ``CallableConcatenation``.
+
+An alternative would be just have a single ``types.CallableType`` with
+fields
+- ``is_async: bool``
+- ``is_any_arguments: bool``
+- ``positional_only: tuple[typing.Type]``
+- ``param_spec: typing.ParamSpec | None``
+
+This would make the implementation simpler. But it becomes harder
+to see which combinations of fields are legal, and to use pattern
+matching or refinement to understand what the type represents.
+
+Since the users of this API would be developers implementing runtime typing
+libraries, our view is that they would likely prefer the more structured API
+we have proposed, in which legal combinations of arguments-related fields can be
+determined by matching against the type of the ``arguments`` field.
+
+Using the plain return type in ``__args__`` for async types
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is debatable whether we are required to preserve backward compatiblity
+of ``__args__`` for async callable types like ``async (int) -> str``. The
+reason is that one could argue they are not expressible directly
+using ``typing.Callable``, and therefore it would be fine to set
+``__args__`` as ``(int, int)`` rather than ``(int, typing.Awaitable[int])``.
+
+But we believe this would be problematic. By preserving the appearance
+of a backward-compatible API while actually breaking its semantics on
+async types, we would cause runtime type libraries that attempt to
+interpret ``Callable`` using ``__args__`` to fail silently.
+
+It is for this reason that we automatically wrap the return type in
+``Awaitable``.
+
 Backward Compatibility
 ======================
 
@@ -1033,10 +1167,11 @@ Open Issues
 Details of the Runtime API
 --------------------------
 
-Once we have finalized all details of the runtime behavior, we
-will need to add a full specification of the behavior to the
-`Runtime Behavior`_ section of this PEP as well as include that
-behavior in our reference implementation.
+We have attempted to provide a complete behavior specification in
+the `Runtime Behavior`_ section of this PEP.
+
+But there are probably more details that we will not realize we
+need to define until we build a full reference implementation.
 
 Optimizing ``SyntaxError`` messages
 -----------------------------------