From c8a66e38e6a51fd52fc068f0b20646e45d069a7a Mon Sep 17 00:00:00 2001 From: Steven Troxler Date: Wed, 15 Dec 2021 21:23:34 -0800 Subject: [PATCH] PEP 677: Readability and organizational edits (#2193) * Address Guido's comments from initial PR * Update functions-as-types based on Pradeep's feedback * Fix some markup, address comments on endpoint example * Rearrange Motivation and Rationale * Move the language comparison subsection to Rational * Fix more markup errors * Rework "Motivation" section again Going with Pradeep's suggested order: - untyped example - correct types - partially-typed code - our proposal * Fix more lint errors * Add a section highlighting a concern about -> readability * Update the add/wrap/flatmap example * Wording change suggested by AlexWaygood * Add missing return annotation Co-authored-by: Alex Waygood * Break up long sentence Co-authored-by: Jelle Zijlstra * Break up long sentence (thanks Jelle) * Reword awkward sentence (thanks Pradeep) * Rearrange paragraphs in Motivation Co-authored-by: Alex Waygood Co-authored-by: Jelle Zijlstra --- pep-0677.rst | 570 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 341 insertions(+), 229 deletions(-) diff --git a/pep-0677.rst b/pep-0677.rst index 7da2be49dd3..150049a624d 100644 --- a/pep-0677.rst +++ b/pep-0677.rst @@ -27,116 +27,130 @@ work as a drop-in replacement. Motivation ========== -Describing Callable Signatures with ``typing.Callable`` -------------------------------------------------------- - One way to make code safer and easier to analyze is by making sure -that functions and classes are well-typed. In Python we have type -annotations defined by PEP 484 to provide type hints that can find -bugs as well as helping with editor tooling like tab completion, -static analysis tooling, and code review. - -One of the types `defined by PEP 484 -`_ is -``typing.Callable``, which describes a callable value such as a -function or a method. It takes two parameters as inputs but with the -first parameter being either a placeholder like ``...`` or a list of -types. For example: - -- ``Callable[..., int]`` indicates a function with arbitrary - parameters returning an integer. -- ``Callable[[str, int], bool]`` indicates a function taking two - positional parameters of types ``str`` and ``int`` and returning a - ``bool``. - -Of the types defined by PEP 484, ``typing.Callable`` is the most -complex because it is the only one that requires two levels of -brackets in the same type. PEP 612 added ``typing.ParamSpec`` and -``typing.Concatenate`` to help describe functions that pass ``*args`` -and ``**kwargs`` to callbacks, which is very common with -decorators. This made ``typing.Callable`` more powerful but also more -complicated. - -Problems with ``typing.Callable`` ---------------------------------- - -Empirically `we have found -`_ -that it is common for library authors to make use of untyped or -partially-typed callables (e.g. ``Callable[..., Any]`` or a bare -``Callable``) which we believe is partially a result of the existing -types being hard to use. But using this kind of partially-typed -callable can negate the benefits of static typing. For example, -consider the following code:: +that functions and classes are well-typed. In Python we have type +annotations, the framework for which is defined in PEP 484, to provide +type hints that can find bugs as well as helping with editor tooling +like tab completion, static analysis tooling, and code review. - from typing import Any, Callable +Consider the following untyped code:: - def with_retries( - f: Callable[..., Any] - ) -> Callable[..., Any]: - def wrapper(retry_once, *args, **kwargs): - if retry_once: - try: return f(*args, **kwargs) - except Exception: pass - return f(*args, **kwargs) - return wrapper + def flat_map(l, func): + out = [] + for element in l: + out.extend(f(element)) + return out - @with_retries - def f(x: int) -> int: - return x + def wrap(x: int) -> list[int]: + return [x] - f(True, z=15) + def add(x: int, y: int) -> int: + return x + y -The decorator above is clearly not intended to modify the type of the -function it wraps, but because it uses ``Callable[..., Any]`` it -accidentally eliminates the annotations on ``f``, and type checkers -will accept the code above even though it is sure to crash at -runtime. A correct version of this code would look like this:: + flat_map(wrap, [1, 2, 3]) # no runtime error, output is [1, 2, 3] + flat_map(add, [1, 2, 3]) # runtime error: `add` expects 2 arguments, got 1 - from typing import Any, Callable, Concatenate, ParamSpec, TypeVar - R = TypeVar("R") - P = ParamSpec("P") +We can add types to this example to detect the runtime error:: - def with_retries( - f: Callable[P, R] - ) -> Callable[Concatenate[bool, P] R]: - def wrapper(retry_once: bool, *args: P.args, **kwargs: P.kwargs): - ... - return wrapper + from typing import Callable + + def flat_map( + l: list[int], + func: Callable[[int], list[int]] + ) -> list[int]: + .... ... -With these changes, the incorrect attempt to pass ``z`` to ``f`` -produces a typecheck error as we would like. -Four usability problems with the way ``typing.Callable`` is -represented may explain why library authors often do not use its full -power: + flat_map(wrap, [1, 2, 3]) # type checks okay, output is [1, 2, 3] + flat_map(add, [1, 2, 3]) # type check error + +There are a few usability challenges with ``Callable`` we can see here: - It is verbose, particularly for more complex function signatures. -- It does not visually represent the way function headers are written, - which can make it harder to learn and use. -- It requires an explicit import, something we no longer require for - most of the other very common types after PEP 604 (``|`` for - ``Union`` types) and PEP 585 (generic collections) -- It relies on two levels of nested square brackets. This can be quite - hard to read, especially when the function arguments themselves have - square brackets. +- It relies on two levels of nested brackets, unlike any other generic + type. This can be expecially hard to read when some of the type + parameters are themselves generic types. +- The bracket structure is not visually similar to how function signatures + are written. +- It requires an explicit import, unlike many of the other most common + types like ``list``. + +Possibly as a result, `programmers often fail to write complete +Callable types +`_. +Such untyped or partially-typed callable types do not check the +parameter types or return types of the given callable and thus negate +the benefits of static typing. For example, they might write this:: + + + from typing import Callable + + def flat_map( + l: list[int], + func: Callable[..., Any] + ) -> list[int]: + .... -With our proposed syntax, the properly-typed decorator example becomes -concise and the type representations are visually descriptive:: + ... - from typing import Any, ParamSpec, TypeVar - R = TypeVar("R") - P = ParamSpec("P") + flat_map(add, [1, 2, 3]) # oops, no type check error! - def with_retries( - f: (**P) -> R - ) -> (bool, **P) -> R: - ... +There's some partial type information here - we at least know that ``func`` +needs to be callable. But we've dropped too much type information to catch +the mistake. + +With our proposal, the example looks like this:: + + def flat_map(l: list[int], func: (int) -> list[int]) -> list[int]: + out = [] + for element in l: + out.extend(f(element)) + return out + + ... + +The type ``(int) -> list[int]`` is more concise, uses an arrow similar +to the one indicating a return type in a function header, avoids +nested brackets, and does not require an import. + + +Rationale +========= + +The ``Callable`` type is widely used. For example, `as of October 2021 +it was +`_ +the fifth most common complex type in typeshed, after ``Optional``, +``Tuple``, ``Union``, and ``List``. + +The others have had their syntax improved and the need for imports +eliminated by either PEP 604 or PEP 585: + +- ``typing.Optional[int]`` is written ``int | None`` +- ``typing.Union[int, str]`` is written ``int | str`` +- ``typing.List[int]`` is written ``list[int]`` +- ``typing.Tuple[int, str]`` is written ``tuple[int, str]`` + +The ``typing.Callable`` type is used almost as often as these other +types, is more complicated to read and write, and still requires an +import and bracket-based syntax. + +In this proposal, we chose to support all the existing semantics of +``typing.Callable``, without adding support for new features. We took +this decision after examining how frequently each feature might be +used in existing typed and untyped open-source code. We determined +that the vast majority of use cases are covered. + +We considered adding support for named, optional, and variadic +arguments. However, we decided against including these features, as +our analysis showed they are infrequently used. When they are really +needed, it is possible to type these using `callback protocols +`_. An Arrow Syntax for Callable Types ---------------------------------- @@ -164,7 +178,7 @@ using the existing ``typing.Callable``:: from app_logic import Response, UserSetting - async def customize_response_for_settings( + def customize_response( response: Response, customizer: Callable[[Response, list[UserSetting]], Awaitable[Response]] ) -> Response: @@ -174,7 +188,7 @@ With our proposal, this code can be abbreviated to:: from app_logic import Response, UserSetting - def make_endpoint( + def customize_response( response: Response, customizer: async (Response, list[UserSetting]) -> Response, ) -> Response: @@ -184,33 +198,171 @@ This is shorter and requires fewer imports. It also has far less nesting of square brackets - only one level, as opposed to three in the original code. -Rationale -========= +Compact Syntax for ``ParamSpec`` +-------------------------------- -The ``Callable`` type is widely used. For example, `as of October 2021 -it was -`_ -the fifth most common complex type in typeshed, after ``Optional``, -``Tuple``, ``Union``, and ``List``. +A particularly common case where library authors leave off type information +for callables is when defining decorators. Consider the following:: -Most of the other commonly used types have had their syntax improved -via either PEP 604 or PEP 585. ``Callable`` is used heavily enough to -similarly justify a more usable syntax. -In this proposal, we chose to support all the existing semantics of -``typing.Callable``, without adding support for new features. We took -this decision after examining how frequently each feature might be -used in existing typed and untyped open-source code. We determined -that the vast majority of use cases are covered. + from typing import Any, Callable -We considered adding support for named, optional, and variadic -arguments. However, we decided against including these features, as -our analysis showed they are infrequently used. When they are really -needed, it is possible to type these using `Callback Protocols -`_. + def with_retries( + f: Callable[..., Any] + ) -> Callable[..., Any]: + def wrapper(retry_once, *args, **kwargs): + if retry_once: + try: return f(*args, **kwargs) + except Exception: pass + return f(*args, **kwargs) + return wrapper + + @with_retries + def f(x: int) -> int: + return x + + + f(y=10) # oops - no type error! + +In the code above, it is clear that the decorator should produce a +function whose signature is like that of the argument ``f`` other +than an additional ``retry_once`` argument. But the use of ``...`` +prevents a type checker from seeing this and alerting a user that +``f(y=10)`` is invalid. + + +With PEP 612 it is possible to type decorators like this correctly +as follows:: + + from typing import Any, Callable, Concatenate, ParamSpec, TypeVar + + R = TypeVar("R") + P = ParamSpec("P") + + def with_retries( + f: Callable[P, R] + ) -> Callable[Concatenate[bool, P] R]: + def wrapper(retry_once: bool, *args: P.args, **kwargs: P.kwargs) -> R: + ... + return wrapper + + ... + + +With our proposed syntax, the properly-typed decorator example becomes +concise and the type representations are visually descriptive:: + + from typing import Any, ParamSpec, TypeVar + + R = TypeVar("R") + P = ParamSpec("P") + + def with_retries( + f: (**P) -> R + ) -> (bool, **P) -> R: + ... + +Comparing to Other Languages +---------------------------- + +Many popular programming languages use an arrow syntax similar +to the one we are proposing here. + +TypeScript +~~~~~~~~~~ + +In `TypeScript +`_, +function types are expressed in a syntax almost the same as the one we +are proposing, but the arrow token is ``=>`` and arguments have names:: + + (x: int, y: str) => bool + +The names of the arguments are not actually relevant to the type. So, +for example, this is the same callable type:: + + (a: int, b: str) => bool + +Kotlin +~~~~~~ + +Function types in `Kotlin `_ permit +an identical syntax to the one we are proposing, for example:: + + (Int, String) -> Bool + +It also optionally allows adding names to the arguments, for example:: + + (x: Int, y: String) -> Bool + +As in TypeScript, the argument names if provided are just there for documentation +and are not part of the type itself. + +Scala +~~~~~ + +`Scala `_ +uses the ``=>`` arrow for function types. Other than that, their syntax is +the same as the one we are proposing, for example:: + + (Int, String) => Bool + +Scala, like Python, has the ability to provide function arguments by name. +Function types can optionally include names, for example:: + + (x: Int, y: String) => Bool + +Unlike in TypeScript and Kotlin, these names are part of the type if +provided - any function implementing the type must use the same names. +This is similar to the extended syntax proposal we described in our +`Rejected Alternatives`_ section. + +Function Definition vs Callable Type Annotations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In all of the languages listed above, type annotations for function +definitions use a ``:`` rather than a ``->``. For example, in TypeScript +a simple add function looks like this:: + + function higher_order(fn: (a: string) => string): string { + return fn("Hello, World"); + } + +Scala and Kotlin use essentially the same ``:`` syntax for return +annotations. The ``:`` makes sense in these languages because they +all use ``:`` for type annotations of +parameters and variables, and the use for function return types is +similar. + +In Python we use ``:`` to denote the start of a function body and +``->`` for return annotations. As a result, even though our proposal +is superficially the same as these other languages the context is +different. There is potential for more confusion in Python when +reading function definitions that include callable types. -See the Rejected Alternatives section for more detailed discussion -about omitted features. +This is a key concern for which we are seeking feedback with our draft +PEP; one idea we have floated is to use ``=>`` instead to make it easier +to differentiate. + + +The ML Language Family +~~~~~~~~~~~~~~~~~~~~~~ + +Languages in the ML family, including `F# +`_, +`OCaml +`_, +and `Haskell `_, all use +``->`` to represent function types. All of them use a parentheses-free +syntax with multiple arrows, for example in Haskell:: + + Integer -> String -> Bool + +The use of multiple arrows, which differs from our proposal, makes +sense for languages in this family because they use automatic +`currying `_ of function arguments, +which means that a multi-argument function behaves like a single-argument +function returning a function. Specification ============= @@ -257,7 +409,8 @@ same:: Grammar and AST --------------- -The proposed new syntax can be described by these AST changes :: +The proposed new syntax can be described by these AST changes to `Parser/Python.asdl +`_:: expr = | AsyncCallableType(callable_type_arguments args, expr returns) @@ -301,8 +454,8 @@ types by modifying the grammar for ``callable_type_positional_argument`` as follows:: callable_type_positional_argument: - | expression ',' - | expression &')' + | !'...' expression ',' + | !'...' expression &')' | '*' expression ',' | '*' expression &')' @@ -340,8 +493,8 @@ Because operators bind more tightly than ``->``, parentheses are required whenever an arrow type is intended to be inside an argument to an operator like ``|``:: - (int) -> bool | () -> bool # syntax error! - (int) -> bool | (() -> bool) # okay + (int) -> () -> int | () -> bool # syntax error! + (int) -> (() -> int) | (() -> bool) # okay We discussed each of these behaviors and believe they are desirable: @@ -400,21 +553,30 @@ want a type annotation and ``...`` is a valid expression. This is never semantically valid and all type checkers would reject it, but the grammar would allow it if we did not explicitly prevent this. -We decided that there were compelling reasons to prevent it: - The -semantics of ``(...) -> bool`` are different from ``(T) -> bool`` for -any valid type T: ``(...)`` is a special form indicating -``AnyArguments`` whereas ``T`` is a type parameter in the arguments -list. - ``...`` is used as a placeholder default value to indicate an -optional argument in stubs and Callback Protocols. Allowing it in the -position of a type could easily lead to confusion and possibly bugs -due to typos. - Since ``...`` is meaningless as a type and there are usability concerns, our grammar rules it out and the following is a syntax error:: (int, ...) -> bool +We decided that there were compelling reasons to do this: + +- The semantics of ``(...) -> bool`` are different from ``(T) -> bool`` + for any valid type T: ``(...)`` is a special form indicating + ``AnyArguments`` whereas ``T`` is a type parameter in the arguments + list. +- ``...`` is used as a placeholder default value to indicate an + optional argument in stubs and callback protocols. Allowing it in + the position of a type could easily lead to confusion and possibly + bugs due to typos. +- In the ``tuple`` generic type, we special-case ``...`` to mean + "more of the same", e.g. a ``tuple[int, ...]`` means a tuple with + one or more integers. We do not use ``...`` in a a similar way + in callable types, so to prevent misunderstandings it makes sense + to prevent this. + + + Incompatibility with other possible uses of ``*`` and ``**`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -508,7 +670,7 @@ described by the existing ``typing.Callable`` semantics: Features that other, more complicated proposals would support account for fewer than 2% of the use cases we found. These are already -expressible using Callback Protocols, and since they are uncommon we +expressible using callback protocols, and since they are uncommon we decided that it made more sense to move forward with a simpler syntax. Extended Syntax Supporting Named and Optional Arguments @@ -551,16 +713,34 @@ We decided against proposing it for the following reasons: community decides after more experience and discussion that we want the additional features, they should be straightforward to propose in the future. -- We realized that because of overloads, it is not possible to replace - all need for Callback Protocols even with an extended syntax. This - makes us prefer proposing a simple solution that handles most use - cases well. +- Even a full extended syntax cannot replace the use of callback + protocols for overloads. For example, no closed form of callable type + could express a function that maps bools to bools and ints to floats, + like this callback protocol.:: + + from typing import overload, Protocol + + class OverloadedCallback(Protocol) + + @overload + def __call__(self, x: int) -> float: ... + + @overload + def __call__(self, x: bool) -> bool: ... + + def __call__(self, x: int | bool) -> float | bool: ... + + + f: OverloadedCallback = ... + f(True) # bool + f(3) # float + + We confirmed that the current proposal is forward-compatible with extended syntax by `implementing `_ -a quick-and-dirty grammar and AST on top of the grammar and AST for -the current proposal. +a quick-and-dirty grammar and AST on top of this grammar and AST for. Syntax Closer to Function Signatures @@ -605,26 +785,35 @@ Other Proposals Considered Functions-as-Types ~~~~~~~~~~~~~~~~~~ -An idea we looked at very early on was to `allow using functions as -types +An idea we looked at very early on was to `allow using functions as types `_. The idea is allowing a function to stand in for its own call signature, with roughly the same semantics as the ``__call__`` method -of Callback Protocols. Think this may be a great idea and worth its -own PEP, but that it is not a good alternative to improving the -usability of callable types: - -- Using functions as types would not give us a new way of describing - function types as first class values. Instead, they would require a - function definition statement that effectively defines a type alias - (much as a Callable Protocol class statement does). -- Functions-as-types would support almost exactly the same features - that Callable Protocols do today: named, optional, and variadic args - as well as the ability to define overloads. - -Another reason we don't view functions-as-types as a good alternative -is that it would be difficult to handle ``ParamSpec``, which we -consider a critical feature to support. +of callback protocols:: + + def CallableType( + positional_only: int, + /, + named: str, + *args: float, + keyword_only: int = ..., + **kwargs: str)` + ) -> bool: ... + + f: CallableType = ... + f(5, 6.6, 6.7, named=6, x="hello", y="world") # typechecks as bool + +This may be a good idea, but we do not consider it a viable +replacement for callable types: + +- It would be difficult to handle ``ParamSpec``, which we consider a + critical feature to support. +- When using functions as types, the callable types are not first-class + values. Instead, they require a separate, out-of-line function + definition to define a type alias +- It would not support more features than callback protocols, and seems + more like a shorter way to write them than a replacement for + ``Callable``. Parenthesis-Free Syntax ~~~~~~~~~~~~~~~~~~~~~~~ @@ -748,83 +937,6 @@ led us to the current proposal. `_ for feedback. -Other Languages ---------------- - -Many popular programming languages use an arrow syntax similar -to the one we are proposing here - -the same ``->`` arrow token we are proposing here. -almost identical to the ones we are proposing here - -TypeScript -~~~~~~~~~~ - -In `TypeScript -`_, -function types are expressed in a syntax almost the same as the one we -are proposing, but the arrow token is ``=>`` and arguments have names:: - - (x: int, y: str) => bool - -The names of the arguments are not actually relevant to the type. So, -for example, this is the same callable type:: - - (a: int, b: str) => bool - -Kotlin -~~~~~~ - -Function types in `Kotlin `_ permit -an identical syntax to the one we are proposing, for example:: - - (Int, String) -> Bool - -It also optionally allows adding names to the arguments, for example:: - - (x: Int, y: String) -> Bool - -As in TypeScript, the argument names if provided are just there for documentation -and are not part of the type itself. - -Scala -~~~~~ - -`Scala `_ -uses the ``=>`` arrow for function types. Other than that, their syntax is -the same as the one we are proposing, for example:: - - (Int, String) => Bool - -Scala, like Python, has the ability to provide function arguments by name. -Funciton types can optionally include names, for example:: - - (x: Int, y: String) => Bool - -Unlike in TypeScript and Kotlin, these names are part of the type if -provided - any function implementing the type must use the same names. -This is similar to the extended syntax proposal we described in our -`Rejected Alternatives`_ section. - -The ML Language Family -~~~~~~~~~~~~~~~~~~~~~~ - -Languages in the ML family, including `F# -`_, -`OCaml -`_, -and `Haskell `_, all use -``->`` to represent function types. All of them use a parentheses-free -syntax with multiple arrows, for example in Haskell:: - - Integer -> String -> Bool - -The use of multiple arrows, which differs from our proposal, makes -sense for languages in this family because they use automatic -`currying `_ of function arguments, -which means that a multi-argument function behaves like a single-argument -function returning a function. - Acknowledgments ---------------