Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-37154][PYTHON] Inline hints for pyspark.rdd #35252

Closed
wants to merge 9 commits into from
Closed

[SPARK-37154][PYTHON] Inline hints for pyspark.rdd #35252

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
87e997e
First pass
zero323 Jan 15, 2022
023f424
Replace T_co with T outside Generic
zero323 Jan 20, 2022
1391f58
Add data tests for simple methods
zero323 Jan 21, 2022
2a37b86
Add more data tests
zero323 Jan 23, 2022
f49b55f
O -> S
zero323 Jan 27, 2022
ee99a79
Change "List[S]" to List["S"] in takeOrdered
zero323 Feb 3, 2022
c4c0eeb
Merge branch 'master' into SPARK-37154
zero323 Feb 10, 2022
87a97df
Merge branch 'master' into SPARK-37154
zero323 Feb 16, 2022
4708a68
Merge branch 'master' into SPARK-37154
zero323 Feb 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions python/pyspark/_typing.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,27 @@
# under the License.

from typing import Callable, Iterable, Sized, TypeVar, Union
from typing_extensions import Protocol
from typing_extensions import Literal, Protocol

from numpy import int32, int64, float32, float64, ndarray

F = TypeVar("F", bound=Callable)
T_co = TypeVar("T_co", covariant=True)

PrimitiveType = Union[bool, float, int, str]

NonUDFType = Literal[0]

class SupportsIAdd(Protocol):
def __iadd__(self, other: SupportsIAdd) -> SupportsIAdd: ...

class SupportsOrdering(Protocol):
def __le__(self, other: SupportsOrdering) -> bool: ...
def __lt__(self, other: SupportsOrdering) -> bool: ...

class SizedIterable(Protocol, Sized, Iterable[T_co]): ...

S = TypeVar("S", bound=SupportsOrdering)

NumberOrArray = TypeVar(
"NumberOrArray", float, int, complex, int32, int64, float32, float64, ndarray
)
12 changes: 12 additions & 0 deletions python/pyspark/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,18 @@ def setAll(self, pairs: List[Tuple[str, str]]) -> "SparkConf":
self.set(k, v)
return self

@overload
def get(self, key: str) -> Optional[str]:
...

@overload
def get(self, key: str, defaultValue: None) -> Optional[str]:
...

@overload
def get(self, key: str, defaultValue: str) -> str:
...

Comment on lines +206 to +217
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are added to clearly indicate which calls can result in None and, in turn, avoid ignores or casts in rdd.py.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qq: Maybe that is a stupid question, but I'd like to ask if
def get(self, key: str, defaultValue: Optional[str]) -> Optional[str]:
can't cover the both
def get(self, key: str, defaultValue: None) -> Optional[str]:
and
def get(self, key: str, defaultValue: str) -> str: ??

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It covers both, but doesn't capture the same relationship between arguments. With (SparkConf, str, str) -> str we know that

conf.get("foo", "42")

is str, which saves as cast / ignores / asserts not None later.

With only (SparkConf, str, Optional[str]) -> Optional[str] we still have to assert that result is not None.

(There might be other way of capturing this through type parameters, i.e. (SparkConf, str, T) -> T where T is TypeVar("T", None, str))

def get(self, key: str, defaultValue: Optional[str] = None) -> Optional[str]:
"""Get the configured value for some key, or return a default otherwise."""
if defaultValue is None: # Py4J doesn't call the right get() if we pass None
Expand Down
3 changes: 2 additions & 1 deletion python/pyspark/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
List,
NoReturn,
Optional,
Sequence,
Tuple,
Type,
TYPE_CHECKING,
Expand Down Expand Up @@ -1421,7 +1422,7 @@ def runJob(
self,
rdd: "RDD[T]",
partitionFunc: Callable[[Iterable[T]], Iterable[U]],
partitions: Optional[List[int]] = None,
partitions: Optional[Sequence[int]] = None,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use range in rdd.py, so we need generic type of collection.

allowLocal: bool = False,
) -> List[U]:
"""
Expand Down
Loading