TextIOWrapper stub does not match stdlib documentation

[SnoopJ]

The official documentation for TextIOWrapper implies that the underlying buffer is an instance of (or implements the interface of) io.BufferedIOBase, but typeshed annotates it as IO[bytes], leading to the following counterintuitive behavior:

# repro.py
import typing
from io import TextIOWrapper, BufferedIOBase

def badfunc(stream: BufferedIOBase) -> TextIOWrapper:
    return TextIOWrapper(stream)

def goodfunc(stream: typing.IO[bytes]) -> TextIOWrapper:
    return TextIOWrapper(stream)

18:12 [snoopjedi@denton ~]
$ python3 -m mypy repro.py 
repro.py:6: error: Argument 1 to "TextIOWrapper" has incompatible type "BufferedIOBase"; expected "IO[bytes]"
Found 1 error in 1 file (checked 1 source file)

This example is reduced from a question a user asked in the #python IRC channel on Libera.net, regarding this user code in rdflib.

Version information

18:12 [snoopjedi@denton ~]
$ python3 -m mypy --version
mypy 0.910
18:12 [snoopjedi@denton ~]
$ python3 -V
Python 3.8.10

[aucampia]

See also: https://stackoverflow.com/questions/58394410/why-does-mypy-complain-about-textiowrapper-receiving-gzipfile-as-argument-1

[aucampia]

Maybe related: #4146

[srittau]

I/O types are a mess. We generally recommend to use tight protocols in argument positions to avoid these kind of issues. That said, I'd be open to an experimental PR that changes the buffer type to BufferedIOBase to judge the impact this change would have.

[aucampia]

Is there some place that the relationship between IO[bytes], IO[str], BufferdIOBase, RawIOBase and StringIOBase is defined? I know IO[...] is marked for deprecation, but I'm actually not sure what the intended replacement is. I would guess it is the ...IOBase types, but this is not made explicit. I think actually the reasoning for using IO[bytes] is that maybe TextIOWrapper works fine for both RawIOBase and BufferedIOBase, and IO[bytes] is less verbose than Union[RawIOBase,BufferedIOBase]. I will try dig into this deeper sometime to confirm.

[srittau]

The general guideline is to use protocols for arguments and concrete types (derived from IO, at least for now) for return types. IO and its sub-classes TextIO and BinaryIO are "protocol-like" type-only classes, so their relationship to the other classes is a bit hard to pin down.

[JelleZijlstra]

See also python/typing#829 for some discussion of this area.

[aucampia]

Maybe somewhat related: python/mypy#11193

[aucampia]

I think maybe the root of all problems here come from #6077, if that was resolved this would be somewhat inconsequential.

[aucampia]

Given the response in #6077, I think this is then not a bug. But I think maybe some documentation to this effect would be good, as it was somewhat surprising behaviour to me at least.

[aucampia]

I think this should be closed. This problem arose because I read the open() documentation, and figured, if it says it returns io.TextIOBase, I should type things like that. It is clear to me now that I should not. I guess there won't be any simple solutions to this conundrum anytime soon though, and probably the best interim solution will come from python/typing#851 - it would be nice if the official python docs for io.*IOBase made it clear they should not be used for typing, but that will maybe be hard to achieve right now.

Thanks for the help and information.