mypy does not honour open() overrides from typeshed.

[aucampia]

Bug Report

It seems like mypy is not honoring the typeshed Path.open overrides exactly, as from these I would expect mypy to think the values returned by open was:

• io.TextIOWrapper for .open("w")
• io.BufferedWriter for .open("wb")
• io.FileIO for .open("wb", buffering=0)
  But instead, mypy thinks the values returned are:
• typing.TextIO for .open("w")
• typing.BinaryIO for .open("wb")
• typing.BinaryIO for .open("wb", buffering=0)

The same is the case for the open() builtin.

I'm reporting this as a mypy issue, because the typing in typeshed looks right to me for the most part, and I can't even tell from typeshed where mypy would be getting typing.TextIO from. I will however report a seperate issue against typeshed for mixing the io. and typing.IO heirarchies in return types, as this is maybe where typing.BinaryIO comes from, but even so the other overrides should take precedence as far as I can tell.

To Reproduce

The problem can be seen when running mypy on the following code which contain unit tests which run with no errors:

import io
import pathlib
import tempfile
import typing
import unittest


class TestOpen(unittest.TestCase):
    def setUp(self) -> None:
        self._tmp_path = tempfile.TemporaryDirectory()
        self.tmp_path = pathlib.Path(self._tmp_path.name)
        self.tmp_file = self.tmp_path / "file"

    def tearDown(self) -> None:
        self._tmp_path.cleanup()

    def test_open_text_stream(self) -> None:
        with self.tmp_file.open("w") as text_stream:
            text_io: typing.TextIO = text_stream  # noqa: F841
            text_io_base: io.TextIOBase = text_stream  # noqa: F841
            assert isinstance(text_stream, io.TextIOBase)

    def test_open_buffered_stream(self) -> None:
        with self.tmp_file.open("wb") as buffered_stream:
            binary_io: typing.BinaryIO = buffered_stream  # noqa: F841
            buffered_io_base: io.BufferedIOBase = buffered_stream  # noqa: F841
            assert isinstance(buffered_stream, io.BufferedIOBase)

    def test_open_raw_stream(self) -> None:
        with self.tmp_file.open("wb", buffering=0) as raw_stream:
            binary_io: typing.BinaryIO = raw_stream  # noqa: F841
            raw_io_base: io.RawIOBase = raw_stream  # noqa: F841
            assert isinstance(binary_io, io.RawIOBase)


if __name__ == "__main__":
    unittest.main()

Expected Behavior

I expect mypy to not find any errors in the shared code.

Actual Behavior

Mypy reports the following type errors:

$ poetry run mypy --show-error-codes --show-error-context  test_open_ut.py
test_open_ut.py: note: In member "test_open_text_stream" of class "TestOpen":
test_open_ut.py:20: error: Incompatible types in assignment (expression has type "TextIO", variable has type "TextIOBase")  [assignment]
test_open_ut.py:21: error: Subclass of "TextIO" and "TextIOBase" cannot exist: would have incompatible method signatures  [unreachable]
test_open_ut.py: note: In member "test_open_buffered_stream" of class "TestOpen":
test_open_ut.py:26: error: Incompatible types in assignment (expression has type "BinaryIO", variable has type "BufferedIOBase")  [assignment]
test_open_ut.py:27: error: Subclass of "BinaryIO" and "BufferedIOBase" cannot exist: would have incompatible method signatures  [unreachable]
test_open_ut.py: note: In member "test_open_raw_stream" of class "TestOpen":
test_open_ut.py:32: error: Incompatible types in assignment (expression has type "BinaryIO", variable has type "RawIOBase")  [assignment]
test_open_ut.py:33: error: Subclass of "BinaryIO" and "RawIOBase" cannot exist: would have incompatible method signatures  [unreachable]

Your Environment

• Mypy version used: 0.910
• Mypy command-line flags: --show-error-codes --show-error-context
• Mypy configuration options from mypy.ini (and other config files):

  [mypy]
  # Kept as seperate config file as some plugins don't support pyproject.toml
  # (e.g pydantic.mypy)
  # https://mypy.readthedocs.io/en/stable/config_file.html
  python_version = 3.7
  strict = True
  warn_unreachable = True
  warn_unused_configs = True
  

• Python version used: 3.7
• Operating system and version: Fedora 34

The code for this can be found here

[erictraut]

Mypy has custom logic for open that predates the current typeshed overloads. This custom logic (implemented in mypy/plugins/default.py/open_callback) appears to be causing the problem. I think the right fix is to delete this old special-case logic and simply use the type information in typeshed.

[aucampia]

Thanks for the heads up @erictraut, in that case I guess the "bug" is more specifically that mypy does not think objects returned by open() can be instances of io.{RawIOBase,TextIOBase,BufferedIOBase}, but I guess that is maybe a problem which goes beyond mypy, and I will open a question regarding this in the python/typing community to get some clarity on this.

This behaviour would not be a problem if the io.IOBase type hierarchy was connected to the typing.IO type hierarchy, but this is not the case at the moment.

[aucampia]

See also python/typeshed#6076

[Akuli]

This would likely be fixed by #9275, but it hasn't gotten merged for some reason.

[hauntsaninja]

Yeah, Akuli is right that that (at this point quite old) PR will fix.
I think there were concerns about what the change would do to Dropbox's codebase. I'm loathe to do anything that will further slow mypy's release cycle, but maybe I'll merge after the next release.