Remove automatic type promotion of bytearray and memoryview to bytes

[erictraut]

PEP 688 aims to remove a poorly-documented "hack" in the type system, namely that bytearray and memoryview should be treated as implicit subtypes of bytes even though they have no runtime subclass relationship.

PEP 688 recommends that type checkers deprecate and remove this support. I don't see a good way to do this gradually, so I'm tempted to "rip the bandaid off" and remove the old support.

I'll do a mypy_primer run to inform the decision.

@JelleZijlstra, what are your thoughts? Too soon?

[JelleZijlstra]

I don't think it's too soon.

When I wrote that section I was thinking of a deprecation mechanism like this:

• The type checker detects when a bytearray/memoryview is passed to a position that requires bytes, and shows a warning
• After some time, the type checker removes the special case completely and shows its standard error for passing the wrong type

I don't know pyright's codebase well, but I imagine it may be hard to implement this warning in the right place, because presumably type compatibility is often checked in places where it's hard to show a warning to the user in the right place. So maybe it's fine to just make the change at once, as you suggest. python/mypy#12661 is a sample of the possible impact; it does affect some projects but it's relatively rare.

[erictraut]

Yeah, reporting errors at the point where pyright checks for type compatibility isn't possible. There are many cases where type incompatibility isn't even an error — e.g. protocol matching or overload matching.

The mypy_primer run shows that this change will have more impact than I feared. I think the options are:

1. Don't do anything for now. Allow code bases to more gradually make the change before cutting over to the new behavior.
2. Add warnings in the obvious locations such as variable and parameter assignment locations and for the most obvious type combinations where the declared type is bytes and the assigned type is bytearray or memoryview (i.e. ignoring nested types or unions).
3. Add a new configuration knob that allows users to enable/disable the new behavior. I don't like adding new configuration knobs — especially those that we know will be deprecated, so this isn't very appealing to me.
4. Go ahead with the change now. Peel the band-aid off.

I'll need to think about this a bit more. All of these are unappealing at some level.

[erictraut]

I'm going to punt on this problem for now. We can revisit in 6 to 12 months.

[erictraut]

This is now getting in the way of additional improvements and bug fixes, so I've decided to "rip the bandaid off". Developers who were relying on promotion of bytes to bytearray and memoryview will now need to explicitly specify those in the type.

[erictraut]

I've added a new configuration option disableBytesTypePromotions which is set to false by default in basic type checking mode. In strict mode, it defaults to true.

[erictraut]

This is addressed in pyright 1.1.329, which I just published. It will also be included in a future release of pylance.

[brianm78]

Note that without this setting, there can be some confusing errors due to the mismatch between the runtime behaviour vs typing assumptions. Eg. I ended up here trying to figure out why:

def encode(val: str | bytes) -> bytes:
    reveal_type(val)
    if isinstance(val, bytes):
        return val
    reveal_type(val)
    return val.encode("utf8")

Is giving:

  /tmp/a.py:2:17 - information: Type of "val" is "str | bytes"
  /tmp/a.py:5:17 - information: Type of "val" is "str | bytearray | memoryview"
  /tmp/a.py:6:16 - error: Cannot access member "encode" for type "bytearray"
    Member "encode" is unknown (reportAttributeAccessIssue)
  /tmp/a.py:6:16 - error: Cannot access member "encode" for type "memoryview"
    Member "encode" is unknown (reportAttributeAccessIssue)

Which seems to be because its assuming the supposed "bytes" argument might really be a bytearray/memoryview, so narrowing includes those types because the runtime isinstance check won't eliminate them. It may make more sense for this setting to default to on even on standard mode.