Dynamic base classes are not handled

[snakescott]

repro:

• set some $BASEDIR for files
• git clone https://github.com/python/typeshed.git $BASEDIR/typeshed
• cd $BASEDIR/typeshed; git checkout f36d7a3; rm -rf third_party/2/sqlalchemy/ (note: sqlalchemy stubs are incomplete, so ignore them)
• put the following code at $BASEDIR/failure.py

from sqlalchemy import Column, String
Base = declarative_base()

class Foo(Base):
  __tablename__ = 'foo'
  key = Column(String, primary_key=True)

• install mypy 0.4.6
• run mypy --py2 -v -v -v --fast-parser -s --custom-typeshed-dir $BASEDIR/typeshed $BASEDIR/failure.py

Output is something like

failure.py:33: error: Invalid type "failure.Base"
failure.py:33: error: Invalid base class

when it should be clean

[gvanrossum]

Simpler repro (without the typeshed/sqlalchemy stuff):

def func(): pass
Base = func()
class C(Base): pass  # errors on this line

Errors (mypy -s x.py):

x.py:3: error: Invalid type "x.Base"
x.py:3: error: Invalid base class

(Update: used def func... instead if import.)

[JukkaL]

The reason this doesn't work is that the type of Base is not yet inferred during semantic analysis, and that's when we process base classes. You can add an annotation as a workaround:

from typing import Any

def func(): pass
Base = func()  # type: Any
class C(Base): pass  # now works

[gvanrossum]

Should we close this then or update the docs somewhere? Maybe the error message could also be better.

[JukkaL]

I think that we should at least see if we can make the error message better.

[snakescott]

improved error message would be helpful. I did discover # type: Any as a workaround but it is good to be aware this is a known issue.

Is there a github issue I should be tracking for the semantic analysis improvement, or just watch the release notes?

[JukkaL]

Let's use this issue to track both the error message and the underlying semantic analysis issue for now.

[Deimos]

I seem to have hit a very similar bug that's a bit more convoluted (involving Optional and importing), but probably has a related base cause. It happened to me originally with sqlalchemy's declarative_base() as well, but this was my attempt to find a simple reproduction (Python 3.6):

file1.py:

from typing import Any

Base: Any

file2.py:

from typing import Optional
from .file1 import Base

class SubClass(Base):
    pass

def optional_return_func() -> Optional[SubClass]:
    return SubClass()

def should_fail() -> int:
    test_var = optional_return_func()
    reveal_type(test_var)

    if not test_var:
        return 1

    return 'not an int'

When I run mypy on this, the result is error: Revealed type is 'builtins.None'

The expected result should be Revealed type is 'Union[file2.SubClass, builtins.None]' along with an Incompatible return value type error, but since mypy is certain that test_var is None, it considers the final line that returns a string unreachable.

Hope that makes sense, let me know if any more information would be helpful.

[Deimos]

@JukkaL sorry for the ping, but I just wanted to check and see if you think the issue I described above is similar enough that leaving it as part of this one is good, or if I should open a new, separate issue for it.

[JukkaL]

@Deimos The builtins.None issue seems to be fixed on github master -- can you verify that it works for you? It's unrelated to the original issue, but no need to create an issue if it already works.

[Deimos]

Oh, yes, it does seem to be fixed using current github master. That's great, thanks (and sorry for bothering you about something that was already fixed).

[libre-man]

But if I add a Any annotation the problem is that it thinks the class can be anything, so doing stuff like

import typing as t
base: t.Any = make_base()
class A(base): pass

def make_list() -> t.Sequence[A]:
    return A()

does not yield an error which makes anything using this class (A in this case) not typesafe anymore.

[gvanrossum]

But what would you expect? Except for a few trivial edge cases, when the base class is computed dynamically, mypy is unable to understand the structure of your class. That's why it gives an error when you use a dynamic base class -- if you shut it up using Any, literally "Any"thing can happen. :-)

[I changed the issue topic because it has nothing to do with sqlalchemy.]

[libre-man]

I understand that mypy can't determine the class, however it might be nice to give mypy a typehint of some sort. Capturing all the possibilities of the base are probably still not possible, however it would be possible to check somethings.

I suspected that doing Type[cls] would do this, but this does not work. This also feels somewhat inconsistent as

class A: pass
Base = A
Base2: Type[A] = A
class B(Base): pass
class C(A): pass
class D(Base2): pass

only gives an error for class D but passing A to a function like def e(cls: Type[A]) works as expected.

[gvanrossum]

If you have a base class that you think would make for a suitable static substitute for the dynamic base, you could try this (I am using object here for the static substitute):

base = object
def make_base(): pass
base = make_base() # type: ignore
class C(base):
  def foo(self): pass
c = C()
c.foo()
c.bar()  # Error, proving static type checking honors the `base = object`.

You might also do something with typing.TYPE_CHECKING:

from typing import TYPE_CHECKING
def make_base(): pass
if TYPE_CHECKING:
  base = object
else:
  base = make_base()
class C(base):
  def foo(self): pass
c = C()
c.foo()
c.bar()  # Error again

[ilevkivskyi]

@libre-man

This also feels somewhat inconsistent

It is important to make a distinction between type aliases and variables with type Type[...]:

B = A  # makes a type alias
C: Type[A]  # creates a variable

The point is that for aliases we know the value statically since type alias cannot be re-bound, so that it can be used in the type context (like an annotation or a base class). In contrast we don't know the value of a variable statically. Consider this example:

class C1: ...
class C2: ...

A: Type[Union[C1, C2]]

class B(A): ...

it is not clear what would this mean, since one cannot subclass a union.

I like the idea of using TYPE_CHECKING proposed by Guido, since it clearly indicates the purpose.

[w0rp]

I was looking for an existing GitHub issue for a problem I was having, and I think this might be the one. I've been trying to define types for Django where the stub files use generic types for Manager and QuerySet, even though the real types currently do not. It works pretty well for me, but I have encountered a problem with trying to define the from_queryset method, which is commonly used for creating Django Manager classes from QuerySet classes, with most of the methods copied across. The code looks roughly like this.

from typing import Any, Generic, TypeVar

ModelType = TypeVar('ModelType')


class Foo:
    pass


class QuerySet(Generic[ModelType]):
    pass


class Manager(Generic[ModelType]):
    @classmethod
    def from_queryset(cls, queryset_class: QuerySet[Any]) -> 'Manager[ModelType]':
        ...


class Bar(Manager.from_queryset(QuerySet)):
    pass

That's enough code to reproduce the problem. mypy will complain with the "invalid base class" error for class Bar.

So far, my workaround is to define Manager and QuerySet classes the old way, by defining methods in two places.

[gvanrossum]

@w0rp: *anything* with an expression that's not a simple class reference will cause that error. Do you have a specific question?

[w0rp]

No sorry, I just thought I'd mention my use case here in case any other Django users notice the same issue I did.

[boompig]

Any updates on this issue?

[germaniumhq]

This also does not work:

def model(base: Any) -> Callable[...,T]:
    class ModelProxy(base):  # fails
        # ....

While this does:

def model(base: Any) -> Callable[...,T]:
    wut: Any = base
    class ModelProxy(wut):  # works
        # ....

[JukkaL]

@germaniumhq Created #5865 to track that particular issue.

[ilevkivskyi]

Currently mypy supports writing plugins, including using get_dynamic_class_hook(), see the docs. In particular, SQLAlchemy plugin already supports the definitions like in the original example. I am not sure about the Django plugin, if it doesn't support this yet, you can open an issue on their tracker.

I don't think we need to keep this open. One should either use plugins for supported frameworks, or the if TYPE_CHECKING: ... workaround for not yet supported ones.

[revmischa]

If mypy can't parse it, it is unhelpful to say it's a type error

[w0rp]

In case others are interested in the Django type checking issues I mentioned here and they find this thread, I'd like to point to this plugin for mypy: https://github.com/mkurnikov/django-stubs I haven't started using it yet, but it should solve a lot of Django type checking issues.

[ghost]

Edit: Please disregard, I was able to use TestCase after all.

I can't get this to work:

if TYPE_CHECKING:
    test_mixin_class = TestCase
else:
    test_mixin_class = object


class CanCreateTestMixin(test_mixin_class):

I get

Invalid base class

on the class line with mypy 0.711 and the following configuration:

[mypy]
check_untyped_defs = true
disallow_untyped_defs = true
ignore_missing_imports = true
no_implicit_optional = true
warn_redundant_casts = true
warn_return_any = true
warn_unused_ignores = true

[blueyed]

@dms-cat
Is this inside of a function?

Note the workaround from Guido above does not work when wrapped in a function in general:

def foo():
    from typing import TYPE_CHECKING
    def make_base(): pass
    if TYPE_CHECKING:
      base = object
    else:
      base = make_base()
    class C(base):  # !! Variable "base" is not valid as a type
      def foo(self): pass
    c = C()
    c.foo()
    c.bar()  # Error again

Simpler:

if True:  # works
    base = object

    class C(base):
        pass


def f():
    base = object

    class C(base):  # Variable "base" is not valid as a type / Invalid base class "base"
        pass

(0.750+dev.a94e649de794ecd169a5bcc42274e5205ffbb1fb)

[micahjsmith]

For anyone who comes across this via the original report, a simple way to address sqlalchemy/mypy issues is to also import DeclarativeMeta.

from sqalalchemy.ext.declarative import declarative_base, DeclarativeMeta
Base: DeclarativeMeta = declarative_base()
...

[kevlarr]

Thanks @micahjsmith, just ran into this for the first time (on a 2-yo project, no clue why it was never reporting before). Base: Declarative = ... is way better than Base = ... # type: ignore