Add type annotation into untyped functions automatically

[ghost]

Pull Request Check List

• A news fragment is added in news/ describing what is new.
• Test cases added for changed code.

Describe what you have changed in this PR

• Add type annotation into untyped functions
• It is because it reduces bug by static type checking such as mypy
• I achieve this by automatically so that I don't missing untyped def
• Related to Typing #261

Remarks

I believe adding type annotations into all variables is anti-pattern.

It is because,

• It occurs many circular import error because python is structured only as parent-to-child. If you import type from other module, but that other module requires others, then it errors out.
• mypy also doesn't have any kind of options to disallow untyped variable. (Ref: https://stackoverflow.com/questions/58881389/is-there-a-way-for-mypy-to-warn-about-missing-explicit-typing-when-declaring-a-v). Instead of it, mypy recommends to disallow untyped function.

So, I only add typing into functions.

How to use pyannotate

pyannotate is made by Guido and his team in dropbox. It's easy and powerful to use compared to monkeytype by instagram because monkeytype doesn't support pytest integration officially

Add below lines into tests/conftest.py. (I don't add this code in this PR, because I think this requires only this time.)

def pytest_collection_finish(session):
    from pyannotate_runtime import collect_types
    collect_types.init_types_collection()

@pytest.fixture(autouse=True)
def collect_types_fixture():
    from pyannotate_runtime import collect_types
    collect_types.start()
    yield
    collect_types.stop()

def pytest_sessionfinish(session, exitstatus):
    from pyannotate_runtime import collect_types
    collect_types.dump_stats("type_info.json")

Then run pytest.
It generates the typing informations of all functions what pytest run through into type_info.json.
It is based on sys.setprofile.

When you want to add typing into actual code, run below.

pdm run pyannotate -w --py3 ./**/*.py

The code will have typing automatically.

[ghost]

Actually, it fails with 1 error currently.

It looks not fatal, but I want to fix in ideal way possibly by another PR.

After it's solved, I want to add mypy into .pre-commit-config.yaml.

[frostming]

Not true, it turns out there are ~500 errors to fix, which is a huge amount

[ghost]

It was actually blocking mypy.
After I excluded that directory -> and removed tests from mypy checking, it shows all errors correctly.

I first of all fixed "missing a type" errors in this PR.
It used to cause "missing a type" errors more than 50 functions, Now all errors of them are gone.

[frostming]

Just a quick look, there must be many more untyped testing functions. But I don't see it necessary to annotate testing funcs, it doesn't seem conventional.

[ghost]

Oh I'm sorry for having you review many things.
I'm going to check the codes where you pointed out again.

[ghost]

there must be many more untyped testing functions

Sorry, I noticed that now.
For example, test_parse_local_directory_metadata is not typed, but pyannotate found it for sure.

type_info.json.log

Imade mistake when applying, so I'm going to add more fix commit in this PR.

[frostming]

Too much to review, I suggest some of them, others should be similar

[frostming]

@ulwlu thanks for your effort in adding typing, but can we just leave tests/ untyped? Most famous projects seem to not annotate tests/ directory.

This will dramatically reduce the work to review this PR.

[ghost]

@frostming

Too much to review

Hi, I am sorry for causing you trouble by adding to many files.

I removed typing from tests, and fixed the all codes with based on what you pointed out.

I believed this would make the project more maintainable, and I found myself hooked into adding typing, but if there are parts you don't want, feel free to point out.

[ghost]

@frostming
Hi, thank you for reviewing.
I fixed where you pointed out and other codes where have similar problem.

I also wrote one question.

Some codes are still using pip._vendor such as from pip._vendor.requests.models import Response, and if you'd like to move all of them into pdm.models, pleate tell me.

[frostming]

That is really huge, if you don't mind I will merge it to a temporary branch and do more modifications based on that.

[ghost]

Sure, thank you very much.

Since I see you start applying suggestions, I stop modifying here.

[frostming]

Thank you, since this PR needs to be resolved first otherwise it will likely cause many merge conflicts.