CI: Upgrade 'pyupgrade' (v2.7.4 --> v2.9.0)

[gunjan-solanki]

• closes CI upgrade pyupgrade #39523
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

---

This PR upgrades pyupgrade and also includes style/formatting changes required after upgrading pyupgrade, namely, removing unused imports (flake8), and running black and isort.

[MarcoGorelli]

Thanks @gunjan-solanki for doing this

@simonjayhawkins @Dr-Irv are we OK with making these PEP604 changes? I just saw that an issue was raised here asottile/pyupgrade#387 (comment) , I don't know if that applies to pandas (I haven't yet looked into "runtime introspection of type hints" and don't know if it's a concern which applies)

If we didn't want these rewrites, we could pass --keep-runtime-typing to pyupgrade

[gunjan-solanki]

I converted it to draft because of failing typing validation. Few other tests are also failing, possibly because I used autoflake to remove unused imports and it also ended up removing pass statements. Although I am not sure about the exact cause (Could be because of PEP604 issue @MarcoGorelli mentioned).
Will investigate further and push the fix.

[Dr-Irv]

@simonjayhawkins @Dr-Irv are we OK with making these PEP604 changes? I just saw that an issue was raised here asottile/pyupgrade#387 (comment) , I don't know if that applies to pandas (I haven't yet looked into "runtime introspection of type hints" and don't know if it's a concern which applies)

I'm not 100% sure here, but it seems that we don't want all these typing changes to happen because it would then break doing mypy checks in a Python 3.7, 3.8 and even 3.9 environment. E.g., the Union[x,y] to x | y is in Python 3.10, and that syntax wouldn't be supported in earlier versions.

So using --keep-runtime-typing on pyupgrade is probably a better way to go, IMHO.

[simonjayhawkins]

I don't think runtime introspection is an issue. I'm generally happy with these changes. This sort of change where changes are made across the codebase can cause autobackports to fail and we don't want to backport style changes. But we will need to take that hit a some point, so no reason imo not to do it sooner rather than later.

[MarcoGorelli]

E.g., the Union[x,y] to x | y is in Python 3.10, and that syntax wouldn't be supported in earlier versions.

Isn't it supported as long as you have from __future__ import annotations?

[MarcoGorelli]

I converted it to draft because of failing typing validation. Few other tests are also failing, possibly because I used autoflake to remove unused imports and it also ended up removing pass statements. Although I am not sure about the exact cause (Could be because of PEP604 issue @MarcoGorelli mentioned).
Will investigate further and push the fix.

There's a number of CI error ATM so these may be unrelated, will come back to this when CI's fixed

[Dr-Irv]

Isn't it supported as long as you have from __future__ import annotations?

Maybe. I guess we would have to test it in different python versions to be sure. My concern with this typing stuff is that if our library goes out with the latest PEP typing enhancements, and then someone is using python 3.7 and a version of mypy on their application, whether everything will work properly. I think we should stick to "least common denominator" on typing things, but I could be convinced otherwise.

[jreback]

definitely ok with the idea here. but is this syntax backwards compatible to 1.2.2 if we need to backport things?

e.g. we have a lower mypy version in 1.2.2

[MarcoGorelli]

That's a good point - maybe this should wait until 1.4.0 then, so that 1.3.x would have a modern enough version of mypy?

[gunjan-solanki]

How should I proceed?

• Should I close the PR for the time being? or
• Update the PR to pass --keep-runtime-typing to pyupgrade ?

Thanks!

[MarcoGorelli]

@gunjan-solanki based on the above (which I hadn't considered when I opened the issue, sorry!), I think we can go with the latter, i.e.

-   repo: https://github.com/asottile/pyupgrade
    rev: v2.9.0
    hooks:
    -   id: pyupgrade
        args: [--py37-plus, --keep-runtime-typing]

This argument can be removed when 1.3.0 is out and autobackports will be a bit easier

If you do

git checkout upstream/master -- .

and then make the above changes to .pre-commit-config.yaml, then I think pre-commit run pyupgrade -a should pass without it reformatting anything (so, the diff should only show .pre-commit-config.yaml)

[gunjan-solanki]

and then make the above changes to .pre-commit-config.yaml, then I think pre-commit run pyupgrade -a should pass without it reformatting anything (so, the diff should only show .pre-commit-config.yaml)

@MarcoGorelli Your guess is spot on. I tested it yesterday and can confirm no reformatting is necessary.

@gunjan-solanki based on the above (which I hadn't considered when I opened the issue, sorry!), I think we can go with the latter, i.e.

Not an issue. Happy to help. I will go ahead and update the PR :)

[MarcoGorelli]

Thanks @gunjan-solanki