TypeInferenceProvider using another tool than pyre - Jedi #451
Labels
Comments
|
I didn't try to implement the |
|
At a glance, this is really cool! I can take a deeper look a bit later |
|
I'd be happy to accept something that adds this functionality to LibCST. My only minor issue is with the name: the metadata provider doesn't really give back types but rather the names of definitions at the line/column, if I understand correctly (so you'll never get back something like |
|
That's s really great. As an extra data point, this is the feature request I reported with the mypy team python/mypy#4868. They are even considering on having a similar API. |
|
Hi all. I’m facing a similar problem. It seems to be harder than I thought. In my case, I want to annotate a XML representing a AST with type information provided by mypy. Now I trying to solve this using mypy cache files, but the cache files doesn't have enough information to relate the inferred types with tokens in the source code or nodes in the original ast. Here is an example of how a FunctionDef node is represented by a mypy cache
Is there anyone working on this issue now? |
|
Nobody's working on this AFAIK. Out of curiosity: what's preventing you from using pyre for this? |
|
Nothing, I already have pyre integration in my project https://github.com/pyastrx/pyastrx . However, most of the projects that I've worked on used mypy...and I believe most of the projects in my new job also uses mypy. |
|
I've started to write a PR for this in But mypy must approve other changes like exposing the |
rominf
pushed a commit
to rominf/LibCST
that referenced
this issue
Dec 7, 2022
This change is RFC (please read whole change message).
Add `MypyTypeInferenceProvider` as an alternative for
`TypeInferenceProvider`. The provider infers types using mypy as
library. The only requirement for the usage is to have the latest mypy
installed. Types inferred are mypy types, since mypy type system is well
designed, to avoid the conversion, and also to keep it simple. For
compatibility and extensibility reasons, these types are stored in
separate field `MypyType.mypy_type`.
Let's assume we have the following code in the file `x.py` which we want
to inspect:
```python
x = [42]
s = set()
from enum import Enum
class E(Enum):
f = "f"
e = E.f
```
Then to get play with mypy types one should use the code like:
```python
import libcst as cst
from libcst.metadata import MypyTypeInferenceProvider
filename = "x.py"
module = cst.parse_module(open(filename).read())
cache = MypyTypeInferenceProvider.gen_cache(".", [filename])[filename]
wrapper = cst.MetadataWrapper(
module,
cache={MypyTypeInferenceProvider: cache},
)
mypy_type = wrapper.resolve(MypyTypeInferenceProvider)
x_name_node = wrapper.module.body[0].body[0].targets[0].target
set_call_node = wrapper.module.body[1].body[0].value
e_name_node = wrapper.module.body[-1].body[0].targets[0].target
print(mypy_type[x_name_node])
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].fullname)
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].mypy_type.type.fullname)
# prints: builtins.list
print(mypy_type[x_name_node].mypy_type.args)
# prints: (builtins.int,)
print(mypy_type[x_name_node].mypy_type.type.bases[0].type.fullname)
# prints: typing.MutableSequence
print(mypy_type[set_call_node])
# prints: builtins.set
print("issuperset" in mypy_type[set_call_node].mypy_type.names)
# prints: True
print(mypy_type[set_call_node.func])
# prints: typing.Type[builtins.set]
print(mypy_type[e_name_node].mypy_type.type.is_enum)
# prints: True
```
Why?
1. `TypeInferenceProvider` requires pyre (`pyre-check` on PyPI) to be
installed. mypy is more popular than pyre. If the organization uses
mypy already (which is almost always the case), it may be difficult
to assure collegues (including security team) that "we need yet
another type checker". `MypyTypeInferenceProvider` requires the
latest mypy only.
2. Even though it is possible to run pyre without watchman installation,
this is not advertised. watchman installation is not always possible
because of system requirements, or because of the security
requirements like "we install only our favorite GNU/Linux
distribution packages".
3. `TypeInferenceProvider` usage requires `pyre start` command to be run
before the execution, and `pyre stop` - after the execution. This may
be inconvenient, especially for the cases when pyre was not used
before.
4. Types produced by pyre in `TypeInferenceProvider` are just strings.
For example, it's not easily possible to infer that some variable is
enum instance. `MypyTypeInferenceProvider` makes it easy:
```
[FIXME: code here]
```
Drawback:
1. Speed. mypy is slower than pyre, so is `MypyTypeInferenceProvider`
comparing to `TypeInferenceProvider`.
How to partially solve this:
1. Implement AST tree caching in mypy. It may be difficult, however
this will lead to speed improvements for all the projects that use
this functionality.
2. Implement inferred types caching inside LibCST. As far as I know,
no caching at all is implemented inside LibCST, which is the
prerequisite for inferred types caching, so the task is big.
3. Implement LibCST CST to mypy AST. I am not sure if this possible
at all. Even if it is possible, the task is huge.
2. Two providers are doing similar things in LibCST will be present,
this can potentially lead to the situation when there is a need
install two typecheckers to get all codemods from the library
running.
Alternatives considered:
1. Put `MypyTypeInferenceProvider` inside separate library (say,
LibCST-mypy or `libcst-mypy` on PyPI). This will explicitly
separate `MypyTypeInferenceProvider` from the rest of LibCST.
Drawbacks:
1. The need to maintain separate library.
2. Limited fame (people need to know that the library exists).
3. Since some codemods cannot be implemented easily without the
library, for example, `if-elif-else` to `match` converter
(it needs powerful type inference), they are doomed to not be
shipped with LibCST, which makes the latter less attractive for
end users.
2. Implement base class for inferred type, which inherits from `str`
(to keep the compatibility with the existing codebase) and
the mechanism for dynamically selecting `TypeInferenceProvider`
typechecker (mypy or pyre; user can do this via enviromental
variable). If the code inside LibCST requires just shallow type
information (so, just `str` is enough), then the code can run with
any typechecker. Ther remaining code (such as `if-elif-else` to
`match` converter) will still require mypy.
Misc:
Code does not lint in my env, by some reason `pyre check` cannot find
`mypy` library.
Related to:
* Instagram#451
* pyastrx/pyastrx#40
* python/mypy#12513
* python/mypy#4868
rominf
pushed a commit
to rominf/LibCST
that referenced
this issue
Dec 7, 2022
This change is RFC (please read whole change message).
Add `MypyTypeInferenceProvider` as an alternative for
`TypeInferenceProvider`. The provider infers types using mypy as
library. The only requirement for the usage is to have the latest mypy
installed. Types inferred are mypy types, since mypy type system is well
designed, to avoid the conversion, and also to keep it simple. For
compatibility and extensibility reasons, these types are stored in
separate field `MypyType.mypy_type`.
Let's assume we have the following code in the file `x.py` which we want
to inspect:
```python
x = [42]
s = set()
from enum import Enum
class E(Enum):
f = "f"
e = E.f
```
Then to get play with mypy types one should use the code like:
```python
import libcst as cst
from libcst.metadata import MypyTypeInferenceProvider
filename = "x.py"
module = cst.parse_module(open(filename).read())
cache = MypyTypeInferenceProvider.gen_cache(".", [filename])[filename]
wrapper = cst.MetadataWrapper(
module,
cache={MypyTypeInferenceProvider: cache},
)
mypy_type = wrapper.resolve(MypyTypeInferenceProvider)
x_name_node = wrapper.module.body[0].body[0].targets[0].target
set_call_node = wrapper.module.body[1].body[0].value
e_name_node = wrapper.module.body[-1].body[0].targets[0].target
print(mypy_type[x_name_node])
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].fullname)
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].mypy_type.type.fullname)
# prints: builtins.list
print(mypy_type[x_name_node].mypy_type.args)
# prints: (builtins.int,)
print(mypy_type[x_name_node].mypy_type.type.bases[0].type.fullname)
# prints: typing.MutableSequence
print(mypy_type[set_call_node])
# prints: builtins.set
print("issuperset" in mypy_type[set_call_node].mypy_type.names)
# prints: True
print(mypy_type[set_call_node.func])
# prints: typing.Type[builtins.set]
print(mypy_type[e_name_node].mypy_type.type.is_enum)
# prints: True
```
Why?
1. `TypeInferenceProvider` requires pyre (`pyre-check` on PyPI) to be
installed. mypy is more popular than pyre. If the organization uses
mypy already (which is almost always the case), it may be difficult
to assure collegues (including security team) that "we need yet
another type checker". `MypyTypeInferenceProvider` requires the
latest mypy only.
2. Even though it is possible to run pyre without watchman installation,
this is not advertised. watchman installation is not always possible
because of system requirements, or because of the security
requirements like "we install only our favorite GNU/Linux
distribution packages".
3. `TypeInferenceProvider` usage requires `pyre start` command to be run
before the execution, and `pyre stop` - after the execution. This may
be inconvenient, especially for the cases when pyre was not used
before.
4. Types produced by pyre in `TypeInferenceProvider` are just strings.
For example, it's not easily possible to infer that some variable is
enum instance. `MypyTypeInferenceProvider` makes it easy, see the
code above.
Drawback:
1. Speed. mypy is slower than pyre, so is `MypyTypeInferenceProvider`
comparing to `TypeInferenceProvider`.
How to partially solve this:
1. Implement AST tree caching in mypy. It may be difficult, however
this will lead to speed improvements for all the projects that use
this functionality.
2. Implement inferred types caching inside LibCST. As far as I know,
no caching at all is implemented inside LibCST, which is the
prerequisite for inferred types caching, so the task is big.
3. Implement LibCST CST to mypy AST. I am not sure if this possible
at all. Even if it is possible, the task is huge.
2. Two providers are doing similar things in LibCST will be present,
this can potentially lead to the situation when there is a need
install two typecheckers to get all codemods from the library
running.
Alternatives considered:
1. Put `MypyTypeInferenceProvider` inside separate library (say,
LibCST-mypy or `libcst-mypy` on PyPI). This will explicitly
separate `MypyTypeInferenceProvider` from the rest of LibCST.
Drawbacks:
1. The need to maintain separate library.
2. Limited fame (people need to know that the library exists).
3. Since some codemods cannot be implemented easily without the
library, for example, `if-elif-else` to `match` converter
(it needs powerful type inference), they are doomed to not be
shipped with LibCST, which makes the latter less attractive for
end users.
2. Implement base class for inferred type, which inherits from `str`
(to keep the compatibility with the existing codebase) and
the mechanism for dynamically selecting `TypeInferenceProvider`
typechecker (mypy or pyre; user can do this via enviromental
variable). If the code inside LibCST requires just shallow type
information (so, just `str` is enough), then the code can run with
any typechecker. Ther remaining code (such as `if-elif-else` to
`match` converter) will still require mypy.
Misc:
Code does not lint in my env, by some reason `pyre check` cannot find
`mypy` library.
Related to:
* Instagram#451
* pyastrx/pyastrx#40
* python/mypy#12513
* python/mypy#4868
rominf
pushed a commit
to rominf/LibCST
that referenced
this issue
Dec 7, 2022
This change is RFC (please read whole change message).
Add `MypyTypeInferenceProvider` as an alternative for
`TypeInferenceProvider`. The provider infers types using mypy as
library. The only requirement for the usage is to have the latest mypy
installed. Types inferred are mypy types, since mypy type system is well
designed, to avoid the conversion, and also to keep it simple. For
compatibility and extensibility reasons, these types are stored in
separate field `MypyType.mypy_type`.
Let's assume we have the following code in the file `x.py` which we want
to inspect:
```python
x = [42]
s = set()
from enum import Enum
class E(Enum):
f = "f"
e = E.f
```
Then to get play with mypy types one should use the code like:
```python
import libcst as cst
from libcst.metadata import MypyTypeInferenceProvider
filename = "x.py"
module = cst.parse_module(open(filename).read())
cache = MypyTypeInferenceProvider.gen_cache(".", [filename])[filename]
wrapper = cst.MetadataWrapper(
module,
cache={MypyTypeInferenceProvider: cache},
)
mypy_type = wrapper.resolve(MypyTypeInferenceProvider)
x_name_node = wrapper.module.body[0].body[0].targets[0].target
set_call_node = wrapper.module.body[1].body[0].value
e_name_node = wrapper.module.body[-1].body[0].targets[0].target
print(mypy_type[x_name_node])
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].fullname)
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].mypy_type.type.fullname)
# prints: builtins.list
print(mypy_type[x_name_node].mypy_type.args)
# prints: (builtins.int,)
print(mypy_type[x_name_node].mypy_type.type.bases[0].type.fullname)
# prints: typing.MutableSequence
print(mypy_type[set_call_node])
# prints: builtins.set
print("issuperset" in mypy_type[set_call_node].mypy_type.names)
# prints: True
print(mypy_type[set_call_node.func])
# prints: typing.Type[builtins.set]
print(mypy_type[e_name_node].mypy_type.type.is_enum)
# prints: True
```
Why?
1. `TypeInferenceProvider` requires pyre (`pyre-check` on PyPI) to be
installed. mypy is more popular than pyre. If the organization uses
mypy already (which is almost always the case), it may be difficult
to assure colleagues (including security team) that "we need yet
another type checker". `MypyTypeInferenceProvider` requires the
latest mypy only.
2. Even though it is possible to run pyre without watchman installation,
this is not advertised. watchman installation is not always possible
because of system requirements, or because of the security
requirements like "we install only our favorite GNU/Linux
distribution packages".
3. `TypeInferenceProvider` usage requires `pyre start` command to be run
before the execution, and `pyre stop` - after the execution. This may
be inconvenient, especially for the cases when pyre was not used
before.
4. Types produced by pyre in `TypeInferenceProvider` are just strings.
For example, it's not easily possible to infer that some variable is
enum instance. `MypyTypeInferenceProvider` makes it easy, see the
code above.
Drawback:
1. Speed. mypy is slower than pyre, so is `MypyTypeInferenceProvider`
comparing to `TypeInferenceProvider`.
How to partially solve this:
1. Implement AST tree caching in mypy. It may be difficult, however
this will lead to speed improvements for all the projects that use
this functionality.
2. Implement inferred types caching inside LibCST. As far as I know,
no caching at all is implemented inside LibCST, which is the
prerequisite for inferred types caching, so the task is big.
3. Implement LibCST CST to mypy AST. I am not sure if this possible
at all. Even if it is possible, the task is huge.
2. Two providers are doing similar things in LibCST will be present,
this can potentially lead to the situation when there is a need
install two typecheckers to get all codemods from the library
running.
Alternatives considered:
1. Put `MypyTypeInferenceProvider` inside separate library (say,
LibCST-mypy or `libcst-mypy` on PyPI). This will explicitly
separate `MypyTypeInferenceProvider` from the rest of LibCST.
Drawbacks:
1. The need to maintain separate library.
2. Limited fame (people need to know that the library exists).
3. Since some codemods cannot be implemented easily without the
library, for example, `if-elif-else` to `match` converter
(it needs powerful type inference), they are doomed to not be
shipped with LibCST, which makes the latter less attractive for
end users.
2. Implement base class for inferred type, which inherits from `str`
(to keep the compatibility with the existing codebase) and
the mechanism for dynamically selecting `TypeInferenceProvider`
typechecker (mypy or pyre; user can do this via enviromental
variable). If the code inside LibCST requires just shallow type
information (so, just `str` is enough), then the code can run with
any typechecker. The remaining code (such as `if-elif-else` to
`match` converter) will still require mypy.
Misc:
Code does not lint in my env, by some reason `pyre check` cannot find
`mypy` library.
Related to:
* Instagram#451
* pyastrx/pyastrx#40
* python/mypy#12513
* python/mypy#4868
rominf
pushed a commit
to rominf/LibCST
that referenced
this issue
Dec 7, 2022
This change is RFC (please read whole change message).
Add `MypyTypeInferenceProvider` as an alternative for
`TypeInferenceProvider`. The provider infers types using mypy as
library. The only requirement for the usage is to have the latest mypy
installed. Types inferred are mypy types, since mypy type system is well
designed, to avoid the conversion, and also to keep it simple. For
compatibility and extensibility reasons, these types are stored in
separate field `MypyType.mypy_type`.
Let's assume we have the following code in the file `x.py` which we want
to inspect:
```python
x = [42]
s = set()
from enum import Enum
class E(Enum):
f = "f"
e = E.f
```
Then to get play with mypy types one should use the code like:
```python
import libcst as cst
from libcst.metadata import MypyTypeInferenceProvider
filename = "x.py"
module = cst.parse_module(open(filename).read())
cache = MypyTypeInferenceProvider.gen_cache(".", [filename])[filename]
wrapper = cst.MetadataWrapper(
module,
cache={MypyTypeInferenceProvider: cache},
)
mypy_type = wrapper.resolve(MypyTypeInferenceProvider)
x_name_node = wrapper.module.body[0].body[0].targets[0].target
set_call_node = wrapper.module.body[1].body[0].value
e_name_node = wrapper.module.body[-1].body[0].targets[0].target
print(mypy_type[x_name_node])
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].fullname)
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].mypy_type.type.fullname)
# prints: builtins.list
print(mypy_type[x_name_node].mypy_type.args)
# prints: (builtins.int,)
print(mypy_type[x_name_node].mypy_type.type.bases[0].type.fullname)
# prints: typing.MutableSequence
print(mypy_type[set_call_node])
# prints: builtins.set
print("issuperset" in mypy_type[set_call_node].mypy_type.names)
# prints: True
print(mypy_type[set_call_node.func])
# prints: typing.Type[builtins.set]
print(mypy_type[e_name_node].mypy_type.type.is_enum)
# prints: True
```
Why?
1. `TypeInferenceProvider` requires pyre (`pyre-check` on PyPI) to be
installed. mypy is more popular than pyre. If the organization uses
mypy already (which is almost always the case), it may be difficult
to assure colleagues (including security team) that "we need yet
another type checker". `MypyTypeInferenceProvider` requires the
latest mypy only.
2. Even though it is possible to run pyre without watchman installation,
this is not advertised. watchman installation is not always possible
because of system requirements, or because of the security
requirements like "we install only our favorite GNU/Linux
distribution packages".
3. `TypeInferenceProvider` usage requires `pyre start` command to be run
before the execution, and `pyre stop` - after the execution. This may
be inconvenient, especially for the cases when pyre was not used
before.
4. Types produced by pyre in `TypeInferenceProvider` are just strings.
For example, it's not easily possible to infer that some variable is
enum instance. `MypyTypeInferenceProvider` makes it easy, see the
code above.
Drawbacks:
1. Speed. mypy is slower than pyre, so is `MypyTypeInferenceProvider`
comparing to `TypeInferenceProvider`.
How to partially solve this:
1. Implement AST tree caching in mypy. It may be difficult, however
this will lead to speed improvements for all the projects that use
this functionality.
2. Implement inferred types caching inside LibCST. As far as I know,
no caching at all is implemented inside LibCST, which is the
prerequisite for inferred types caching, so the task is big.
3. Implement LibCST CST to mypy AST. I am not sure if this possible
at all. Even if it is possible, the task is huge.
2. Two providers are doing similar things in LibCST will be present,
this can potentially lead to the situation when there is a need
install two typecheckers to get all codemods from the library
running.
Alternatives considered:
1. Put `MypyTypeInferenceProvider` inside separate library (say,
LibCST-mypy or `libcst-mypy` on PyPI). This will explicitly
separate `MypyTypeInferenceProvider` from the rest of LibCST.
Drawbacks:
1. The need to maintain separate library.
2. Limited fame (people need to know that the library exists).
3. Since some codemods cannot be implemented easily without the
library, for example, `if-elif-else` to `match` converter
(it needs powerful type inference), they are doomed to not be
shipped with LibCST, which makes the latter less attractive for
end users.
2. Implement base class for inferred type, which inherits from `str`
(to keep the compatibility with the existing codebase) and
the mechanism for dynamically selecting `TypeInferenceProvider`
typechecker (mypy or pyre; user can do this via enviromental
variable). If the code inside LibCST requires just shallow type
information (so, just `str` is enough), then the code can run with
any typechecker. The remaining code (such as `if-elif-else` to
`match` converter) will still require mypy.
Misc:
Code does not lint in my env, by some reason `pyre check` cannot find
`mypy` library.
Related to:
* Instagram#451
* pyastrx/pyastrx#40
* python/mypy#12513
* python/mypy#4868
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
[What follows is probably heresy given that Pyre is another Instagram project, so please don't "throw me to the pyre" 🔥 - I mean no offence to that project]
I've found setting up a working pyre environment somewhat painful (building watchman from source, then getting a core dump because my project path was too long, etc.) and the documentation in LibCST of how to actually setup a
TypeInferenceProviderandFullRepoManagerto be lacking an example. Indeed, the best I've found was a screenshot of a notebook in #179, which I've diligently transformed into indexable form below (for my future self who wants to be able to google an example of doing it):As a result, I've also looked at other means of getting inference data given a node...
Jedi's Script.inferis an interesting and seemingly simple option despite Jedi and LibCST working on a different level (a Script in Jedi has the ability to look through a virtual environment / PYTHONPATH to find references etc., much like Pyre and theFullRepoManagerconcept in LibCST).A quick prototype later, I have a means to get hold of the Jedi inference for a node through a metadata provider (I think this is a testament to the LibCST code that this is so simple to do 👍):
Which is used as:
Output:
Given the knowledge that Jedi and LibCST are both using parso under the hood, I didn't look into trying to avoid multiple parsing stages (performance isn't so critical to me, and the performance was good enough).
I just wanted to write this down so that others can benefit - I don't expect this will make its way into the LibCST codebase. (please feel free to close the issue!)
The text was updated successfully, but these errors were encountered: