[Feature Request] Provide an API to get the type of the expressions in the AST #4868
Comments
|
If you're doing refactoring, wouldn't it be better to use lib2to3's AST? I've got something that resolves the non-dynamic names in lib2to3's AST and am (slowly) working on resolving the dynamic names (e.g., |
|
I'm interested in this - particularly for convenient inspection of source code in an IDE. Most other typed languages can support "type reveal on hover" for expressions. |
|
This is easier said than done. Of course it would be helpful if it
magically existed, but this is not a simple addition to the stdlib ast
module.
…On Sun, Jun 10, 2018 at 3:10 PM Daniel ***@***.***> wrote:
I'm interested in this - particularly for convenient inspection source
code from an IDE. Most other typed languages support a "type reveal on
hover" for expressions.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#4868 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ACwrMnsV9tamGE_09Ro8wTEGyFtWVZVOks5t7Zk2gaJpZM4TKh4B>
.
--
--Guido van Rossum (python.org/~guido)
|
|
Do you need an annotated AST, or is it sufficient to have the tokens in the file mapped to fully qualified names (FQNs) and types? I have some code for mapping all the tokens to FQNs and partial code for their type information (the main thing missing is for imports, which I'm working on but it's summer and I'm doing it in my spare time). |
|
@kamahen What do you mean by tokens? I'm guessing you refer to names/bindings or do you refer to the tokens as outputted by the That would still be very helpful, as it would allow to easily refactor calls to a modules without having to make assumptions on how it is imported. |
|
@sk- Yes, it's more-or-less the tokens as output by the Eventually, I hope to have inferred types with all the tokens, but that code is currently missing some features, such as proper handling of If you want to play around with my code, I can give you my latest version (which isn't yet on github). Are you planning on using the AST in |
|
@kamahen That'd be perfect, as I'm also using I'd be happy to play with the code you have so far. |
|
OK, let me get things into a slightly better shape, then I'll send it to you (or put it on github if it's not too awful). This week is rather busy, but hopefully some time next week. There are two parts of the code -- the first produces a simplified AST with fully qualified names (currently, it outputs in JSON); the second takes that simplified AST and figures out how to resolve (BTW, the most expensive part of the code seems to be the JSON marshaling/unmarshaling). |
|
I've pushed an interim version of my code to https://github.com/kamahen/pykythe Its outputs will take some explaining, so (assuming you want to play with it), I suggest you follow the setup instructions and run the test ( The main things that are missing:
|
|
I'm closing this since there is no concrete proposal and there doesn't seem to be much active interest in this issue. |
|
Just wanted to mention that It'd be great if we could use Mypy instead of Pyre. |
|
Hm, dmypy (mypy's daemon mode) has much of the same information available, there's just no API for it yet. We do have an experimental API that suggests the signature for an unannotated function based on how it's called ( Maybe we should develop something similar to Pyre's API? Maybe we could even just copy the same API style, to make it easier for clients to switch. |
|
Okay, let's open this since there is renewed interest.
This would a reasonable thing to have. At least most of the Pyre API features should be easy enough to implement on top of dmypy. The core team doesn't have a lot of spare cycles, but if somebody wants to look into this, I'm happy to give some help. |
|
I have made a tool to enhance import sys
a = 1
b = 2
print(a is b)Output: I am going to release it soon. |
|
@sobolevn did you release that? |
Hi @sobolevn |
I think the most relevant query from
There are many more queries in pyre. But being able to get a list of types direct from dmypy or mypy will be enough. |
This was referenced May 31, 2022
I working on something to address that But this is just a workaround, later on, I'll try to create a better and faster way to do this.
|
Hi @sobolevn , have you release it? It will be really useful for one of my projects. |
|
I've something already doing that but I forgot about this Issue. I can work on this issue and send a PR. |
rominf
pushed a commit
to rominf/LibCST
that referenced
this issue
Dec 7, 2022
This change is RFC (please read whole change message).
Add `MypyTypeInferenceProvider` as an alternative for
`TypeInferenceProvider`. The provider infers types using mypy as
library. The only requirement for the usage is to have the latest mypy
installed. Types inferred are mypy types, since mypy type system is well
designed, to avoid the conversion, and also to keep it simple. For
compatibility and extensibility reasons, these types are stored in
separate field `MypyType.mypy_type`.
Let's assume we have the following code in the file `x.py` which we want
to inspect:
```python
x = [42]
s = set()
from enum import Enum
class E(Enum):
f = "f"
e = E.f
```
Then to get play with mypy types one should use the code like:
```python
import libcst as cst
from libcst.metadata import MypyTypeInferenceProvider
filename = "x.py"
module = cst.parse_module(open(filename).read())
cache = MypyTypeInferenceProvider.gen_cache(".", [filename])[filename]
wrapper = cst.MetadataWrapper(
module,
cache={MypyTypeInferenceProvider: cache},
)
mypy_type = wrapper.resolve(MypyTypeInferenceProvider)
x_name_node = wrapper.module.body[0].body[0].targets[0].target
set_call_node = wrapper.module.body[1].body[0].value
e_name_node = wrapper.module.body[-1].body[0].targets[0].target
print(mypy_type[x_name_node])
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].fullname)
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].mypy_type.type.fullname)
# prints: builtins.list
print(mypy_type[x_name_node].mypy_type.args)
# prints: (builtins.int,)
print(mypy_type[x_name_node].mypy_type.type.bases[0].type.fullname)
# prints: typing.MutableSequence
print(mypy_type[set_call_node])
# prints: builtins.set
print("issuperset" in mypy_type[set_call_node].mypy_type.names)
# prints: True
print(mypy_type[set_call_node.func])
# prints: typing.Type[builtins.set]
print(mypy_type[e_name_node].mypy_type.type.is_enum)
# prints: True
```
Why?
1. `TypeInferenceProvider` requires pyre (`pyre-check` on PyPI) to be
installed. mypy is more popular than pyre. If the organization uses
mypy already (which is almost always the case), it may be difficult
to assure collegues (including security team) that "we need yet
another type checker". `MypyTypeInferenceProvider` requires the
latest mypy only.
2. Even though it is possible to run pyre without watchman installation,
this is not advertised. watchman installation is not always possible
because of system requirements, or because of the security
requirements like "we install only our favorite GNU/Linux
distribution packages".
3. `TypeInferenceProvider` usage requires `pyre start` command to be run
before the execution, and `pyre stop` - after the execution. This may
be inconvenient, especially for the cases when pyre was not used
before.
4. Types produced by pyre in `TypeInferenceProvider` are just strings.
For example, it's not easily possible to infer that some variable is
enum instance. `MypyTypeInferenceProvider` makes it easy:
```
[FIXME: code here]
```
Drawback:
1. Speed. mypy is slower than pyre, so is `MypyTypeInferenceProvider`
comparing to `TypeInferenceProvider`.
How to partially solve this:
1. Implement AST tree caching in mypy. It may be difficult, however
this will lead to speed improvements for all the projects that use
this functionality.
2. Implement inferred types caching inside LibCST. As far as I know,
no caching at all is implemented inside LibCST, which is the
prerequisite for inferred types caching, so the task is big.
3. Implement LibCST CST to mypy AST. I am not sure if this possible
at all. Even if it is possible, the task is huge.
2. Two providers are doing similar things in LibCST will be present,
this can potentially lead to the situation when there is a need
install two typecheckers to get all codemods from the library
running.
Alternatives considered:
1. Put `MypyTypeInferenceProvider` inside separate library (say,
LibCST-mypy or `libcst-mypy` on PyPI). This will explicitly
separate `MypyTypeInferenceProvider` from the rest of LibCST.
Drawbacks:
1. The need to maintain separate library.
2. Limited fame (people need to know that the library exists).
3. Since some codemods cannot be implemented easily without the
library, for example, `if-elif-else` to `match` converter
(it needs powerful type inference), they are doomed to not be
shipped with LibCST, which makes the latter less attractive for
end users.
2. Implement base class for inferred type, which inherits from `str`
(to keep the compatibility with the existing codebase) and
the mechanism for dynamically selecting `TypeInferenceProvider`
typechecker (mypy or pyre; user can do this via enviromental
variable). If the code inside LibCST requires just shallow type
information (so, just `str` is enough), then the code can run with
any typechecker. Ther remaining code (such as `if-elif-else` to
`match` converter) will still require mypy.
Misc:
Code does not lint in my env, by some reason `pyre check` cannot find
`mypy` library.
Related to:
* Instagram#451
* pyastrx/pyastrx#40
* python/mypy#12513
* python/mypy#4868
rominf
pushed a commit
to rominf/LibCST
that referenced
this issue
Dec 7, 2022
This change is RFC (please read whole change message).
Add `MypyTypeInferenceProvider` as an alternative for
`TypeInferenceProvider`. The provider infers types using mypy as
library. The only requirement for the usage is to have the latest mypy
installed. Types inferred are mypy types, since mypy type system is well
designed, to avoid the conversion, and also to keep it simple. For
compatibility and extensibility reasons, these types are stored in
separate field `MypyType.mypy_type`.
Let's assume we have the following code in the file `x.py` which we want
to inspect:
```python
x = [42]
s = set()
from enum import Enum
class E(Enum):
f = "f"
e = E.f
```
Then to get play with mypy types one should use the code like:
```python
import libcst as cst
from libcst.metadata import MypyTypeInferenceProvider
filename = "x.py"
module = cst.parse_module(open(filename).read())
cache = MypyTypeInferenceProvider.gen_cache(".", [filename])[filename]
wrapper = cst.MetadataWrapper(
module,
cache={MypyTypeInferenceProvider: cache},
)
mypy_type = wrapper.resolve(MypyTypeInferenceProvider)
x_name_node = wrapper.module.body[0].body[0].targets[0].target
set_call_node = wrapper.module.body[1].body[0].value
e_name_node = wrapper.module.body[-1].body[0].targets[0].target
print(mypy_type[x_name_node])
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].fullname)
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].mypy_type.type.fullname)
# prints: builtins.list
print(mypy_type[x_name_node].mypy_type.args)
# prints: (builtins.int,)
print(mypy_type[x_name_node].mypy_type.type.bases[0].type.fullname)
# prints: typing.MutableSequence
print(mypy_type[set_call_node])
# prints: builtins.set
print("issuperset" in mypy_type[set_call_node].mypy_type.names)
# prints: True
print(mypy_type[set_call_node.func])
# prints: typing.Type[builtins.set]
print(mypy_type[e_name_node].mypy_type.type.is_enum)
# prints: True
```
Why?
1. `TypeInferenceProvider` requires pyre (`pyre-check` on PyPI) to be
installed. mypy is more popular than pyre. If the organization uses
mypy already (which is almost always the case), it may be difficult
to assure collegues (including security team) that "we need yet
another type checker". `MypyTypeInferenceProvider` requires the
latest mypy only.
2. Even though it is possible to run pyre without watchman installation,
this is not advertised. watchman installation is not always possible
because of system requirements, or because of the security
requirements like "we install only our favorite GNU/Linux
distribution packages".
3. `TypeInferenceProvider` usage requires `pyre start` command to be run
before the execution, and `pyre stop` - after the execution. This may
be inconvenient, especially for the cases when pyre was not used
before.
4. Types produced by pyre in `TypeInferenceProvider` are just strings.
For example, it's not easily possible to infer that some variable is
enum instance. `MypyTypeInferenceProvider` makes it easy, see the
code above.
Drawback:
1. Speed. mypy is slower than pyre, so is `MypyTypeInferenceProvider`
comparing to `TypeInferenceProvider`.
How to partially solve this:
1. Implement AST tree caching in mypy. It may be difficult, however
this will lead to speed improvements for all the projects that use
this functionality.
2. Implement inferred types caching inside LibCST. As far as I know,
no caching at all is implemented inside LibCST, which is the
prerequisite for inferred types caching, so the task is big.
3. Implement LibCST CST to mypy AST. I am not sure if this possible
at all. Even if it is possible, the task is huge.
2. Two providers are doing similar things in LibCST will be present,
this can potentially lead to the situation when there is a need
install two typecheckers to get all codemods from the library
running.
Alternatives considered:
1. Put `MypyTypeInferenceProvider` inside separate library (say,
LibCST-mypy or `libcst-mypy` on PyPI). This will explicitly
separate `MypyTypeInferenceProvider` from the rest of LibCST.
Drawbacks:
1. The need to maintain separate library.
2. Limited fame (people need to know that the library exists).
3. Since some codemods cannot be implemented easily without the
library, for example, `if-elif-else` to `match` converter
(it needs powerful type inference), they are doomed to not be
shipped with LibCST, which makes the latter less attractive for
end users.
2. Implement base class for inferred type, which inherits from `str`
(to keep the compatibility with the existing codebase) and
the mechanism for dynamically selecting `TypeInferenceProvider`
typechecker (mypy or pyre; user can do this via enviromental
variable). If the code inside LibCST requires just shallow type
information (so, just `str` is enough), then the code can run with
any typechecker. Ther remaining code (such as `if-elif-else` to
`match` converter) will still require mypy.
Misc:
Code does not lint in my env, by some reason `pyre check` cannot find
`mypy` library.
Related to:
* Instagram#451
* pyastrx/pyastrx#40
* python/mypy#12513
* python/mypy#4868
rominf
pushed a commit
to rominf/LibCST
that referenced
this issue
Dec 7, 2022
This change is RFC (please read whole change message).
Add `MypyTypeInferenceProvider` as an alternative for
`TypeInferenceProvider`. The provider infers types using mypy as
library. The only requirement for the usage is to have the latest mypy
installed. Types inferred are mypy types, since mypy type system is well
designed, to avoid the conversion, and also to keep it simple. For
compatibility and extensibility reasons, these types are stored in
separate field `MypyType.mypy_type`.
Let's assume we have the following code in the file `x.py` which we want
to inspect:
```python
x = [42]
s = set()
from enum import Enum
class E(Enum):
f = "f"
e = E.f
```
Then to get play with mypy types one should use the code like:
```python
import libcst as cst
from libcst.metadata import MypyTypeInferenceProvider
filename = "x.py"
module = cst.parse_module(open(filename).read())
cache = MypyTypeInferenceProvider.gen_cache(".", [filename])[filename]
wrapper = cst.MetadataWrapper(
module,
cache={MypyTypeInferenceProvider: cache},
)
mypy_type = wrapper.resolve(MypyTypeInferenceProvider)
x_name_node = wrapper.module.body[0].body[0].targets[0].target
set_call_node = wrapper.module.body[1].body[0].value
e_name_node = wrapper.module.body[-1].body[0].targets[0].target
print(mypy_type[x_name_node])
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].fullname)
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].mypy_type.type.fullname)
# prints: builtins.list
print(mypy_type[x_name_node].mypy_type.args)
# prints: (builtins.int,)
print(mypy_type[x_name_node].mypy_type.type.bases[0].type.fullname)
# prints: typing.MutableSequence
print(mypy_type[set_call_node])
# prints: builtins.set
print("issuperset" in mypy_type[set_call_node].mypy_type.names)
# prints: True
print(mypy_type[set_call_node.func])
# prints: typing.Type[builtins.set]
print(mypy_type[e_name_node].mypy_type.type.is_enum)
# prints: True
```
Why?
1. `TypeInferenceProvider` requires pyre (`pyre-check` on PyPI) to be
installed. mypy is more popular than pyre. If the organization uses
mypy already (which is almost always the case), it may be difficult
to assure colleagues (including security team) that "we need yet
another type checker". `MypyTypeInferenceProvider` requires the
latest mypy only.
2. Even though it is possible to run pyre without watchman installation,
this is not advertised. watchman installation is not always possible
because of system requirements, or because of the security
requirements like "we install only our favorite GNU/Linux
distribution packages".
3. `TypeInferenceProvider` usage requires `pyre start` command to be run
before the execution, and `pyre stop` - after the execution. This may
be inconvenient, especially for the cases when pyre was not used
before.
4. Types produced by pyre in `TypeInferenceProvider` are just strings.
For example, it's not easily possible to infer that some variable is
enum instance. `MypyTypeInferenceProvider` makes it easy, see the
code above.
Drawback:
1. Speed. mypy is slower than pyre, so is `MypyTypeInferenceProvider`
comparing to `TypeInferenceProvider`.
How to partially solve this:
1. Implement AST tree caching in mypy. It may be difficult, however
this will lead to speed improvements for all the projects that use
this functionality.
2. Implement inferred types caching inside LibCST. As far as I know,
no caching at all is implemented inside LibCST, which is the
prerequisite for inferred types caching, so the task is big.
3. Implement LibCST CST to mypy AST. I am not sure if this possible
at all. Even if it is possible, the task is huge.
2. Two providers are doing similar things in LibCST will be present,
this can potentially lead to the situation when there is a need
install two typecheckers to get all codemods from the library
running.
Alternatives considered:
1. Put `MypyTypeInferenceProvider` inside separate library (say,
LibCST-mypy or `libcst-mypy` on PyPI). This will explicitly
separate `MypyTypeInferenceProvider` from the rest of LibCST.
Drawbacks:
1. The need to maintain separate library.
2. Limited fame (people need to know that the library exists).
3. Since some codemods cannot be implemented easily without the
library, for example, `if-elif-else` to `match` converter
(it needs powerful type inference), they are doomed to not be
shipped with LibCST, which makes the latter less attractive for
end users.
2. Implement base class for inferred type, which inherits from `str`
(to keep the compatibility with the existing codebase) and
the mechanism for dynamically selecting `TypeInferenceProvider`
typechecker (mypy or pyre; user can do this via enviromental
variable). If the code inside LibCST requires just shallow type
information (so, just `str` is enough), then the code can run with
any typechecker. The remaining code (such as `if-elif-else` to
`match` converter) will still require mypy.
Misc:
Code does not lint in my env, by some reason `pyre check` cannot find
`mypy` library.
Related to:
* Instagram#451
* pyastrx/pyastrx#40
* python/mypy#12513
* python/mypy#4868
rominf
pushed a commit
to rominf/LibCST
that referenced
this issue
Dec 7, 2022
This change is RFC (please read whole change message).
Add `MypyTypeInferenceProvider` as an alternative for
`TypeInferenceProvider`. The provider infers types using mypy as
library. The only requirement for the usage is to have the latest mypy
installed. Types inferred are mypy types, since mypy type system is well
designed, to avoid the conversion, and also to keep it simple. For
compatibility and extensibility reasons, these types are stored in
separate field `MypyType.mypy_type`.
Let's assume we have the following code in the file `x.py` which we want
to inspect:
```python
x = [42]
s = set()
from enum import Enum
class E(Enum):
f = "f"
e = E.f
```
Then to get play with mypy types one should use the code like:
```python
import libcst as cst
from libcst.metadata import MypyTypeInferenceProvider
filename = "x.py"
module = cst.parse_module(open(filename).read())
cache = MypyTypeInferenceProvider.gen_cache(".", [filename])[filename]
wrapper = cst.MetadataWrapper(
module,
cache={MypyTypeInferenceProvider: cache},
)
mypy_type = wrapper.resolve(MypyTypeInferenceProvider)
x_name_node = wrapper.module.body[0].body[0].targets[0].target
set_call_node = wrapper.module.body[1].body[0].value
e_name_node = wrapper.module.body[-1].body[0].targets[0].target
print(mypy_type[x_name_node])
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].fullname)
# prints: builtins.list[builtins.int]
print(mypy_type[x_name_node].mypy_type.type.fullname)
# prints: builtins.list
print(mypy_type[x_name_node].mypy_type.args)
# prints: (builtins.int,)
print(mypy_type[x_name_node].mypy_type.type.bases[0].type.fullname)
# prints: typing.MutableSequence
print(mypy_type[set_call_node])
# prints: builtins.set
print("issuperset" in mypy_type[set_call_node].mypy_type.names)
# prints: True
print(mypy_type[set_call_node.func])
# prints: typing.Type[builtins.set]
print(mypy_type[e_name_node].mypy_type.type.is_enum)
# prints: True
```
Why?
1. `TypeInferenceProvider` requires pyre (`pyre-check` on PyPI) to be
installed. mypy is more popular than pyre. If the organization uses
mypy already (which is almost always the case), it may be difficult
to assure colleagues (including security team) that "we need yet
another type checker". `MypyTypeInferenceProvider` requires the
latest mypy only.
2. Even though it is possible to run pyre without watchman installation,
this is not advertised. watchman installation is not always possible
because of system requirements, or because of the security
requirements like "we install only our favorite GNU/Linux
distribution packages".
3. `TypeInferenceProvider` usage requires `pyre start` command to be run
before the execution, and `pyre stop` - after the execution. This may
be inconvenient, especially for the cases when pyre was not used
before.
4. Types produced by pyre in `TypeInferenceProvider` are just strings.
For example, it's not easily possible to infer that some variable is
enum instance. `MypyTypeInferenceProvider` makes it easy, see the
code above.
Drawbacks:
1. Speed. mypy is slower than pyre, so is `MypyTypeInferenceProvider`
comparing to `TypeInferenceProvider`.
How to partially solve this:
1. Implement AST tree caching in mypy. It may be difficult, however
this will lead to speed improvements for all the projects that use
this functionality.
2. Implement inferred types caching inside LibCST. As far as I know,
no caching at all is implemented inside LibCST, which is the
prerequisite for inferred types caching, so the task is big.
3. Implement LibCST CST to mypy AST. I am not sure if this possible
at all. Even if it is possible, the task is huge.
2. Two providers are doing similar things in LibCST will be present,
this can potentially lead to the situation when there is a need
install two typecheckers to get all codemods from the library
running.
Alternatives considered:
1. Put `MypyTypeInferenceProvider` inside separate library (say,
LibCST-mypy or `libcst-mypy` on PyPI). This will explicitly
separate `MypyTypeInferenceProvider` from the rest of LibCST.
Drawbacks:
1. The need to maintain separate library.
2. Limited fame (people need to know that the library exists).
3. Since some codemods cannot be implemented easily without the
library, for example, `if-elif-else` to `match` converter
(it needs powerful type inference), they are doomed to not be
shipped with LibCST, which makes the latter less attractive for
end users.
2. Implement base class for inferred type, which inherits from `str`
(to keep the compatibility with the existing codebase) and
the mechanism for dynamically selecting `TypeInferenceProvider`
typechecker (mypy or pyre; user can do this via enviromental
variable). If the code inside LibCST requires just shallow type
information (so, just `str` is enough), then the code can run with
any typechecker. The remaining code (such as `if-elif-else` to
`match` converter) will still require mypy.
Misc:
Code does not lint in my env, by some reason `pyre check` cannot find
`mypy` library.
Related to:
* Instagram#451
* pyastrx/pyastrx#40
* python/mypy#12513
* python/mypy#4868
Hi @sobolevn, this would be extremely useful for me and probably others. Have you released this? If not, can you release the (partial) source code? Thanks! |
|
@GideonBear no, it is too unreliable. |
@sobolevn Is it on private? https://github.com/wemake-services/typed-linter is a 404 for me. |
|
Maybe this can help you @GideonBear , https://github.com/pyastrx/pyastrx/tree/main/pyastrx/inference . Also you use this after installing pyastrx |
|
@devmessias what is your recommended approach to get the inferred type information within the AST? Is there a viable solution now? I saw that you've been busy getting related PRs approved across various projects. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Knowing the type of the expressions in the AST is useful for IDEs (and IDE plugins) to support autocompletion, for static analyzers and for refactoring tools. As a matter of fact, both
jediandpylinthave their own heuristics for type inference.In my specific use case I would like to use the type information to write a safe refactoring tool. The idea is to be able to say that you want to refactor specific methods. For example one could want to refactor
string.findintostring.index, as:where
expris any string expression, like'string',string_var,(var + 'foo'),string_var.replace(' ', '-'), etc.into
To safely do this refactoring, one needs to be able to query the type of sub expressions, given their location in the source file. Otherwise, given that
findis a common name present in many different classes, the refactoring, would blindly be applied to all of them.Note: this is the feature request version of issue #4713.
The text was updated successfully, but these errors were encountered: