-
-
Notifications
You must be signed in to change notification settings - Fork 31k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-35113: Fix inspect.getsource to return correct source for inner classes #10307
Conversation
Lib/inspect.py
Outdated
|
||
class ClassVisitor(ast.NodeVisitor): | ||
|
||
def recursive(func): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This decorator is only used once, and it is not general. I think the code would be simpler if just adds corresponding code in visit_ClassDef()
.
Lib/inspect.py
Outdated
set_top_level_class = True | ||
self.stack.append(node.name) | ||
|
||
qualname = '.'.join(self.stack) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just inline it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel this is more readable that I am joining the stack to form qualname than if self.qualname == '.'join(self.stack)
. Let me know if it needs to be inlined.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is needed if use local variables from the outer scope. Then the condition will be if qualname == '.'join(stack)
.
Next step. The classes can be nested not only in other classes, but in functions, which can be nested in classes or functions, etc.
def func123():
class class234:
pass
return class234
class class345:
def func456(self):
class class567:
pass
return class567 etc (and similar for async functions). |
Lib/inspect.py
Outdated
|
||
class ClassVisitor(ast.NodeVisitor): | ||
|
||
def __init__(self, qualname): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the class is not used outside of findsource()
you can simplify the code by removing __init__
. Inside the visitor you can just use local variables from the outer scope. This will save several lines. Add nonlocal line_number
because it is set inside the visitor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried removing the constructor and using local variables which removes use of self
as in self.stack
. But since stack is changed in three functions now I have to declare nonlocal
on three and make other changes. After conversion I think using class variables is more self-contained and I have kept them. Let me know if you need to change them to use local variables.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, you don't have to use nonlocal
with a stack, because you don't assign to this variable in these methods. Only line_number
needs it, at only in a single place. I prefer more compact code, but it is up to you using the construct and instance attributes.
Lib/inspect.py
Outdated
set_top_level_class = True | ||
self.stack.append(node.name) | ||
|
||
qualname = '.'.join(self.stack) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is needed if use local variables from the outer scope. Then the condition will be if qualname == '.'join(stack)
.
Thanks, I will work on them. It seems there is no test for similar to your examples since my PR raises OSError for your examples meanwhile master works. I guess my code breaks these cases which was handled by the previous regex based approach. def func123():
class class234:
def func(self):
pass
return class234
class class345:
def func456(self):
class class567:
pass
return class567
if __name__ == "__main__":
import inspect
print(inspect.getsource(func123()))
print(inspect.getsource(class345().func456()))
I will fix the rest of the PR comments too in next commit. |
One side effect of the change is that pydoc's Line 94 in 3e28eed
getcomments() though there is a valid comment # 124 . This causes the comment to be rendered on pydoc. In the linked test_pydoc function there was a comment which was also rendered along with the doc for A() causing test failure. So I have added an empty line between the class A and comment at 4d11679a9ab3997280a02ce6b5de37566d984444 to fix it.
import pydoc
def func123():
# 124
class class234:
pass
return class234
class class234:
pass
if __name__ == "__main__":
import inspect
print(inspect.getcomments(func123()))
print(pydoc.render_doc(func123()))
Seems there is an altered environment failure with my test I will look into it. I think I am running the asyncdef version of the test correctly at https://github.com/python/cpython/pull/10307/files#diff-0258c7716015b67c231e0d2a60ebbd31R711. Please add in your thoughts. Thanks! |
Looking at the failure and other tests I think I need to call Current test
set event loop policy in the end def test_nested_class_definition_inside_async_function(self):
import asyncio
self.assertSourceEqual(asyncio.run(mod2.func226()), 227, 229)
self.assertSourceEqual(mod2.cls227, 233, 237)
self.assertSourceEqual(asyncio.run(mod2.cls227().func234()), 235, 236)
asyncio.set_event_loop_policy(None)
|
Lib/inspect.py
Outdated
|
||
class ClassVisitor(ast.NodeVisitor): | ||
|
||
def __init__(self, qualname): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, you don't have to use nonlocal
with a stack, because you don't assign to this variable in these methods. Only line_number
needs it, at only in a single place. I prefer more compact code, but it is up to you using the construct and instance attributes.
Lib/inspect.py
Outdated
self.stack.pop() | ||
self.stack.pop() | ||
|
||
def visit_AsyncFunctionDef(self, node): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just visit_AsyncFunctionDef = visit_FunctionDef
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I will add this. I will also refactor class to use local variables. My assumption was that stack.append
requires stack variable to be nonlocal since it mutates stack
. I was wrong on that part. Thanks for clearing up and I will change it as suggested to use variables from outer scope making it more compact 👍
Lib/test/inspect_fodder2.py
Outdated
class cls205: | ||
pass | ||
class cls207: | ||
class cls208: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe use cls205
for testing classes with same name but different __qualname__
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Used cls205
in the test. Thanks.
Lib/test/inspect_fodder2.py
Outdated
# line 211 | ||
def func212(): | ||
class cls213: | ||
def func(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method is not needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed. Thanks
Lib/test/inspect_fodder2.py
Outdated
# line 225 | ||
async def func226(): | ||
class cls227: | ||
def func(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method is not needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed. Thanks
Lib/test/test_inspect.py
Outdated
self.assertSourceEqual(asyncio.run(mod2.func226()), 227, 229) | ||
self.assertSourceEqual(mod2.cls227, 233, 237) | ||
self.assertSourceEqual(asyncio.run(mod2.cls227().func234()), 235, 236) | ||
asyncio.set_event_loop_policy(None) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use addCleanup()
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems this test class inherits from unittest.Test
and self.addCleanup(asyncio.set_event_loop_policy, None)
doesn't work. Should I use asyncio.set_event_loop_policy(None)
or change the base class from which it's inherited?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried adding class TestBuggyCases(GetSourceBase, test_utils.TestCase):
but it seems it expects all the test cases to setup an event loop and gives below error. Any other better way to do this?
======================================================================
ERROR: test_with_comment_instead_of_docstring (test.test_inspect.TestBuggyCases)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_asyncio/utils.py", line 533, in tearDown
self.unpatch_get_running_loop()
File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_asyncio/utils.py", line 525, in unpatch_get_running_loop
events._get_running_loop = self._get_running_loop
AttributeError: 'TestBuggyCases' object has no attribute '_get_running_loop'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why it doesn't work? There is no unittest.Test
, this class inherits from unittest.TestCase
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, it works. I think I was having test failures and tested it during that time causing confusion that it's an environment change. Added the change as part of c6e5e2c133069bcb45568b141636b6b78e2348d3 and verified locally.
./python.exe -m test --fail-env-changed test_inspect
Run tests sequentially
0:00:00 load avg: 2.57 [1/1] test_inspect
== Tests result: SUCCESS ==
1 test OK.
Total duration: 2 sec 345 ms
Tests result: SUCCESS
@@ -0,0 +1,3 @@ | |||
:meth:`inspect.getsource` now returns correct source code for inner class | |||
with same name as module level class. class decorators are also returned as | |||
part of the source. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add "Path by ...".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added patch attribute. Thanks :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just few nitpicks.
But see also the comment on the tracker.
Lib/inspect.py
Outdated
class_visitor.visit(tree) | ||
|
||
if line_number is not None: | ||
line_number = line_number |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed. Thanks.
Lib/test/test_inspect.py
Outdated
self.assertSourceEqual(asyncio.run(mod2.func225()), 226, 227) | ||
self.assertSourceEqual(mod2.cls226, 231, 235) | ||
self.assertSourceEqual(asyncio.run(mod2.cls226().func232()), 233, 234) | ||
self.addCleanup(asyncio.set_event_loop_policy, None) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be called before the first asyncio.run()
. The purpose of using addCleanup
is that the clean up code should be executed if any code above fails.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks I moved it up before first asyncio.run
and tested the same.
Lib/inspect.py
Outdated
# it's not used for other than this part. | ||
import ast | ||
|
||
class ClassVisitor(ast.NodeVisitor): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ClassVisitor
is not very informative name. It visits not only classes. Maybe name it ClassFinder
or something like?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed to ClassFinder
and the corresponding variable below to class_finder
. Thanks.
@@ -0,0 +1,3 @@ | |||
:meth:`inspect.getsource` now returns correct source code for inner class | |||
with same name as module level class. class decorators are also returned as |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
with same name as module level class. class decorators are also returned as | |
with same name as module level class. Class decorators are also returned as |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed. Thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for your contributions
return cls213 | ||
|
||
# line 217 | ||
class cls213: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be cls218
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This to check that the inner class which has the same name cls213
inside func212
doesn't conflict with this class cls213
. Previously regex was used and hence there were cases where two classes with same name defined under different scopes might conflict with each other returning wrong results.
return cls226 | ||
|
||
# line 230 | ||
class cls226: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be cls231
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as my above comment but this is to check for async def
related code.
This to check that the inner class which has the same name cls226
inside func225
doesn't conflict with this class cls226
. Previously regex was used and hence there were cases where two classes with same name defined under different scopes might conflict with each other returning wrong results.
Lib/inspect.py
Outdated
else: | ||
line_number = node.lineno | ||
# decrement by one since lines starts with indexing by zero | ||
line_number = line_number - 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not "line_number -= 1" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No specific reason to be honest :) I will update it along with other review comments in the second round of review. Thanks.
@serhiy-storchaka The AST module takes some noticeable hit for very large files like machine generated python code with a lot of classes. The only optimization I tried was to use I just want to add it here as a known tradeoff and maybe I am benchmarking on a highly pathological case with 60k classes with 160k lines. The largest I came across in the wild was generating Python code classes from XML schema using generateDS that generated 35k line file with around 4k classes. I prefer correctness over wrong results given the trade off but if there are any optimizations I am missing then I can add them and @1st1 might also have thoughts on the approach. For general files like using
I generated a large file with around 20000 classes with each 3 layer of nesting. On an optimized build the timings were as below : with open('foo_perf.py', 'a') as f:
f.write('import inspect')
for i in range(20000):
r = '''
class cls{0}:
class cls{0}:
class cls{0}:
def func{0}(self):
pass
'''.format(i)
f.write(r)
f.write('print(inspect.getsource(cls1999.cls1999))') On master (Incorrect result but performant)
With patch (Correct result but very slow)
|
I don't think a deque has advantages over a list here. We push and pop from one end. Stopping when the first candidate is found will make finding Yes, parsing to AST is much slower than using a regex, and uses much more memory, but I don't think this is very large issue. In common cases the performance should be good enough. |
@1st1 It would be great if you can review this |
Looks good to me in general. Here're a few questions:
|
Thanks @1st1 for the review.
I have modified the PR to behave like the current implementation to return the first class definition inside conditional clauses. Added a test at https://github.com/python/cpython/pull/10307/files#diff-0258c7716015b67c231e0d2a60ebbd31R697
I guess @serhiy-storchaka brought this up since functions return decorators as part of |
It's an interesting discrepancy. For the sake of making the API consistent we might consider breaking it slightly for classes and start returning decorators too in 3.8. I honestly don't think it's going to be a huge deal. I'm curious what @serhiy-storchaka thinks about that though. |
I think it is good to start returning decorators too. |
Agreed with @serhiy-storchaka that this is a useful enhancement and makes the API consistent with functions. Since the decorator related code is reverted @1st1 please let me know on your decision so that I can add it back if needed. Thanks |
Adding decorators to classes too was an enhancement proposed by https://bugs.python.org/issue15856 . I have currently reverted the decorator related code but I think it would be good to add it back to return decorators for consistency. |
@1st1 It will be helpful if you can review the changes. I have changed the code to return decorators too to resolve https://bugs.python.org/issue15856 and also it makes it consistent with source code for functions. Not sure if this can be merged after beta release given the decorator enhancement but I would be very happy if this makes it part of 3.8 beta 1. Thanks |
…. Remove decorator for class source.
@serhiy-storchaka I have rebased the branch with latest master since it was old. It will be helpful to have your review and re-approval so that I can merge this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
I just have found a way to optimize the visitor, but it may complicate the implementation. Will try after merging this PR.
@tirkarthi: Please replace |
Thanks Serhiy, I have merged the PR. Unfortunately, I had an issue with GitHub UI that my squashed message in the textbox didn't pass through and it just added the intermediate commit messages even after edit :( |
FWIW the performance regression due to this PR made |
Previous approach used regex along with indentation of the match to return the source code. This caused below issues. This PR proposes usage of ast module to do correct parsing.
Issues
Examples of incorrect output can be found with https://bugs.python.org/issue35113.
This PR also adds returning class decorators as part of the source code instead of just returning the class definition.
Inner class example :
On master
IPython
inspect.getsource
is also used in lot of tools like IPython and pdb which also return incorrect output due to thisWith PR
Class decorator example
On master
With PR
https://bugs.python.org/issue35113