-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PEP 626:Guarantee stability of line table during release and define API for out-of-process tools. #1555
PEP 626:Guarantee stability of line table during release and define API for out-of-process tools. #1555
Conversation
…r out-of-process tools.
|
||
:: | ||
|
||
void print_address_ranges(char *linetable, int firstlineno) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This won't work as py-spy
cannot execute code in the process being traced. Profilers that trace out-of-process must rely on static data, basically, they only are allowed to read memory chunks from the remote process, not to execute any code. There are many tools that rely on this (not only py-spy) so this is a general problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That doesn't make sense to me.
py-spy
must execute code; it's software.
Or are you saying that py-spy
cannot use PyLineTable_InitAddressRange
, etc. for licensing reasons, or similar?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Presumably py-spy is something like gdb attaching to a different process. It can easily read any data in the process, but it is not so easy to call a function. (I think gdb can, but I presume it requires borrowing a thread, and given that py-spy is a profiling tool I think it would rather not do so.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly: these tools attach to a process using ptrace
and only read memory from it using the ptrace
or process_vm_readv
system calls. They will not execute code in the remote process due to security concerns (and also because is orders of magnitude more difficult than just reading memory). The setup is basically you have process A running Python and process B running the profiler. Process B can be implemented in any other language and will not be linked against libpython.so
: is it's own separate thing. Then process B attaches to process A and reads its memory (the location of the memory and the layout are obtaining using DWARF information or hardcoded offsets, there are many techniques).
These tools can also analyze core files to get the python stack trace (as opposed to the C native stack trace) by reading the core file memory and in these core files, you cannot execute functions either (because is a core file not a living process).
As another example, https://github.com/P403n1x87/austin is another profiler tool that does the same thing but there are multiple other tools that use the same techniques such as memory profilers and custom debuggers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe you that these tools can't execute code in a different process.
But why can't they execute code in their own process, like all other programs? That's the bit I don't get.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
co_linetable
is a Python object, for now.
The layout of code objects is not part of the C-API, so might well change in future versions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Can I treat co_linetable like a PyVarObject to figure out the number of bytes to copy?
Yes, for now. It may change in future versions.
- Will the PyLineTable functions access any pointer values inside of co_linetable?
No
- Will the PyLineTable functions have any other dependencies on other code in the python runtime?
No.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that PyLineTable_InitAddressRange
expects a `char *. It knows nothing about object layout.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to clarify, the PyLineTable functions
will only access memory inside the array of bytes passed to PyLineTable_InitAddressRange
, but nothing outside of that array. It knows nothing of the original code object.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
awesome - that sounds good thanks!
it will not change during a release unless absolutely necessary for a bug fix. | ||
|
||
To reduce the work required to implement these tools, the following C struct and utility functions are provided. | ||
Note that these functions are not part of the C-API, so will be need to be linked into any code that needs to use them. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I think this line clarifies the proposal much better 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thanks @markshannon for the patience going through this with us and for all the clarifications!
…PI for out-of-process tools. (python#1555) * Guarantee stability of line table during release and implement API for out-of-process tools.
…PI for out-of-process tools. (python#1555) * Guarantee stability of line table during release and implement API for out-of-process tools.
No description provided.