Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 626:Guarantee stability of line table during release and define API for out-of-process tools. #1555

Merged
merged 3 commits into from
Aug 10, 2020

Conversation

markshannon
Copy link
Member

No description provided.

@markshannon markshannon changed the title PEP 626:Guarantee stability of line table during release and implement API for out-of-process tools. PEP 626:Guarantee stability of line table during release and define API for out-of-process tools. Aug 7, 2020

::

void print_address_ranges(char *linetable, int firstlineno)
Copy link
Member

@pablogsal pablogsal Aug 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work as py-spy cannot execute code in the process being traced. Profilers that trace out-of-process must rely on static data, basically, they only are allowed to read memory chunks from the remote process, not to execute any code. There are many tools that rely on this (not only py-spy) so this is a general problem.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't make sense to me.
py-spy must execute code; it's software.

Or are you saying that py-spy cannot use PyLineTable_InitAddressRange, etc. for licensing reasons, or similar?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably py-spy is something like gdb attaching to a different process. It can easily read any data in the process, but it is not so easy to call a function. (I think gdb can, but I presume it requires borrowing a thread, and given that py-spy is a profiling tool I think it would rather not do so.)

Copy link
Member

@pablogsal pablogsal Aug 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly: these tools attach to a process using ptrace and only read memory from it using the ptrace or process_vm_readv system calls. They will not execute code in the remote process due to security concerns (and also because is orders of magnitude more difficult than just reading memory). The setup is basically you have process A running Python and process B running the profiler. Process B can be implemented in any other language and will not be linked against libpython.so: is it's own separate thing. Then process B attaches to process A and reads its memory (the location of the memory and the layout are obtaining using DWARF information or hardcoded offsets, there are many techniques).

These tools can also analyze core files to get the python stack trace (as opposed to the C native stack trace) by reading the core file memory and in these core files, you cannot execute functions either (because is a core file not a living process).

As another example, https://github.com/P403n1x87/austin is another profiler tool that does the same thing but there are multiple other tools that use the same techniques such as memory profilers and custom debuggers.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you that these tools can't execute code in a different process.

But why can't they execute code in their own process, like all other programs? That's the bit I don't get.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

co_linetable is a Python object, for now.
The layout of code objects is not part of the C-API, so might well change in future versions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Can I treat co_linetable like a PyVarObject to figure out the number of bytes to copy?

Yes, for now. It may change in future versions.

  1. Will the PyLineTable functions access any pointer values inside of co_linetable?

No

  1. Will the PyLineTable functions have any other dependencies on other code in the python runtime?

No.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that PyLineTable_InitAddressRange expects a `char *. It knows nothing about object layout.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, the PyLineTable functions will only access memory inside the array of bytes passed to PyLineTable_InitAddressRange, but nothing outside of that array. It knows nothing of the original code object.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome - that sounds good thanks!

it will not change during a release unless absolutely necessary for a bug fix.

To reduce the work required to implement these tools, the following C struct and utility functions are provided.
Note that these functions are not part of the C-API, so will be need to be linked into any code that needs to use them.
Copy link
Member

@pablogsal pablogsal Aug 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I think this line clarifies the proposal much better 👍

pep-0626.rst Show resolved Hide resolved
Copy link
Member

@pablogsal pablogsal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @markshannon for the patience going through this with us and for all the clarifications!

@markshannon markshannon merged commit 8dd402b into python:master Aug 10, 2020
@markshannon markshannon deleted the pep-626-out-of-process branch August 10, 2020 13:33
AA-Turner pushed a commit to AA-Turner/peps that referenced this pull request Aug 15, 2020
…PI for out-of-process tools. (python#1555)

* Guarantee stability of line table during release and implement API for out-of-process tools.
mnm678 pushed a commit to mnm678/peps that referenced this pull request Oct 22, 2020
…PI for out-of-process tools. (python#1555)

* Guarantee stability of line table during release and implement API for out-of-process tools.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants