Support visualizing tags as nodes

[nolanmar511]

The pprof CLI has some support for visualizing profiles with tags (the -tags output format allows one to see all tags in the profile, tag filtering allows one to view only or exclude portions of the profile without tags, and the graph view(s) show some tags as nodeletes), but it's not always possible to visualize which tags are associated with which stacks when using pprof.

On a few occasions now, internal users have hacked pprof to show certain tags as nodes at the root of the call stack, so it's something that has been useful. For example, if samples are tagged such that they can be associated with particular chunks of work (for example, work done for a particular URL endpoint), it can be useful to add a pseudo-node for a particular tag toward the root of the call stack to group stacks associated with a particular chunk of work in the flame graph view.

On possibility for doing this would be to introduce the following options to pprof:

• -tagroot: takes in a comma-separated list of label keys and adds the associated key/value pairs as pseudo-nodes on the stack for samples which have the listed labels. For each sample and label key, if the sample has a label with the specified key, a pseudo-node will be added. Pseudo-nodes will be added at the top of the call stack (closest to the root), with pseudo-node for the label that is first in the list being highest in the callstack and the pseudo-node for the label that is last in the list appearing just before frames in the original profile.
• -tagleaf: takes in a comma-separated list of label keys and adds the associated labels as pseudo-nodes on the stack. For each sample and label key, if the sample has a label with the specified key, a pseudo-node will be added. Pseudo-nodes will be added at the bottom of the call stack (after leaf nodes), with pseudo-node for the label that is first in the list being the child of what was the leaf in the original profile and the pseudo-node for the label that is last in the list appearing as the new leaf node.

A key-value pair with key “” and value “value” could be represented as a Function with a function name “:”, and a filename of “”.

[mhansen]

Turns out @felixge may have already implemented and documented a variant of this: https://github.com/felixge/pprofutils#labelframes. Linking these up as prior art on this issue.