Add ability to associate phase & graph with ops & time

[jxtps]

The tensorboard profiling capabilities are great, but it can be hard to tell precisely where in your computational graph the slowness comes from. To help with that, it would be fantastic if it were possible for the profiling tool to include some additional categorization of the performance data:

Associate ops & specific time spent with each phase (forward, loss, backward, optimizer) and each layer (Conv2d, BatchNorm, ReLU, etc) in the graph hierarchy.

Ideally the block diagram of the graph is hierarchically captured so that you can view ops & timings for e.g. your ResNetBlock1, then drill down to whatever constituent Conv/BN/ReLU/other blocks that may be within, recursively.

The training & inference time consumption can be very non-uniform across a graph, and having this breakdown would allow practitioners to pin-point specific problem-blocks in their networks, hopefully assisting in producing more efficient networks.

Thanks!

[bmd3k]

Hi jxtps,
The profiler plugin actually exists in a different repo now: https://github.com/tensorflow/profiler
Would you mind reopening this request there? Hopefully it'll help you get a more informed response.