General performance testing tooling improvements

[skottmckay]

Description:
Various changes to aid performance testing

• Add access to parent node from a subgraph so details from that node can be used
• Default to level 3 optimization for onnxruntime_perf_test
• Add code to create Concurrency Visualizer markers so it's easy for anyone to use that tool.
  • Disabled by #define.
• Add support for parallel requests to onnxruntime_perf_test
  • use threadpool with one thread per concurrent request
  • keep all threads running until all requests are done
    • previously issued n concurrent requests, waited for all threads to be done, and repeated which means you're always returning to a state where there are no requests being processed by ORT before new ones are issued
  • issue the specified number of requests (-r parameter value) without multiplying by the number of concurrent requests.
• When statistics are requested write them to std::cout as well as the output file so they're easily viewed
• Add overall time value
  • when issuing concurrent requests this is meaningful

Motivation and Context
General changes based on recent performance testing needs

[snnn]

Why do you removed one level of loop?

[skottmckay]

This way we do the number of requests that the '-r' parameter specified. It's confusing to run with something like '-r 1000 -c 2' and have that send 2000 requests vs '-r 1000 -c 4' and have that send 4000 requests. Maybe 'concurrent_session_runs' should be renamed to 'concurrent_requests'.