Parallel tools and compiling

[nonhermitian]

Fix #706

Summary

Introduces the ability to run functions in parallel.

The following routines have been added:

• Parallel tools parallel_map.

This is supposed to replace #679.

Details and comments

parallel_map uses the multiprocessing library that is not efficient on Windows, and therefore implements a serial loop on Win platforms. In addition, it checks if the input has len(value)==1 and just returns the single execution result.

The parallel_map is smart in the sense that it checks for the QISKIT_IN_PARALLEL env flag and will not execute in parallel if this flag is 'TRUE', as set when the parallel_map is being executed. So you can nest calls to parallel_map, and only the outer one will execute in parallel. This flag is set to 'FALSE' at init, and is reset when a QISKitError is raised.

The number of processes to spawn is set automatically to the number of physical cpus on the local machine. Discovering this value on Windows requires the use of the comtypes module. Although this is an extra dependency, it is installed by default in Anaconda on Windows.

[nonhermitian]

@diego-plan9, @atilag, and @ajavadia you probably have thoughts on this PR. Please let me know.

@chunfuchen Does this look like what you aimed to do in #679? Don't want to step on your toes here.

[chunfuchen]

@nonhermitian Yes, this way is excellent :), it is what I want to do in #679 .

[atilag]

Thanks @nonhermitian
This is going to be actually a great improvement in our compilation times. It won't improve anything on Windows though but this is something we can't fix in the short term, and definitely nothing that the Python concurrent API could tackle.
As commented by @jaygambetta in #704 , let's merge #704 first and then tackle this one.
Anyway, we may need to work out some refactoring of this PR. Splitting it into two PRs could make sense because there are actually two features introduced here:

1. • Parallel compilation
2. • Jupyter notebook progress bar

So a PR with the first point using some sort of event mechanism (Pub/Sub?) to decouple the progress bar from the compilation could be an option (we can follow up on issue #706)
And another PR implementing the Progress bar into the wrapper, and, possibly, as part of a new Jupyter Notebook tooling (following up on #708?)

Does it make sense @nonhermitian?

[nonhermitian]

The refactor is fine by me. I will make a new PR for the parallel parts, and let you handle the progress bar stuff as I am sure you will want to rework that as part of the Pub/Sub framework.

This will also allow me to merge #383 with the parallel components, as that PR gives a nice way of returning the number of physical CPU cores, rather than the number of cores + hyperthreads that is returned by the multiprocessing library. Thus we can avoid dispatching too many processes.

[nonhermitian]

@atilag, This has been updated in the direction that you suggested. In addition, the HTMLProgressBar uses Jupyter widgets, and there are new magic functions to make life easier in the notebook env.

This review is a bit out of date with our current plans.

[nonhermitian]

@mtreinish, @ajavadia please take a look.

[jaygambetta]

@nonhermitian can you put all the utils inside the transpiler folder and then we are good to merge this PR. How it talks to other parts of qiskit-terra we will discuss later.