Add Host Manager

[gcatron]

Description: This is a resurrected PR for the HostManager. It is almost ready for merging, once Executor, Partitioner, and Provisioner are merged this will be updated and be ready.
The HostManager will serve as the entry point for execution calls and handle orchestrating the rest of the runtime components.
Testing: Will write Unit Test before merge.
Documentation: N/A
Related to #2045

[nadavrot]

Why are you using std::string to represent nodes? Use pointers or something that can be uniqued.

[gcatron]

Strings are being used because that's the key to the map that stores the nodes. I could use pointers instead and then pointer->name to get the iterator to remove. Using a string seemed cleaner.

[nadavrot]

Please make sure to squash the commits.

[jackm321]

@gcatron this is looking good, can't wait to have this interface to the runtime!

[rdzhabarov]

CMake changes missing
Need to look into concurrency issues.

[rdzhabarov]

We really need multithreaded tests for those methods that can be executed concurrently (potentially enable thread sanitizer to catch issues).

I'm going to unblock the progress here, but need better verification around concurrency here in general.

[jackm321]

Looking good! How close are we to this being in a working state atm? Asking so I can judge whether I should parallelize some more onnxifi work or just wait til this is in.

[jackm321]

does this lock need to be held while callback() is being called?

[gcatron]

In the callback we are good since activeRequestCount_ is atomic. It's needed here since we are accessing roots_.

[jackm321]

does lock need to be held while stopping devices?

[gcatron]

For the device no, but this way we can't add a function to a device we are stopping.

[gcatron]

I currently have the concurrent test disabled. This is because the compiled function is owned by the provisioner not the DeviceManager. Since there's no callback on evictNetwork The HostManager can free the compiledFunction out from under the DeviceManager. We either need to add a callback/make evict blocking, or move ownership of the CompiledFunction to the deviceManager.

[nickgg]

@gcatron I think it makes sense to add a callback to the evict, we can probably use the same one from addNetwork.

[gcatron]

@gcatron I think it makes sense to add a callback to the evict, we can probably use the same one from addNetwork.

That sounds good to me. I created #2357 once we do that we should be able to re-enable the concurrent test.