Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

PVF validation host: move artifacts states into memory #3720

Closed
Tracked by #3211
pepyakin opened this issue Aug 25, 2021 · 0 comments · Fixed by #3907
Closed
Tracked by #3211

PVF validation host: move artifacts states into memory #3720

pepyakin opened this issue Aug 25, 2021 · 0 comments · Fixed by #3907
Assignees

Comments

@pepyakin
Copy link
Contributor

Initially we designed PVF validation host so that it is persistent. That is, results of validation is always persisted on disk be it a compiled artifact or an error status. One example is that the prepare queue doesn't even communicate the result back to the host directly but instead does it through a file, when execute worker pick it up it will read the result and return it. Even if it is not the most efficient way it is a simple one.

However, after deploying initial version of pvf validation host we pivoted to clear the cache on the startup and refill it. That made the current code structure suboptimal. Moreover, bugs like #3581 or #3044 become possible.

We can restructure the system so that the results of preparation will be stored directly in the memory. That is, instead of storing the success flag or error status in the file we will store it in the artifacts map.

However, since the compiled artifacts can be rather heavy we have to continue storing them in files.

From 10k ft view, it looks like we could do it by doing the following steps:

  1. the prepare worker will return detailed outcome of the compilation. It will continue saving compiled artifacts into a file
  2. the prepare pool will pass the outcome to the queue via FromPool::Concluded
  3. in the prepare queue we handle handle_worker_concluded and pass the outcome to the host via FromQueue::Prepared
  4. in host's handle_prepare_done we save the prepared variant and supply the outcome there.
  5. in host's handle_execute_pvf we will have to check the artifacts preparation outcome and act on it eagerly.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants