You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
Initially we designed PVF validation host so that it is persistent. That is, results of validation is always persisted on disk be it a compiled artifact or an error status. One example is that the prepare queue doesn't even communicate the result back to the host directly but instead does it through a file, when execute worker pick it up it will read the result and return it. Even if it is not the most efficient way it is a simple one.
However, after deploying initial version of pvf validation host we pivoted to clear the cache on the startup and refill it. That made the current code structure suboptimal. Moreover, bugs like #3581 or #3044 become possible.
We can restructure the system so that the results of preparation will be stored directly in the memory. That is, instead of storing the success flag or error status in the file we will store it in the artifacts map.
However, since the compiled artifacts can be rather heavy we have to continue storing them in files.
From 10k ft view, it looks like we could do it by doing the following steps:
the prepare worker will return detailed outcome of the compilation. It will continue saving compiled artifacts into a file
the prepare pool will pass the outcome to the queue via FromPool::Concluded
in the prepare queue we handle handle_worker_concluded and pass the outcome to the host via FromQueue::Prepared
in host's handle_prepare_done we save the prepared variant and supply the outcome there.
in host's handle_execute_pvf we will have to check the artifacts preparation outcome and act on it eagerly.
The text was updated successfully, but these errors were encountered:
Initially we designed PVF validation host so that it is persistent. That is, results of validation is always persisted on disk be it a compiled artifact or an error status. One example is that the prepare queue doesn't even communicate the result back to the host directly but instead does it through a file, when execute worker pick it up it will read the result and return it. Even if it is not the most efficient way it is a simple one.
However, after deploying initial version of pvf validation host we pivoted to clear the cache on the startup and refill it. That made the current code structure suboptimal. Moreover, bugs like #3581 or #3044 become possible.
We can restructure the system so that the results of preparation will be stored directly in the memory. That is, instead of storing the success flag or error status in the file we will store it in the artifacts map.
However, since the compiled artifacts can be rather heavy we have to continue storing them in files.
From 10k ft view, it looks like we could do it by doing the following steps:
FromPool::Concluded
handle_worker_concluded
and pass the outcome to the host viaFromQueue::Prepared
handle_prepare_done
we save the prepared variant and supply the outcome there.handle_execute_pvf
we will have to check the artifacts preparation outcome and act on it eagerly.The text was updated successfully, but these errors were encountered: