You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, each Step of the Pipeline gets executed in a single process. It would be good to achieve parallelization at Step level too, i.e. one step of the pipeline uses multiple processes to produce outputs faster. This could be really beneficial for steps that are slow, like GenerateEmbeddings, which usually causes a bottleneck in the pipelines.
The text was updated successfully, but these errors were encountered:
Currently, each
Step
of thePipeline
gets executed in a single process. It would be good to achieve parallelization atStep
level too, i.e. one step of the pipeline uses multiple processes to produce outputs faster. This could be really beneficial for steps that are slow, likeGenerateEmbeddings
, which usually causes a bottleneck in the pipelines.The text was updated successfully, but these errors were encountered: