How does this compare to Snakemake?

[Hoeze]

Hi, I'd like to understand the capabilities of Prefect to decide whether it fits my needs.
Usually, I use Snakemake to build pipelines, because:

• Very easy running of python, R, jupyter notebooks or shell code
• Allows to annotate how many resources each task requires
• Easy deployment in our Slurm cluster

However, it is not very well suited for cases where the input or output of rules are not files.
For example, imagine that I want to step-wise import some dataframes into a database, because it is too big to handle at once in memory.
As I understood, it's quite simple to define a pipeline like this in Prefect:

for partition in range(100):
    "read_dataframe(partition)" -> "apply transformations" -> "store in database"

However, the things that I do not understand on Prefect:

• How can I require certain resources? E.g. that "apply transformations" requires 20GB of RAM?
  Or in case of running hyperparameter optimization for some ML model: How can I request a GPU for the training task?
• How do I set up a cluster via Slurm? Does Prefect have some support for auto-scaling, e.g. by submitting Slurm-jobs?
• How flexible is the task graph? Is it possible that tasks dynamically require additional tasks?
  (Consider e.g. a task that creates an unknown number of output files that all need to be processed individually by another task)

[Hoeze]

Is there some example of a machine learning pipeline with hyperparameter tuning built using Prefect?

[SultanOrazbayev]

I've also been trying to see if prefect can be used as a substitute for snakemake, but it seems that prefect is not as good for HPC-based workflows (so resource management, etc. are better defined in snakemake). Also, not directly related, but: #2949

[Hoeze]

Thanks for your answer @SultanOrazbayev.
In other words, I'd have to keep my tasks small enough to run them with a Dask cluster.

Do you know what would happen when two flows require the same task?
E.g. imagine a task that creates some SQL table if it does not exist yet.
Would running the flow twice at the same time then cause the table to be written 2x?

Or is Prefect smart enough to not run the same task twice with the same input in this case?

[Hoeze]

Maybe some members of Prefect (@joshmeek ?) could try answer these questions?

[SultanOrazbayev]

Prefect is smart to not run the same flow twice, it's also possible to cache results in memory or on disk.

Right now prefect seems to be geared towards applications/environments that have more or less comparable resource requirements per task, where it helps with scaling the number of executed tasks across workers. But as I understand this is less about prefect and more about the difficulty/inconvenience of setting up heterogeneous workers on dask and then tracking worker resource availability, there are people working on this, though: Quansight-Labs/dask-jobqueue#1

[dariober]

@Hoeze

However, [snakemake] is not very well suited for cases where the input or output of rules are not files.

I'm a snakemake user and I landed here by accident. I'm curious, would this pass variables between rules suit you?

Sometimes it might be handy to have a kind of persistent storage for simple values between jobs and rules. Using plain python objects like a global dict for this will not work as each job is run in a separate process by snakemake. What helps here is the PersistentDict from the pytools package.

[Hoeze]

Thanks for the hint @dariober. I did not know this one yet :)
This could be indeed quite helpful in certain cases 👍

I went a slightly different route using embedded databases (sqlite, tiledb and RocksDB) together with Ray:

• Ray allows to specify how much resources a certain task needs.
• Data can be shared in-memory without copying (Plasma object store)
• The embedded databases allow to check whether a certain key was already computed or it still needs to be done

I'm quite happy with this approach until now.
It needs some boilerplate code but it does what I want and its only requirement is a shared filesystem like NFS or CephFS.