Programmatic Flow Orchestration

[iamyojimbo]

Hi all!

Firstly, just have to say, I just got my hands on Prefect and really enjoying it so far!

I have just created my first couple of flows, one which downloads a file from the web and saves in Google Storage, and another one that imports that file to Google BigQuery.

Now, I have a bunch of files around the web that I do this for. Our current setup could be defined as an array of each of these respective mappings.

Web to Google Cloud Storage Flow

{
  "url": ....
  "gs_key": ...
}

Google Cloud Storage to BigQuery Flow

{
  "gs_key": ....
  "table_id": ...
  "data_types: {
     "col1": "INT64"
     "col2": "FLOAT"
  }
}

My goal is to have this orchestration version controlled.

My question is, what is the best way to describe this setup within Prefect?

• Do I create one big flow each where the parameters are an array of each of the mappings?
• Do I define these parameters in some config file, import them, and then loop over them to register a discrete flow per config (i.e. if I were downloading and importing 5 files from he web, then I'd have 10 flows)?
• Do I register the simple flows, and then programatically run the flows with the parameters from the config file.
• Is there some other way? Have I misunderstood Prefect?