Best Practice for Executing Native Binary Jobs at Scale

[jbertman]

Loving Prefect so far, a huge thanks to the maintainers for making it a reality.

I'm developing a system that requires executing native binaries (Golang) at scale. The initial idea was to put them in containers, develop tasks that encapsulate those containers, and execute them as necessary via Flows. But I was left wondering what the best practice is here.

We have EKS in an auto-scaling group, and a Kubernetes agent properly permissioned to execute namespaced jobs (what I assume is best used with runnamespacedjob). So, a couple questions:

• Is there a way to encapsulate a separate container as a task? And call that task from any flow?
• How do we bridge the gap between job output (e.g. JSON) and Prefect's native Pythonic outputs? Is there a way to grab kubernetes logs and convert them to Prefect Results? Or would we basically have to use native Python if we want that functionality?

I apologize if the terminology is incorrect, this is my first exposure to Prefect. Thanks in advance!

[jbertman]

So far I have a version of it working by adding the binaries as dependencies to our Docker storage inside of each flow (using the files kwarg). Then, when each task executes, it simply executes the binary and returns the output as a string (ShellTask). I'm trying to use a Dask executor to scale out those task executions. Still looking forward to other suggestions.