Dependency management with a stored_as_script
storage backend
#4172
Unanswered
sm-Fifteen
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have a number of machines I can't install Docker on that I would like to use as agents for my prefect setup. I'm using a git (Bitbucket) storage backend and just realized that none of the
stored_as_script
storage backends appear to have any way to handle dependencies. Even something really simple:Even if I happen to have
cx_Oracle
installed on all my agents (which is easy to forget), Bitbucket storage only fetches the actual file the flow is defined in, so theOracleFetch
task I'm defining in thisshared.tasks.oracle
module is not going to be available unless I have--import-path
set as a launch option on all my agents and then separately make sure to keep that separate module (which I happen to keep in the same git repository, though it could also be a git submodule) up to date in case I make any changes to it. If I need to install all the pip dependencies and keep those up to date as well, that's an extra layer of maintenance on top.The Docker storage backend already lets you specify additional
files
to include andpython_dependencies
to preload, so it doesn't have that issue.Besides copy-pasting those shared tasks in any flow that might need them or getting them upstreamed, is there any way to currently manage the runtime dependencies of a flow in Prefect without using Docker?
EDIT: I get the feeling issues #3513 and #3859 are among the missing parts of the documentation that would be helpful, here.
EDIT2: See also #4175
Beta Was this translation helpful? Give feedback.
All reactions