Update docs to recommend creating a bind mount for airflow scheduler …

…logs. #3622
Flowminder · Jun 23, 2022 · 553da80 · 553da80
1 parent 2278d3e
commit 553da80
Show file tree

Hide file tree

Showing 5 changed files with 5 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -17,6 +17,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - In and out flows no longer return location columns with to/from suffix.
 - FlowDB now always creates a role named `flowmachine.`
 - Flowmachine will set the state of a query being stored to cancelled if interrupted while the store is running.
+- Docs now recommend creating a separate bind mount for airflow scheduler logs, and include this in the secrets quickstart. [#3622](https://github.com/Flowminder/FlowKit/issues/3622)
 
 ### Fixed
 - Flowmachine now makes the built in `flowmachine` role owner of cache tables as a post-action when a query is `store`d. [#4714](https://github.com/Flowminder/FlowKit/issues/4714)

diff --git a/docs/source/administrator/deployment.md b/docs/source/administrator/deployment.md
@@ -115,6 +115,8 @@ We recommend running FlowETL using the celery scheduler, in which case you will
 
 See also the [airflow documentation](https://airflow.apache.org/docs/stable/) for other configuration options which you can provide as environment variables.
 
+When deploying to production you are strongly advised to create a bind mount for logs generated by the airflow scheduler, as these can grow very large. This should bind mount `/opt/airflow/logs` inside the container to a directory where the flowetl user has _read-write access_. 
+
 The [ETL](management/etl/etl.md) documentation gives detail on how to use FlowETL to load data into FlowDB.
 
 ##### Sample stack files

diff --git a/flowetl/mounts/logs/.keep b/flowetl/mounts/logs/.keep
diff --git a/secrets_quickstart/flowetl.yml b/secrets_quickstart/flowetl.yml
@@ -46,6 +46,7 @@ x-flowetl-common:
 
   volumes:
     - ${FLOWETL_HOST_DAG_DIR:?Must set FLOWETL_HOST_DAG_DIR env var}:/opt/airflow/dags:ro
+    - ${FLOWETL_HOST_LOGS_DIR:?}/opt/airflow/logs:rw
 
   environment:
     &flowetl-envs

diff --git a/secrets_quickstart/secrets-quickstart.sh b/secrets_quickstart/secrets-quickstart.sh
@@ -216,6 +216,7 @@ export FLOWDB_DATA_DIR=./flowdb_pgdata
 mkdir $FLOWDB_DATA_DIR || true
 export FLOWDB_ETL_DIR=./../flowetl/mounts/files/
 export FLOWETL_HOST_DAG_DIR=./../flowetl/mounts/dags/
+export FLOWETL_HOST_LOGS_DIR=./../flowetl/mounts/logs/
 
 
 # Deploy the stack