Update docs to recommend creating a bind mount for airflow scheduler …

…logs. #3622
Flowminder · Apr 29, 2022 · 957aeb8 · 957aeb8
1 parent 5251046
commit 957aeb8
Show file tree

Hide file tree

Showing 5 changed files with 5 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Tables created under the cache schema in FlowDB will automatically be set to be owned by the `flowmachine` user. [#4714](https://github.com/Flowminder/FlowKit/issues/4714)
 - `Query.explain` will now explain the query even where it is already stored. [#1285](https://github.com/Flowminder/FlowKit/issues/1285)
 - `unstored_dependencies_graph` no longer blocks until dependencies are in a determinate state. [#4949](https://github.com/Flowminder/FlowKit/issues/4949)
+- Docs now recommend creating a separate bind mount for airflow scheduler logs, and include this in the secrets quickstart. [#3622](https://github.com/Flowminder/FlowKit/issues/3622)
 
 ### Fixed
 - FlowDB trigger to alter ownership of cache tables is now triggered when a flowmachine query is `store`d. [#4714](https://github.com/Flowminder/FlowKit/issues/4714)

diff --git a/docs/source/administrator/deployment.md b/docs/source/administrator/deployment.md
@@ -110,6 +110,8 @@ To run FlowETL, you will need to provide the following secrets:
 
 See also the [airflow documentation](https://airflow.apache.org/docs/stable/) for other configuration options which you can provide as environment variables.
 
+When deploying to production you are strongly advised to create a bind mount for logs generated by the airflow scheduler, as these can grow very large. This should bind mount `/opt/airflow/logs` inside the container to a directory where the flowetl user has _read-write access_. 
+
 The [ETL](management/etl/etl.md) documentation gives detail on how to use FlowETL to load data into FlowDB.
 
 ##### Sample stack files

diff --git a/flowetl/mounts/logs/.keep b/flowetl/mounts/logs/.keep
diff --git a/secrets_quickstart/flowetl.yml b/secrets_quickstart/flowetl.yml
@@ -41,6 +41,7 @@ services:
       - ${FLOWETL_HOST_PORT:?}:8080
     volumes:
       - ${FLOWETL_HOST_DAG_DIR:?}:/opt/airflow/dags:ro
+      - ${FLOWETL_HOST_LOGS_DIR:?}/opt/airflow/logs:rw
     environment:
       AIRFLOW__CORE__EXECUTOR: LocalExecutor
     networks:

diff --git a/secrets_quickstart/secrets-quickstart.sh b/secrets_quickstart/secrets-quickstart.sh
@@ -145,6 +145,7 @@ export FLOWDB_DATA_DIR=./flowdb_pgdata
 mkdir $FLOWDB_DATA_DIR || true
 export FLOWDB_ETL_DIR=./../flowetl/mounts/files/
 export FLOWETL_HOST_DAG_DIR=./../flowetl/mounts/dags/
+export FLOWETL_HOST_LOGS_DIR=./../flowetl/mounts/logs/
 
 
 # Deploy the stack