config.yaml - problem with reading variable (v.1.16.3)

[kamandir]

Hello!

after the update from 15.1 to 16.3 all DAGs fail to start.
The error message is like the following:
time=2025-02-24T16:25:23.800+01:00 level=ERROR msg="failed to initialize log file" DAG=my_job_01 err="failed to setup log directory: failed to initialize directory ${-logs/my_job_01: mkdir ${DAGU_HOME}/stdout-logs: permission denied"
Error: failed to initialize log file for DAG my_job_01: failed to setup log directory: failed to initialize directory ${DAGU_HOME}/stdout-logs/my_job_01: mkdir ${DAGU_HOME}/stdout-logs: permission denied

workarouded it by changing the line in the config.yaml:

from: logDir: ${DAGU_HOME}/stdout-logs
to: logDir: /some/folder/DAGU_HOME/stdout-logs

please help to get this fixed.

PS: folders are there, perm-s are fine, old version works fine

thanks and appreciation for creating a great scheduler!

[yottahmd]

Hey @kamandir, thank you so much for reporting the issue. Sorry for the inconvenience, I'll try to fix this asap!

[yottahmd]

@kamandir I'm trying to reproduce the issue in v1.16.3, but I haven’t been able to get it to fail yet.
Would you mind sharing a minimal config.yaml that actually triggers the problem?
I've tried this setup, but everything seems to work fine on my end:

Here's what I did:

# config.yaml
logDir: ${TMP_LOG_DIR}/stdout

Then I ran this command:

TMP_LOG_DIR=/tmp/foo go run ./cmd start /Users/hamadayouta/dev/dagu/basic.yaml

[kamandir]

@yottahmd it is nothing special:

$ cat config.yaml

# directory path to save logs from standard output
logDir: ${DAGU_HOME}/stdout-logs/
#logDir: /opt/dagu/DAGU_HOME/stdout-logs/

# history retention days (default: 30)
histRetentionDays: 30

# Eamil notification settings
mailOn:
  failure: true
  success: false

# SMTP server settings
smtp:
  host: "*******.com"
  port: "25"
  username: ""
  password: ""

errorMail:
  from: "***************@***.com"
  to: "***************@***.com"
  prefix: "[JOB Error]"

infoMail:
  from: "***************@***.com"
  to: "***************@***.com"
  prefix: "[JOB Info]"

# user parameters
params:
  LANG=en_US.UTF-8

and the variable is set on the Linux (OS) level and it was working before the update to 16:

$ echo  ${DAGU_HOME}
/opt/dagu/DAGU_HOME

maybe it is some 'magic' combination of my environment settings and this DAGu version...
I don't find this problem as critical and can workaround it, so maybe one day the root cause will be surfaced.

anyway, thanks for looking into it!

Here are some fresh lines from the logs after i changed the config back to a variable and tried to start DAG from WEB interface:

time=2025-02-25T20:27:08.969+08:00 level=INFO msg="Request: GET /api/v1/dags"
time=2025-02-25T20:27:08.974+08:00 level=INFO msg="Request: POST /api/v1/dags/temp_check_python_path"
time=2025-02-25T13:27:08.983+01:00 level=ERROR msg="failed to initialize log file" DAG=temp_check_python_path err="failed to setup log directory: failed to initialize directory ${DAGU_HOME}/stdout-logs/temp_check_python_path: mkdir ${DAGU_HOME}/stdout-logs: permission denied"
Error: failed to initialize log file for DAG temp_check_python_path: failed to setup log directory: failed to initialize directory ${DAGU_HOME}/stdout-logs/temp_check_python_path: mkdir ${DAGU_HOME}/stdout-logs: permission denied
time=2025-02-25T20:27:08.984+08:00 level=ERROR msg="DAG start operation failed" err="exit status 1"
time=2025-02-25T20:27:11.278+08:00 level=INFO msg="Request: GET /api/v1/dags"

[yottahmd]

@kamandir Thanks for sharing those extra details! That really helps. I just spotted an issue; if you specify logDir field in DAG file (individual DAGs or base.yaml). I'll fix this right away!

[yottahmd]

@kamandir Fixed in v1.16.4! Could you take a look when you have a moment? Thanks a lot!