--environment <docker|systemd> takes precedence over "logging.to_files: true"

[martinmorch-rsyd]

Edit

Since #41186 got merged, --environment <docker|systemd> is taking precedence over logging.to_files: true. The old behaviour was for the logging configuration in the configuration file to take precedence over the --environment flag.

For reference, v8.14.3 is the latest release with the correct behaviour of the --environment CLI argument.

More details on #41767 (comment)

Original post

Filebeat versions of 8.15.4 and higher no longer logs to "logging.files.path" when "logging.to_files: true" is set, unless "logging.to_stderr: false" is explicitly set.

In versions <=8.15.3, setting "logging.to_stderr: false" was not required for "logging.to_files: true" to be honored.

• Filebeat >=8.15.4:
• Operating System: Rocky Linux 8.10, Ubuntu 22.04.5
• Steps to Reproduce:

1. Run Filebeat version >=8.15.4 with the following logging configuration:

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640

2. Log files are not created under "logging.file.path".
3. Revert to running Filebeat <=8.15.3 with the same configuration.
4. Log files are created under "logging.file.path".

I see nothing in the release notes about this behavioral change, is it expected?

[amardeep-s]

I am facing the same issue on all nodes where Filebeat was upgraded.

• Filebeat >=8.15.4
• Operating System: Rocky Linux 8.10, Rocky Linux 9

Spend hours to troubleshoot and trying different combinations, despite no changes being made in filebeat.yml before upgrading filebeat.

The current workaround is downgrade to filebeat-8.13.2 which is working as expected, or change logging.level: error instead of info.

[belimawr]

Folks, I've just tried reproducing it with v8.15.4, however it does not reproduce :/

The filebeat.yml I used is:

filebeat.inputs:
  - type: benchmark
    eps: 1
    message: "foo bar"

output.discard:
  enabled: true

logging.level: info
logging.to_files: true
logging.files:
  path: /tmp/logs
  name: filebeat
  keepfiles: 7
  permissions: 0640

My execution output

tiago@millennium-falcon /tmp/filebeat-8.15.4-linux-x86_64 % ll /tmp/logs
ls: cannot access '/tmp/logs': No such file or directory
tiago@millennium-falcon /tmp/filebeat-8.15.4-linux-x86_64 % ./filebeat &
[1] 473020
tiago@millennium-falcon /tmp/filebeat-8.15.4-linux-x86_64 % ll /tmp/logs
total 12K
-rw-r----- 1 tiago tiago 12K Dec  3 17:42 filebeat-20241203.ndjson
tiago@millennium-falcon /tmp/filebeat-8.15.4-linux-x86_64 % ./filebeat version
filebeat version 8.15.4 (amd64), libbeat 8.15.4 [bb01339f5a453be79bda545a40c863353134e88a built 2024-11-07 08:46:38 +0000 UTC]
tiago@millennium-falcon /tmp/filebeat-8.15.4-linux-x86_64 % 

I only changed the path to /tmp/logs so I don't need to run Filebeat as root.

How are you folks running Filebeat? Are you passing any CLI argument?

[belimawr]

Could you folks share the output of ps aux |grep filebeat when running Filebeat?

[martinmorch-rsyd]

@belimawr

$ filebeat version
filebeat version 8.16.1 (amd64), libbeat 8.16.1 [f17e0828f1de9f1a256d3f520324fa6da53daee5 built 2024-11-14 14:57:13 +0000 UTC]
$ cat /etc/redhat-release
Rocky Linux release 8.10 (Green Obsidian)

$ ps aux | grep filebeat
root     3971550  9.7  0.2 2025520 79332 ?       Ssl  07:32   0:00 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

filebeat.yml:

logging:
  level: info
  to_files: true
  to_syslog: false
  files:
    path: /var/log/filebeat
    name: filebeat.log
    keepfiles: 3
    permissions: 0644

$ ls -la /var/log/filebeat/
total 8
drwxr-xr-x.  2 root root    6 Dec  4 07:35 .
drwxr-xr-x. 17 root root 4096 Dec  2 15:00 ..

[neeej]

I see the same issue.

$ filebeat version
filebeat version 8.16.1 (arm64), libbeat 8.16.1 [f17e0828f1de9f1a256d3f520324fa6da53daee5 built 2024-11-14 14:57:06 +0000 UTC]

$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.6.20241121"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2028-03-15"

$ ps aux | grep filebeat
root        2031  0.0  2.8 2087104 111796 ?      Ssl  10:19   0:04 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

$ rpm -qa |grep filebeat
filebeat-8.16.1-1.aarch64

filebeat.yml

logging:
  to_syslog: false
  to_files: true
  redirect_stderr: true
  files:
    path: /var/log/filebeat
    name: filebeat
    rotateeverybytes: 10485760
    keepfiles: 10
  selectors: ["*"]
  level: warning
  metrics.enabled: false
  json: true

$ ls -la /var/log/filebeat
ls: cannot access '/var/log/filebeat': No such file or directory

[belimawr]

Thanks for the information folks!

So the CLI argument --environment systemd is affecting the logging behaviour. We use --environment to tell Filebeat (well, any Beat) the environment they're running at so they can adjust their behaviour accordingly, which includes logging. There are a few environments we default to logging to stderr: docker, systemd, etc.

Some updates on our logging introduced a bug on v8.15.0 (#41118) that made Beats to not log to stderr when running on docker, systemd, etc. It got fixed recently (elastic/elastic-agent-libs#236) and released in v8.15.4, however we missed the changelog entry 🤦‍♂ and now --environment [systemd|docker] is taking precedence over setting logging.to_files: true in the configuration YAML 🤦‍♂ 🤦‍♂

[belimawr]

As a wokaround, you'll need to set logging.to_stderr: false, which disables logging to stderr and set whichever other output you want to true.

logging.level: info
logging.to_files: true
logging.to_stderr: false

I'll edit the original post to be a bit more clear about it.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[redbaron4]

@belimawr Does this mean that from now on --environment systemd will take precedence over logging configuration in config file? In that case docs will have to be updated because currently the doc page for command line options says This setting is used to select a default log output when no log output is configured

[belimawr]

@belimawr Does this mean that from now on --environment systemd will take precedence over logging configuration in config file? In that case docs will have to be updated because currently the doc page for command line options says This setting is used to select a default log output when no log output is configured

No, the precedence order should not change, we need to fix the implementation.