Systemd service race causes incorrect hostname on Amazon Linux 2

[pdehlke]

Bug Report

Describe the bug
init/systemd.in contains this:

[Unit]
Description=@FLB_PROG_NAME@
Documentation=https://docs.fluentbit.io/manual/
Requires=network.target
After=network.target

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/@FLB_OUT_NAME@
EnvironmentFile=-/etc/default/@FLB_OUT_NAME@
ExecStart=@CMAKE_INSTALL_FULL_BINDIR@/@FLB_OUT_NAME@ -c /@CMAKE_INSTALL_SYSCONFDIR@/@FLB_OUT_NAME@/@[email protected]
Restart=always

[Install]
WantedBy=multi-user.target

Unfortunately, Amazon Linux 2 still uses an old hostnamectl invocation like so:

[root@ip-10-0-0-235 ~]# systemctl cat sethostname.service

# /usr/lib/systemd/system/sethostname.service
[Unit]
Description=Set Hostname Workaround coreos/bugs#1272
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c "/usr/bin/hostnamectl set-hostname $(curl -s http://169.254.169.254/latest/meta-data/hostname)"

[Install]
WantedBy=multi-user.target
[root@ip-10-0-0-235 ~]#

This is a race condition; often times if not virtually always, fluentd starts before hostnamectl can do its thing, and any output specifiers that contain ${HOSTNAME} end up with an incorrect value.

To Reproduce

• Build a fluentbit AMI based on amazon linux 2 using packer/ansible/chef/puppet/what have you. Include fluentbit from https://packages.fluentbit.io/amazonlinux/2/
• Have an output (say to S3) that has ${HOSTNAME} in it: s3_key_format /logs/%Y/%m/%d/%H/$TAG/${HOSTNAME}/$UUID.log.gz
• Launch an instance of this AMI.
• Watch as the HOSTNAME portion of the output path is incorrect until either fluentbit is restarted or the system is rebooted.

Expected behavior
${HOSTNAME} should match the hostname configured by hostnamectl

Additional context
I worked around this by adding this to /etc/systemd/system/fluent-bit.service.d/override.conf:

[Unit]
Requires=sethostname.service
After=sethostname.service

But I don't think folks should have to do that :(

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.