[Filebeat] Can't manage to get Journald logs

[johnny550]

Hello.
Been banging my head for a while around this issue. I would truly appreciate any helping hand.
Using Filebeat, I cant get logs using the type JOURNALD offered by filebeat.

ENV
OS: FCOS 38
Systemd version:

systemd 249 (v249.9-1.fc35)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

Filebeat version: 8.5.3, updated to 8.10.0
Platform: Kubernetes 1.28.2
Filebeat config

filebeat.inputs:
  - type: journald
    id: all-journald
    seek: tail
    paths:
    - '/var/log/journal'
    - '/run/log/journal'
    fields:
      geap_log_group: my_journal_logs

name: $NODE_NAME
...

output:
  logstash:
    hosts:
      - "host_1"
      - ...
    compression_level: 9
    worker: 3
    loadbalance: true
    bulk_max_size: 2048

Logs I get are unfortunately not very explicit. No mention of journald except with the following lines

{"log.level":"warn","@timestamp":"2023-10-23T06:33:10.221Z","log.logger":"input","log.origin":{"file.name":"v2/loader.go","file.line":91},"message":"EXPERIMENTAL: The journald input is experimental","service.name":"filebeat","input":"journald","stability":"Experimental","deprecated":false,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-23T06:33:10.221Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 7895873006178960417)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-23T06:33:10.221Z","log.logger":"input.journald","log.origin":{"file.name":"compat/compat.go","file.line":121},"message":"Input 'journald' starting","service.name":"filebeat","id":"all-journald","ecs.version":"1.6.0"}

Images I used

• docker.elastic.co/beats/filebeat-oss:8.10.4
• custom image with the following config (Dockerfile)

FROM golang:1.17.6
 
RUN \
    apt-get update \
      && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
         libsystemd-dev \
         netcat \
         rsync \
         python3 \
         python3-pip \
         python3-venv \
         libpcap-dev \
      && rm -rf /var/lib/apt/lists/*
 
RUN apt update -y  \
  && apt install -y wget \
    &&  apt install -y ca-certificates \
    &&  apt install -y gnupg \
    &&  apt install -y openssl
     
  
# Install elasticSearch. for Debian? 
RUN wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
RUN  apt-get install apt-transport-https
RUN echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-8.x.list


RUN apt-get update && apt-get install filebeat=8.10.4       ## from 8.5.3
RUN apt-cache showpkg filebeat


RUN /usr/share/filebeat/bin/filebeat version
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
 
RUN chmod 600 /usr/share/filebeat/filebeat.yml
RUN uname -a
RUN uname -r

ENTRYPOINT [ "/usr/share/filebeat/bin/filebeat", "-e", "--path.home", "/usr/share/filebeat", "--path.data", "/usr/share/filebeat/data" ]

#build cmd
# podman build -f Dockerfile -t fcos-filebeat-1710:v8 --arch x86_64 . --no-cache 

No matter which of those images I used, the result is still the same. The logs in the filebeat pods are the same, and the journald logs are not exported by the latter.

Thanks for taking the time to read this through. If you see something I am missing, please chime in.

Regards

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

Did you mount the host's /var/log and /run/log inside the container?

Does journalctl inside the container read the logs you are expected Filebeat to read? If not, perhaps the version of systemd installed inside the container is in some way not compatible with the version from the host.

[johnny550]

thanks for your comment. You're right. it was due to a discrepancy between systemd versions.

Cheers