[noble] syslog release breaks rsyslog

[ramonskie]

in syslog_storer in the syslog release
it overwrite rsyslog see: https://github.com/cloudfoundry/syslog-release/blob/main/jobs/syslog_storer/templates/rsyslog.conf.erb

this causes rsyslog to fail.
it seems to have to do with write permissions.

as we had also problems with writing the bosh-agent logs to the /var/vcap/bosh/log from systemd due to permissions.
and we agreed that logs should be in the logs directory.

more investigation is needed

[ramonskie]

noticed when ci tests where failing https://bosh.ci.cloudfoundry.org/teams/stemcell/pipelines/stemcells-ubuntu-noble/jobs/test-stemcells-master-ipv4/builds/47#L667487cc:926

systemctl status rsyslog output:

un 28 09:05:38 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension.>
Jun 28 09:05:38 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2312.0 try https://www.rsyslog.com/e/2359 ]
Jun 28 09:05:38 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension.>
Jun 28 09:05:50 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: rsyslogd: file '/var/vcap/data/syslog_storer/buffered/agg_backlog.00000001': open error: Permission denied [v8.2312.0 try https://www.rsyslog.com/e/2433 ]
Jun 28 09:05:50 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: file '/var/vcap/data/syslog_storer/buffered/agg_backlog.00000001': open error: Permission denied [v8.2312.0 try https://www.rsyslog.com/e/2433 ]

[ctlong]

This reads like rsyslog does not have access to /var/vcap/data/syslog_storer in Noble. Was there a permissions change from Jammy to Noble related to this directory?

[rkoster]

Nobel stemcells can be found here:

• storage.googleapis.com/bosh-core-stemcells-candidate/google/bosh-stemcell-0.59-google-kvm-ubuntu-noble-go_agent.tgz
• storage.googleapis.com/bosh-core-stemcells-candidate/aws/bosh-stemcell-0.59-aws-xen-hvm-ubuntu-noble-go_agent.tgz
• storage.googleapis.com/bosh-core-stemcells-candidate/azure/bosh-stemcell-0.59-azure-hyperv-ubuntu-noble-go_agent.tgz

Source: https://bosh.ci.cloudfoundry.org/teams/stemcell/pipelines/stemcells-ubuntu-noble/

[ctlong]

Sorry, I lost track of this for a while.

❓ Is this error still occurring?
❓ Has the new rsyslog package for Noble been pulled in?

[ramonskie]

@ctlong yes this error is still occuring.
the problem is not rsyslog but systemd and its new security measures.
it does not allow logs writing outside of /var/log/**/*

According to the Rsyslog documentation and user experiences, rsyslog has a default behavior that restricts log writing to directories outside of /var/log. This limitation is in place for security and compatibility reasons.

Why is this limitation in place?
Security: By default, rsyslog runs as the syslog user and group, which has limited privileges. Writing logs to directories outside of /var/log could potentially allow unauthorized access or tampering with logs.
Compatibility: Rsyslog is designed to work with the traditional Linux logging infrastructure, which assumes logs are stored in /var/log. Writing logs to other directories might break compatibility with other system components or tools that rely on the standard log location.

[ctlong]

I don't appear to be able to view the pipeline, but I just wanted to confirm what tests you're running to see the syslog_storer failures? Are you running the tests in the syslog-release repo?

[ramonskie]

the tests that we run are here https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/src/github.com/cloudfoundry/stemcell-acceptance-tests/ipv4director/syslogrelease/smoke_test.go#L18

[ramonskie]

syslog-release 12.3.4 sovled the issue