Linux service package - adapt to older systemd versions #454

qrkourier · 2023-11-15T20:01:06Z

If the version of systemd predates the features used by the service then the service should adapt itself to the available features.

The features used to drop privileges are not new, but give this issue a +1 if you encounter the issue of STATE_DIRECTORY not defined and want to run the zrok-share service with an older version of systemd.

The workaround is to modify the service unit to run as root, or another user, and to manually create that user's HOME for the service to use, replacing the directives DynamicUser or StateDirectory if they're not supported by the older systemd.

The text was updated successfully, but these errors were encountered:

scareything · 2023-11-15T21:38:37Z

What are the boundaries in terms of distro versions?

qrkourier · 2023-11-15T21:57:30Z

Currently, zrok-share.service requires systemd v232 or newer because that's when DynamicUser was added, which allows us to avoid managing users, groups, permissions, and more.

scareything · 2023-11-16T01:24:31Z

What does that translate to in distro versions for Debian/ubuntu and centos?

qrkourier · 2023-11-16T02:24:23Z

It excludes Ubuntu Bionic and CentOS 7, but all the newer distros I've tested were good.

qrkourier · 2023-11-16T02:36:54Z

~~I included in the testing matrix~~ Successfully tested Fedora 37-38 , Rockylinux 8-9, Ubuntu 20-22, Debian 11-12.

from the Vagrantfile

    # config.vm.box = "generic/ubuntu2204"   # good
    # config.vm.box = "generic/ubuntu2004"   # good
    # config.vm.box = "rockylinux/9"         # good

    # config.vm.box = "rockylinux/8"         # good
    # config.vm.box_version = "5.0.0"

    # config.vm.box = "generic/debian10"     # golang panic: user: unknown userid 64342 (the dynamic UID)
    # config.vm.box = "generic/debian11"     # good
    # config.vm.box = "generic/debian12"     # good

    # config.vm.box = "generic-x64/fedora38" # good
    # config.vm.box = "generic-x64/fedora37" # good

    # config.vm.box = "generic/ubuntu1804"   # undefined STATE_DIRECTORY
    # config.vm.box = "centos/7"             # undefined STATE_DIRECTORY

EDIT: add Debian test results

qrkourier · 2023-11-16T03:35:12Z

Tried Debian Buster (10), which has v241, but zrok panicked.

Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]: INFO: running: zrok enable ...
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]: panic: user: unknown userid 64342
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]: goroutine 1 [running]:
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]: main.(*enableCommand).run(0xc000090d20, 0x0?, {0xc000322fc0, 0x1, 0x0?})
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]:         /home/runner/work/zrok/zrok/cmd/zrok/enable.go:61 +0x1268
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]: github.com/spf13/cobra.(*Command).execute(0xc000396300, {0xc000322f40, 0x4, 0x4})
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]:         /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:944 +0x863
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]: github.com/spf13/cobra.(*Command).ExecuteC(0x4d7e2c0)
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]:         /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x3a5
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]: github.com/spf13/cobra.(*Command).Execute(...)
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]:         /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:992
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]: main.main()
Nov 16 03:32:40 debian10.localdomain zrok-enable.bash[6397]:         /home/runner/work/zrok/zrok/cmd/zrok/main.go:99 +0x1a
Nov 16 03:32:40 debian10.localdomain systemd[1]: zrok-share.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Nov 16 03:32:40 debian10.localdomain systemd[1]: zrok-share.service: Failed with result 'exit-code'.

qrkourier · 2024-02-05T14:07:32Z

I'll revisit this after the zrok daemon mode is introduced because that will entail a refactor of the zrok-share Linux package that provides the systemd service.

tristanryerparke · 2024-03-10T21:14:06Z

Hi,
I'm getting a similar error when trying to start zrok frontdoor with this command from the guide:
sudo systemctl enable --now zrok-share.service
Error:
Mar 10 16:42:37 user zrok-enable.bash[296364]: panic: user: unknown userid 64342
I'm on a raspberry pi 4 with 64bit os:
6.1.0-rpi8-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25) aarch64 GNU/Linux
systemd version is: systemd 252 (252.22-1~deb12u1)
Any idea why this wouldn't be working even though the version is > v232
Thanks,
T

qrkourier · 2024-03-11T15:01:51Z

Hey there @tristanryerparke! I'll need your help to diagnose this one because I don't have a representative environment to test with.

Let's debug zrok-enable.bash to identify which command emitted the unknown userid message. The message appears to have originated from a zrok CLI command, because it appears similar to what I'd expect from a Go program failing to resolve a UID.

Please add this line to the other shell options that are set near the top of /opt/openziti/bin/zrok-enable.bash.

set -o xtrace

The next time the service runs it will use the modified script.

Are you allowing systemd to manage the file content, owner, and mode of the files in /var/lib/zrok-share, or do you need to modify those managed files' contents, owner, or mode at all?

I expect systemd to set the file owner of all files in /var/lib/zrok-share with the new dynamic UID for each run.

This means the dynamic UID from the last run no longer exists, so it raises a question about whether there are still some files owned by the prior run's dynamic UID inside that directory.

tristanryerparke · 2024-03-13T22:41:45Z

@qrkourier here is the full output from journalctl -u zrok-share after adding the xtrace line:

Mar 10 16:42:34 tristan-pm4 systemd[1]: zrok-share.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Mar 10 16:42:34 tristan-pm4 systemd[1]: zrok-share.service: Failed with result 'exit-code'.
Mar 10 16:42:34 tristan-pm4 systemd[1]: Failed to start zrok-share.service - zrok reserved public share service.
Mar 10 16:42:37 tristan-pm4 systemd[1]: zrok-share.service: Scheduled restart job, restart counter is at 2.
Mar 10 16:42:37 tristan-pm4 systemd[1]: Stopped zrok-share.service - zrok reserved public share service.
Mar 10 16:42:37 tristan-pm4 systemd[1]: Starting zrok-share.service - zrok reserved public share service...
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: INFO: reading enable parameters from /opt/openziti/etc/zrok/zrok-share.env
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296369]: zrok configuration updated
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: INFO: running: zrok enable ...
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: panic: user: unknown userid 64342
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: goroutine 1 [running]:
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: main.(*enableCommand).run(0x40002fc000, 0xd4d4380100000000?, {0x40003b6540, 0x1>
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]:         /home/runner/work/zrok/zrok/cmd/zrok/enable.go:61 +0xe3c
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: github.com/spf13/cobra.(*Command).execute(0x40006de908, {0x40003b64c0, 0x4, 0x4>
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]:         /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:987 +0>
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: github.com/spf13/cobra.(*Command).ExecuteC(0x4d725c0)
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]:         /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +>
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: github.com/spf13/cobra.(*Command).Execute(...)
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]:         /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
Mar 10 16:42:37 tristan-pm4 zrok-enable.bash[296364]: main.main()

I'm not that familiar with linux, but when I try cd /var/lib/zrok-share I get a permission denied error, so I'm guessing that is something I don't need to modify.

Funnily enough, I can run a command on that same system like zrok share private --headless --backend-mode localhost:80 and use zrok normally, only when running the frontdoor systemd service do I have issues.

Is there good way to do a full uninstall and reinstall of all zrok things on the system, and try again?
Maybe that would fix the UID issues.
Best,
T

qrkourier · 2024-03-14T04:05:26Z

I did find that the Go library in use will sometimes fail to resolve the UID when running in a systemd dynamic user sandbox. I've sent a patch that resolves this for Debian 10 (Buster), which is where I first encountered the same error you reported. I think that will fix it for Raspbian too. Can you determine whether your version of Raspbian is in fact a descendant of Debian 10? I didn't have the same issue on Debian 11 or 12.

# inspect distribution info
cat /etc/*-release

Meanwhile, you may wish to disable your /lib/systemd/system/zrok-share.service and copy it instead to ~/.config/systemd/user/zrok-share.service with the following modifications to run as your login user.

sudo systemctl disable --now zrok-share.service

Substitute a line like this in place of DynamicUser=yes where /home/pi is your login user's homedir.

Environment=STATE_DIRECTORY=/home/pi

With that modification to the user service, you may load and enable it with these commands.

systemctl --user daemon-reload
systemctl --user enable --now zrok-share.service

Now you should see a status.

systemctl --user status zrok-share.service

And logs

journalctl --user -xeu zrok-share.service

If you don't see logs, you may need to enable journal persistence in /etc/systemd/journald.conf by setting this directive.

[Journal]
Storage=persistent

After changing journald.conf you must also reload the configuration.

sudo systemctl restart systemd-journald.service

In this configuration, the systemd service runs as user "pi" and so the enabled zrok environment is located in ~/.zrok.

tristanryerparke · 2024-03-17T14:38:06Z

this worked, thank you.

Debian info:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

qrkourier self-assigned this Nov 15, 2023

qrkourier mentioned this issue Mar 14, 2024

fall back to env var USER when /etc/passwd and stdlib faculties fail … #583

Merged

qrkourier closed this as completed in #583 Mar 14, 2024

qrkourier mentioned this issue Mar 20, 2024

Linux server - install existing env to service #453

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linux service package - adapt to older systemd versions #454

Linux service package - adapt to older systemd versions #454

qrkourier commented Nov 15, 2023

scareything commented Nov 15, 2023

qrkourier commented Nov 15, 2023

scareything commented Nov 16, 2023

qrkourier commented Nov 16, 2023

qrkourier commented Nov 16, 2023 •

edited

Loading

qrkourier commented Nov 16, 2023

qrkourier commented Feb 5, 2024

tristanryerparke commented Mar 10, 2024

qrkourier commented Mar 11, 2024

tristanryerparke commented Mar 13, 2024 •

edited

Loading

qrkourier commented Mar 14, 2024

tristanryerparke commented Mar 17, 2024

Linux service package - adapt to older systemd versions #454

Linux service package - adapt to older systemd versions #454

Comments

qrkourier commented Nov 15, 2023

scareything commented Nov 15, 2023

qrkourier commented Nov 15, 2023

scareything commented Nov 16, 2023

qrkourier commented Nov 16, 2023

qrkourier commented Nov 16, 2023 • edited Loading

qrkourier commented Nov 16, 2023

qrkourier commented Feb 5, 2024

tristanryerparke commented Mar 10, 2024

qrkourier commented Mar 11, 2024

tristanryerparke commented Mar 13, 2024 • edited Loading

qrkourier commented Mar 14, 2024

tristanryerparke commented Mar 17, 2024

qrkourier commented Nov 16, 2023 •

edited

Loading

tristanryerparke commented Mar 13, 2024 •

edited

Loading