Issues on FreeBSD

[leo-arch]

A few issues when attempting to install and use AM on FreeBSD:

1. When installing am (git method), wget(1) isn't found (and thereby nothing was actually installed), but I still get this message: "AM" successfully installed.

2. I installed wget(1). After am installation (git method), I get this message:

Installing "AM" on /opt/am
WARNING: Couldn't link am to "/usr/local/bin/am!"

The thing is that /usr/local/bin/am was actually generated and properly points to /opt/am/APP-MANAGER.

3. I tried to install clifm using the am -i clifm command. Here's the output:

? WARNING: ACCESS TO USER NAMESPACES IS RESTRICTED!

Some apps may to run, you need to enable access to user namespaces,
please visit https://github.com/ivan-hc/AM#ubuntu-mess to know more

-----------------------------------------------------------------------------------------------------

START OF INSTALLATION PROCESS

==========================================================

? ERROR: "clifm" does NOT exist in the "AM" database, please check the list,
run the "am -l" command

==========================================================

END OF ALL INSTALLATION PROCESS

https://github.com/ivan-hc/AM#ubuntu-mess redirects to the main AM github page: no ubuntu-mess section.

The namespaces messages is triggered by APP-MANAGER:258: There's no unshare(1) command in FreeBSD. I've modified APP-MANAGER to only run unshare(1) if it exists:

if type unshare >/dev/null 2>&1 && ! unset --user -p /bin/true >/dev/null 2>&1

Still get the above error screen, but without the namespaces warning

Ok. Let's see what am -l has for us:

grep: /home/user/.cache/am/version-args: No such file or directory

YOU HAVE INSTALLED -1 PROGRAM OUT OF 622 AVAILABLE, AND     0 LIBRARIES

LIST OF THE 1 PROGRAMS AVAILABLE IN THE "AM" DATABASE:

To list all installable programs, use the "--all" flag

404: Not found.

Oops!

Let's run am -l --all. I only see a bunch of numbers, like these:

>Public Access</a

d="M103.074 142.831C103.048 ...

And the thing goes on. It seems to be parsing an HTML file, and failing.

Summarizing

1. What about namespaces?
2. Cannot parse the apps database file. The file seems to be reachable, but cannot be properly parsed. Now, since this is a parsing issue, you most likely have some non-supported parameter in some of the parsing utilities, like sed(1) or something like that (but I might be wrong).

[Samueru-sama]

When installing am (git method), wget(1) isn't found (and thereby nothing was actually installed), but I still get this message: "AM" successfully installed.

The INSTALL script should check for wget 🤔 or you mean that you got the For Installation to work... error? that's expected if wget isn't installed.

The namespaces messages is triggered by APP-MANAGER:258: There's no unshare(1) command in FreeBSD. I've modified APP-MANAGER to only run unshare(1) if it exists:

if type unshare >/dev/null 2>&1 && ! unset --user -p /bin/true >/dev/null 2>&1

Still get the above error screen, but without the namespaces warning

unshare is a linux kernel only feature as far as I know, which we use to check for access to usernamespaces which a lot of appimages and electron apps depend on, ubuntu recently decided to disable that so that check was put in place.

I guess we will only run that check if unshare is present.

[leo-arch]

And what about the parsing issue? If I'm not wrong, you're trying to parse https://portable-linux-apps.github.io/apps. Is that right? In this case, why not to autogenerate a simple text file filled with only app names and app descriptions, something like this:

appname|appdesc
...

That's is. It's much straightforward, easier to parse, faster, and more portable.
This file needs to be regenerated automatically every time the main html page is updated.

[Samueru-sama]

And what about the parsing issue? If I'm not wrong, you're trying to parse https://portable-linux-apps.github.io/apps. Is that right? In this case, why not to autogenerate a simple text file filled with only app names and app descriptions, something like this:

That's something for @ivan-hc

But we already have such file

[ivan-hc]

@leo-arch since you are using git, use this script instead

https://github.com/ivan-hc/AM/blob/dev/INSTALL

before running it, there is a variable AM_BRANCH="main" on top of the script, change the value to "AM_BRANCH="dev"" and then retry

Changes are now on the "dev" branch of this repo https://github.com/ivan-hc/AM/tree/dev

[leo-arch]

Installed using the suggested script, made requested change, and still nothing.

While the installation was sweat, the database is still a problem. am -l warns: 404: Not found, and the total amount of installed program is -1.

[ivan-hc]

Installed using the suggested script, made requested change, and still nothing.

While the installation was sweat, the database is still a problem. am -l warns: 404: Not found, and the total amount of installed program is -1.

this is really strange

[leo-arch]

What's the module taking care of am -i APP?

[ivan-hc]

install.am

[leo-arch]

Ok. I guess _install_normally() is the function we're looking for, isn't it? Maybe the curl line is the culprit. Let's inspect that.

[ivan-hc]

yes, _install_normally take care of the scripts in the database, _install_local_scripts does what you asked today about scripts downloaded via -d and how to test them

[leo-arch]

Great. Let's start debugging.

[leo-arch]

Located! There's something wrong with INSTALL:423. The curl line is taking the false branch.

[ivan-hc]

that line tests if the script exists in the online database

[ivan-hc]

what version of curl you have?

[leo-arch]

Well, here is it: curl is trying to access a non-existent file:

https://raw.githubusercontent.com/ivan-hc/AM/main/programs/FreeBSD/clifm

As you can see, the FreeBSD path component is the culprit. It should be x86_64 instead.

[Samueru-sama]

Well, here is it: curl is trying to access a non-existent file:

https://raw.githubusercontent.com/ivan-hc/AM/main/programs/FreeBSD/clifm

As you can see, the FreeBSD path component is the culprit. It should be x86_64 instead.

what do you get when you run uname -m?

[leo-arch]

amd64

[Samueru-sama]

amd64

and what about $HOSTTYPE? (Do this in bash btw)

[leo-arch]

FreeBSD

[leo-arch]

Ok. Where is the APPSDB variable generated. At this point it seems clear that the issue is there.

EDIT: I think APP-MANAGER:20 is our guy. It reads:

arch="$HOSTTYPE"

[Samueru-sama]

Ok. Where is the APPSDB variable generated. At this point it seems clear that the issue is there.

EDIT: I think APP-MANAGER:20 is our guy. It reads:

arch="$HOSTTYPE"

Yeah it was that. I forgot we used HOSTTYPE instead of uname originally

but still even uname -m gives you amd64 instead of x86_64, so it would have break anyway lol

[ivan-hc]

just merged the PR by @Samueru-sama

in APP-MANAGER, replace

# Determine system architecture and current user
arch="$HOSTTYPE"
export ARCH="$arch"

with

# Determine system architecture and current user
ARCH="$(uname -m)"
[ "$ARCH" = "amd64" ] && ARCH=x86_64
arch="$ARCH"
export ARCH arch

[ivan-hc]

Also, you can keep changes from "deb" by entering the Developer Mode

am --devmode-enable
am -s

to exit the developer mode

am --devmode-disable

[leo-arch]

Gone through the install process successfully. However, the binary won't run: it's Linux only. But this is a different issue.

[ivan-hc]

what program have you installed?

[leo-arch]

clifm

[leo-arch]

I'll try with something else.

Tried with nnn and puddletag. Same thing: Exec format error. Binary file not executable.

These binaries were compiled in Linux, and won't run on BSD.

[Samueru-sama]

I'll try with something else.

Can you see what happens if you try to run puddletag? am -i puddletag

that's an appimage that I have that bundles everything so it even works on musl libc systems, I want to know if it works on FreeBSD.

[Samueru-sama]

According with distrowatch

# Determine alternative operating systems
if echo "$(uname)" | grep -q "BSD\|DragonFly\|DynFi\|FreeNAS\|helloSystem\|OPNsense\|pfSense\|TrueNAS\|XigmaNAS"; then
	sed_cmd="gsed"
	TERM=xterm-clear
	export TERM
else
	sed_cmd="sed"
fi

You can pipe uname directly if uname | grep -q "BSD\| ...

Also I don't know if we can just set the TERM variable always when empty, maybe this problem also happens with some obscure linux distro as well, something like [ -n "$TERM" ] || TERM=xterm-clear

[ivan-hc]

Just finished to finalize all left fixes and added some improvements to some options.

Also, thanks to @Samueru-sama 's suggestions, even more improvement have been done not to hardcode symlinking.

Last release is 9.3-10, in "dev".

I have read some documentation and it seems that ways to run Linux apps nativelly on freeBSD already exist

https://docs.freebsd.org/en/books/handbook/linuxemu/

@leo-arch as you said, this must be tested on other BSD-based systems. The function that does the check is the following, in APP-MANAGER

# Determine alternative operating systems
if uname | grep -q "BSD\|DragonFly\|DynFi\|FreeNAS\|helloSystem\|OPNsense\|pfSense\|TrueNAS\|XigmaNAS"; then
	sed_cmd="gsed"
	TERM=xterm-clear
	export TERM
else
	sed_cmd="sed"
fi

[ ... ]

# Create new data directory and move important files there
AMDATADIR="$DATADIR/AM"
mkdir -p "$AMDATADIR"
if [ "$sed_cmd" = gsed ]; then
	mkdir -p "$AMDATADIR/bin"
	export PATH="${AMDATADIR}"/bin/:"${PATH}"
	[ ! -f "${AMDATADIR}"/bin/sed ] && ln -s "$(command -v gsed)"  "${AMDATADIR}"/bin/sed
fi

One think left that I would like to add is bash-completion support to FISH, this is the functio that take care of this support in BASH and ZSH

# BASH AND ZSH COMPLETION
completion_file="$DATADIR/bash-completion/completions/$AMCLI"
mkdir -p "$DATADIR/bash-completion/completions" || exit 1
[ -f "$HOME"/.bash_completion ] && sed -i "/ $AMCLI$/d" "$HOME"/.bash_completion
if ! grep -o " $AMCLI$" "$completion_file" >/dev/null 2>&1; then
	echo "complete -W \"\$(cat $AMDATADIR/list 2>/dev/null)\" $AMCLI" >> "$completion_file"
	if [ -f "${ZDOTDIR:-$HOME}"/.zshrc ] && echo "$SHELL" | grep -q "zsh"; then
		cat <<-HEREDOC >> "${ZDOTDIR:-$HOME}"/.zshrc
		autoload bashcompinit
		bashcompinit
		source "$completion_file"
		HEREDOC
	fi
	echo "Shell completion has been enabled!"
fi

and here I don't know how to move on.

[ivan-hc]

@leo-arch version 9.3-11 in "dev", thanks to the hard work of @Samueru-sama also the sed issue has been solved.

Now AM checks if sed is GNU variant, if not, you will receiven a warn that if you don't install a GNU compatible version of sed, so gsed, something will break

Now AM works n BSD like it was on GNU/Linux.

The only difference is that installed programs will not work. We can solve it by running an installation script that will configure everything as it should... but we are not BSD experts. I wait your Pull Request for this. If you want join or you want to suggest others to get involved, you are wellcome.

About FISH/DASH completion... we don't know anything.

For me, AM is ready to be released as 9.4 stable.

We will keep test it normally on our systems as well. If you can team up with others by spreading this program, we would improve it fastrer and better.

[ivan-hc]

@leo-arch if you have Discord you can join our community https://discord.gg/djJUs48Zbu

[ivan-hc]

@leo-arch There are some important things you need to know:

1. the programs from the Toolpacks repository (installable with the command am -i {program}.toolpack or with the flag --toolpack, also in -l and -q for your searches) are the only ones that seem to work without problems

2. the programs are installed in /usr/local, and I noticed that the programs installed with PKG are installed there too. The only problem you will encounter will be the symlink in /usr/local/bin, which could conflict with pre-existing ones, and when removing an application installed with AM you risk removing the binary installed with PKG

   • 2a. about point 2, if you want to install the programs system-wide, it would be better to "divert" via a patch (in "AM") the symbolic links of the programs to an alternative $PATH

   • 2b. the safest and easiest way to solve point 2 is to use AppMan, or install AM and use the --user flag during installations, only for local use... or run the command am --user to use AM permanently as if it were AppMan

Miscellaneous will be listed on the 9.4 release page when it's ready, and on a README page for BSD systems if I have the time and expertise to provide more information.

I can't do more than this. Starting next week I'll be busy with some important commitments that will surely keep me away from github for a long time. However, I hope to solve this problem as soon as possible and to release this release full of new features as soon as possible.

[ivan-hc]

@leo-arch

Version 9.3-12, gsed is no more a core dependency, but if you have it, it will be used where its needed

Thanks @Samueru-sama

[leo-arch]

the programs from the Toolpacks repository (installable with the command am -i {program}.toolpack or with the flag --toolpack, also in -l and -q for your searches) are the only ones that seem to work without problems

Really good to know.

the programs are installed in /usr/local, and I noticed that the programs installed with PKG are installed there too. The only problem you will encounter will be the symlink in /usr/local/bin, which could conflict with pre-existing ones, and when removing an application installed with AM you risk removing the binary installed with PKG

This one is bad.

the safest and easiest way to solve point 2 is to use AppMan

Maybe you should automate this so that the user's binaries are not at risk (which, again, is really bad).

Btw, I don't use Discord.

[Samueru-sama]

This one is bad.
Maybe you should automate this so that the user's binaries are not at risk (which, again, is really bad).

Note this is an FHS violation by ghostbsd.

Archlinux is also notorious on this issue by putting packages in /opt.

[ivan-hc]

I have solved the issue of symlinks in $PATH here a7ef8e9

[leo-arch]

Note this is an FHS violation by ghostbsd.

As a matter of fact, most Linux distros silently break this rule by installing most package binaries in /usr/bin. BSDs are in this sense much more FHS compliant.

[ivan-hc]

I have never seen a distro that installs from its official repos to /usr/local, only under /usr directly. /usr/local and /opt are for third party programs.

For what I know, Arch uses these paths only for AUR

[leo-arch]

I have never seen a distro that installs from its official repos to /usr/local

This is exactly what BSDs do most of the time (not sure about MacOS, but it probably follows the same pattern).

[ivan-hc]

How third party programs are installed not to conflift with PKG?

[leo-arch]

Well, as far as I know, /usr/bin is for system binaries, while /usr/local/bin is for user installed binaries. As to how to avoid conflicts, binaries shouldn't have conflicting names in the first place (this is why, for example, Archlinux PKGBUILDs provide a conflict field).

Maybe, and just maybe, AM should refuse to install packages already installed (like pacman does).

[ivan-hc]

this is how AM installs programs

/opt/$PROGRAM/
/opt/$PROGRAM/$PROGRAM
/opt/$PROGRAM/AM-updater
/opt/$PROGRAM/remove
/opt/$PROGRAM/icons/$ICON-NAME
/usr/local/bin/$PROGRAM
/usr/local/share/applications/$PROGRAM-AM.desktop

Maybe, and just maybe, AM should refuse to install packages already installed.

the issue was only with /usr/local/bin, but as I said before...

I have solved the issue of symlinks in $PATH here a7ef8e9

[ivan-hc]

Now, what is left

1. FISH/DASH completion... for BASH/ZSH AM uses all words listed in $HOME/.local/share/AM/list. We don't know how FISH/DASH does this (I tried and tried... but in the end I replaced FISH with BASH in the VM to prevent headaches)
2. you can solve compatibility for linux apps by creating a script that installs from PKG the needed dependences and enables the needed flags... it must include also a script to revert these changes in any case. Basically, you can get any installation script as an example. We don't knoe how to do this without causing troubles (I had to reinstall the VM after I compromised it too much)

Apart this, we, on the AM side, have finished. For me the release can come out this evening (for me in Italy its 9:15 am).

Me and @Samueru-sama have only finalized thwe last fixes fr the release. All its left are tests in normal usage. No more issues for now, so the release is basically ready.

[xplshn]

dash is not an interactive shell, it doesn't have completion. Not even tab completion for file paths or commands. (not only does it not have any interactive features, it is way slower than busybox's ash, which has tab completion of commands and paths, as well as all interactive features enabled.)

[Samueru-sama]

Well, as far as I know, /usr/bin is for system binaries, while /usr/local/bin is for user installed binaries

That would be /bin /sbin and /usr/bin /usr/sbin,

/usr/local is for the system administrator to add packages locally on the distro.

cheat sheet

user binaries would be more like $HOME/.local/bin which the XDG Base dir spec mentions that distros should put in PATH, I think only debian does it, Arch does not lol.

For what I know, Arch uses these paths only for AUR

Most aur packages use /usr/bin, I don't know a single official arch package that uses /usr/local, the directory is empty by default as well. (However what I meant with opt is that some official arch packages do use /opt and they don't care about it lol)

[leo-arch]

/usr/local is for the system administrator to add packages locally on the distro.

Yes, according to the standard. But implementations differ:

1. On Linux, (almost) all packages installed by the user go to /usr/bin, while /usr/sbin is for system administration binaries (Debian, for example, but not Arch, where /usr/sbin is made a symlink to /usr/bin). /usr/local/bin is mostly ignored, and /opt is very rarely used.
2. On BSD systems, OS packages go to either /usr/bin (binaries for regular users) or /usr/sbin (binaries for administrators, like daemons, file system stuff, and so on). Packages installed by the user (added to the base system) go to /usr/local/bin. OpenBSD even suggests to use a separate partition for /usr/local, so that the base system is as isolated from the userland as possible (which just makes sense). What about /opt? very rarely used as well.

Let's suppose you install the coreutils package. On Linux, the binaries provided by the package (say gsed, gcp, etc) would go to /usr/bin, while on BSD to /usr/local/bin.

For more info consult hier(7) on both Linux and BSD.

Summarizing, it is quite safe to copy your binaries into /usr/local/bin if you are on Linux. Don't do this blindly on BSD systems.

user binaries would be more like $HOME/.local/bin which the XDG Base dir spec mentions that distros should put in PATH, I think only debian does it, Arch does not lol.

Not on Debian either (at least not by default). Haven't seen this on BSD either (though FreeBSD does have /home/user/bin at the end of its $PATH).

[grahamperrin]

Cross-reference: Trying to run AppImages on *BSD based systems, using "AM" package manager in /r/freebsd

[ivan-hc]

I'll start by saying that I'm a total ignoramus on the subject, but here's what an unlikely installation script named "linux4bsd" for a Linux app compatibility layout on freeBSD might contain (based on what I understand from this guide https://docs.freebsd.org/en/books/handbook/linuxemu/ )

#!/bin/sh

# Determine wheter to use between "sudo" or "doas"
if command -v sudo >/dev/null 2>&1; then
	export SUDOCMD="sudo"
elif command -v doas >/dev/null 2>&1; then
	export SUDOCMD="doas"
else
	echo 'ERROR: No sudo or doas found'
	exit 1
fi

# Determine if PKG package manager exists in $PATH
if ! command pkg >/dev/null 2>&1; then
	echo 'ERROR: Command  "pkg" not found'
	exit 1
fi

APP=linux4bsd

# CREATE DIRECTORIES AND ADD REMOVER
[ -n "$APP" ] && mkdir -p "/opt/$APP" && cd "/opt/$APP" || exit 1
printf "#!/bin/sh\nset -e\n$SUDOCMD pkg remove -y linux-sublime-text4" > ./remove
printf '\n%s' "$SUDOCMD service linux stop" >> ./remove
printf '\n%s' "$SUDOCMD sysrc linux_enable=\"NO\"" >> ./remove
chmod a+x ./remove || exit 1

# DOWNLOAD AND PREPARE THE LINUX COMPATIBILITY LAYOUT
"$SUDOCMD" sysrc linux_enable="YES" || exit 1
"$SUDOCMD" service linux start || exit 1
"$SUDOCMD" pkg install -y linux-sublime-text4 || exit 1

and that's all.

All of this would be installed with one of the following two commands

am -i linux4bsd
am install linux4bsd

...and would be removed with one of the following three commands

am -R linux4bsd
am -r linux4bsd
am remove linux4bsd

If the installation fails, the first script created, called "remove" will be run to reverse what has been done so far.

NOTE! I have only written an example of what a script might look like. The commands I have listed are just an example. Do not take what I have written literally. THIS IS JUST A PROOF OF CONCEPT!

[xplshn]

IMO its better to do:
"$SUCMD" sh -c '
# all root commands here
'

Why? Because if the user has doas set up without persistance, he'll get prompted for his password 3 times.

[ivan-hc]

I think it's time to release the version.

Even if the apps don't work (most of them, I haven't found any "working" ones yet), at least we've laid the groundwork for a way to support them on *BSD.

What do you think @leo-arch ? Does this solve the issue?

[ivan-hc]

In the meantime I finally managed to run AppImages on GhostBSD using this guide https://unixdigest.com/tutorials/how-to-install-signal-desktop-on-freebsd-using-the-linux-binary-compatibility.html

You can see some screenshots here #1209 (comment)

[leo-arch]

I think it's time to release the version.

Provided AM itself works as expected, installing, listing, uninstalling, and so on (haven't tested the latest modifications though), that's enough for a release (maybe adding something like "Initial BSD support" to the release notes would be fine). As I said before, it's not AM's business to provide BSD compatible binaries.

In this sense this issue can be considered solved: as a user, I wouldn't blame AM for not providing BSD binaries. Even the linux emulation workaround should be considered an add-on, not a core feature.