AUR packaging

[ghbrown]

Adds packaging and build/test instructions for the Arch (Linux) User Repository (AUR), as per the discourse discussion.

Additions:

• The chapel package for AUR tracks the static 1.26 release, but is non-functional due to issues in 1.26 that cause build issues on Arch. These issues have been resolved on main, but the chapel AUR package will be left broken until 1.27 releases (the online description of the broken AUR points to functioning chapel-git package).
• The chapel-git package for AUR tracks the main branch of the chapel repo and is fully functional.

Outstanding issues:

• updating chapel AUR to use 1.27 upon its release
• fixing the git describe --long --tags --always behavior for the main branch (currently main reports as version 1.24, while it is really 1.26 at the time of writing).

Possible future work:

• automated copying of packaging files from their remote sources (so maintainers don't need to manually copy and make PRs, etc), hopefully such a tool would work for many different packaging formats, OSes, etc. and not be a one-off for Arch
• testing of packaging; it is important to test both successful install and a functioning compiler after build and install, as we ran into multiple situations where the compiler built and checked, but did not actually work (Don't use libtool .la files to get 3p lib flags #19913).

[ghbrown]

@mppf I also ran a build from source with --prefix=/usr that produces a working compiler. However, I noticed that one step of make check is to compile hello6 to an executable in ~/.chpl. However, this hidden directory is not around after make check, and even when I create said directory (in case there was an error in that step) I can't find this file anywhere in ~/.
Part of the reason I was digging into this was because the make check run by makepkg passes (this happens for you too, right?). This is very confusing/concerning because part of the check is a test compilation, but we know know the compiler that's actually built seems to fail.
Not directly related to this PR, but it came up when debugging, so I thought I'd see if this was expected behavior or there was a more complicated explanation for what I'm observing.

[ghbrown]

I've done a bit more poking around and have the following thoughts:

• makepkg gives

WARNING: Package contains reference to $srcdir

Anecdotally, this is often a benign warning, but the Arch Wiki suggests running grep -R "$PWD/src" pkg/. I did, and got some matches on files in directories including hwloc, including pkg/chapel/usr/lib/chapel/1.26/third-party/hwloc/install/linux64-x86_64-native-llvm-pic-flat/lib/libhwloc.a, which may connect to the many linker errors about undefined references to chpl_hwloc_XX.

• my error at compile time begins with something like

/usr/bin/ld: /home/ghbrown/Temporary/chpl-ghbrown.deleteme-VR0PAv/chpl__module.o: in function `chpl_task_reset_spawn_order':                                                           
root:(.text+0x73175): undefined reference to `qthread_chpl_reset_spawn_order'

The first 3 directories in this path exist, but the rest don't exist and I can't find this file. What do you see here if anything? I do have an environment variable called ${TEMP}, perhaps you all use a similar variable and things get mixed up? How this path ends up having anything to do with Chapel I don't know.

[mppf]

Part of the reason I was digging into this was because the make check run by makepkg passes (this happens for you too, right?). This is very confusing/concerning because part of the check is a test compilation, but we know know the compiler that's actually built seems to fail.

I think it's expected for make check to remove everything. It just runs chpl examples/hello6-taskpar-dist.chpl so I'm used to just running that command when debugging the compiler. But of course we could consider having it not remove on a failure (but then ... when would it be removed?) or having it give output that is more clear about how to manually do the same thing. But as you said, this is all a side issue here.

• my error at compile time begins with something like

/usr/bin/ld: /home/ghbrown/Temporary/chpl-ghbrown.deleteme-VR0PAv/chpl__module.o: in function `chpl_task_reset_spawn_order':                                                           
root:(.text+0x73175): undefined reference to `qthread_chpl_reset_spawn_order'

The first 3 directories in this path exist, but the rest don't exist and I can't find this file. What do you see here if anything? I do have an environment variable called ${TEMP}, perhaps you all use a similar variable and things get mixed up? How this path ends up having anything to do with Chapel I don't know.

Right the compiler is just creating a temporary directory in $TEMP to store some intermediate results during compilation. It is expected that it deletes this temporary directory. If you want to keep it, compile with --savec=tmp (to save in the ./tmp directory).

[mppf]

Not sure if it is relevant to you but we also support arm

[ghbrown]

Great! I think I can add arm to that array then.

[mppf]

So, this checks that chpl runs before it is installed, right? I think the problem we are seeing is that something about the install process is not working right, so that's why this can pass even though the package is not working. Is there a post-install check available in some way (maybe just for people editing the AUR package and not necessarily something that runs when somebody installs the AUR package)?

[ghbrown]

Yes, that's my understanding as well, and I think that could be the case.

It is possible for me to include a post install script, but I'm not sure how integrated it is in the build and install process, since the package will already be installed. The one example I have hear is people using this script to bring up end user license agreement and so on, but I'm sure we can get it do more. What exactly were you thinking would go in here?

[mppf]

I was just thinking it would be nice to have some scripting so that the maintainer of the Chapel AUR package can easily check that the installation is working. I.e. it could just be a script that you know how to run after installing. If we have that, we can (eventually) get some nightly testing config to check it as well.

I don't know whether or not it would be reasonable to do this with an AUR post install script but I suspect you wouldn't want people just installing the package to always run it. (So then not a post-install).

[ghbrown]

I agree post-install is probably not the right tool here, but something like you describe would be nice.

I think the typical approach for AUR packages is to do verification with make check before install, but that didn't work for us. I'm sure there are issues with this approach, but the way I was going to verify that the installation is working is just by building and running it myself (then addressing user issues as they appear in comments on AUR). This is less of a problem for chapel since it's frozen until the next release, but perhaps would be problematic for chapel-git which changes (and possibly breaks) with every PR.

[mppf]

I'm trying to debug the problem on Arch and the instructions you added have been very useful to me. I'm running chpl hello.chpl --print-commands with the installed version (that does not work) and compared to a normal source build the -lhwloc line seems to be missing - even though we do have -L paths for hwloc. In fact, if I compile with chpl hello.chpl --print-commands -lhwloc -lqthread, then I get a program that compiles and runs. So, the question is, why are these link arguments not automatically added?

[mppf]

I found a problem when .la files are removed in the install process. We had a fallback but it wasn't working. PR #19846 should fix it.

[ghbrown]

Thanks for answering the questions I raised a few days ago. I agree it doesn't make sense to keep a bunch of test-compile executables around. As for the $TEMP I suppose I saw it as a possible environment variable namespace issue, and didn't expect to see anything being edited or changed in "user-land", but since you're deleting everything anyway I suppose it could be seen as a feature to hook into the users $TEMP.

I'm glad you found the instructions useful and have narrowed down the issue! I'm not sure why these arguments aren't added automatically, but if I can dig into anything on my side or run any tests just let me know. I can look into them after the packaging and readme improvements from your comments.

[ghbrown]

The old AUR packaging used to have a dependence on libtool that I removed since you didn't include it in the installs of the Arch packages you depended on to build Chapel in your Arch VM.
I'm pretty sure removing this dependence isn't what's causing the issue (since we can both follow the PKGBUILD's instructions without makepkg and get working compilers, but should I put back the dependence on libtool?

Great, glad you found it! Is there a way for me to get around this at build-time when packaging 1.26.0? Your PR will fix things automatically for chapel-git (which we'll get up and running soon after this PR merges, I hope), but for the chapel package I'll have to do the fixes after downloading the release. I suppose I could do patches, but is there a quicker and simpler way via configure and make flags, etc.?

[mppf]

I suppose I could do patches, but is there a quicker and simpler way via configure and make flags, etc.?

I think it has to be a patch, for this one, unfortunately. The only thing I can think of is somehow getting the pkgbuild stuff to install the .la files but presumably there is a reason it does not.

[mppf]

should I put back the dependence on libtool?

Would you be able to do an experiment to see if libqthread.la is installed if the package has a dependence on libtool? (It might be that makepkg is doing something more clever than we first expected, here).

I don't think we need libtool for any other reason, but could be forgetting/missing something.

[ghbrown]

I'm relatively inexperienced at patching manually, but I'm not optimistic about it in this case.
First, just patching the files changed in PR #19846 gives build errors because I assume other files also changed in a way that breaks things. Chasing down all of these dependencies and patching them seems like a daunting task.
On the other hand, completely patching the 1.26 release with your branch produces an error-free build (not an error free compiler though!) and includes an almost 200MB patch file. That's larger than I'm comfortable putting in the AUR chapel repo, and I assume it wouldn't be welcome in this repo either.

Now onto the compile-time errors (though you are apparently not getting any?). Both the unpatched release with the compiler flags you suggested and the fully patched release with no compiler flags give errors like this

/usr/bin/ld: /usr//lib/chapel/1.26/third-party/hwloc/install/linux64-x86_64-native-llvm-pic-flat/lib/libhwloc.a(topology-linux.o): in function `hwloc_linux_set_thisthread_membind':
(.text+0x112e): undefined reference to `set_mempolicy'
/usr/bin/ld: (.text+0x1166): undefined reference to `set_mempolicy'
/usr/bin/ld: (.text+0x1200): undefined reference to `migrate_pages'
/usr/bin/ld: /usr//lib/chapel/1.26/third-party/hwloc/install/linux64-x86_64-native-llvm-pic-flat/lib/libhwloc.a(topology-linux.o): in function `hwloc_linux_get_thisthread_membind':
(.text+0x12b3): undefined reference to `get_mempolicy'
/usr/bin/ld: (.text+0x1362): undefined reference to `get_mempolicy'
/usr/bin/ld: /usr//lib/chapel/1.26/third-party/hwloc/install/linux64-x86_64-native-llvm-pic-flat/lib/libhwloc.a(topology-linux.o): in function `hwloc_linux_get_area_membind':
(.text+0x1506): undefined reference to `get_mempolicy'
/usr/bin/ld: (.text+0x16a2): undefined reference to `get_mempolicy'
/usr/bin/ld: (.text+0x17e9): undefined reference to `get_mempolicy'
/usr/bin/ld: /usr//lib/chapel/1.26/third-party/hwloc/install/linux64-x86_64-native-llvm-pic-flat/lib/libhwloc.a(topology-linux.o): in function `hwloc_linux_set_area_membind':
(.text+0x1b09): undefined reference to `mbind'
/usr/bin/ld: (.text+0x1b43): undefined reference to `mbind'
/usr/bin/ld: /usr//lib/chapel/1.26/third-party/hwloc/install/linux64-x86_64-native-llvm-pic-flat/lib/libhwloc.a(topology-linux.o): in function `hwloc_linux_get_area_memlocation':
(.text+0x1cad): undefined reference to `move_pages'
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
error: Make Binary - Linking

Perhaps I'm doing something wrong?

As for package dependencies, I added GMP and libtool for now. I tried to do some A/B testing to see if it made a difference, but wasn't sure exactly what to look for (please let me know and I'll retry). For the libtool test I ran # find /usr/* -name libqthread.la after install, but got no matches.

[ghbrown]

@mppf I started work on a chapel-git package for AUR since I thought it might be nice to have a working package while we work on getting the main one shored up via patches, etc. However, even cloning and building from your unmerged adjust-install PR branch appears to result in the same errors as my previous comment.

I also tried to spot differences in compilation commands with --print-commands. The logfiles for the failing chapel-git and working version from your adjust-install branch (which I just manually configured and built) are attached, where I tried to compile hello2.
chapel-git.log
chapel_mppf.log
I don't know exactly what to look for, but a search for hwloc reveals that the working version has two more instances (8 compared to 6) than the failing version. However, I think the extra pair is just a duplicate of an already existing -L -l pair, so this may be a red herring.

[mppf]

The symbols you mentioned ought to be brought in by -lnuma which I would have imagined would be required by hwloc. Not sure what is going wrong though.

[mppf]

Oh, maybe the difference between our two systems is that you have libnuma installed somewhere and hwloc is auto-detecting it? E.g., for me, I get this:

$ gcc test.c -lnuma
/usr/bin/ld: cannot find -lnuma: No such file or directory
collect2: error: ld returned 1 exit status

I think I can install numactl with pacman -S numactl.
If I do that, third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib/libhwloc.la includes dependency_libs=' -lm -lnuma' and our build system normally figures out to add the -lnuma by looking at this .la file. But the Arch install process likes to delete the .la files, for some reason.

Given that the -lnuma shouldn't be added if hwloc didn't configure itself to work with libnuma, I don't think we can just always add -lnuma. However you could make the AUR package depend on numactl and then the AUR package could always add -lnuma (but I think that would have to be through a patch or some sort of custom setting to activate it).

I'm relatively inexperienced at patching manually, but I'm not optimistic about it in this case. First, just patching the files changed in PR #19846 gives build errors because I assume other files also changed in a way that breaks things.

I could probably create a series of patches against 1.26 but let's figure out how to get main working first.

[mppf]

It looks like (from my previous AUR install) that it does install the hwloc.pc file. Given that, I think that we could use it instead of libhwloc.la - potentially in all cases - but we could at least do that if libhwloc.la is missing.

An aside, I am curious why the AUR install process deletes the .la files, if you are able to figure that out.

[mppf]

@ghbrown - I've created PR #19913 to hopefully address the .la files issue (which you were observing as a link errors related to missing -lnuma). It worked in my testing on an Arch system to resolve the problems when building and installing the AUR package. In order to test it, I needed to create an AUR package that tested my development branch. (I have no idea what I am doing, but I tried to base it off of https://aur.archlinux.org/packages/yay-git ). Files I used are in the details section below in case they are useful to you.

.SRCINFO

pkgbase = chapel-git
	pkgdesc = Programming language designed for productive parallel computing at scale
	pkgver = 11.1.2.r0.gae01f8e
	pkgrel = 1
	url = https://chapel-lang.org/
	arch = x86_64
	license = Apache
	makedepends = git
	makedepends = cmake
	depends = python
	depends = perl
	depends = llvm
	depends = clang
        source = chapel::git+https://github.com/mppf/chapel.git#branch=less-la
	sha256sums = SKIP

pkgname = chapel-git

PKGBUILD

#    Maintainer: J. Emiliano Deustua <[email protected]>
# Co-maintainer: Gabriel Brown <[email protected]>

pkgname="chapel-git"
_pkgname="chapel"
pkgver=c579b55d1a
pkgrel=1
pkgdesc="Programming language designed for productive parallel computing at scale"
url="https://chapel-lang.org/"
arch=('x86_64')
license=('Apache')
depends=('python' 'perl' 'llvm' 'clang')
makedepends=('git' 'cmake')
source=("chapel::git+https://github.com/mppf/chapel.git#branch=less-la")
sha256sums=("SKIP")

pkgver() {
  cd "$srcdir/$_pkgname"
  git describe --long --tags --always | sed 's/^v//;s/\([^-]*-g\)/r\1/;s/-/./g'
}

build() {
        cd "$srcdir/$_pkgname"
        #export CHPL_LIB_PIC=pic # remove on next release a la
        # https://github.com/chapel-lang/chapel/pull/19785
        ./configure --prefix=/usr
        make
}

check() {
        cd "$srcdir/$_pkgname"
        export PATH="$srcdir/$_pkgname/bin/linux64-x86_64:$PATH"
        make check
}

package() {
        cd "$srcdir/$_pkgname"
        make DESTDIR="${pkgdir}" install
}

Next steps are:

• More testing and review of PR Don't use libtool .la files to get 3p lib flags #19913 (which I will take care of)
• Creating an AUR package for chapel-git based off of https://github.com/chapel-lang/chapel so that we can start to verify everything is working once my PR merges. Can you help with this part?
• Once we have a chapel-git AUR install working, we need to decide what to do with the main chapel AUR package. Could you suggest an approach here?
  • We could add patches to the AUR package. If needed, I would be happy to help rebase the patches I have developed to support Arch/AUR against 1.26. Or...
  • We are about a month out from releasing 1.27, so we could decide to drop the 1.26 AUR package and focus on 1.27
  • Or we could make the 1.26 version use fifo tasking instead of qthreads, which will avoid libhwloc and the -lnuma problem altogether, even though it creates a less-than-ideal configuration. But that might be OK if it is just for one version.

Thanks for your continuing help with this @ghbrown !

[ghbrown]

Apologies for the delay, I was out of town and unavailable for a bit. I plan to catch up on Chapel issues and PRs starting tonight.

Creating an AUR package for chapel-git based off of https://github.com/chapel-lang/chapel so that we can start to verify everything is working once my PR merges. Can you help with this part?

Sure, I'd be happy to. Your example PKGBUILD already looks about right, which is great.

We are about a month out from releasing 1.27, so we could decide to drop the 1.26 AUR package and focus on 1.27

I think this is probably the best use of time. I can temporarily change the description of AUR chapel to direct users to chapel-git (once it's up and running) and note that it will revert on upcoming 1.27.

[ghbrown]

I've created chapel-git based on your less-la branch until it's merged, and added the files here. I also added the redirect to the description of standard chapel AUR (though I haven't committed those minor changes here).

[mppf]

@ghbrown - quick question - is chapel-git working for you at this point?

[ghbrown]

Yes, but I am working on my personal machine. Let me know if you find any issues.

[mppf]

Hi @ghbrown - I've just merged #19913, so when you are able it's a good time to update the AUR package with the main branch and check that it works. Once we have those things working OK I'll give this PR a real review and we'll seek to merge it.

[ghbrown]

Great, thanks. I can make the change tomorrow morning, although it will only be to chapel-git while this PR now contains both non-git and git versions.

[ghbrown]

@mppf When I use git describe --long --tags --always on main I get 1.24.0-7648-gbcc4ac3109, but I expected it to begin with 1.26 or 1.27.
When I was building based on your branch there was no version number at the beginning.
This is my first time using git describe, but this output doesn't seem quite right to me. Perhaps I'm missing something?

[mppf]

@mppf When I use git describe --long --tags --always on main I get 1.24.0-7648-gbcc4ac3109, but I expected it to begin with 1.26 or 1.27. When I was building based on your branch there was no version number at the beginning. This is my first time using git describe, but this output doesn't seem quite right to me. Perhaps I'm missing something?

Huh, that is weird. As you are observing, git describe --long --tags --always worked with my main branch but not with upstream:

# in a checkout of mppf/chapel main
$ git describe --long --tags --always
1.26.0-1346-gbcc4ac3109

# different behavior for upstream main
$ cd /tmp
$ git clone https://github.com/chapel-lang/chapel
$ cd chapel
$ git describe --long --tags --always
1.24.0-7648-gbcc4ac3109

The tag 1.26.0 corresponds to commit d80a127 but that commit is not in the git log history for the main branch. But AFAIK that is also true of my branch. So I don't know what's going on here.

You could consider doing some scripting to parse compiler/main/version_num.h or you could build chpl and use chpl --version.

[ghbrown]

Thanks for confirming and checking it out.
I think I need to do this before build, so I'll go for the scripting to combine the stored text with the hash at the end of git describe. Hopefully that will only be a temporary solution though.

[mppf]

If you are just getting the hash maybe another command (not git describe) would be better. E.g. git log --pretty=format:'%h' -n 1 give a short hash or git rev-parse HEAD gives the long hash.

[ghbrown]

I think I'm just going to replace the leading part of git describe with a hardcoded version or a version from the header. I can split the git describe on the first hypen and use the rest.

[mppf]

Please let me know if you feel this version is ready to merge into the Chapel repo or if there is something else you want to do first. I think it's ready to merge once you are satisfied with it. Thanks again for your efforts on the Chapel AUR package!

[mppf]

Do you know what the stuff after the 1.27.0 here means? I copied it from something else but I have no idea what it's purpose is. I guess I'm just checking that you know more than I do about it :)

[ghbrown]

Unsure exactly what you mean by "copied" it from something else?
The output of the pkgver() function automatically overwrites the pkgver line in PKGBUILD (I think the makepkg command probably feeds it to sed or something behind the scenes?).
As for what it means, I'm unsure on the central 4-digit part, but the very last part is g<beginning of most recent commit hash.

[mppf]

When I made a hacky chapel-git for my own testing I just copied the version number from the other AUR project I was basing it on. Anyway since it will be overwritten in the PKGBUILD file by running pkgver(), it might be nice to have a comment to that effect next to the pkgver= line.

[ghbrown]

Thanks for clarifying. Just added that comment.

[ghbrown]

Overall, I'm happy with the state of the PR, but just to make sure we are on the same page about its outstanding deficiencies:

• the chapel AUR package is currently broken since it is based directly on 1.26, which has issues that prevent it from building on Arch; its description will point to chapel-git until 1.27
• the AUR version is being generated in a robust (but non-ideal) way, since git describe gives confusing outputs involving older versions at present; ideally this will be resolved and we can use more traditional commands in pkgver() in the future

[mppf]

@ghbrown - I forgot to ask earlier, but the PR message is now out of date. Would you mind updating it? It would be a good place to call out those deficiencies you just mentioned.

[ghbrown]

I hope that was the correct approach?

[mppf]

Oh I'm not sure what you did. I was just talking about the text box at the top of the PR that starts with

As per the discourse discussion, I provide AUR packaging files and instructions to test the build.

We (as a matter of practice) turn that into the commit message for the merge when we merge a PR, so its content matters. (Besides browsing PRs when understanding history).

Anyway this line is no longer true, right?

Currently, the included recipe results in a no-error build, but the compiler gives linker errors when chpl is used on even simple programs.

[ghbrown]

Ah, thanks. I changed the final git commit message to be more substantial and detailed (following the trend of some others I'd seen in the git log), but didn't change the PR comment here.
To be clear, I should update and correct the first comment in this PR, then use that as the git commit message? Should I use markdown syntaxed version or just plain text?

[mppf]

Ah, thanks. I changed the final git commit message to be more substantial and detailed (following the trend of some others I'd seen in the git log), but didn't change the PR comment here.

The commit messages can be whatever they were at the time - no need to update them all. The detailed commit messages you are seeing are from merge commits when we paste the PR message (that appears at the top of the PR webpage) into the merge commit message. For the most part our project focuses on PRs as the unit of archeology.

To be clear, I should update and correct the first comment in this PR, then use that as the git commit message? Should I use markdown syntaxed version or just plain text?

No need to worry about the git commit messages for this. Just update the PR message (which I think you are calling the 1st comment in the PR but it's at the top of the PR webpage either way). Markdown or plain text are both fine - up to you. I do tend to use markdown myself most of the time.

[ghbrown]

The commit messages can be whatever they were at the time - no need to update them all. The detailed commit messages you are seeing are from merge commits when we paste the PR message

Yes, thankfully I only changed the last commit message (which was probably for the best anyway). The PR message should be changed too.

[ghbrown]

@mppf Thank you for all your help and support on this, it was a pleasure!