Feature Request: Official ARM support

[blitzcode]

The current process of getting a Stack Haskell application build on ARM is rather... tricky:

https://github.com/blitzcode/hue-dashboard/#raspberry-pi

I'd love for Stack to have an official ARM bindist and be able to install GHC/LLVM with stack setup.

[borsboom]

The main thing we'd need to start to start generating ARM binaries for releases is a Vagrantfile that sets up a build environment similar to the ones found in https://github.com/commercialhaskell/stack/tree/master/etc/vagrant. I guess that Vagrantfile would have to launch a VM on an Arm-supporting VPS provider (or in an emulator, but that might be really slow).

From your link, it sounds like we may also want to add support for installing LLVM to stack setup. Although I guess if you already have LLVM, stack setup might already be able to take care of installing GHC (at least once we add a link to the correct GHC bindist for ARM to https://github.com/fpco/stackage-content/blob/master/stack/stack-setup-2.yaml).

[blitzcode]

I initially tried building on QEMU, but had little success with it. The Raspberry Pi with its 1GB of RAM might be problematic for reliable builds. As you suggest, a VPS provider that offers server-grade ARM systems with enough memory seems like the best option.

I filed a issue #1742 a while ago to suggest stack should install LLVM as it's often difficult to pick the right version and different GHC versions need different LLVMs. Wasn't there talk of GHC8 bundling the right LLVM anyway?

[quyse]

@blitzcode Just a few notes, I have set up an ARM machine on Scaleway few months ago for CI purposes, and it's doing great. I would advise to use Arch Linux distribution instead of Debian/Ubuntu, as it has up-to-date binary packages, in particular GHC 7.10.3 (on ARM too) and LLVM, which work together flawlessly. As I remember the setup was relatively straightforward, just install ghc, llvm35, cabal-install from packages, add some swap space (2Gb RAM will not be enough), compile stack from sources, and that's it. Don't do parallel builds, as it will run out of memory. Unfortunately I don't know other good ARM cloud providers, I would love to hear about any providing bigger ARM machines in terms of memory.

[blitzcode]

@quyse Yes, I was checking them out, seems really affordable. 2GB really wasn't enough? I got everything to build with the 1GB RPi and 100MB swap, I figure 2GB should be smooth sailing. Maybe only with -j1 / -j2, though. I'm a bit wary of Arch simply because I don't like the idea of being forced to upgrade and I want reproduceability. If 1 year from now the SD card in a RPi I deployed dies, I'd want to be able to set up the exact same system again. If Arch has already moved to GHC8 by then, I might be in trouble with my LTS 5.x code. At least that's my worry without ever having tried Arch. It seems otherwise much easier to get going than Debian Jessie with its dated packages...

[quyse]

@blitzcode I've checked again what I'm doing there, actually I'm not specifying -j1, so I guess it's actually -j4 by default (because of 4 cores), no wonder it was running out of memory :) With additional swap space it's fine though. -j1 is just too slow.
Good point about Arch, yes, you are usually forced to upgrade. You can freeze your system's state by simply not upgrading, you also can make a backup of your pacman cache, so you can restore packages in the future without contacting distribution servers. But, yes, probably easier to stick with Debian in your case. I'm just not worrying too much about reproduceability at the moment, as I'm upgrading regularly to the latest LTS.

[borsboom]

I'm trying to build a stack binary on a Scaleway VM running Debian Jessie (to match the GHC bindist), but am having trouble with LLVM. I downloaded an LLVM bindist as recommended in here, but when I try to use it I get:

# opt --version
Illegal instruction

Any ideas where I can easily get an LLVM that will work?

[blitzcode]

@borsboom That's puzzling, last time I checked their servers were armv7 / 32 bit, same as the Raspberry Pi with Raspbian. So it's really strange that the LLVM bindist would not work.

Could you maybe do a lscpu, uname -a and maybe run file or whatever on the LLVM bindist and see if the architectures etc. match? I also sometimes hear that SoC vendors have a lot of discretion on what to include in their ARM cores, like NEON, 32/64 bit, etc, so maybe there's something different about the server-grade SoCs Scaleway uses. No idea.

[borsboom]

@blitzcode:

root@scw-476b18:~# uname -a
Linux scw-476b18 4.3.5-std-1 #1 SMP Fri Feb 19 11:52:18 UTC 2016 armv7l GNU/Linux

root@scw-476b18:~# lscpu
Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
Model name:            ARMv7 Processor rev 2 (v7l)
CPU max MHz:           1333.0000
CPU min MHz:           666.5000

root@scw-476b18:~# /usr/local/bin/opt --version
Illegal instruction

root@scw-476b18:~# file /usr/local/bin/opt
/usr/local/bin/opt: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=f99b13805d70643bc8a3485026fb7ab9f6a37f04, stripped

Oddly enough, the file output matches that for ghc (which does run):

root@scw-476b18:~# /opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3

root@scw-476b18:~# file /opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc
/root/opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=c2002709f062e00b92536c9c6770e7fa9d96f7d8, not stripped

[varosi]

@borsboom, you need at least LLVM 3.5.2 or greater. There is a fix related to GHC in it.

[varosi]

@blitzcode, I'm running GHC 7.10.2 with LLVM 3.5.2 on ARMv7 (32bit) and it is working well.

[blitzcode]

@borsboom I'm terribly sorry, I said 3.5.2 in my document and then linked 3.5.1, dammit...

[mgsloan]

Note that another thing that will likely need to be resolved is support for byte alignment in store - mgsloan/store#37 - as ARM cares about this.

[varosi]

Related: haskell-crypto/cryptonite#90

[varosi]

This temporarily workarounds this problem:

cabal install --dependencies-only --constraint=cryptonite==0.15

[andreyk0]

Jumping here from #1332

I've finally taken some time to untangle haskell/arm stuff from my personal files, here's the result:
https://github.com/andreyk0/haskell-on-arm

Hope it helps somebody.

@varosi cryptonite used to be an issue for me at some point but haven't had any problems recently. Something got fixed, I imagine.

[varosi]

@andreyk0, fix is not yet pushed in Hackage and I still have this problem on my ARM.

[andreyk0]

@varosi hmm, could be my environment then. I don't remember everything but (it's in the readme in that repo and my images are on dockerhub) I've installed LLVM from the project's downloads page and hacked 7.10.2 to be a symlink to 7.10.3, so there's no 7.10.2 in my system.

[varosi]

with cryptonite-0.15 it is working fine on IMX.53 board. So I'm building project successfully with Stack. It'll be great if Stack have official distro for ARM and support stack setup.

[borsboom]

I did some more experimenting with this. I got Debian Jessie for armhf running in qemu (note: the LLVM bindists are not compatible with armel or with Wheezy for armhf which I tried first) and things seem to be working, but very slow. I'm building cabal-install and it's been running for about 10 hours now, and looks like it might be finished in the next hour or so. At that rate, Stack will take at least a couple of days to build, and I expect the integration tests to take another day or so to run. At that point I'll at least know whether Stack works properly on ARMv7.

The situation with LLVM bindists is annoying though; they only seem to support fairly specific versions of distros (unlike GHC bindists, which tend to work on a wide range). And since each GHC version needs a specific version of LLVM, we can't leave it up to the Linux distro to install LLVM for us. So we could officially support ARMv7 on Jessie armhf, but not necessarily on any other distributions or versions.

[blitzcode]

@borsboom Good to hear, but I'm surprised that the build times are so much worse than on the RPi. I've read several times about good results with QEMU. Hope you build Cabal with EXTRA_CONFIGURE_OPTS="" ./bootstrap.sh, it helps quite a bit!

In the meantime, Stack v1.1 has been working absolutely without issues for me on ARM after the initial setup pains.

Debian Wheezy only seems to have LLVM 3.1 available, Wheezy only goes to 3.5, had to download the LLVM binaries. Where on the LLVM page does it actually say with which distros the binaries have been tested?

[borsboom]

Thanks for the tip about EXTRA_CONFIGURE_OPTS; I didn't know that!

I didn't see anywhere on the LLVM page where it mentioned the distros. But when I tried clang+llvm-3.7.1-armv7a-linux-gnueabihf (I'm using GHC 8.0.1, so the newer LLVM is required) on Wheezy, this happened:

manny@debarm:~$ clang
clang: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by clang)
clang: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.15' not found (required by clang)

[blitzcode]

Oh, it's gated by glibc, that makes sense. That's an unfortunate problem...

Once you have Cabal be sure to use --disable-library-stripping --disable-executable-stripping and / or set executable-stripping: False + library-stripping: False in Cabal's ~/.cabal/config. I had to start over and rebuild all of Stack's dependencies on my first try :/

[varosi]

I have tested latest cryptonite-0.17 on ARMv7 and the compilation issue from 0.16 is fixed (haskell-crypto/cryptonite#90) there. So Stack will be building alright now.

[andreyk0]

@varosi yeah, I must've hacked flags long time ago and forgot about it, sorry for the noise, you were right. Had the same problem with a clean ghc8.0.1 build, luckily it's fixed now!

[andreyk0]

@borsboom " ARMv7 on Jessie armhf" is probably the way to go, too much pain to get this stuff to work anywhere else for now.
Regarding QEMU build times: I only used cabal-install once, about a year ago to build stack itself. Since then using stack to build packages and newer versions of stack itself. Most packages and my tiny projects compile in a few minutes, which is reasonable. There are outliers though, e.g. compiling regex-tdfa the other day with ghc8.0.1 took over 2h. Don't know why. At least stack will reuse the binaries, which helps a lot.

[borsboom]

Replying to myself here, regarding #2103 (comment): I've seen some issues in other projects that need ARM CPUs with NEON support and have the same Illegal Instruction error on Scaleway (where the CPU doesn't have NEON support), so I'm guessing that's what is going on with the LLVM bindist. Annoying, because this would be much faster/easier if I could build on Scaleway.

[varosi-chaosgroup]

For small and slow machines like Raspberry Pi, it would be great if there is a binary distribution.

[fosskers]

@nh2 would your static compilation Nix magic make this more possible?

[vielmetti]

On arm64, I'm trying to build Cilium. Cilium depends on Istio and Envoy, and several of those packages use hadolint to lint their Dockerfiles. Building hadolint depends on stack. & that brings me here.

From what I can see, the CI for stack is Azure Pipelines, and there is an external agent support for Azure recently that can support arm64. See microsoft/azure-pipelines-agent#1911 for that issue.

[ryandesign]

Apple will likely migrate from Intel to ARM processors for their computers in following years?

With the first models scheduled to appear before the end of this year, in fact. So hopefully someone is working rapidly toward getting ghc and stack ready for that.

[varosi]

If Stack developers need someone to test Stack on such a machine - I'll test it.

@borsboom @fuszenecker

[fosskers]

Yes please.

[jmatsushita]

@Fleuv I got the same error message you did using Debian Buster which uses tinfo6. I could circumvent the issue by using an ubuntu:18.04 base image which allowed stack to find an aarch64 GHC bindist (since this version of ubuntu seems to use tinfo5). I could probably have used Debian 9 instead, since that's the bindist that stack used anyway.

[tolysz]

Hi Guys,

I am trying to bootstrap some later stack on aarm64 starting with stack-1.6.6-be415545029d1bcafddb9fc13b8283e3d1b1abb1-linux-aarch64.gz (I moved the urls to http://ghcjs.tolysz.org/ backed by s3).
Then I can build stack-1.9.3.1 (you need to fix aarm64 support first, by adding extra line to setup... otherwise it you will not be able to use any of the compilers after you build it).

I build my own ghc's(how to build it from git) but you should be able to grab some of the official ones, you might need to install some additional llvm compilers it will complain with opt check which version does it need.

It is not this hard to set it on your own, just the bootstrapping is tedious...
Basically you check which ghcyou need for each lts then you build them from sources starting low enough so the native ghc works. Then you extend setup-info.yaml with that compiler, then use stack to stack exec -- build-ghc... sometimes you need to stack exec env and manually setup env vars...

Basically for Nvidia Jetson(Xaviers, nano) 8.0.2(ubuntu18:04) -> 8.2.2 -> 8.6.3; 8.2.2 -> 8.6.5 (need to play with env)
then stack-1.6.6 -> stack-1.9.3.1(patch) -> stack-2.5.0.1

I am not sure how portable all those binaries are... but it should be safe to run them on some scratch servers.
For ghc-8.6.5 you might need libgmp-dev llvm-6.0
PATH=/usr/lib/llvm-6.0/bin/:$PATH stack install happy

UPDATE:
start with the latest link, put it on your path(unpack, rename)...
stack --resolver nightly-2020-10-07 setup
will do the job... if you are lucky(and have libtinfo.so.6)... the people behind stack made some magic... otherwise build your own ghc's (or use the ones I did) (once It builds beyond 8.6.5 I will let you know... it just takes a while to compile)
Just create stack.yaml and you can install custom ghc via:
stack setup --resolver=lts-11.22 --stack-yaml ~/.stack/global-project/stack.yaml

UPDATE2:
Looks like there is a nice task to add nopie libtinfo etc... classification for arm64...

[robwithhair]

Hi there. I’m struggling to build my project on ARM64 on raspberry pi 4 in 64bit. We were hoping to migrate to ARM64 on AWS too for some of our data processing tasks. Are there plans to add ARM64 support back into Stack at all?

[johndpope]

supposedly - there.s a 2.7.1 build in releases - (the 2.7.3 didn't spit out an arm build.)

➜ ~ wget https://github.com/commercialhaskell/stack/releases/download/v2.7.1/stack-2.7.1-linux-aarch64.bin
--2021-11-17 23:48:03-- https://github.com/commercialhaskell/stack/releases/download/v2.7.1/stack-2.7.1-linux-aarch64.bin
Resolving github.com (github.com)... 140.82.112.4
Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/34817922/b1942380-c167-11eb-9ffa-b4f266ae41ce?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211117T234803Z&X-Amz-Expires=300&X-Amz-Signature=eb00f0b69ea9dd6f459416cf19b98ebc15d5ba1ed130ccf38397af4d059f9df1&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=34817922&response-content-disposition=attachment%3B%20filename%3Dstack-2.7.1-linux-aarch64.bin&response-content-type=application%2Foctet-stream [following]
--2021-11-17 23:48:03-- https://github-releases.githubusercontent.com/34817922/b1942380-c167-11eb-9ffa-b4f266ae41ce?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211117T234803Z&X-Amz-Expires=300&X-Amz-Signature=eb00f0b69ea9dd6f459416cf19b98ebc15d5ba1ed130ccf38397af4d059f9df1&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=34817922&response-content-disposition=attachment%3B%20filename%3Dstack-2.7.1-linux-aarch64.bin&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.110.154, 185.199.109.154, 185.199.111.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.110.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 70088040 (67M) [application/octet-stream]
Saving to: ‘stack-2.7.1-linux-aarch64.bin’

100%[============================================================================================================================================>] 70,088,040 46.6MB/s in 1.4s

2021-11-17 23:48:05 (46.6 MB/s) - ‘stack-2.7.1-linux-aarch64.bin’ saved [70088040/70088040]

➜ ~ mv stack-2.7.1-linux-aarch64.bin stack
➜ ~ ls
cabal ghc-8.2.2 hosts ngrok PostGUI stack-2.7.1-linux-aarch64.tar
cabal-install-3.6.0.0-aarch64-linux-deb10.tar.xz ghc-8.2.2-aarch64-deb8-linux.tar.xz index.html postgrest quickstart temp
fundfi go nginx.conf postgrest-binary.tar.gz stack tunnels
➜ ~ chmod +x stack
➜ ~ ./stack
./stack: /lib64/libm.so.6: version GLIBC_2.29' not found (required by ./stack) ./stack: /lib64/libm.so.6: version GLIBC_2.27' not found (required by ./stack)
./stack: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by ./stack)

GLIBC - 2.26 is out of the box with AWS - is it possible to make stack work just on that version for ARM64?

[mpilgrem]

I am going to close this issue because the first release candidate for Stack 2.9.1 does include the Linux AArch64 platform, and the GitHub workflow does build Stack for Linux AArch64. I have opened a separate issue for macOS AArch64.

[wbadart]

I have opened a separate issue for macOS AArch64

Link for bookkeeping: #5854