OpenWRT

This page attempts to explain various strategies for getting Haskell programs to run on OpenWRT. Since (AFAIK?) no attempt has been made to integrate ghc into OpenWRT's buildroot, the strategies here are pretty much synonymous with cross-compiling and static linking Haskell programs in general, and are not specific to OpenWRT.

Cross-compiling for MIPS

wereHamster has a github repository of Dockerfiles for building a GHC cross-compiler for the Onion Omega.

Cross-compiling for x86_64

(Apologies for the first-person nature of the text; I originally wrote it as a blog post. Please feel free to edit it to be more third-person and wiki-appropriate.)

I'd like to be able to run Haskell programs on my OpenWRT box. Since my OpenWRT box has an x86_64 processor, it could almost run the same binaries that my desktop Linux machine does. However, the reason it can't is because it uses a different C library. Desktop Linux (specifically Ubuntu in my case) uses glibc. OpenWRT used uclibc in past versions, and uses musl on trunk. Since binaries are normally linked dynamically (and in the case of glibc, they pretty much have to be), the target machine needs to have the same C library that the executable was built with.

One approach would be to dynamically link against the C library that OpenWRT uses. However, an even more robust approach is to link the executable statically. Then it can run on any x86_64 Linux machine, regardless of what libraries it has installed.

Unfortunately, glibc doesn't really work correctly with fully static linking. Therefore, another C library needs to be used. The best choice seems to be musl. Coincidentally, musl is also the C library used by OpenWRT trunk, but since we're going to link statically, it wouldn't have to be.

So, how do we compile and link Haskell programs against musl? One approach is to use a musl version of ghc. Unfortunately, that approach requires a full musl-based Linux system to run on. The author of that post recommends a musl-based Gentoo distribution, but that seemed kind of icky to me. I almost could use the OpenWRT box as the musl-based build system, but that would require a working gcc on the OpenWRT box. Currently, on OpenWRT trunk, the gcc package is broken.

What I really want is a cross-compiler, which will run on my glibc-based Ubuntu box, but will build executables that use statically-linked musl. That, in turn, requires having a cross-compiling toolchain (gcc, binutils, etc.) before building the ghc cross-compiler. There are several ways to satisfy this requirement, but after a recommendation on the haskell-cafe mailing list, I decided to give Crosstool-NG a try.

I discovered that, in addition to the packages already installed on my system, I needed to install these packages in order to get Crosstool-NG to work:

ppelletier@patrick64:~/src$ sudo apt-get install gperf bison texinfo help2man python-dev

To avoid confusion, keep in mind that there are four directories relevant to Crosstool-NG:

1. The directory you build Crosstool-NG in. In this example, it is ~/src/crosstool-ng
2. The directory you install Crosstool-NG in. In this example, it is /usr/local
3. The directory you build the toolchain in. In this example, it is ~/crosstool-ng
4. The directory you install the toolchain in. In this example, it is ~/x-tools/x86_64-unknown-linux-musl (which I did not have to specify explicitly, because it is the default)

(Once you've installed Crosstool-NG, you can delete the directory in (1), and once you've installed the toolchain, you can delete the directory in (3), to save space. The end result you care about is in directory (4).)

So, I got started with Crosstool-NG:

ppelletier@patrick64:~/src$ curl -O http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.22.0.tar.xz
ppelletier@patrick64:~/src$ tar -xf crosstool-ng-1.22.0.tar.xz
ppelletier@patrick64:~/src$ cd crosstool-ng/
ppelletier@patrick64:~/src/crosstool-ng$ ./configure --prefix=/usr/local
ppelletier@patrick64:~/src/crosstool-ng$ make
ppelletier@patrick64:~/src/crosstool-ng$ sudo make install
ppelletier@patrick64:~/src/crosstool-ng$ mkdir ~/crosstool-ng
ppelletier@patrick64:~/src/crosstool-ng$ cd ~/crosstool-ng
ppelletier@patrick64:~/crosstool-ng$ ct-ng x86_64-unknown-linux-gnu
ppelletier@patrick64:~/crosstool-ng$ ct-ng menuconfig

In the menuconfig, I selected C-library > C library > musl, and it's also a good idea to de-select Paths and misc options > Render the toolchain read-only, because I'm going to install the GHC cross-compiler in the same directory tree, so the directories need to be writable.

Then, I was able to do:

ppelletier@patrick64:~/crosstool-ng$ ct-ng build

And a little over an hour later, I could do:

ppelletier@patrick64:~/x-tools$ export PATH="${PATH}:${HOME}/x-tools/x86_64-unknown-linux-musl/bin"
ppelletier@patrick64:~/x-tools$ x86_64-unknown-linux-musl-gcc --version
x86_64-unknown-linux-musl-gcc (crosstool-NG crosstool-ng-1.22.0) 5.2.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Next, I read the instructions for cross-compiling ghc, and then I did:

ppelletier@patrick64:~/src$ curl -O http://downloads.haskell.org/~ghc/7.10.3/ghc-7.10.3b-src.tar.xz
ppelletier@patrick64:~/src$ tar -xf ghc-7.10.3b-src.tar.xz
ppelletier@patrick64:~/src$ cd ghc-7.10.3/

And then I attempted to build GHC. I ran into some issues along the way, so I'm going to present the fixes to those issues now, rather than reconstructing the exact order in which I did things.

The main issue is that GHC needs ncurses on the target. I'm not sure why, but it does. So, I had to go and cross-compile ncurses:

ppelletier@patrick64:~/src/ncurses-6.0$ ./configure --host=x86_64-unknown-linux-musl --prefix=${HOME}/x-tools/x86_64-unknown-linux-musl
ppelletier@patrick64:~/src/ncurses-6.0$ make

Well, that ended up failing with:

x86_64-unknown-linux-musl-gcc -DHAVE_CONFIG_H -I. -I../include -DNDEBUG -O2 --param max-inline-insns-single=1200 -c ../ncurses/lib_gen.c -o ../objects/lib_gen.o
In file included from ./curses.priv.h:325:0,
                 from ../ncurses/lib_gen.c:19:
_20591.c:843:15: error: expected ')' before 'int'
../include/curses.h:1631:56: note: in definition of macro 'mouse_trafo'
 #define mouse_trafo(y,x,to_screen) wmouse_trafo(stdscr,y,x,to_screen)
                                                        ^
make[1]: *** [../objects/lib_gen.o] Error 1
make[1]: Leaving directory `/home/ppelletier/src/ncurses-6.0/ncurses'
make: *** [all] Error 2

I found the answer in wereHamster's Dockerfile. In ncurses/base/MKlib_gen.sh, line 65 needs to be changed from:

preprocessor="$1 -DNCURSES_INTERNALS -I../include"

to:

preprocessor="$1 -P -DNCURSES_INTERNALS -I../include"

Okay, now the make succeeds, and after make, I do:

ppelletier@patrick64:~/src/ncurses-6.0$ make install

However, this is not enough. I thought that by installing it under the cross toolchain's prefix, gcc would automatically pick up the ncurses includes. But it didn't. So, I need to specify --with-curses-includes and --with-curses-libraries to ghc's configure. Unfortunately, there doesn't seem to be a single value that works for --with-curses-includes, and it doesn't seem to be possible to specify more than one value for --with-curses-includes.

If I use --with-curses-includes=/home/ppelletier/x-tools/x86_64-unknown-linux-musl/include, then the build is unable to find ncurses.h. However, if I use --with-curses-includes=/home/ppelletier/x-tools/x86_64-unknown-linux-musl/include/ncurses, then ncurses.h includes ncurses/ncurses_dll.h, which is not found. My really ugly solution was:

ppelletier@patrick64:~/x-tools/x86_64-unknown-linux-musl/include/ncurses$ ln -s . ncurses

Now we're ready to actually build ghc successfully. In the ~/src/ghc-7.10.3/mk directory, I copied build.mk.sample to build.mk, and then uncommented BuildFlavour = quick-cross. Next I did:

ppelletier@patrick64:~/src/ghc-7.10.3$ ./configure --target=x86_64-unknown-linux-musl --with-curses-includes=/home/ppelletier/x-tools/x86_64-unknown-linux-musl/include/ncurses --with-curses-libraries=/home/ppelletier/x-tools/x86_64-unknown-linux-musl/lib --prefix=/home/ppelletier/x-tools/x86_64-unknown-linux-musl
ppelletier@patrick64:~/src/ghc-7.10.3$ make
ppelletier@patrick64:~/src/ghc-7.10.3$ make install

It turns out that this toolchain still links dynamically by default, so I have to specify -static -optl-static when compiling Haskell programs. For example:

ppelletier@patrick64:~/programming/haskell$ x86_64-unknown-linux-musl-ghc -O -static -optl-static contact.hs
[1 of 1] Compiling Main             ( contact.hs, contact.o )
Linking contact ...
ppelletier@patrick64:~/programming/haskell$ file contact
contact: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), statically linked, not stripped
ppelletier@patrick64:~/programming/haskell$ ldd contact
        not a dynamic executable

The contact executable can now be run on either my Ubuntu box or my OpenWRT box.

However, in order to build nontrivial programs, I need cabal. The GHC cross-compiling page recommends:

cabal --with-ghc=<cross-ghc> --with-ld=<ld> ...

So I expanded this approach to:

alias mcabal="cabal --with-ghc=x86_64-unknown-linux-musl-ghc --with-ld=x86_64-unknown-linux-musl-ld --with-gcc=x86_64-unknown-linux-musl-gcc --with-ghc-pkg=x86_64-unknown-linux-musl-ghc-pkg"

I'm able to install some packages with mcabal. (For example, mcabal install hourglass.)

But with cryptonite, I get:

ppelletier@patrick64:~/src/lifx/lifx-programs$ mcabal install --flags=-integer-gmp cryptonite
Resolving dependencies...
Configuring memory-0.12...
Building memory-0.12...
Preprocessing library memory-0.12...
running dist/dist-sandbox-5f352b7e/build/Data/Memory/MemMap/Posix_hsc_make failed (exit code 127)
command was: dist/dist-sandbox-5f352b7e/build/Data/Memory/MemMap/Posix_hsc_make >dist/dist-sandbox-5f352b7e/build/Data/Memory/MemMap/Posix.hs
Failed to install memory-0.12
cabal: Error: some packages failed to install:
cryptonite-0.15 depends on memory-0.12 which failed to install.
memory-0.12 failed during the building phase. The exception was:
ExitFailure 1

I haven't been able to figure out exactly what's going on there.

Or, mcabal install aeson fails with:

Building aeson-0.11.1.4...
Preprocessing library aeson-0.11.1.4...
ghc: Data/Aeson/TH.hs:9:14-28: Template Haskell requires GHC with interpreter support
    Perhaps you are using a stage-1 compiler?
Usage: For basic information, try the `--help' option.
Failed to install aeson-0.11.1.4
cabal: Error: some packages failed to install:
aeson-0.11.1.4 failed during the building phase. The exception was:
ExitFailure 1

Native GHC for musl

Given the issues with cabal and Template Haskell when cross-compiling, the best solution (for x86_64, at least) may be a native GHC compiler which is linked against musl.

Marios Titas has a haskell-cafe post and a github repository about a musl-native ghc toolchain.

If you prefer Docker over chroot, there is a Docker image based on Alpine Linux, and the corresponding Dockerfile is on github.

Here is an example of using that Docker image to build a static executable with cabal:

ppelletier@patrick64:~$ sudo docker run -it nilcons/ghc-musl:cabal sh
Unable to find image 'nilcons/ghc-musl:cabal' locally
cabal: Pulling from nilcons/ghc-musl
c862d82a67a2: Pull complete
00b183080618: Pull complete
7193868286e6: Pull complete
a3ed95caeb02: Pull complete
bbfc077cc684: Pull complete
Digest: sha256:27657bba49ed54b57d10c964ff59fc275d47d26e3e3a8e5991c61edc9f5f40b2
Status: Downloaded newer image for nilcons/ghc-musl:cabal
/ # cabal update
Downloading the latest package list from hackage.haskell.org
/ # cabal install --ghc-option=-optl-static hello
Resolving dependencies...
Configuring hello-1.0.0.2...
Building hello-1.0.0.2...
Installed hello-1.0.0.2
/ # which hello
/root/.cabal/bin/hello
/ # hello
Hello, World!
/ # file /root/.cabal/bin/hello
/root/.cabal/bin/hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
/ # ldd /root/.cabal/bin/hello
ldd: /root/.cabal/bin/hello: Not a valid dynamic program