Editing OpenWRT (section)

== Cross-compiling for x86_64 ==

(Apologies for the first-person nature of the text; I originally wrote it as a blog post.  Please feel free to edit it to be more third-person and wiki-appropriate.)

I'd like to be able to run Haskell programs on my OpenWRT box.
Since my OpenWRT box has an x86_64 processor, it could almost run the
same binaries that my desktop Linux machine does.  However, the reason
it can't is because it uses a different C library.  Desktop Linux
(specifically Ubuntu in my case) uses glibc.  OpenWRT used uclibc in
past versions, and uses [http://www.musl-libc.org/ musl] on trunk.  Since binaries are
normally linked dynamically (and in the case of glibc, they pretty
much have to be), the target machine needs to have the same C library
that the executable was built with.

One approach would be to dynamically link against the C library that
OpenWRT uses.  However, an even more robust approach is to link the
executable statically.  Then it can run on ''any'' x86_64 Linux machine,
regardless of what libraries it has installed.

Unfortunately, glibc doesn't really work correctly with fully static
linking.  Therefore, another C library needs to be used.  The best
choice seems to be musl.  Coincidentally, musl is also the C library
used by OpenWRT trunk, but since we're going to link statically, it
wouldn't have to be.

So, how do we compile and link Haskell programs against musl?  One
approach is to use [https://mail.haskell.org/pipermail/haskell-cafe/2015-May/119843.html a musl version of ghc].  Unfortunately, that
approach requires a full musl-based Linux system to run on.  The
author of that post recommends a musl-based Gentoo distribution, but
that seemed kind of icky to me.  I almost could use the OpenWRT box as
the musl-based build system, but that would require a working gcc on
the OpenWRT box.  Currently, on OpenWRT trunk,
[https://github.com/openwrt/packages/issues/2244 the gcc package is broken].

What I really want is a cross-compiler, which will run on my
glibc-based Ubuntu box, but will build executables that use
statically-linked musl.  That, in turn, requires having a
cross-compiling toolchain (gcc, binutils, etc.) before building the
ghc cross-compiler.  There are several ways to satisfy this
requirement, but after a
[https://mail.haskell.org/pipermail/haskell-cafe/2016-April/123610.html recommendation on the haskell-cafe mailing list], I
decided to give [http://crosstool-ng.org/ Crosstool-NG] a try.

I discovered that, in addition to the packages already installed on my
system, I needed to install these packages in order to get
Crosstool-NG to work:

<pre>
ppelletier@patrick64:~/src$ sudo apt-get install gperf bison texinfo help2man python-dev
</pre>

To avoid confusion, keep in mind that there are four directories relevant to Crosstool-NG:

# The directory you build Crosstool-NG in.  In this example, it is <code>~/src/crosstool-ng</code>
# The directory you install Crosstool-NG in.  In this example, it is <code>/usr/local</code>
# The directory you build the toolchain in.  In this example, it is <code>~/crosstool-ng</code>
# The directory you install the toolchain in.  In this example, it is <code>~/x-tools/x86_64-unknown-linux-musl</code> (which I did not have to specify explicitly, because it is the default)

(Once you've installed Crosstool-NG, you can delete the directory in (1), and once you've installed the toolchain, you can delete the directory in (3), to save space.  The end result you care about is in directory (4).)

So, I got started with Crosstool-NG:

<pre>
ppelletier@patrick64:~/src$ curl -O http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.22.0.tar.xz
ppelletier@patrick64:~/src$ tar -xf crosstool-ng-1.22.0.tar.xz
ppelletier@patrick64:~/src$ cd crosstool-ng/
ppelletier@patrick64:~/src/crosstool-ng$ ./configure --prefix=/usr/local
ppelletier@patrick64:~/src/crosstool-ng$ make
ppelletier@patrick64:~/src/crosstool-ng$ sudo make install
ppelletier@patrick64:~/src/crosstool-ng$ mkdir ~/crosstool-ng
ppelletier@patrick64:~/src/crosstool-ng$ cd ~/crosstool-ng
ppelletier@patrick64:~/crosstool-ng$ ct-ng x86_64-unknown-linux-gnu
ppelletier@patrick64:~/crosstool-ng$ ct-ng menuconfig
</pre>

In the menuconfig, I selected '''C-library > C library > musl''', and
it's also a good idea to de-select
'''Paths and misc options > Render the toolchain read-only''',
because I'm going to install the GHC cross-compiler in the same
directory tree, so the directories need to be writable.

Then, I was able to do:

<pre>
ppelletier@patrick64:~/crosstool-ng$ ct-ng build
</pre>

And a little over an hour later, I could do:

<pre>
ppelletier@patrick64:~/x-tools$ export PATH="${PATH}:${HOME}/x-tools/x86_64-unknown-linux-musl/bin"
ppelletier@patrick64:~/x-tools$ x86_64-unknown-linux-musl-gcc --version
x86_64-unknown-linux-musl-gcc (crosstool-NG crosstool-ng-1.22.0) 5.2.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
</pre>

Next, I read [https://ghc.haskell.org/trac/ghc/wiki/Building/CrossCompiling the instructions for cross-compiling ghc], and then I
did:

<pre>
ppelletier@patrick64:~/src$ curl -O http://downloads.haskell.org/~ghc/7.10.3/ghc-7.10.3b-src.tar.xz
ppelletier@patrick64:~/src$ tar -xf ghc-7.10.3b-src.tar.xz
ppelletier@patrick64:~/src$ cd ghc-7.10.3/
</pre>

And then I attempted to build GHC.  I ran into some issues along the
way, so I'm going to present the fixes to those issues now, rather
than reconstructing the exact order in which I did things.

The main issue is that GHC needs ncurses on the target.  I'm not sure
why, but it does.  So, I had to go and cross-compile ncurses:

<pre>
ppelletier@patrick64:~/src/ncurses-6.0$ ./configure --host=x86_64-unknown-linux-musl --prefix=${HOME}/x-tools/x86_64-unknown-linux-musl
ppelletier@patrick64:~/src/ncurses-6.0$ make
</pre>

Well, that ended up failing with:

<pre>
x86_64-unknown-linux-musl-gcc -DHAVE_CONFIG_H -I. -I../include -DNDEBUG -O2 --param max-inline-insns-single=1200 -c ../ncurses/lib_gen.c -o ../objects/lib_gen.o
In file included from ./curses.priv.h:325:0,
                 from ../ncurses/lib_gen.c:19:
_20591.c:843:15: error: expected ')' before 'int'
../include/curses.h:1631:56: note: in definition of macro 'mouse_trafo'
 #define mouse_trafo(y,x,to_screen) wmouse_trafo(stdscr,y,x,to_screen)
                                                        ^
make[1]: *** [../objects/lib_gen.o] Error 1
make[1]: Leaving directory `/home/ppelletier/src/ncurses-6.0/ncurses'
make: *** [all] Error 2
</pre>

I found the answer in [https://github.com/wereHamster/onion-omega/blob/master/ghc/Dockerfile wereHamster's Dockerfile].  In
<code>ncurses/base/MKlib_gen.sh</code>, line 65 needs to be changed from:

<pre>
preprocessor="$1 -DNCURSES_INTERNALS -I../include"
</pre>

to:

<pre>
preprocessor="$1 -P -DNCURSES_INTERNALS -I../include"
</pre>

Okay, now the <code>make</code> succeeds, and after <code>make</code>, I do:

<pre>
ppelletier@patrick64:~/src/ncurses-6.0$ make install
</pre>

However, this is not enough.  I thought that by installing it under
the cross toolchain's prefix, gcc would automatically pick up the
ncurses includes.  But it didn't.  So, I need to specify
<code>--with-curses-includes</code> and <code>--with-curses-libraries</code> to ghc's
<code>configure</code>.  Unfortunately, there doesn't seem to be a single value
that works for <code>--with-curses-includes</code>, and it doesn't seem to be
possible to specify more than one value for <code>--with-curses-includes</code>.

If I use
<code>--with-curses-includes=/home/ppelletier/x-tools/x86_64-unknown-linux-musl/include</code>,
then the build is unable to find <code>ncurses.h</code>.  However, if I use
<code>--with-curses-includes=/home/ppelletier/x-tools/x86_64-unknown-linux-musl/include/ncurses</code>,
then <code>ncurses.h</code> includes <code>ncurses/ncurses_dll.h</code>, which is not
found.  My really ugly solution was:

<pre>
ppelletier@patrick64:~/x-tools/x86_64-unknown-linux-musl/include/ncurses$ ln -s . ncurses
</pre>

Now we're ready to actually build ghc successfully.  In the
<code>~/src/ghc-7.10.3/mk</code> directory, I copied <code>build.mk.sample</code> to
<code>build.mk</code>, and then uncommented <code>BuildFlavour  = quick-cross</code>.  Next
I did:

<pre>
ppelletier@patrick64:~/src/ghc-7.10.3$ ./configure --target=x86_64-unknown-linux-musl --with-curses-includes=/home/ppelletier/x-tools/x86_64-unknown-linux-musl/include/ncurses --with-curses-libraries=/home/ppelletier/x-tools/x86_64-unknown-linux-musl/lib --prefix=/home/ppelletier/x-tools/x86_64-unknown-linux-musl
ppelletier@patrick64:~/src/ghc-7.10.3$ make
ppelletier@patrick64:~/src/ghc-7.10.3$ make install
</pre>

It turns out that this toolchain still links dynamically by default,
so I have to specify <code>-static -optl-static</code> when compiling Haskell
programs.  For example:

<pre>
ppelletier@patrick64:~/programming/haskell$ x86_64-unknown-linux-musl-ghc -O -static -optl-static contact.hs
[1 of 1] Compiling Main             ( contact.hs, contact.o )
Linking contact ...
ppelletier@patrick64:~/programming/haskell$ file contact
contact: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), statically linked, not stripped
ppelletier@patrick64:~/programming/haskell$ ldd contact
        not a dynamic executable
</pre>

The <code>contact</code> executable can now be run on either my Ubuntu box or my
OpenWRT box.

However, in order to build nontrivial programs, I need cabal.  The [https://ghc.haskell.org/trac/ghc/wiki/Building/CrossCompiling GHC cross-compiling page] recommends:

<pre>
cabal --with-ghc=<cross-ghc> --with-ld=<ld> ...
</pre>

So I expanded this approach to:

<pre>
alias mcabal="cabal --with-ghc=x86_64-unknown-linux-musl-ghc --with-ld=x86_64-unknown-linux-musl-ld --with-gcc=x86_64-unknown-linux-musl-gcc --with-ghc-pkg=x86_64-unknown-linux-musl-ghc-pkg"
</pre>

I'm able to install some packages with mcabal.  (For example, <code>mcabal install hourglass</code>.)

But with cryptonite, I get:

<pre>
ppelletier@patrick64:~/src/lifx/lifx-programs$ mcabal install --flags=-integer-gmp cryptonite
Resolving dependencies...
Configuring memory-0.12...
Building memory-0.12...
Preprocessing library memory-0.12...
running dist/dist-sandbox-5f352b7e/build/Data/Memory/MemMap/Posix_hsc_make failed (exit code 127)
command was: dist/dist-sandbox-5f352b7e/build/Data/Memory/MemMap/Posix_hsc_make >dist/dist-sandbox-5f352b7e/build/Data/Memory/MemMap/Posix.hs
Failed to install memory-0.12
cabal: Error: some packages failed to install:
cryptonite-0.15 depends on memory-0.12 which failed to install.
memory-0.12 failed during the building phase. The exception was:
ExitFailure 1
</pre>

I haven't been able to figure out exactly what's going on there.

Or, <code>mcabal install aeson</code> fails with:

<pre>
Building aeson-0.11.1.4...
Preprocessing library aeson-0.11.1.4...
ghc: Data/Aeson/TH.hs:9:14-28: Template Haskell requires GHC with interpreter support
    Perhaps you are using a stage-1 compiler?
Usage: For basic information, try the `--help' option.
Failed to install aeson-0.11.1.4
cabal: Error: some packages failed to install:
aeson-0.11.1.4 failed during the building phase. The exception was:
ExitFailure 1
</pre>