Author: Mugabi Siro

Category: GNU/Linux Toolchain

Summary:

This page includes a few examples on using C/C++ GNU/Linux cross-toolchains

Tags: gnu/linux toolchain gnu gnu autotools linux configure cross-compiling arm

Pre-requisites

  • Your system already has a cross-toolchain installed. For purposes of illustration, a GNU/Linux ARM cross-toolchain will be used throughout this entry1. The cross-toolchain tuple used here is arm-linux-gnueabi[hf]-. Replace accordingly.

  • It is assumed that the path to the cross-toolchain binaries already exists in $PATH. For example, if:

    $ ls -l /usr/bin | grep arm-linux-gnueabi-
    arm-linux-gnueabi-addr2line
    arm-linux-gnueabi-ar
    arm-linux-gnueabi-as
    arm-linux-gnueabi-c++filt
    arm-linux-gnueabi-cpp -> arm-linux-gnueabi-cpp-4.6
    arm-linux-gnueabi-cpp-4.6
    arm-linux-gnueabi-elfedit
    arm-linux-gnueabi-gcc -> arm-linux-gnueabi-gcc-4.6
    arm-linux-gnueabi-gcc-4.6
    ...
    

    then /usr/bin should be found in $PATH:

    $  echo $PATH
    /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
    
  • It is also assummed that the cross-compiler driver is properly installed such that it can find the respective toolchain component programs (including the compiler proper, assembler and linker) along with the toolchain's (standard) libraries and headers. For example:

    • Program and library search path:

      $ arm-linux-gnueabihf-gcc -print-search-dirs
      install: /usr/lib/gcc/arm-linux-gnueabihf/4.6/
      
      programs: =/usr/lib/gcc/arm-linux-gnueabihf/4.6/:/usr/lib/gcc/arm-linux-gnueabihf/4.6/: ... arm-linux-gnueabihf/bin/
      
      libraries: =/usr/lib/gcc/arm-linux-gnueabihf/4.6/:/usr/lib/gcc/arm-linux-gnueabihf/4.6/ ... arm-linux-gnueabihf/lib/
      
    • Header file search patch:

      $ arm-linux-gnueabihf-gcc -v -E - < /dev/null 2>&1 | awk '/^#include/,/^End of search/ { print }'
      #include "..." search starts here:
      #include <...> search starts here:
       /usr/lib/gcc/arm-linux-gnueabihf/4.6/include
       /usr/lib/gcc/arm-linux-gnueabihf/4.6/include-fixed
       /usr/arm-linux-gnueabihf/include
       /usr/include
      End of search list.
      

Helloworld Example

$ cat vipi.c
#include <stdio.h>

int main(void)
{
    printf("Dunia, vipi?\n");
    return 0;
}

$ arm-linux-gnueabi-gcc -Wall vipi.c

$ file a.out
a.out: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.31, ...

For simple application build scenarios such as these, i.e. where only the standard cross-toolchain headers and libraries are required, no CFLAGS and LDFLAGS, respectively, need to be supplied to the cross-compiler driver (arm-linux-gnueabi-gcc, in this case). It automatically searches for these standard headers and libraries.

Now, running the above build command with the -v option:

$ arm-linux-gnueabi-gcc -Wall vipi.c -v
Target: arm-linux-gnueabi
...
gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) 
COLLECT_GCC_OPTIONS='-Wall' '-v' '-march=armv7-a' '-mfloat-abi=softfp' '-mfpu=vfpv3-d16' '-mthumb'

/usr/lib/gcc/arm-linux-gnueabi/4.6/cc1 -quiet ...
...
ignoring nonexistent directory "/usr/include/arm-linux-gnueabi"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/arm-linux-gnueabi/4.6/include
 /usr/lib/gcc/arm-linux-gnueabi/4.6/include-fixed
 /usr/arm-linux-gnueabi/include
 /usr/include
End of search list.
...
COLLECT_GCC_OPTIONS='-Wall' '-v' '-march=armv7-a' '-mfloat-abi=softfp' '-mfpu=vfpv3-d16' '-mthumb'

 /usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/bin/as -march=armv7-a -mfloat-abi=softfp

COMPILER_PATH=/usr/lib/gcc/arm-linux-gnueabi/4.6/:/usr/lib/gcc/arm-linux-gnueabi/4.6/:/usr/lib/gcc/arm-linux-gnueabi/:/usr/lib/gcc/arm-linux-gnueabi/4.6/:/usr/lib/gcc/arm-linux-gnueabi/:/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/bin/

LIBRARY_PATH=/usr/lib/gcc/arm-linux-gnueabi/4.6/:/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/lib/../lib/:/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/lib/

COLLECT_GCC_OPTIONS='-Wall' '-v' '-march=armv7-a' '-mfloat-abi=softfp' '-mfpu=vfpv3-d16' '-mthumb'
 /usr/lib/gcc/arm-linux-gnueabi/4.6/collect2 --build-id ...
...

... reveals several interesting things (in addition to the include file search path). To highlight a few:

  • Cross-tools involved:

    The cross-compiler proper /usr/lib/gcc/arm-linux-gnueabi/4.6/cc12. the cross-assembler (arm-linux-gnueabi/bin/as) and the cross-linker (arm-linux-gnuabi/bin/collect2)3

    Also note that:

    $ md5sum /usr/arm-linux-gnueabi/bin/as
    bb5c559bd24f5795799938872bd0ca73  /usr/arm-linux-gnueabi/bin/as
    
    $ md5sum /usr/bin/arm-linux-gnueabi-as
    bb5c559bd24f5795799938872bd0ca73  /usr/bin/arm-linux-gnueabi-as
    

    Omitted, for the sake of brevity, is the full listing of options passed to each tool by the cross-compiler driver. Run a similar instance of the command to view the full output.

  • The default configuration settings of the cross-compiler driver including the Instruction Set Architecture, ISA (-march=armv7-a, -mthumb), floating-point ABI (-mfloat-abi=softfp), etc.

To run the freshly cross-built a.out4 binary on the (x86 based) development platform, the services of an emulator could be employed. For instance (and in this case)5:

$ qemu-arm -L /usr/arm-linux-gnueabi a.out
Dunia, vipi?

Alternatively, if using a statically linked executable build approach:

$ arm-linux-gnueabi-gcc -Wall vipi.c -static

$ file a.out 
a.out: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.31, ...

$ qemu-arm-static a.out
Dunia, vipi?

Or if the the QEMU host's binfmt misc settings are properly configured6:

$ ./a.out # statically linked
Dunia, vipi?

Cross-Compiling via GNU configure

GNU Autools Cross-Compilation Contexts

The GNU Autotools support cross-compilation in two different contexts and it is important to distinguish between them:

  • Configuring and building a cross-toolchain

  • Building ordinary binaries using a cross-toolchain.

The terms build, host, and target take special meanings when using the GNU Autotools. Generally, they describe the different systems on which the compilation and execution of the package will occur. However, depending on the compilation context, these terms may take different meanings or relevance.

Configuring and Building a Cross-Toolchain

In this context:

  • build The system on which the cross-toolchain is built.

  • host The system on which the cross-toolchain will run.

  • target The system for which the cross-toolchain generates code.

Typically, build will be of the same architecture type as host. In this scenario, no cross-compilation is actually performed when producing the cross-toolchain binaries. But since a cross-toolchain is being produced, target will be of a different architecture. Below is the output of a cross-compiler driver that was built with this approach:

$ arm-linux-gnueabi-gcc -v 
...
Configured with: ... --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-linux-gnueabi ...
...

Nevertheless, there exist some exotic scenarios that actually involve cross-compilation during toolchain build:

When building a cross-native toolchain

Here, the toolchain binaries are built on one architecture, but run on, and generate code for, another. In other words, while host and target are of the same architecture type, build is different.

When cross-compiling a cross-toolchain

There are two cases here:

  1. Canadian Cross7

    Here, the cross-toolchain built on one machine, will run on a different architecture to generate binaries for yet another. In this case, build, host and target are each of different architecture types.

  2. Cross-back
    This is a rare scenario where build and target are of the same type and differ from host. In other words, the cross-toolchain built on one machine, is deployed and run on another to generate code for architectures of the same type as the build machine.

Building Ordinary Binaries Using a Cross-Toolchain

In this context:

  • build The system on which the binaries are built.
  • host The system on which the binaries will run.

When building normal binaries, the term target becomes meaningless. It only makes sense to speak of a target when building toolchain binaries which, in turn, generate code.

Summary

Regardless of the cross-compilation context:

  • build and host are never the same. In other words, whenever and only if build and host differ will cross-compilation be effected.

  • By extension, the target setting (whenever relevant) has no influence in effecting cross-compilation during package build.

Enabling Cross-Compilation via configure

The --build, --host and --target options by configure are used to specify the build, host and target systems, respectively. If configure is run with none of these options specified, then --target is set to the value of --host, which assumes the value of --build, which, in turn, defaults to the canonical8 name of the current system.

So, in order to enable cross-compilation, the user should at least specify --host. This will implicitly set --build not equal to --host, and effect cross-compilation mode.

Note that, for historic reasons, whenever --host is specified, --build should also be specified. However, this requirement has seemingly been fixed i.e. no need to specify --build; configure will automatically set it to the canonical value of the current system.

GNU Configuration Names

The --build, --host and --target options of Autoconf -generated configure scripts expect values that take the form of canonical names or GNU Triplets. Both GNU triplets and canonical names take the form of cpu-vendor-os, where os can be system or kernel-system. The following table describes the meanings of these fields:

cpu The type of processor used on the system e.g. x86_64 or arm.
vendor This may take any (reasonable) value. Typically, the name of the manufacturer of the system, for example example pc, or simply unknown e.g. x86_64-unkown-linux-gnu. In fact, this field is often omitted e.g. arm-linux-gnueabi.
kernel This is used mainly for GNU/Linux systems e.g. i686-pc-linux-gnu. In other words, linux specifies kernel.
system Typically gnu* for GNU based systems.

There are a number of Autotools related utilities that facilitate deriving GNU configuration names:

  • config.guess(1) : guesses the build system triplet e.g.

    $ /usr/share/misc/config.guess 
    x86_64-unknown-linux-gnu
    
  • config.sub(1) : validates and canonicalizes a configuration triplet e.g.

    $ ls /usr/bin/ | grep x86_64 | grep gcc
    x86_64-linux-gnu-gcc
    x86_64-linux-gnu-gcc-4.6
    
    $ /usr/share/misc/config.sub x86_64-linux-gnu
    x86_64-pc-linux-gnu
    

These scripts may be installed via packages such as libtool or, at least with Debian based systems, by running:

$ sudo apt-get install autotools-dev

Alternatively, a canonical name can be obtained by running the config.sub script (sometimes) supplied in the source-level package. In addition, the config.guess script is also sometimes included in a source-level package distribution and could be used to obtain the build system triplet. These scripts use a combination of uname commands or symbols predefined by the C preprocessor to eventually derive a set of canonical values or triplets for cpu, vendor, and os. Note that, by default, the configure script also usually guesses correctly the canonical name for the type of build system.

The following snippet illustrates configuring a package of an ordinary9 application for cross-compilation (on, say, an x86 based host) for an ARM runtime host using GNU triplets:

$ ./configure --host=arm-linux-gnueabi --build=`config.guess`

where:

  • The config.guess script is present in the current working directory.

  • arm-linux-gnueabi is the GNU triplet corresponding to the cross-toolchain's tuple. In this case:

    $ ls /usr/bin/ | grep arm-linux-gnueabi-gcc
    arm-linux-gnueabi-gcc
    arm-linux-gnueabi-gcc-4.6
    

Also check out Specifying GNU Triplets.

Specifying The Cross-Compiler Driver

In addition to specifying the GNU configuration names, it is also advisable to explicitly specify the cross-compiler driver. This can be done in several ways including:

  • via configure

    $ CC=arm-linux-gnueabi-gcc ./configure --host=arm-linux-gnueabi --build=`config.guess` ...
    $ make
    $ make install
    
  • against make:

    $ ./configure --host=arm-linux-gnueabi --build=`config.guess` ...
    $ make CROSS_COMPILE=arm-linux-gnueabi-
    $ make CROSS_COMPILE=arm-linux-gnuebi- install
    

    In this case, instead of specifying the cross-compiler driver, the tuple of the cross-toolchain, arm-linux-gnueabi- (i.e. trailing hyphen included), is specified against CROSS_COMPILE.

Finally, when cross-compiling, configure will warn about any tools (compilers, binutils, etc) whose names are not prefixed with the value specified against --host. This can lead to subtle build (or even runtime) fails. Be sure to inspect configure's output for such warnings.

Examples

Cross-Compiling a Simple Program

This case study involves the cross-compilation of a simple database application that relies on the GNU dbm library.

libgdbm Cross-Compilation
$ KAZI_DIR=$PWD
$ wget -c ftp://ftp.gnu.org/gnu/gdbm/gdbm-1.11.tar.gz
$ tar xzf gdbm-1.11.tar.gz
$ mkdir {gdbm-temp,gdbm-staging}
$ GDBM_SOURCE=${PWD}/gdbm-1.11
$ GDBM_STAGING=${PWD}/gdbm-staging
$ cd gdbm-temp/
$ CC=arm-linux-gnueabihf-gcc ${GDBM_SOURCE}/configure --host=arm-linux-gnueabihf --build=`${GDBM_SOURCE}/build-aux/config.guess` --prefix=/usr

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for arm-linux-gnueabihf-strip... arm-linux-gnueabihf-strip
...

$ make
$ make STRIP=arm-linux-gnueabihf-strip DESTDIR=${GDBM_STAGING} install-strip

Now, verify that the build is indeed ARM:

$ cd $KAZI_DIR
$ file ${GDBM_STAGING}/usr/lib/libgdbm.so.4.0.0 
/home/siro/tmp/gdbm-staging/usr/lib/libgdbm.so.4.0.0: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x3bc00..., stripped
Application Cross-Compilation

Create the following file. (Sample source code taken from Brian Gough, An Introduction To GCC: for the GNU Compilers gcc and g++, ISBN-13: 978-0954161798):

$ cd $KAZI_DIR
$ cat dbmain.c

#include <stdio.h>
#include <gdbm.h>
int main (void)
{
  GDBM_FILE dbf;
  datum key = { "testkey", 7 };
  /* key, length */
  datum value = { "testvalue", 9 }; /* value, length */
  printf ("Storing key-value pair... ");
  dbf = gdbm_open ("test", 0, GDBM_NEWDB, 0644, 0);
  gdbm_store (dbf, key, value, GDBM_INSERT);
  gdbm_close (dbf);
  printf ("done.\n");
  return 0;
}

Since this cross-build uses non-standard headers and libraries, specify the CFLAGS and LDFLAGS. In this particular case, gcc options -I, -L and -l are specified on the command line:

$ arm-linux-gnueabihf-gcc -Wall dbmain.c -o dbmain -I ${GDBM_STAGING}/usr/include -L ${GDBM_STAGING}/usr/lib -lgdbm
$ arm-linux-gnueabihf-strip dbmain
$ file dbmain
dbmain: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked    (uses shared libs), for GNU/Linux 2.6.31, BuildID[sha1]=0x751..., stripped

The cross-compiled ARM executable may now be tested by way of a processor emulator, for example10:

$ qemu-arm -L /usr/arm-linux-gnueabihf/ -E LD_LIBRARY_PATH=${GDBM_STAGING}/usr/lib dbmain
Storing key-value pair... done.

Alternatively, using the statically linked program build approach:

$ arm-linux-gnueabihf-gcc -Wall dbmain.c -o dbmain-static -I ${GDBM_STAGING}/usr/include -L ${GDBM_STAGING}/usr/lib -lgdbm -static
$ file dbmain-static 
dbmain-static: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.31 ...
$ qemu-arm dbmain-static 
Storing key-value pair... done.

or, if the QEMU VM host has proper binfmt misc settings:

$ ./dbmain-static 
Storing key-value pair... done.

The application is now ready for transfer to the embedded runtime host. There are several ways to perform installation of an application (and its runtime dependencies) in a runtime host. In this particular instance (where the runtime host's IP address was 192.168.2.27)11:

$ cd $KAZI_DIR
$ scp dbmain root@192.168.2.27:/usr/local/bin
$ scp ${GDBM_STAGING}/usr/lib/libgdbm.so.4.0.0 root@192.168.2.27:/usr/local/lib

$ ssh -l root 192.168.2.27
root@genericarmv7a:~# ls /usr/local/bin/
dbmain
root@genericarmv7a:~# ls /usr/local/lib
libgdbm.so.4.0.0
root@genericarmv7a:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@genericarmv7a:~# cat /etc/ld.so.conf 
root@genericarmv7a:~# echo '/usr/local/lib' > /etc/ld.so.conf
root@genericarmv7a:~# ldconfig -v
/usr/local/lib:
    libgdbm.so.4 -> libgdbm.so.4.0.0 (changed)
/lib:
    libpthread.so.0 -> libpthread-2.18-2013.10.so
    libcrypto.so.1.0.0 -> libcrypto.so.1.0.0
    libgcc_s.so.1 -> libgcc_s.so.1
...

root@genericarmv7a:~# dbmain 
Storing key-value pair... done.

Real world Embedded Application Cross-Build

This page includes a case study on cross-compiling the DirectFB framework for an ARM target.

Cross-Kbuild

The kernel build system facilitates cross-compilation via the ARCH and CROSS_COMPILE variable references in its Makefiles. ARCH controls selection of the target architecture. Available architectures can be viewed in arch/* . CROSS_COMPILE specifies the prefix used for the cross-toolchain components used - specifically, gcc and the binutils utilities. For example:

$ cat Makefile | grep CROSS_COMPILE -A2
# CROSS_COMPILE specify the prefix used for all executables used
# during compilation. Only gcc and related bin-utils executables
# are prefixed with $(CROSS_COMPILE).
# CROSS_COMPILE can be set on the command line
# make CROSS_COMPILE=ia64-linux-
# Alternatively CROSS_COMPILE can be set in the environment.
# A third alternative is to store a setting in .config so that plain
# "make" in the configured kernel build directory always uses that.
# Default value for CROSS_COMPILE is not to prefix executables
# Note: Some architectures assign CROSS_COMPILE in their arch/*/Makefile
ARCH        ?= $(SUBARCH)
CROSS_COMPILE   ?= $(CONFIG_CROSS_COMPILE:"%"=%)

# Architecture as present in compile.h
--
AS      = $(CROSS_COMPILE)as
LD      = $(CROSS_COMPILE)ld
CC      = $(CROSS_COMPILE)gcc
CPP     = $(CC) -E
AR      = $(CROSS_COMPILE)ar
NM      = $(CROSS_COMPILE)nm
STRIP       = $(CROSS_COMPILE)strip
OBJCOPY     = $(CROSS_COMPILE)objcopy
OBJDUMP     = $(CROSS_COMPILE)objdump
AWK     = awk
GENKSYMS    = scripts/genksyms/genksyms
--
export ARCH SRCARCH CONFIG_SHELL HOSTCC HOSTCFLAGS CROSS_COMPILE AS LD CC
export CPP AR NM STRIP OBJCOPY OBJDUMP

and

$ cat Makefile | grep '\<ARCH\>' -A2
[...]
# When performing cross compilation for other architectures ARCH shall be set
# to the target architecture. (See arch/* for the possibilities).
# ARCH can be set during invocation of make:
# make ARCH=ia64
# Another way is to have ARCH set in the environment.
# The default ARCH is the host where make is executed.
[...]

In other words, alternative ways of setting values to the ARCH and CROSS_COMPILE variables include:

  • Directly in the make process' environment. For instance:

    $ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-
    
  • Setting and using the shells environment e.g:

    $ export ARCH=arm
    $ export CROSS_COMPILE=arm-linux-gnueabi-
    $ make
    

    Notice that, unlike the previous approach, the settings here make persistent modifications to the shell's environment via its export command, and so subsequent make commands need not specify ARCH= and CROSS_COMPILE=12. The settings in the previous method only affect the environment of the corresponding instance of make and not the shell's.

  • Via .config so that, and like the second method above, a plain make command now automatically uses the configured ARCH and CROSS_COMPILE settings.

Note that if the path to the cross-toolchain binaries does not exist in the PATH shell environment variable, then it must be explicitly specified, e.g. CROSS_COMPILE=/usr/bin/arm-linux-gnueabi-, or first set accordingly in the shell environment.

Kernel Cross-Build

Typical Linux config and build steps during kernel cross-compilation are:

$ export ARM=arm
$ export CROSS_COMPILE=arm-linux-gnueabi-
$ make menuconfig
$ make [-jN]
$ make modules_install INSTALL_MOD_PATH=${INSTALLDIR}

Cross-Compiling External Modules

Consider the following helloworld kernel module:

$ cat extern_module.c

#define pr_fmt(fmt) "%s:%s:%d " fmt, KBUILD_MODNAME, __func__, __LINE__
#include <linux/module.h>

int xmod_init(void)
{
    pr_info("\n");
    return 0;
}

void xmod_fini(void)
{
    pr_info("\n");
}
module_init(xmod_init);
module_exit(xmod_fini);
MODULE_LICENSE("GPL");

with its correspondingly simple Makefile13

$ cat Makefile

ifneq ($(KERNELRELEASE),)
obj-m := extern_module.o
else
KDIR ?= /home/${USER}/linux-linaro-tracking

default :
    $(MAKE) -C $(KDIR) M=$$PWD modules

clean:
    $(MAKE) -C $(KDIR) M=$$PWD clean

.PHONY : clean
endif

This makefile builds against a local copy of the Linaro Linux GIT respository14. Replace this path accordingly.

Once again, using the cross-toolchain is simply a matter of appropriately setting ARCH= and CROSS_COMPILE= , e.g.

$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- [clean]

Footnotes

1. See link for examples on obtaining and installing ARM cross-toolchains. [go back].

2. gcc will, by default, use cc1's internal preprocessor. [go back]

3. collect2 is a GCC linker utility that eventually invokes the real linker, ld(1). [go back]

4. Actually an ELF executable object file [go back]

5. See QEMU Linux User Mode Emulation [go back]

6. See QEMU Linux User Mode Emulation for an example of configuring a host's binfmt misc. [go back]

7. Refered to that way due to the historical context of the GNU Autotools. [go back]

8. See Section link [go back]

9. i.e. in contrast to a toolchain package. [go back]

10. See QEMU Linux User Mode Emulation for more details on using QEMU in Linux User Mode Emulation [go back]

11. The ARM host was a QEMU emulated Vexpress Cortex-A9 machine running Linaro OpenEmbedded release 13.11 [go back].

12. Child processes spawned by the shell inherit its environment. Execute, say, unset ARCH and unset CROSS_COMPILE (in bash) to delete these environment variables. [go back].

13. See Documentation/kbuild/modules.txt for details on writing external module makefiles [go back]

14. Check out this page for a kernel build example with this tree. [go back]