Author: Siro Mugabi

Category: qemu

Summary:

This page presents a few examples on using the QEMU usermode emulator.

Tags: qemu linux arm

Introduction

QEMU Emulation Modes

QEMU provides two modes of emulation:

  • Full System Emulation

    A complete and unmodified OS is run in a virtual machine environment. This mode is instantiated when the qemu-system-$ARCH command is executed. This mode of emulation will also be referred to as qemu-system in this entry.

  • Linux User Mode Emulation

    Here, a (set of) Linux process(es) compiled for one target CPU can be executed in a host environment based on another CPU. The qemu-$ARCH commands provide this emulation mode. The term qemu-user will also be used in this entry to refer to this mode of emulation.

QEMU Linux User Mode Emulation

Unlike full system emulation mode, user mode emulation provides only CPU emulation and is only available for Linux hosts. This mode features:

  • Generic Linux system call converter, including most IOCTLs.

  • clone() emulation using native CPU clone() to use the Linux scheduler for threads.

  • Accurate signal handling by remapping host signals to target signals.

See QEMU intro for details on installing qemu-user.

qemu-user Options and Associated Environment Variables

The list of options and associated environment variables can be viewed by running, say:

$ qemu-arm -h

Among other things, these options can be used to control the behaviour of the emulator as well as modifying the environment of the target process. Usage of a few of these options will be covered here, namely:

Option Environ Var Description
-L path QEMU_LD_PREFIX set the ELF interpreter prefix to path
-E var=value QEMU_SET_ENV sets target's environment variable
-r uname QEMU_UNAME set QEMU uname release string to uname

An ARM cross-toolchain is used here to build the foreign binaries in order to demonstrate qemu-user usage on the x86 host. Details on acquiring an ARM toolchain can be found here and there.

If running an Ubuntu system1:

$ sudo apt-get install gcc-arm-linux-gnueabi gcc-arm-linux-gnueabihf

This command should install all other cross-toolchain components including binutils and eglibc for both the armel and armhf architectures.

QEMU_LD_PREFIX

For a dynamically linked foreign executable, the path to its dynamic linker/loader can be specified in the qemu-user commandline via the -L option or the QEMU_LD_PREFIX environment variable.

For instance:

$ cat vipi.c

#include <stdio.h>

int main(void)
{
    printf("Dunia, vipi?\n");
    return 0;
}

$ arm-linux-gnueabi-gcc -Wall vipi.c 
$ qemu-arm -L /usr/arm-linux-gnueabi a.out 
Dunia, vipi?

Where:

$ ls -l /usr/arm-linux-gnueabi/lib/ld-*
/usr/arm-linux-gnueabi/lib/ld-2.15.so
/usr/arm-linux-gnueabi/lib/ld-linux.so.3 -> ld-2.15.so

Alternatively,

$ QEMU_LD_PREFIX=/usr/arm-linux-gnueabi qemu-arm a.out 
Dunia, vipi?

or

$ export QEMU_LD_PREFIX=/usr/arm-linux-gnueabi

$ qemu-arm a.out 
Dunia, vipi?

$ unset QEMU_LD_PREFIX

Statically Linked Build

If performing statically linked program build of the executable, then specifying QEMU_LD_PREFIX is not required:

$ arm-linux-gnueabi-gcc -Wall vipi.c -static

$ file a.out 
a.out: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.31,...

$ qemu-arm a.out 
Dunia, vipi?

or, if the QEMU VM host has proper binfmt misc settings (See Section link), simply run:

$ ./a.out 
Dunia, vipi?

QEMU_SET_ENV

Specifying this qemu-user environment variable provides a way of modifying the target's environment. It comes in handy when testing foreign executables that use non-standard runtime libraries e.g. not located within the cross-toolchain sysroot.

For example, perform the following library build steps:

$ KAZI_DIR=$PWD
$ wget -c ftp://ftp.gnu.org/gnu/gdbm/gdbm-1.11.tar.gz
$ tar xzf gdbm-1.11.tar.gz 
$ cd gdbm-1.11/
$ CC=arm-linux-gnueabihf-gcc ./configure --host=arm-linux-gnueabihf --build=`build-aux/config.guess` --prefix=${PWD}/../gdbm-install
$ make
$ make install

This procedure should result in the following installation:

$ cd $KAZI_DIR

$ tree -L 2 gdbm-install/
gdbm-install/
├── bin
│   ├── gdbm_dump
│   ├── gdbm_load
│   └── gdbmtool
├── include
│   └── gdbm.h
├── lib
│   ├── libgdbm.a
│   ├── libgdbm.la
│   ├── libgdbm.so -> libgdbm.so.4.0.0
│   ├── libgdbm.so.4 -> libgdbm.so.4.0.0
│   └── libgdbm.so.4.0.0
└── share
    ├── info
    ├── locale
    └── man

Now, verify that the installation is indeed ARM:

$ file gdbm-install/lib/libgdbm.so.4.0.0 
gdbm-install/lib/libgdbm.so.4.0.0: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x38..., not stripped

Then create the following file (Sample source code taken from Brian Gough, An Introduction To GCC: for the GNU Compilers gcc and g++, ISBN-13: 978-0954161798):

$ cd $KAZI_DIR
$ cat dbmain.c
#include <stdio.h>
#include <gdbm.h>

int main (void)
{
    GDBM_FILE dbf;
    datum key = { "testkey", 7 };
    /* key, length */
    datum value = { "testvalue", 9 }; /* value, length */
    printf ("Storing key-value pair... ");
    dbf = gdbm_open ("test", 0, GDBM_NEWDB, 0644, 0);
    gdbm_store (dbf, key, value, GDBM_INSERT);
    gdbm_close (dbf);
    printf ("done.\n");
    return 0;
}

and compile against the just built libgdbm.so.4 library:

$ arm-linux-gnueabihf-gcc -Wall dbmain.c -I gdbm-install/include/ -L gdbm-install/lib/ -lgdbm

Now, the following attempt to execute the foreign a.out (actually, an ELF executable):

$ qemu-arm -L /usr/arm-linux-gnueabihf a.out 
a.out: error while loading shared libraries: libgdbm.so.4: cannot open shared object file: No such file or directory

fails, since the target dynamic linker/loader cannot find the required libgdbm.so.4 soname. In other words, the path to this runtime library has to be passed on to the target environment:

$ qemu-arm -L /usr/arm-linux-gnueabihf -E LD_LIBRARY_PATH=gdbm-install/lib a.out 
Storing key-value pair... done.

Alternatively,

$ export QEMU_LD_PREFIX=/usr/arm-linux-gnueabihf
$ export QEMU_SET_ENV="LD_LIBRARY_PATH=gdbm-install/lib"
$ qemu-arm a.out 
Storing key-value pair... done.

$ unset QEMU_SET_ENV
$ unset QEMU_LD_PREFIX

Note:

  • You can use the -U option or the QEMU_UNSET_ENV environment variable to unset environment variables for the target process.

  • You could use the statically linked program build shortcut to avoid the need for specifying QEMU_LD_PREFIX and QEMU_SET_ENV.

QEMU_UNAME

As noted in Section link, qemu-user emulates a CPU and provides the Linux system call interface for the foreign userspace application(s). Now, the toolchain's C library - against which the foreign binaries are compiled - supports a certain mininum Linux kernel version. In other words, these applications will not run on kernels older than this minimum supported kernel version2. For instance, consider the following crosstool-NG built ARM cross-toolchain3 whose minimum supported kernel version is v3.1.14

$ file ${CT_PREFIX_DIR}/arm-linaro-linux-gnueabihf/sysroot/lib/libc-2.18-2013.10.so 
ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 3.1.1, ...

But the qemu-user version used in this entry exported a kernel interface for Linux v2.6.32 (see the uname -r output in the section discussing foreign chroot below). So, compiling the simple program, vipi.c, above with this toolchain and attempting to qemu-user emulate it:

$ ${CT_PREFIX_DIR}/bin/arm-linaro-linux-gnueabihf-gcc -Wall vipi.c

$ qemu-arm -L ${CT_PREFIX_DIR}/arm-linaro-linux-gnueabihf/sysroot/ a.out
FATAL: kernel too old

failed with a rather bleak error message. The -r commandline option, or the QEMU_UNAME environment variable, could be used to quickly remedy this:

$ qemu-arm -r 3.1.1 -L ${CT_PREFIX}/arm-linaro-linux-gnueabihf/sysroot/ a.out
Dunia, vipi?

Alternatively,

$ export QEMU_UNAME=3.1.1

$ qemu-arm -L ${CT_PREFIX_DIR}/arm-linaro-linux-gnueabihf/sysroot/ a.out
Dunia, vipi?

$ unset QEMU_UNAME

Nevertheless, beware that this fix might introduce subtle bugs when testing more complex programs.

Specifying Multiple Variable Assignments to a qemu-user Option or Environment Variable

When specifying multiple variable assignments to a given qemu-user option or environment variable, either:

  • specify the option multiple times, once for each variable assignment i.e:

    -E var1=val2 -E var2=val2 -U LD_PRELOAD -U LD_DEBUG
    

    However, using this approach is generally not feasible with qemu-user environment variables. If several assignments are specified this way to a single environment variable, only the last specified assignment will take effect.

  • specify the qemu-user option or environment variable once and then separate the variable assignments by commas i.e:

    -E var1=val2,var2=val2 -U LD_PRELOAD,LD_DEBUG
    QEMU_SET_ENV=var1=val2,var2=val2 QEMU_UNSET_ENV=LD_PRELOAD,LD_DEBUG
    

Note that all of the above assignments are equivalent.

Foreign chroot

It is possible to chroot to a foreign or non-native root filesystem with QEMU Linux user mode. For example, performing an ARM chroot on an x86-based host.

Essentially, a successful foreign chroot will require proper BINFMT MISC settings on the host. The binfmt misc feature of the kernel allows execution of foreign binaries by way of a specified interpreter.

Host binfmt misc Configuration

The following instructions were carried out on an Ubuntu 12.04 AMD64 host.

Kernel Support

The distro kernel already had BINFMT_MISC configured:

$ cat /boot/config-`uname -r` | grep BINFMT_MISC
CONFIG_BINFMT_MISC=m

System configuration

Running the:

$ sudo apt-get install ... qemu-user-static binfmt-support

command (as described in QEMU intro) results in an automagic binfmt misc configuration of the host system:

$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/bin/qemu-arm-static
flags: OC
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

$ cat /var/lib/binfmts/qemu-arm
qemu-user-static
magic
0
\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00
\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
/usr/bin/qemu-arm-static

yes

$ cat /usr/share/binfmts/qemu-arm
package qemu-user-static
interpreter /usr/bin/qemu-arm-static
credentials yes
offset 0
magic \x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00
mask \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff

$ cat /etc/mtab | grep binfmt
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,noexec,nosuid,nodev 0 0

$ ls /etc/init | grep binfmt
binfmt-support.conf

Pay special attention to the interpreter /usr/bin/qemu-arm-static entry of the /proc output.

Performing Foreign chroot

This section assumes that a disk (image file) partition containing an ARM root filesystem has already been mounted under ./rfs. This particular case study was performed against the Linaro Ubuntu raring server distro release 13.11;

$ ls rfs
bin   etc         lib         mnt   root  selinux     sys  var
boot  home        lost+found  opt   run   SHA256SUMS  tmp  vmlinuz
dev   initrd.img  media       proc  sbin  srv         usr

$ ls rfs/bin | grep sh # which shells are available for chroot?
bash
dash
rbash
sh
...

Attempting to chroot with the following command failed with a rather misleading/confusing error message:

$ sudo chroot rfs bin/bash
chroot: failed to run command `bin/bash': No such file or directory

To fix this:

$ sudo cp /usr/bin/qemu-arm-static rfs/usr/bin/

Note that the location of the statically linked interpreter, /usr/bin/qemu-arm-static, in the target should correspond to the path listed in the interpreter /usr/bin/qemu-arm-static entry of the /proc output in Section link. Now, the chroot command should succeed:

$ sudo chroot rfs bin/bash
root@linaroubuntu:/# ls
SHA256SUMS  dev     initrd.img  media   proc    sbin        sys var
bin                 etc     lib                 mnt     root    selinux tmp vmlinuz
boot                home    lost+found  opt     run     srv         usr

root@linaroubuntu:/# lsb_release -a
No LSB modules are available.
Distributor ID: Linaro
Description:    Linaro 13.09
Release:        13.09
Codename:   `   raring

Checking the kernel version exported by qemu-arm-static:

root@linaroubuntu:/# uname -a
Linux linaroubuntu 2.6.32 #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 armv7l armv7l armv7l GNU/Linux

which (incidentally) is the minimum supported kernel version by the C library of the chroot environment:

root@linaroubuntu:/# file lib/arm-linux-gnueabihf/libc-2.17.so 
lib/arm-linux-gnueabihf/libc-2.17.so: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), BuildID[sha1]=0x1eaa32..., for GNU/Linux 2.6.32, stripped

The /proc and /sys virtual filesystems may now be mounted/umounted either:

  • From within the chroot environment:

    root@linaroubuntu:/ # mount -t proc none /proc
    root@linaroubuntu:/ # mount -t sysfs none /sys
    ...
    root@linaroubuntu:/ # umount /sys
    root@linaroubuntu:/ # umount /proc
    root@linaroubuntu:/ # exit
    
  • From the host environment, i.e. prior to entering ...

    $ sudo mount --bind /proc rfs/proc
    $ sudo mount --bind /sys rfs/sys
    $ sudo chroot rfs bin/sh
    root@linaroubuntu:/ # ischroot
    root@linaroubuntu:/ # echo $?
    0
    

    ... and after exiting the chroot environment:

    root@linaroubuntu:/ # exit
    $ sudo umount rfs/proc
    $ sudo umount rfs/sys
    

Resources

  • QEMU wiki
  • http://qemu.weilnetz.de/qemu-tech.html

Footnotes

1. As of Maverick (10.10), the *-arm-linux-gnueabi package is available from the Ubuntu universe repositories. Ubuntu 12.04 AMD64 used here [go back]

2. Refer to link for a discussion on the (e)glibc and the minimum support kernel version [go back]

3. See ARM Toolchain with crosstool-NG [go back]

4. file(1) reads the .note.ABI-tag section in an ELF file. See src/readelf.c of its source code. [go back]