Author: Mugabi Siro

Category: qemu

Summary:

This entry presents a number of (non-standard) configurations for redirecting the QEMU Human Monitor Interface (HMI), and the guest serial port system console and login terminal. Tested with QEMU v1.x, v2.1, v2.2, and v2.7

Tags: qemu linux openssh

Preliminaries

Prerequisites

The configurations presented here assume background knowledge of Linux guest configuration for serial port system console and login terminal, in addition to the basic QEMU command line for specifying serial line terminal devices and backends. Check out QEMU Serial Port System Console for this background coverage. The examples presented here illustrate redirection of QEMU serial line terminals to non-default backends i.e. other than a QEMU (SDL) virtual console (VC), or the stdio of the lauch terminal.

QEMU Serial Port Device Alternatives

To keep illustration simple, the Linux guest and QEMU command lines presented here will consistently make use of QEMU standard serial port emulation, i.e. the venerable Universal Asynchronous Receiver Transmitter (UART) device, on qemu-system-x86_64. The Linux command line option for UART on x86 is /dev/ttySn. It also exports options for standard UART devices on other machine architectures e.g. /dev/AMAn for ARM. Replace accordingly.

QEMU includes support for serial port device alternatives such USB-to-UART emulation and virtio-serial which can as well be used in place of UART. For examples of Linux guest configuration and QEMU command line options for these devices, check out Using QEMU Character Devices (for USB-serial) and QEMU-virtio (for virtio-serial's virtconsole device option).

Redirecting to Host Terminals

Using a non-default terminal on the host is generally specified via the tty QEMU option. The term terminal here will apply to Linux virtual terminals (/dev/ttyN) and terminal emulators e.g. xterms which operate on pseudoterminal devices (PTY's). In the case of using pseudoterminal devices, it is important to distinguish between use of the tty and pty options. See A Note on pty vs. tty QEMU Options.

The following example illustrates redirecting the guest serial port system console to a "target" xterm on the host other than the QEMU launch terminal:

  • Obtain the (UNIX 98 PTY) pseudoterminal slave (PTS) device name for the "target" xterm:

    $ tty
    /dev/pts/2
    

    Since the QEMU serial device will be opening this host PTS device that is already a controlling terminal for a shell, the following command (or similar) need also be executed on this "target" xterm:

    $ sleep 10d
    

    in order for keyboard input to now get redirected to stdin of the connected QEMU serial line terminal. Otherwise, while the stdout of the QEMU serial line terminal will appear on this xterm's window, it will not be possible to login or use keyboard input. A sleep of 10 days is specified here.

  • On the QEMU launch xterm:

    $ qemu-system-x86_64 -enable-kvm -smp 2 -m 1G \
        -kernel vmlinuz -initrd initrd.img \
        -drive file=demo.img,if=virtio \
        -append "root=/dev/vda1 rw console=ttyS0" \
        -chardev tty,id=pts2,path=/dev/pts/2 \
        -device isa-serial,chardev=pts2 [-nographic]
    

    At this point, guest boot messages via system console on emulated UART should appear on the "target" xterm (/dev/pts/2) and it should also be possible to login via that interface. If booting in nographic mode, HMI will be on stdio.

To redirect both the HMI and guest serial port system console to separate "target" xterms:

  • Open an extra xterm on the host for the HMI:

    $ tty
    /dev/pts/3
    
    $ sleep 10d
    
  • Then launch QEMU with, say:

    • Update Note: The QEMU commandline for HMI redirection via -monitor is legacy. See QEMU Character Devices, HowTo for HMI redirection by way of -chardev/-mon.

    $ qemu-system-x86_64 -enable-kvm -smp 2 -m 1G \
        -kernel vmlinuz -initrd initrd.img \
        -drive file=demo.img,if=virtio \
        -append "root=/dev/vda1 rw console=ttyS0" \
        -monitor /dev/pts/3 \
        -chardev tty,id=pts2,path=/dev/pts/2 \
        -device isa-serial,chardev=pts2  [-nographic]
    

Notes

These configurations can be quite useful in Setups for Debugging QEMU with GDB

Redirecting to Net Consoles

A number of Net Console options are available based on different protocols including UDP, TCP, Telnet and Net Consoles based on Unix domain sockets. With the exception of UDP, Net Consoles have two modes of operation:

  • Client mode: This is the default mode. A connection request is sent to a listening server.

  • Server mode: In this mode, QEMU boot will wait for a client socket application to connect to the port before continuing unless the nowait option was specified.

Usage of the udp:, tcp:, telnet: and unix: legacy command line options is well documented in qemu(1). See docs/qdev-device-use.txt or Using QEMU Character Devices for differences between this legacy and the standard command line syntax.

Client Mode

Consider a case where the local machine (IP address 192.168.2.1) is to be used for the guest serial port system console via nc(1). The remote machine will be the QEMU host.

  • Fire up a listening nc(1) server instance on the local machine:

    $ nc -l 4555
    
  • Instantiate QEMU on the remote machine with -net or -netdev:

    • Update Note: The QEMU commandline shown below uses the qemu_if*.sh scripts for a QEMU TAP network configuration on the host's Linux bridge interface. However, it is recommended that the libexec/qemu-bridge-helper program be used instead of these scripts. See A QEMU TAP Networking Setup.

    $ sudo qemu-system-x86_64 -kernel bzImage -hda brdisk-img.raw \
      -append "root=/dev/sda2 rw console=ttyS0" \
      -net nic \
      -net  tap,ifname=tap1,script=qemu_br0_ifup.sh,downscript=qemu_br0_ifdown.sh \
      -chardev socket,host=192.168.2.1,port=4555,id=gnc0 \
      -device isa-serial,chardev=gnc0 \
      -monitor stdio [-nographic]
    
    qemu_br0_ifup.sh: adding tap interface "tap1" to bridge "br0"
    QEMU 1.6.0 monitor - type 'help' for more information
    (qemu)
    
  • Back on the local machine, the QEMU guest boot messages and serial port terminal login prompt should get displayed on the console of the listening nc(1):

    $ nc -l 4555    
    [    0.000000] Initializing cgroup subsys cpuset
    [    0.000000] Initializing cgroup subsys cpu
    [    0.000000] Linux version 3.5.1
    (...)
    Starting SMB services: done
    Starting NMB services: done
    Starting atd: OK
    INIT: Entering runlevel: 1
    
    Welcome to Buildroot
    buildroot login: root
    root
    [root@buildroot ~]# tty
    tty
    /dev/ttyS0
    

To obtain the HMI via Net Console, specify e.g. -monitor tcp:${IP}:${PORT}.

Server Mode

For instance, launching QEMU with:

  • Update Note: The QEMU commandline shown below uses the qemu_if*.sh scripts for a QEMU TAP network configuration on the host's Linux bridge interface. However, it is recommended that the libexec/qemu-bridge-helper program be used instead of these scripts. See A QEMU TAP Networking Setup.

$ sudo qemu-system-x86_64 -enable-kvm -m 1G -kernel vmlinuz -initrd initrd.img  \
    -drive file=demo.img,if=virtio -append "root=/dev/vda1 rw console=ttyS0" \
    -net nic \
    -net tap,ifname=tap0,script=qemu_br0_ifup.sh,downscript=qemu_br0_ifdown.sh \
    -monitor tcp:127.0.0.1:4555,server,nodelay \
    -chardev socket,host=127.0.0.1,port=4556,id=gnc0,server,nowait \
    -device isa-serial,chardev=gnc0 [-nographic]

will result in the following prompt for a remote connection to the Net Console for the HMI before QEMU commences with Linux guest boot execution:

QEMU waiting for connection on: tcp:127.0.0.1:4555,server

Making a remote connection e.g.

$ nc 127.0.0.1 4555
QEMU 2.1.3 monitor - type 'help' for more information
(qemu)

will allow QEMU to continue and begin guest boot. Notice that courtesy of the nowait flag, QEMU will not stop to prompt for a remote TCP Net console connection for the guest's serial port system console.

If running in nographic mode, this situation will result in a "blind" boot. A remote connection to this "background" Net Console backend can then be made via:

$ nc 127.0.0.1 4556

Notes

Net Consoles provide an easy way of connecting QEMU serial line terminals across a network. Nevertheless, their purely socket-based Inter Process Communication (IPC) mechanisms introduce certain limitations with respect to conventional terminal operation. These issues are discussed in more detail in the next section.

Net Consoles vs. Host Terminals

Net Consoles provide a rather straightforward way of connecting a QEMU serial line to a remote terminal-like interface across a network. However, one notable limitation with Net Consoles (which rely purely on sockets for IPC) stems from their character input buffering scheme, e.g. their stdin is line buffered and sent to the connected QEMU serial line when the user presses RETURN. This input scheme holds a number of implications. For instance, consider the following screenshot of a Net Console netcat(1) session connected to a QEMU guest serial port terminal:

QEMU NetConsole Implications

  • Double echo. Notice that the first few user commands before the stty -echo command takes effect result in double echoes. The first echo is due to stdin character input on the Net Console interface, nc(1) in this case, until the user presses RETURN. The second echo is due to the default action taken by the guest kernel's terminal driver, i.e. to append a copy of any input character to the end of its output queue. Here, the line buffered input from the Net Console is seen as stdin by the guest kernel's terminal driver. To (s)quash this nuisance, echoing could be disabled in the guest kernel terminal driver via the stty(1) command as shown.

    The stty(1) program permits retrieving and modifying terminal attributes. Other useful commands include:

    $ stty echo   ## (re-)enabling echoing
    $ stty -a     ## viewing all terminal attributes
    $ stty sane   ## restoring all special characters to their default state
    
  • User password echo against login(1) prompt. As explained above, since the Net Console interface first buffers its stdin, then user input for login(1) passwords will now get echoed, like any other input. However, notice that there is no double echo for user input at the login(1) password prompt. This follows from the explanation above that the second echo is due to the guest kernel terminal driver which disables echo for login(1) password prompt input.

  • Command Line Editing. Working with convenient shell features, such as TAB key completion for commands and "arrow" keys for browsing command history, now becomes an awkward enterprise. For example, on the command line ls /dev/ttyS, the user actually typed:

    $ ls /dev/ttyS<TAB><TAB><RETURN>
    

    The two consecutive TAB key presses caused the shell in the guest to correctly interpret the user's intention to list all ttyS* device files under /dev/ -- which it promptly did. However, since the shell still hadn't processed the incomplete command (and from a user's point of view, the ls /dev/ttyS command is "returned" to the shell prompt after it lists the possibilities), the line buffered RETURN key press caused the shell to now attempt to list the non-existent /dev/ttyS file -- which failed with the error:

    ls: cannot access /dev/ttyS: No such file or directory
    

    Browsing command history required the user to blindly buffer "up-arrow" keys (each echoed as ^[[A on the Net Console interface) then press RETURN before the shell in the guest could receive and process the user's "mentally indexed" command (tty(1) in this case) from command history.

Terminal-oriented programs need to be connected to a terminal device (rather than a socket) in order to be able to operate in true "character-by-character" mode (rather than a purely line-buffered mode). In this mode, terminal operations and features by the terminal-oriented programs will work as expected. One way of obtaining true terminal functionality across a network is by integrating services of a network facility on the host which operates with pseudoterminal devices, e.g. the Secure Shell (SSH), into a QEMU serial-line-to-host-terminal configuration. The next section presents an example of one such configuration.

Redirecting to Remote Host Terminals over SSH

Combining redirection for QEMU serial line terminals with (Open)SSH services on the host provides a powerful feature that enables using true remote terminals (at least with respect to functionality) across a network. This approach presents a number of advantages over the purely socket-based Net Console configurations including:

  • True terminal operation support over a network. In addition, convenient command line editing features by the HMI and/or the shell on the guest serial port terminal, e.g. command history browsing and TAB key completion for commands, will work just as they do on a local terminal interface on the QEMU host.

  • Operations over a network independent of QEMU -net or -netdev options. No QEMU networking configuration/support is involved/required.

  • Added cryptographic services -- confidentiality (encryption), authentication, integrity, etc.

SSH clients provide true remote terminals courtesy of pseudoterminal slave (PTS) devices (/dev/pts/N, UNIX 98 PTY) on the SSH server side. A successful ssh(1) connection results in the creation of a /dev/pts/N file on the machine running sshd(8). As far as QEMU is concerned, it will be connecting to an ordinary PTS on the VM host - as described in Section Redirecting To Host Terminals. The user only has to initiate SSH connections to the (soon-to-be) QEMU host, identify the corresponding and newly created PTS devices on that machine, prior to executing a QEMU instance on the remote host with the respective /dev/pts/N options.

Consider a case of accessing the HMI and guest serial port system console of a QEMU instance on a remote machine from two separate xterm interfaces on the local machine via SSH:

  • From the local machine, make new ssh(1) connections to the remote (soon-to-be) QEMU host machine (e.g. ssh -l $USER $IP) from two separate xterms. One xterm on the local machine will be used as the HMI terminal, while the other will serve as the guest serial port system console and login terminal. Then prepare these xterm interfaces for exclusive access of local keyboard for the soon-to-be connected QEMU serial line terminals. For instance:

    $ ssh ${USER}@${IP}
    
    $ tty
    /dev/pts/12
    
    $ sleep 10d
    

    and,

    $ ssh ${USER}@${IP}
    
    $ tty
    /dev/pts/13
    
    $ sleep 10d
    

    The tty(1) command information helps identify which xterm on the local machine is connected to which PTS device on the remote (soon-to-be) QEMU host machine.

  • Fire up a QEMU instance on the remote machine (say, from a third remote termial) with the respective /dev/pts/N options e.g:

    • Update Note: The QEMU commandline for HMI redirection via -monitor is legacy. See QEMU Character Devices, HowTo for HMI redirection by way of -chardev/-mon.

    $ ssh -l ${USER} ${IP}
    
    $ qemu-system-x86_64 -enable-kvm -smp 2 -m 1G \
        -kernel vmlinuz -initrd initrd.img \
        -drive file=demo.img,if=virtio \
        -append "root=/dev/vda1 rw console=ttyS0" \
        -monitor /dev/pts/12 \
        -chardev tty,id=pts13,path=/dev/pts/13 \
        -device isa-serial,chardev=pts13 \
        -nographic
    

At this point, the HMI terminal should be accessible on local xterm interface for /dev/pts/12, while the guest serial port system console and login terminal prompt should appear across the network on /dev/pts/13. After guest shutdown, the following can be executed (from the third remote terminal) to return control to the respective shells of the /dev/pts/12 and /dev/pts/13 remote terminals:

$ ps ax | grep 'sleep 10d'
20596 pts/12    S+     0:00 sleep 10d
21006 pts/13    S+     0:00 sleep 10d
$ kill -15 20596
$ kill -15 21006

Other Options

Redirecting Guest Serial Port System Console Output to a Disk File

$ qemu-system-i386 -kernel bzImage -hda brdisk-img.raw \
    -append "root=/dev/sda2 rw console=ttyS0" \
    -serial file:bootlog

Note that in nographic mode, you may have to include, say, network options in the QEMU command line, or an extra serial line option on both the Linux and QEMU command lines, to allow (remote) login since it will now not be possible to login via the (single) guest serial port system console.

More options

Several other host devices are available for use as the guest serial port system console. Consult qemu(1) for the complete listing.

Also See

  • The Linux Programming Interface: A Linux and UNIX System Programming Handbook, Micheal Kerrisk, No Starch Press, Inc. 2010. The definitive guide to Linux System Programming. Respect.