Author: Siro, Mugabi

Category: qemu

Summary:

Example QEMU TAP network configs/setups. Tested with QEMU v1.x and v2.x on Ubuntu 10.04, 12.04, 13.04, and 14.04 host platforms. UPDATE: 2016 Dec

Tags: qemu linux

Physical Machine/LAN Setup

The host computer should at least have one physical network interface. For purposes of this guide, it is assumed that this network interface, eth0, connects to a development LAN that is served by a DHCP server on some other machine. If, on the other hand, the host machine uses a Wirless NIC interface (e.g. wlan0), this other entry might be more relevant.

With respect to QEMU, the host's network interface is to be configured such that it is capable of bridging the network interfaces of multiple instances of VMs, with each VM directly recieving a unique IP address from the external DHCP server.

Required Packages

  • QEMU Full System Emulation Support

    See QEMU intro for details on obtaining QEMU full system emulation support.

  • Linux bridge support

    In this setup, the QEMU VM tap interfaces work in conjuction with a Linux bridge. The bridge-utils package contains the necessary utilities for configuring a Linux Ethernet bridge.

    $ sudo apt-get install bridge-utils
    

Host Network Interface Configuration

The following configuration is such that both the host and each VM (via TAP interfaces) connect to the external network by way of Linux bridge br0. This enables both the host and each VM instance to recieve unique IP addresses from the external DHCP server.

$ cat /etc/network/interfaces 
auto lo eth0 br0

iface lo inet loopback

iface eth0 inet manual

iface br0 inet dhcp
    bridge_ports eth0
    bridge_maxwait 0
    bridge_fd 0

$ sudo invoke-rc.d networking stop
$ sudo invoke-rc.d networking start

$ ifconfig
br0       Link encap:Ethernet  HWaddr ec:9a:78:56:34:12   
        inet addr:192.168.2.23  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                    ...
eth0      Link encap:Ethernet  HWaddr ec:9a:78:56:34:12   
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  ...

lo        Link encap:Local Loopback  
        inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
        UP LOOPBACK RUNNING  MTU:65536  Metric:1
                    ...

With these settings, both the host and QEMU VM instances can connect to the development LAN network and independently recieve IP address assignments from the external DHCP server.

The crucial setting in /etc/network/interfaces is bridge_ports eth0. It adds eth0 to br0. Roughly speaking, that single line replaces the following untidy (but more illustrative) pre-up and post-down settings:

$ cat /etc/network/interfaces
...
iface br0 inet dhcp
  pre-up ifconfig eth0 0.0.0.0 up
  pre-up brctl addbr br0
  pre-up brctl addif br0 eth0
  bridge_stp off
  bridge_maxwait 0
  bridge_fd 0
  post-down brctl delif br0 eth0
  post-down brctl delbr br0
  post-down ifconfig eth0 down

eth0 is setup as manual to avoid conflicts with services such as Network Manager. The bridge_stp off setting disables the Spanning Tree Protocol (STP) (assuming that this is the master bridge) and bridge_fd 0 turns off forwarding delay. Another useful setting would be bridge_hw (See Section link).

You may experiment with other bridge_* settings e.g. bridge_waitport 0. See bridge-utils-interfaces(5) for details on the bridge_* options, brctl(8) for Linux bridge adminstration and interfaces(5) for the /etc/network/interfaces manpage.

QEMU TAP Interfaces and the Linux Host Bridge

The libexec/qemu-bridge-helper helper program allows a non-privileged user to setup QEMU/TAP networking. This is unlike the case with QEMU/TAP setups that use the qemu_ifup.sh/qemu_ifdown.sh scripts, i.e. root privileges are generally required. The libexec/qemu-bridge-helper program obtains a TAP file descriptor, attaches it to a Linux bridge, and passes it back to QEMU. The binary is setuid root -- allowing it to perform these privileged network operations when invoked by a non-privileged user. Note that once the TAP/bridge operations complete, privileges are immediately dropped to CAP_NET_ADMIN (see capabilities(7)).

This helper program is defined in qemu-bridge-helper.c of the QEMU tree. An ordinary qemu-system build will compile and install this program along with the qemu-system-$ARCH binaries. For example:

$ ./configure [--target-list=...] --prefix=${QEMU_INSTALL}
$ make -jN
$ make install
$ cd ${QEMU_INSTALL}
$ ls -CF
bin/  libexec/  share/  var/
$ ls libexec/
qemu-bridge-helper

To prepare libexec/qemu-bridge-helper:

  • change ownership to root:

    $ ls -l libexec/qemu-bridge-helper 
    -rwxr-xr-x 1 user user libexec/qemu-bridge-helper
    
    $ sudo chown root libexec/qemu-bridge-helper 
    $ ls -l libexec/qemu-bridge-helper 
    -rwxr-xr-x 1 root user libexec/qemu-bridge-helper
    
  • add the setuid attribute:

    $ sudo chmod u+s libexec/qemu-bridge-helper 
    $ ls -l libexec/qemu-bridge-helper 
    -rwsr-xr-x 1 root user libexec/qemu-bridge-helper
    

Note that changing ownership to root is required even with setuid. Otherwise QEMU might complain and exit with a message similar to:

failed to create tun device: Operation not permitted
qemu-system-x86_64: -net bridge,name=tap0,bridge=br0,helper=libexec/qemu-bridge-helper: bridge helper failed

The qemu-bridge-helper expects an Access Control List (ACL) file. By default, it assumes the etc/qemu/bridge.conf file. The path is relative to the root of the current QEMU installation, so:

$ mkdir -p etc/qemu/

The following file will allow qemu-bridge-helper to work with br0:

$ cat etc/qemu/bridge.conf
allow br0

$ ls -l etc/qemu/bridge.conf 
-rw-rw-r-- 1 user user etc/qemu/bridge.conf

i.e. no special permissions or ownership is required for the ACL file. If the Linux host bridge is not listed in the ACL file, libexec/qemu-bridge-helper will complain (and abort QEMU boot) with a message similar to:

access denied by acl file
bridge helper failed

To use this br0 configuration:

  • -netdev bridge

    -netdev bridge,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper,id=hn0,name=tap0,br=br0
    -device rtl8139,netdev=hn0,id=nic0
    
    -netdev bridge,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper,id=hn1,name=tap1,br=br0
    -device e1000,netdev=hn1,id=nic1
    (etc)
    

    Alternatively, via QEMU HMI:

    (qemu) netdev_add bridge,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper,id=hn0,name=tap0,br=br0
    (qemu) device_add rtl8139,netdev=hn0,id=nic0
    

    See Hotplugging Devices for more details.

    Network related device properties for a QEMU NIC can be viewed via, say:

    $ qemu-system-x86_64 -device rtl8139,?
    ...
    rtl8139.vlan=int32 (Integer VLAN id to connect to)
    rtl8139.mac=str (Ethernet 6-byte MAC Address, example: 52:54:00:12:34:56)
    rtl8139.netdev=str (ID of a netdev to use as a backend)
    
    $ qemu-system-x86_64 -device e1000,?
    ...
    e1000.vlan=int32 (Integer VLAN id to connect to)
    e1000.mac=str (Ethernet 6-byte MAC Address, example: 52:54:00:12:34:56)
    e1000.netdev=str (ID of a netdev to use as a backend)
    
  • -net bridge

    -net nic[,model=...,macaddr=...] -net bridge,ifname=tap0,br=br0,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper
    
    -net nic[,model=...,macaddr=...] -net bridge,ifname=tap1,br=br0,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper
    

The -netdev bridge,br=$BR or -net bridge,br=$BR command passes $BR to libexec/qemu-bridge-helper (see net/tap.c:net_init_bridge()). So, to use an additional Linux host bridge, say, br1:

  • Update etc/qemu/bridge.conf:

    $ cat etc/qemu/bridge.conf 
    allow br0
    allow br1
    
  • specify -netdev bridge,br=br1,... or -net bridge,br=br1,....

Using qemu_ifup.sh and qemu_ifdown.sh

This configuration generally requires a QEMU boot with root privileges. The following set of scripts take care of adding/deleting a QEMU VM tapN interface to/from virtual bridge br0.

$ cat qemu_br0_ifup.sh
#!/bin/sh
switch=br0
echo "$0: adding tap interface \"$1\" to bridge \"$switch\""
ifconfig $1 0.0.0.0 up
brctl addif ${switch} $1
exit 0

$ cat qemu_br0_ifdown.sh
#!/bin/sh
switch=br0
echo "$0: deleting tap interface \"$1\" from bridge \"$switch\""
brctl delif $switch $1
ifconfig $1 0.0.0.0 down
exit 0

Do not forget to make them executable:

$ chmod u+x qemu_br0_if*

Along with specifying unique tapN interfaces for each NIC of (a) VM instance(s), each NIC must have a unique MAC. For this reason, create the following script:

$ cat genmac.sh
#!/bin/sh
printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

Then fire-up qemu instances. Only the TAP-related options are shown here1:

$ sudo qemu-system-$ARCH ... -net nic,macaddr=`source genmac.sh` -net tap,ifname=tap0,script=qemu_br0_ifup.sh,downscript=qemu_br0_ifdown.sh

$ sudo qemu-system-$ARCH ... -net nic,macaddr=`source genmac.sh` -net tap,ifname=tap1,script=qemu_br0_ifup.sh,downscript=qemu_br0_ifdown.sh

(etc)

Connectivity and Troubleshooting

Each VM guest, if configured for DHCP, should recieve a unique IP address from the external DHCP server upon boot. At this point, properly configured services such as remote login (e.g. ssh), data transfer (e.g. scp/sftp), and even file sharing (e.g. sshfs) can be performed from any computer on the same subnet - whether VM-to-VM, host-to-VM, or even between a separate physical machine on the same subnet and the QEMU VM instances.

The showmacs command can be used to view the interfaces added to the virtual br0 bridge. For instance, the following was performed on the host running two VM instances:

$ sudo brctl showmacs br0
port no mac addr        is local?   ageing timer
1   12:34:56:78:9a:bc   no         0.04     # external DHCP server NIC
  3 32:0d:99:eb:35:52   yes        0.00     
  2 da:22:b8:d3:8e:7d   yes        0.00   
  3 de:ad:be:ef:d0:c5   no        12.35    
  2 de:ad:be:ef:fd:d1   no         6.17     
  1 ec:9a:78:56:34:12   yes        0.00     # eth0


$ /sbin/ifconfig | egrep '(br0|eth|tap)'
br0       Link encap:Ethernet  HWaddr 32:0d:99:eb:35:52  
eth0      Link encap:Ethernet  HWaddr ec:9a:78:56:34:12  
tap0      Link encap:Ethernet  HWaddr 32:0d:99:eb:35:52  
tap1      Link encap:Ethernet  HWaddr da:22:b8:d3:8e:7d

The port no values in the output of the showmacs command can be used to link a particular TAP interface to the corresponding VM network interface i.e.:

  • VM interface de:ad:be:ef:d0:c5 uses tap0
  • VM interface de:ad:be:ef:fd:d1 uses tap1

Verify this by running:

$ /sbin/ifconfig | grep HWaddr

in the respective VM guests.

Now, upon quiting the VM instance using tap0:

$ sudo brctl showmacs br0
port no mac addr        is local?   ageing timer
  1 12:34:56:78:9a:bc   no         0.00
  2 da:22:b8:d3:8e:7d   yes        0.00
  2 de:ad:be:ef:fd:d1   no        43.09
  1 ec:9a:78:56:34:12   yes        0.00

$ /sbin/ifconfig | egrep '(br0|eth|tap)'
br0       Link encap:Ethernet  HWaddr da:22:b8:d3:8e:7d  
eth0      Link encap:Ethernet  HWaddr ec:9a:78:56:34:12  
tap1      Link encap:Ethernet  HWaddr da:22:b8:d3:8e:7d

Notice how the MAC value of br0 has changed (ifconfig output). And when all VMs are down:

$ sudo brctl showmacs br0
[sudo] password for siro: 
port no mac addr        is local?   ageing timer
1   12:34:56:78:9a:bc   no         0.02
  1 bc:9a:78:56:34:12   yes        0.00

$ /sbin/ifconfig | egrep '(br0|eth|tap)'
br0       Link encap:Ethernet  HWaddr ec:9a:78:56:34:12  
eth0      Link encap:Ethernet  HWaddr ec:9a:78:56:34:12

the MAC address of br0 defaults to that of eth0. In otherwords, the Linux bridge interfaces use the lowest MAC address among the local ports. Also note that the assignment of the MAC addresses of the tap interfaces in this setup is random.

When working with QEMU VM instances, e.g. using QEMU machines as client nodes for testing and debugging netboot configurations, the VM's may have to be turned on/off repeatedly and frequently. Temporary loss in connectivity to the VM host may occur occassionally since other computers in the LAN now have have to update their ARP caches with br0's frequently changing MAC address. This temporary loss in connectivity may take several seconds and is noticable, for instance, when using ssh or synergy connections.

To remedy this problem, the bridge MAC address could be explicitly set and fixed to one of its (permanent) ports. In this case, the following bridge_hw option could be added in /etc/network/interfaces:

$ cat /etc/network/interfaces 
...

iface br0 inet dhcp
    bridge_ports eth0
    bridge_maxwait 0
    bridge_fd 0
    bridge_hw ec:9a:78:56:34:12

where ec:9a:78:56:34:12 is eth0's MAC address.

Also See

Resources

  • http://wiki.qemu.org/Documentation/Networking
  • http://wiki.qemu.org/Features/HelperNetworking
  • http://en.wikibooks.org/wiki/QEMU/Networking
  • http://en.wikibooks.org/wiki/QEMU/Devices/Network
  • http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge
  • https://wiki.debian.org/BridgeNetworkConnections
  • http://backreference.org/2010/07/28/linux-bridge-mac-addresses-and-dynamic-ports/

Footnotes

1. For performance reasons, consider using virio-net for the NIC device model i.e. -net nic,model=virtio. See QEMU Virtio [go back]