Author: Siro, Mugabi

Category: Development Setups

Summary:

Setting up a complete virtual LAN made entirely of QEMU VM nodes (Gateway and clients). Tested with QEMU v1.x and v2.x on Ubuntu 10.04, 12.04, 13.04, and 14.04 Host and VM platforms.

Tags: qemu linux development setups

Background

Machine emulators and virtualization solutions generally present a convinient working platform during development. Although virtual platforms do not fully replace physical hardware, virtual machine setups make (to a large extent) development and experimenting possible when access to actual hardware is not possible. This is especially true for setups that require more than one computer but don't need the processing power of multiple computers.

The following QEMU-based VLAN setup is presented in the hopes that it will be of some use especially for educational/lab sessions. The setup can serve as a plaform for (the study of) various network-related applications including experimenting with different network topologies and firewall configurations, simulating different bandwidth shaping or traffic control scenarios, etc.

The setup presents a VLAN made up of multiple VM-nodes behind a VM-gateway on the host machine. A screenshot of this setup is shown below:

qemu vlan setup

This particular setup has been tested with Ubuntu 10.04, 12.04 and 13.04 (hosts and VMs alike).

Host Setup

Linux Bridge Configuration

The VLAN setup presented below will consist of one VM-gateway serving two VM clients. The VM-gateway is configured to have two Ethernet interfaces. Interface eth0 will connect to an external physical LAN via the host, while interface eth1 will serve the internal virtual LAN of the QEMU nodes. In the context of this entry:

  • The eth0 interface of the VM-gateway connects to a Linux bridge on the host. The host machine is assumed to have at least one physical network interface:

    • If the physical network interface is an ordinary ethernet port, then the VM's eth0 will connect to it via Linux bridge, say, br0 (via a tap interface). Since the br0 setup used here is the very same as the one in A QEMU TAP Networking Setup, a description of its configuration will not be repeated here.

    • If, on the other hand, the host machine uses a Wireless NIC interface to connect to an external physical LAN via an Access Point, check out the Linux bridge and corresponding VM-gateway interface configuration here.

  • The eth1 interface of the VM-gateway will connect to a private VLAN that will serve the VM client nodes. A virtual bridge br1 will be configured on the host for this purpose. To set up br1, the following is performed:

    $ sudo brctl addbr br1 
    $ sudo ifconfig br1 up
    
  • Disabling (default) ethernet filtering in the host kernel

    To allow traffic in the private VLAN between the QEMU VM interfaces on br1, ethernet filtering in the kernel should be disabled. One quick fix is:

    $ for i in `ls /proc/sys/net/bridge/bridge-nf*` ; do sudo sh -c "echo 0 > $i" ; done
    

QEMU Disk Images

Ubuntu distro disk images are used here by the VMs. Refer to link for details of preparing Ubuntu disk images for QEMU VM. Several disk images, one per guest, could be prepared. Alternatively, a single base image with derived images, one per VM, could be used. It should also be possible to use a single image with each QEMU VM instance started with the -snapshot option. However, with this last approach, any changes made in the VM instances will be lost upon VM shutdown. The base and derived images approach is considered here:

$ ls
prestine-ubuntu-13.04-desktop-amd64.qcow2

$ qemu-img create -f qcow2 -o backing_file=prestine-ubuntu-13.04-desktop-amd64.qcow2 vm_gateway-img.qcow2

$ qemu-img create -f qcow2 -o backing_file=prestine-ubuntu-13.04-desktop-amd64.qcow2 vm_node0-img.qcow2

$ qemu-img create -f qcow2 -o backing_file=prestine-ubuntu-13.04-desktop-amd64.qcow2 vm_node1-img.qcow2

$ ls
prestine-ubuntu-13.04-desktop-amd64.qcow2
vm_gateway-img.qcow2
vm_node0-img.qcow2
vm_node1-img.qcow2

See Manipulating Disk Images with qemu-img for more info on working with base and derived QEMU disk images. Refer to qemu-img(1) for the manpage.

VM Setup

QEMU Commandline

The -net, -netdev, and -device options can be combined in a variety of ways when setting up a QEMU VLAN. A few examples are presented below. Essentially, by associating different -net tap|bridge,vlan=N parameters on the VM-gateway command line with separate brN Linux host bridges, isolated virtual subnets are created. The vlan=N parameter is interpreted by the QEMU instance as a hub port. Note that this interpretation is internal to the QEMU instance and has no meaning outside the VM environment. Thus, the vlan=N parameter is only required by the -net bridge|tap command line switches of the VM-gateway in order to create separate subnets for the respective virtual NICs, i.e. with each NIC connected to a separate Linux host bridge.

NOTE:

  • There doesn't seem to be a straightforward way of combining -netdev with explicit VLAN assignments and so the VM-gateway command lines stick with the legacy -net switches. qemu(1) does make mention of a -netdev hubport switch -- but either that discussion is vague (unlikely) or I'm not too bright (likely). In any case, brief code inspection (at least for QEMU versions up until v2.7) shows that net/net.c:net_client_init1() does not invoke net/hub.c:net_hub_add_port() for the -netdev switch, or if the netdev= property of (the associated) -device was specified on the command line. Consult with the experts on the QEMU mailing lists for an authoritative discussion.

  • The VM-nodes (clients) do not require the vlan=N parameter. Rather, their command lines should point to the appropriate Linux host bridge. This also means that their command lines can (sanely) make use of either -net or -netdev.

The examples presented below are categorised in two. The first set makes use of the libexec/qemu-bridge-helper helper program to avoid the need for a QEMU boot with root privileges in order to bring up the TAP interfaces. The second group employs the qemu_if*.sh scripts and requires booting QEMU with root privileges.

For purposes of illustration, the following are the general (i.e. non-network specific) QEMU command line options that were used:

  • "vm_gateway":

    $ qemu-system-x86_64 -machine accel=kvm:tcg -smp N -m $MSIZE \
      -name "vm_gateway" -kernel vmlinuz -initrd initrd.img \
      -drive file=vm_gateway-img[,OPTS] \
      ... [ -vga std -usbdevice tablet -daemonize ]
    
  • "vm_node0":

    $ qemu-system-x86_64 -machine accel=kvm:tcg -smp N -m $MSIZE \
        -name "vm_node0" -kernel vmlinuz -initrd initrd.img \
        -drive file=vm_node0-img[,OPTS] \
        ... [ -vga std -usbdevice tablet -daemonize ]
    
  • "vm_node1":

    $ qemu-system-x86_64 -machine accel=kvm:tcg -smp N -m $MSIZE \
        -name "vm_node1" -kernel vmlinuz -initrd initrd.img \
        -drive file=vm_node1-img[,OPTS] \
        ... [ -vga std -usbdevice tablet -daemonize ]
    

Using libexec/qemu-bridge-helper, Recommended

This configuration does not require a QEMU boot with root privileges in order to bring up the TAP interfaces. libexec/qemu-bridge-helper was already discussed in A QEMU TAP Networking Setup. In this case, the etc/qemu/bridge.conf ACL file will resemble:

$ cat ${QEMU_INSTALL}/etc/qemu/bridge.conf 
allow br0
allow br1

You may also consider using the virtio-net NIC model for performance reasons (see QEMU Virtio).

  • "vm_gateway" network options:

    -net nic,vlan=0,macaddr=DE:AD:BE:EF:00:11 \
    -net bridge,vlan=0,ifname=tap0,br=br0,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper \
    -net nic,vlan=2,macaddr=DE:AD:BE:EF:51:12 \
    -net bridge,vlan=2,ifname=tap1,br=br1,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper \
    

    The HMI network info should resemble:

    (qemu) info network 
    hub 2
     \ hub2port1: bridge.1: index=0,type=tap,helper=./libexec/qemu-bridge-helper,br=br1
     \ hub2port0: e1000.1: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:51:12
    hub 0
     \ hub0port1: bridge.0: index=0,type=tap,helper=./libexec/qemu-bridge-helper,br=br0
     \ hub0port0: e1000.0: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:00:11
    
  • "vm_node0" network options:

    • -net

      -net nic,macaddr=DE:AD:BE:EF:B1:AB \
      -net bridge,ifname=tap2,br=br1,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper
      

      The HMI network info should resemble:

      (qemu) info network 
      hub 0
       \ hub0port1: bridge.0: index=0,type=tap,helper=${QEMU_INSTALl}/libexec/qemu-bridge-helper,br=br1
       \ hub0port0: e1000.0: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:b1:ab
      
    • -netdev

      -netdev bridge,ifname=tap2,br=br1,id=ndev,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper \
      -device e1000,mac=DE:AD:BE:EF:B1:AB,id=nic0,netdev=ndev
      

      The HMI network info should resemble:

      (qemu) info network 
      nic0: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:b1:ab
       \ ndev: index=0,type=tap,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper,br=br1
      
  • "vm_node1" network options:

    • -net

      -net nic,macaddr=DE:AD:BE:EF:31:CA \
      -net bridge,ifname=tap3,br=br1,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper
      

      The HMI network info should resemble:

      (qemu) info network 
      hub 0
       \ hub0port1: bridge.0: index=0,type=tap,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper,br=br1
       \ hub0port0: e1000.0: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:31:ca
      
    • -netdev

      -netdev bridge,ifname=tap3,br=br1,id=ndev,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper \
      -device e1000,mac=DE:AD:BE:EF:31:CA,id=nic0,netdev=ndev
      

      The HMI network info should resemble:

      (qemu) info network 
      nic0: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:31:ca
       \ ndev: index=0,type=tap,helper=${QEMU_INSTALL}/libexec/qemu-bridge-helper,br=br1
      

Using qemu_if*.sh

In order to bring up the TAP interfaces with this configuration, a QEMU boot with root privileges is required. You may also consider using the virtio-net NIC model for performance reasons (see QEMU Virtio).

  • "vm_gateway" network options:

    -net nic,vlan=0,macaddr=DE:AD:BE:EF:00:11 \
    -net tap,vlan=0,ifname=tap0,br=br0,script=qemu_br0_ifup.sh,downscript=qemu_br0_ifdown.sh \
    -net nic,vlan=2,macaddr=DE:AD:BE:EF:51:12 \
    -net tap,vlan=2,ifname=tap1,br=br1,script=qemu_br1_ifup.sh,downscript=qemu_br1_ifdown.sh
    

    Run the info network HMI command to view the network "hub schema":

    (qemu) info network 
    hub 2
     \ hub2port1: tap.1: index=0,type=tap,ifname=tap1,script=qemu_br1_ifup.sh,downscript=qemu_br1_ifdown.sh
     \ hub2port0: e1000.1: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:51:12
    hub 0
     \ hub0port1: tap.0: index=0,type=tap,ifname=tap0,script=qemu_br0_ifup.sh,downscript=qemu_br0_ifdown.sh
     \ hub0port0: e1000.0: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:00:11
    
  • "vm_node0" network options:

    -net nic,macaddr=DE:AD:BE:EF:B1:AB \
    -net tap,ifname=tap2,br=br1,script=qemu_br1_ifup.sh,downscript=qemu_br1_ifdown.sh
    

    and

    (qemu) info network 
    hub 0
     \ hub0port1: tap.0: index=0,type=tap,ifname=tap2,script=qemu_br1_ifup.sh,downscript=qemu_br1_ifdown.sh
     \ hub0port0: e1000.0: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:b1:ab
    
  • "vm_node1" network options:

    -net nic,macaddr=DE:AD:BE:EF:31:CA \
    -net tap,ifname=tap3,br=br1,script=qemu_br1_ifup.sh,downscript=qemu_br1_ifdown.sh
    

    and

    (qemu) info network 
    hub 0
     \ hub0port1: tap.0: index=0,type=tap,ifname=tap3,script=qemu_br1_ifup.sh,downscript=qemu_br1_ifdown.sh
     \ hub0port0: e1000.0: index=0,type=nic,model=e1000,macaddr=de:ad:be:ef:31:ca
    

    where the scripts qemu_br* resided in the present working directory and were defined as:

    $ cat qemu_br0_ifup.sh
    
    #!/bin/sh
    switch=br0
    echo "$0: adding tap interface \"$1\" to bridge \"$switch\""
    ifconfig $1 0.0.0.0 up
    brctl addif ${switch} $1
    exit 0
    
    $ cat qemu_br1_ifup.sh
    
    #!/bin/sh
    switch=br1
    echo "$0: adding tap interface \"$1\" to bridge \"$switch\""
    ifconfig $1 0.0.0.0 up
    brctl addif ${switch} $1
    exit 0
    
    $ cat qemu_br0_ifdown.sh
    
    #!/bin/sh
    switch=br0
    echo "$0: deleting tap interface \"$1\" from bridge \"$switch\""
    brctl delif $switch $1
    ifconfig $1 0.0.0.0 down
    exit 0
    
    $ cat qemu_br1_ifdown.sh
    
    #!/bin/sh
    switch=br1
    echo "$0: deleting tap device \"$1\" from bridge \"$switch\""
    brctl delif $switch $1
    ifconfig $1 0.0.0.0 down
    exit 0
    

Miscellaneous

Note that different MAC address are explicitly assigned to each VM network interface to guarantee that conflicts do not occur. As an alternative to static assignments, the following script:

$ cat genmac.sh
#!/bin/sh
printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

could be sourced againt the macaddr option i.e.

-net nic,macaddr=`source genmac.sh`[,...]

VM Network Interface Configuration

The /etc/network/interfaces file for the VM-gateway is configured as:

# order of interface activation
auto lo eth0 eth1 br0

iface lo inet loopback

iface eth0 inet dhcp

iface eth1 inet manual

iface br0 inet static
    address 192.168.3.1
    netmask 255.255.255.0
    bridge_ports eth1
    bridge_maxwait 0
    bridge_fd 0

If the VM-gateway machine will provide DHCP services to its VM clients, then DHCP will have to be installed and configured. In this particular setup, DHCP was used and the setup for the VM-gateway was:

$ sudo apt-get install isc-dhcp-server

$ sudo vim /etc/dhcp/dhcpd.conf

ddns-update-style none;
default-lease-time 600;
max-lease-time 7200;
log-facility local7;
allow booting;
allow bootp;

# QEMU vLAN
subnet 192.168.3.0 netmask 255.255.255.0 {
   range dynamic-bootp 192.168.3.20 192.168.3.50;
   option broadcast-address 192.168.3.255;
   option routers 192.168.3.1;
   option domain-name-servers 192.168.2.1;
}

host vm_node0 {
   hardware ethernet DE:AD:BE:EF:B1:AB;
   fixed-address 192.168.3.2;
}

host vm_node1 {
   hardware ethernet DE:AD:BE:EF:31:CA;
   fixed-address 192.168.3.3;
}
  • NOTE: The QEMU VM host machine recieved DNS (and DHCP) services from an external machine, 192.168.2.1, via its eth0 on the physical LAN - hence the line option domain-name-servers 192.168.2.1.

The client VM /etc/network/interface was then configured as:

$ cat /etc/network/interfaces 
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo eth0

iface lo inet loopback

iface eth0 inet dhcp

VM Gateway Config

The VM-gateway may now be configured for IP routing, forwarding and SNAT (just like a traditional physical gateway machine) such that its VM client nodes can reach an external network e.g. the Internet.

  • Enabling IP Forwarding

    To forward packets between the private VLAN and the internet:

    $ sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
    
  • Enabling Source NAT (SNAT)

    In order for the VM client nodes on VLAN to be able to recieve replies from the internet, masquerading is enabled:

    $ sudo iptables -A POSTROUTING -t nat -o eth0 -j MASQUERADE
    
  • DNS

    In this setup, the line:

    option domain-name-servers 192.168.2.1;
    

    in the /etc/dhcp/dhcpd.conf file presented above took care of passing DNS information to the VM clients from an external DNS server (192.168.2.1) on the physical LAN connected to the host machine's eth0.

    Below is an instance of the setup with Internet access enabled:

    qemu vlan setup inet

Connectivity and Troubleshooting

The showmacs option can be passed to the brctl command to view the list of MAC addresses added to the respective virtual bridges.

  • On the host machine, a running instance of this VM setup should yeild an ouput similar to:

    $ sudo brctl showmacs br0
    port no mac addr        is local?   ageing timer
     1  12:34:56:78:9a:bc   no         0.09 # remote DHCP/DNS server
     2  b2:44:3e:3f:87:f0   yes        0.00
     2  de:ad:be:ef:00:11   no        23.42
     1  ec:9a:78:56:34:12 yes          0.00 # eth0
    
    $ sudo brctl showmacs br1
    port no mac addr        is local?   ageing timer
     2  26:d3:20:a7:80:2f   yes        0.00
     1  a6:19:18:73:b3:d6   yes        0.00
     2  de:ad:be:ef:31:ca   no         8.33
     1  de:ad:be:ef:51:12   no         4.28
     3  de:ad:be:ef:b1:ab   no         4.28
     3  e2:af:e2:60:ba:52   yes        0.00
    
    $ /sbin/ifconfig | egrep '(br|eth|tap)'
    br0       Link encap:Ethernet  HWaddr b2:44:3e:3f:87:f0  
    br1       Link encap:Ethernet  HWaddr 26:d3:20:a7:80:2f  
    eth0      Link encap:Ethernet  HWaddr ec:9a:78:56:34:12    
    tap0      Link encap:Ethernet  HWaddr b2:44:3e:3f:87:f0  
    tap1      Link encap:Ethernet  HWaddr a6:19:18:73:b3:d6  
    tap2      Link encap:Ethernet  HWaddr e2:af:e2:60:ba:52  
    tap3      Link encap:Ethernet  HWaddr 26:d3:20:a7:80:2f
    
  • On the VM-gateway:

    lumumba@lumumba-virtual-machine:~ $ sudo brctl showmacs br0
    port no mac addr        is local?   ageing timer
     1  26:d3:20:a7:80:2f   no        37.79 # br1
     1  de:ad:be:ef:31:ca   no        21.95
     1  de:ad:be:ef:51:12   yes        0.00 
     1  de:ad:be:ef:b1:ab   no        32.93
    
    lumumba@lumumba-virtual-machine:~$ ifconfig | egrep '(br|eth)'
    br0       Link encap:Ethernet  HWaddr de:ad:be:ef:51:12  
    eth0      Link encap:Ethernet  HWaddr de:ad:be:ef:00:11  
    eth1      Link encap:Ethernet  HWaddr de:ad:be:ef:51:12
    

Next Steps

Now that the VLAN configuration is essentially complete, you may proceed with network-related testing/experimenting just as you would on a physical LAN (to the extent that the VM based infrastructure will allow, of course).

Also See

Resources

  • http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge
  • http://wiki.qemu.org/Documentation/Networking
  • http://en.wikibooks.org/wiki/QEMU/Networking