Author: Siro Mugabi

Category: qemu

Summary:

This page documents QEMU disk image manipulation via the qemu-img(1) command. This material derives heavily from the OpenSUSE QEMU/KVM online documentation (See Section "Resources"). NOTE: This material was written for QEMU v1.x (Ubuntu 12.04 AMD64) and requires some updating.

Tags: qemu linux

Introduction

The qemu-img(1) command allows you to create, convert and modify disk images offline. It can handle all disk image formats supported by QEMU.

Warning: Never use qemu-img to modify images in use by a running virtual machine or any other process; this may destroy the image. Also, beware that querying an image that is being modified by another process may encounter inconsistent state.

The qemu-img command supports a number of operations or sub commands including:

  • check perform consistency checks against disk images

  • create create disk images

  • convert create a disk image of a specified format from a disk image of another format.

  • info display useful info of a disk image

  • snapshot perform operations on disk image snapshots.

  • resize grow/shrink disk images.

To view the list of all supported disk image formats:

$ qemu-img --help
...
Supported formats: vvfat vpc vmdk vdi sheepdog rbd raw host_cdrom host_floppy host_device file qed qcow2 qcow parallels nbd dmg tftp ftps ftp https http cow cloop bochs blkverify blkdebug

also,

$ man 1 qemu-img
...
Supported image file formats:
raw ...
qcow2 ...
...

Choice of an image format will depend on your particular requirements. For example, the raw format has the advantage of being simple and easily portable across other emulators. Unlike most other formats, it fully supports the qemu-img resize operation. The qcow2 format, on the other hand, is the most versatile format. Among other things, it allows working with disk image snapshots. However, it may not fully support shrinking in resize operations. Nevertheless, there exists the qemu-img convert command that allows conversion between image formats whenever necessary.

Creating a disk image file

General Syntax:

$ qemu-img create [-f fmt] [-o options] fname [size]

Typical usage:

$ qemu-img create -f qcow2 disk-img.qcow2 10G
Formatting 'disk-img.qcow2', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536

$ qemu-img create -f raw disk-img.raw 10G
Formatting 'disk-img.raw', fmt=raw size=10737418240

These commands create disk images with:

  • a maximum capacity of 10GB. Suffixes of the image size are k (kilobyte), M(megabyte), G(gigabyte), and T(terabyte).

  • the default option settings of the respective format. Use -o option to specify different settings.

Note that:

$ du disk-img.*
140 disk-img.qcow2
0   disk-img.raw

$ qemu-img info disk-img.qcow2 
image: disk-img.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 140K
cluster_size: 65536

$ qemu-img info disk-img.raw 
image: disk-img.raw
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: 0

while the virtual size (i.e. maximum capacity of the disk images) is reported as 10G, the actual space occupied on host filesystem is 140K and 0K by the qcow2 and raw images respectively. The value of the disk size: entry will grow as data is actually written into the disk image.

Converting between disk image file formats

This operation supports several options (see qemu-img(1)). Nevertheless, basic conversion a between image formats can be achieved with a single option, -O output_fmt. For instance, creating a qcow2 image from a qcow image:

$ qemu-img convert -O qcow2 disk-img.qcow disk-img.qcow2

Consistency check on a disk image

Example:

$ qemu-img check -f qcow2 disk-img.qcow2

Note that not all disk image formats support this operation. Refer to the check command entry under Command description in qemu-img(1) for a listing of formats that support consistency checks.

Mounting a disk image on the host

qemu-img created disk images can be inspected and manipulated with conventional tools such as fdisk, kpartx etc. Once partition offsets have been determined, mount can be used to mount the disk image partitions on directories on the host filesystem. Depending on the disk image format, different approaches are used.

Mounting a raw disk image

Linux can mount an internal partition of a raw disk image using a loopback device. Assuming that brdisk-img.raw is the raw disk image in question containing Buildroot compiled filesystem data:

$ qemu-img info brdisk-img.raw
image: brdisk-img.raw
file format: raw
virtual size: 2.5G (2684354560 bytes)
disk size: 695M

$ mkdir {boot,rfs}
$ sudo losetup /dev/loop0 brdisk-img.raw

$ sudo fdisk -u -l /dev/loop0

Disk /dev/loop0: 2684 MB, 2684354560 bytes
255 heads, 63 sectors/track, 326 cylinders, total 5242880 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x56a8c587

    Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1            2048     1026047      512000   83  Linux
/dev/loop0p2         1026048     5242879     2108416   83  Linux

$ sudo losetup -o $((512*2048)) /dev/loop1 /dev/loop0
$ sudo losetup -o $((512*1026048)) /dev/loop2 /dev/loop0
$ sudo mount /dev/loop1 boot
$ sudo mount /dev/loop2 rfs
$ ls boot
grub
$ ls rfs
bin  etc   lib      lost+found  mnt  proc  run   sys  usr
dev  home  linuxrc  media       opt  root  sbin  tmp  var

$ sudo umount boot 
$ sudo umount rfs
$ sudo losetup -d /dev/loop2
$ sudo losetup -d /dev/loop1
$ sudo losetup -d /dev/loop0

Mounting a qcow2 disk image

Assuming that brdisk-img.qcow2 is a qcow2 disk image containing Buildroot compiled filesystem data:

$ qemu-img info brdisk-img.qcow2 
image: brdisk-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 701M
cluster_size: 65536

$ mkdir {boot,rfs}
$ sudo modprobe nbd
$ sudo qemu-nbd -c /dev/nbd0 ${PWD}/brdisk-img.qcow2

$ sudo fdisk -l -u /dev/nbd0

Disk /dev/nbd0: 2684 MB, 2684354560 bytes
255 heads, 63 sectors/track, 326 cylinders, total 5242880 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x56a8c587

   Device Boot      Start         End      Blocks   Id  System
/dev/nbd0p1            2048     1026047      512000   83  Linux
/dev/nbd0p2         1026048     5242879     2108416   83  Linux

$ sudo mount /dev/nbd0p1 boot # i.e. mount -o offset=$((512*2048)) /dev/nbd0 boot/
$ sudo mount /dev/nbd0p2 rfs  # i.e. mount -o offset=$((512*1026048)) /dev/nbd0 rfs

$ ls boot/
grub
$ ls rfs/
bin  etc   lib      lost+found  mnt  proc  run   sys  usr
dev  home  linuxrc  media       opt  root  sbin  tmp  var

$ sudo umount rfs
$ sudo umount boot
$ sudo qemu-nbd -d /dev/nbd0
/dev/nbd0 disconnected

Resizing an existing disk image file

Note that this example was performed against a raw disk image. Other disk image formats may not fully support or work with the resize option. Continuing with the brdisk-img.raw example:

$ qemu-img resize brdisk-img.raw +1G
Image resized.

$ sudo losetup /dev/loop0 brdisk-img.raw
$ sudo fdisk /dev/loop0

Command (m for help): p

Disk /dev/loop0: 3758 MB, 3758096384 bytes
255 heads, 63 sectors/track, 456 cylinders, total 7340032 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x56a8c587

    Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1            2048     1026047      512000   83  Linux
/dev/loop0p2         1026048     5242879     2108416   83  Linux

Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
Select (default p): p
Partition number (1-4, default 3): 
Using default value 3
First sector (5242880-7340031, default 5242880): 
Using default value 5242880
Last sector, +sectors or +size{K,M,G} (5242880-7340031, default 7340031): 
Using default value 7340031

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

$ sudo losetup -o $((512*5242880)) /dev/loop3 /dev/loop0
$ sudo mkfs.ext3  /dev/loop3

At this point, the guest will be able to detect the newly created third partition and mount it.

Reversing the resize:

$ qemu-img resize brdisk-img.raw -1G
Image resized.

$ fdisk -u -l brdisk-img.raw

Disk brdisk-img.raw: 2684 MB, 2684354560 bytes
255 heads, 63 sectors/track, 326 cylinders, total 5242880 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x56a8c587

       Device Boot      Start         End      Blocks   Id  System
brdisk-img.raw1            2048     1026047      512000   83  Linux
brdisk-img.raw2         1026048     5242879     2108416   83  Linux
brdisk-img.raw3         5242880     7340031     1048576   83  Linux

So, although fdisk reports three partitions, note the value in the Disk brdisk-img.raw: 2684 ... line has shrunk. In otherwords, the partition table was not updated in qemu-img resize operation.

An attempt to mount this third partition from the VM will fail, e.g:

[root@buildroot ~]# mount /dev/sda3 /mnt/
mount: special device /dev/sda3 does not exist

While the update of the partition table can also be made from within the live instance of the VM (as is done below), care should be taken with respect to consistency in use of partitioning tools and environment to avoid data loss:

[root@buildroot ~]# fdisk /dev/sda

Command (m for help): d
Partition number (1-4): 3

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@buildroot ~]# fdisk -u -l /dev/sda

Disk /dev/sda: 2684 MB, 2684354560 bytes
255 heads, 63 sectors/track, 326 cylinders, total 5242880 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x56a8c587

 Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048     1026047      512000   83  Linux
/dev/sda2         1026048     5242879     2108416   83  Linux

And checking the disk image from the host environment - upon VM shutdown:

$ qemu-img info brdisk-img.raw 
image: brdisk-img.raw
file format: raw
virtual size: 2.5G (2684354560 bytes)
disk size: 696M

As noted, other disk image formats may not (fully) support resizing. For instance:

$ sudo qemu-img resize brdisk-img.qcow2 +1G
Image resized.

$ sudo qemu-img info brdisk-img.qcow2 
image: brdisk-img.qcow2
file format: qcow2
virtual size: 3.5G (3758096384 bytes)
disk size: 728M
cluster_size: 65536

$ sudo qemu-img resize brdisk-img.qcow2 -1G
qemu-img: This image format does not support resize

Cloning an image to a physical storage device

For images in raw format, transfer can be done directly:

# dd if=brdisk-img.raw of=/dev/sdc

For other images, format conversion to raw will be required. For instance:

# qemu-img convert -O raw brdisk-img.qcow2 /dev/sdc

Recall that the image should also contain the necessary drivers for the embedded target.

Working with base and derived images

Basically, a base image is a disk image that serves as a template for (disposable) derived images. Typically, the base image will contain a freshly installed distro with no additional applications or third-party software installed. Derived images get created from the base image. It is against these derived images that additional software, updates, modifications are done - leaving the base image (or "backing file") in its prestine state. In other words, all booting should be done against the derived images and not the base image. When running a VM instance against a derived image, writes get re-directed to the derived image while the base image is used only for reads.

Generally, the state of the backing file (i.e. the base image) should remain prestine as future derived images will refer to it.

A somewhat similar mechanism is provided by the qemu-system -snapshot option. See Using Temporary Files in the Appendix Section. This mechanism is not to be confused with that provided by the qemu-img snapshot command discussed in Section Working with Disk Image Snapshots.

Creating Derived Images

Disk image formats that support base/derived images include qcow2, qcow and vmdk. See qemu-img(1) for a complete listing.

These formats support the -o backing_file= option to the qemu-img create command. This option can be used to specify the base image (i.e. backing file) when creating the derived image:

$ qemu-img create -f qcow2 -o backing_file=brdisk-img.qcow2 br-derivd0-img.qcow2
Formatting 'br-derivd0-img.qcow2', fmt=qcow2 size=2684354560 backing_file='brdisk-img.qcow2' encryption=off cluster_size=65536

$ qemu-img info br-derivd0-img.qcow2 
image: br-derivd0-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 140K
cluster_size: 65536
backing file: brdisk-img.qcow2 (actual path: brdisk-img.qcow2)

Alternatively, the base image can be specified via the -b backing_file option to qemu-img create:

$ qemu-img create -f qcow2 -b brdisk-img.qcow2 br-derivd0-img.qcow2
Formatting 'br-derivd0-img.qcow2', fmt=qcow2 size=2684354560 backing_file='brdisk-img.qcow2' encryption=off cluster_size=65536

$ qemu-img info br-derivd0-img.qcow2 
image: br-derivd0-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 140K
cluster_size: 65536
backing file: brdisk-img.qcow2 (actual path: brdisk-img.qcow2)
  • Side note: A raw disk image can be used as a backing file by a derived image in qcow2 or any other format that supports backing files:
    $ qemu-img create -f qcow2 -o backing_file=brdisk-img.raw tmp-img.qcow2
    Formatting 'tmp-img.qcow2', fmt=qcow2 size=2684354560 backing_file='brdisk-img.raw' encryption=off cluster_size=65536
    
    $ qemu-img info tmp-img.qcow2 
    image: tmp-img.qcow2
    file format: qcow2
    virtual size: 2.5G (2684354560 bytes)
    disk size: 140K
    cluster_size: 65536
    backing file: brdisk-img.raw (actual path: brdisk-img.raw)
    

Note that although the virtual size reported by the derived images is the same as that of the base image, their actual size is only 140k. This is because only the changes made to the sysem inside the derived image are saved.

Boot against the derived image. Your QEMU command line may vary - the important thing to note here is use of the root=/dev/XXX option1:

$ sudo qemu-system-i386 -smp 2 -enable-kvm -kernel bzImage \
    -drive file=br-derivd0-img.qcow2,cache=writeback \
    -append "root=/dev/sda2 quiet fbcon=map:-1 rw"

Then transfering some random vmlinux file of size 165MB into the /root directory2 of the VM instance:

before shutting down and examining via qemu-img:

$ qemu-img info br-derivd0-img.qcow2 
image: br-derivd0-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 168M
cluster_size: 65536
backing file: brdisk-img.qcow2 (actual path: brdisk-img.qcow2)

i.e. br-derivd0-img.qcow2 now occupies 167MB from 140KB. Also note that the size of the backing file (base image) remains unchanged3:

$ qemu-img info brdisk-img.qcow2 
image: brdisk-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 701M
cluster_size: 65536

Mounting Derived Images

Sooner or later, the need to retrieve some of the changes made in a derived image without instantiating a VM boot against it may arise. To do so:

$ sudo modprobe nbd
$ sudo qemu-nbd -c /dev/nbd0 ${PWD}/br-derivd0-img.qcow2 
$ sudo fdisk -l -u /dev/nbd0

Disk /dev/nbd0: 2684 MB, 2684354560 bytes
255 heads, 63 sectors/track, 326 cylinders, total 5242880 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x56a8c587

   Device Boot      Start         End      Blocks   Id  System
/dev/nbd0p1            2048     1026047      512000   83  Linux
/dev/nbd0p2         1026048     5242879     2108416   83  Linux

$ sudo mount /dev/nbd0p2 rfs/
$ ls rfs/root/
vmlinux

As expected, the file vmlinux is present in this derived image and may now be copied out with a simple cp(1) operation.

when done, cleanup:

$ sudo umount rfs
$ sudo qemu-nbd -d /dev/nbd0
/dev/nbd0 disconnected

As an exercise, try mounting the base image to manually compare the differences.

Rebasing Derived Images

It is possible to create a new base image from a derived image. This new base image will contain the contents of the backing file of the derived image plus those of the derived image itself. This new base image will be independent of the derived image's backing file.

$ qemu-img convert -O qcow2 br-derivd0-img.qcow2 brdisk-rebase0-img.qcow2

$ qemu-img info brdisk-rebase0-img.qcow2 
image: brdisk-rebase0-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 865M
cluster_size: 65536

$ qemu-img info br-derivd0-img.qcow2 
image: br-derivd0-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 168M
cluster_size: 65536
backing file: brdisk-img.qcow2 (actual path: brdisk-img.qcow2)

$ qemu-img info brdisk-img.qcow2 
image: brdisk-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 701M
cluster_size: 65536

The rebased image (brdisk-rebase0-img.qcow2) is 165MB bigger than the orginal base image (brdisk-img.qcow2). It uses no backing file and can support its own derived images.

NOTE: The rebased image can also be in raw format:

$ qemu-img convert -O raw br-derivd0-img.qcow2 brdisk-rebase0-img.raw
$ qemu-img info brdisk-rebase0-img.raw 
image: brdisk-rebase0-img.raw
file format: raw
virtual size: 2.5G (2684354560 bytes)
disk size: 860M

Working with disk image snapshots

Snapshots are helpful when you need to save your virtual machine in a particular state. This is especially true during development where snapshots are saved progressively. Whenever something breaks, reverting is simply a matter of booting against the snapshot with the "last known working configuration".

Disk image snapshots are created offline4 via the qemu-img snapshot command and contain only disk image info. In contrast, snapshots created via the savevm command of the QEMU monitor contain the entire live VM state including CPU, RAM, devices and all writable disks.

The table below describes the meaning of the various options to the qemu-img snapshot command:

  • -a applies a snapshot (revert disk to saved state)
  • -c creates a snapshot
  • -d deletes a snapshot
  • -l lists all snapshots

Note that while all qemu-img snapshot commands should always be run against disk images of powered off VMs, an exepction lies with the qemu-img snapshot -l variant.

Creating disk image snapshots

To create and view snapshots of the disk image brdisk-rebase0-img.qcow25:

$ qemu-img snapshot -c br-rabase0-snpsht0 brdisk-rebase0-img.qcow2

$ qemu-img snapshot -l brdisk-rebase0-img.qcow2 
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         br-rabase0-snpsht0        0                    23:24:29   00:00:00.000

$ qemu-img info brdisk-rebase0-img.qcow2 
image: brdisk-rebase0-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 865M
cluster_size: 65536
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         br-rabase0-snpsht0        0                    23:24:29   00:00:00.000

Booting, say, with:

$ sudo qemu-system-i386 -smp 2 -enable-kvm -kernel bzImage -drive file=brdisk-rebase0-img.qcow2,cache=writeback -append "root=/dev/sda2 fbcon=map:-1 rw"

and then deleting the file /root/vmlinux6

before halting the VM and creating a second snapshot of the disk image:

$ qemu-img snapshot -c br-rabase0-snpsht1 brdisk-rebase0-img.qcow2

$ qemu-img snapshot -l brdisk-rebase0-img.qcow2 
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         br-rabase0-snpsht0        0                       23:24:29   00:00:00.000
2         br-rabase0-snpsht1        0                       23:37:27   00:00:00.000

$ qemu-img info brdisk-rebase0-img.qcow2 
image: brdisk-rebase0-img.qcow2
file format: qcow2
virtual size: 2.5G (2684354560 bytes)
disk size: 869M
cluster_size: 65536
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         br-rabase0-snpsht0        0                       23:24:29   00:00:00.000
2         br-rabase0-snpsht1        0                       23:37:27   00:00:00.000

Booting (with the same qemu-system commandline above) and checking:

i.e. /root/vmlinux is indeed abscent. Then intentionally executing:

closing the QEMU window, then rebooting (with the same qemu-system command line), results in the following instance:

i.e. /sbin no longer exists

Reverting to a previous disk image snapshot

So, to revert to the initial state of brdisk-rebase0-img.qcow2:

$ qemu-img snapshot -a 1 brdisk-rebase0-img.qcow2

Booting with this disk image, both the /sbin directory and the 165MB /root/vmlinux file are present:

Deleting disk image snapshots

To delete a snapshot, list the disk image snapshots then delete a snapshot using its snapshot ID, for instance:

$ qemu-img snapshot -l brdisk-rebase0-img.qcow2 
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         br-rabase0-snpsht0        0                       23:24:29   00:00:00.000
2         br-rabase0-snpsht1        0                       23:37:27   00:00:00.000

$ qemu-img snapshot -d 2 brdisk-rebase0-img.qcow2

$ qemu-img snapshot -l brdisk-rebase0-img.qcow2 
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         br-rabase0-snpsht0        0                       23:24:29   00:00:00.000

Resources

  • http://doc.opensuse.org/products/draft/SLES/SLES-kvm_sd_draft/cha.qemu.guest_inst.html#cha.qemu.guest_inst.qemu-img

Footnotes

1. Consider use of the virtio-blk block device interface for performance reasons. See QEMU Virtio [go back]

2. We will refer to this vmlinux file and location in Section link [go back]

3. Compare these values to those in of the same brdisk-img.qcow2 image in Section link [go back]

4. VM powered off [go back]

5. The qcow2 format supports snapshots. [go back]

6. See Section link on the previous installation of this file in the image [go back]

Appendix

Using Temporary Files

If a boot is done against a disk image with the -snapshot option, all writes to the disk from within the live instance of the VM will be directed to the temporary files on the host (leaving the base image unmodified). For instance:

    $ qemu-system-x86_64 -enable-kvm -smp 2 -snapshot -drive file=virt-ubuntu-13.04-amd64.qcow2,cachewriteback ...

Note that these temporary files get discarded upon VM shutdown. If, on the other hand, you wish to maintain these disposable changes somewhere, then consider using QEMU's copy-on-write mechanism for Base and Derived Images instead.