I have a guest which has a 10G root disk as /dev/vda, and I would like to expand the root disk from 10G to 15G to offer more room to the /var filesystem.

The typical method would be the following:

shutdown the guest
find the backing store on the host (in this case it's a LVM logical volume)
copy the LV for backup (lvcreate --name LVNew --size=10G; dd if=LVOld of=LVNew;)
expand the LV (lvresize --size=+5G LVOld;)
use kpartx on the host (or start the guest) and use parted to expand the guest disk parition
expand the guest physical volumes (pvresize /dev/vda2), logical volumes (lvresize), and filesystems (resize2fs)

But this can be both automated and generalized with libvirt and libguestfs-tools. These tools provide a common vocabulary across many different storage back-ends such as LVM, iSCSI, file, NFS, etc. Let's see how that works.

The state of virtualization on Fedora moves fast. The following is based on a Fedora 13 host running libvirt-0.8.2-1.fc13.x86_64 and libguestfs-tools-1.6.2-1.fc13.4.x86_64.

Overview

It's actually quite simple and mostly automated. Here are the steps, unexplained.

virsh vol-create-as VGRAID LV_f12-bacula-larger 15G
virt-resize --expand /dev/sda2 /dev/VGRAID/LV_f12-bacula /dev/VGRAID/LV_f12-bacula-larger
virsh edit f12-bacula

That glossed over the details, and it's always the details that are intersting.

Find the backing storage for the guest
Using libvirt storage pools and volumes
Create a larger storage volume with virsh
Migrate to the larger volume with virt-resize
Point guest configuration to larger volume
Restore grub
Boot the guest and expand filesystems
Explore the image with virt-rescue
Cleanup
P.S. Mac OS X tweaks needed for virt-manager

Find the backing storage for the guest

The first disk inside of the guest (/dev/vda) is acutally a logical volume (/dev/VGRAID/LV_f12-bacula) on the host. This can be confirmed in a couple of ways.

[root@sammy ~]# virsh dumpxml f12-bacula | grep -3 disk
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/VGRAID/LV_f12-bacula'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdd'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <interface type='bridge'>
      <mac address='52:54:00:74:78:e8'/>
      <source bridge='br200'/>

[root@sammy ~]# virsh dumpxml f12-bacula | xpath /domain/devices/disk/source
Found 2 nodes:
-- NODE --
<source dev="/dev/VGRAID/LV_f12-bacula" />
-- NODE --
<source dev="/dev/sdd" />

This host LV is 10G and holds 2 partitions. The first one (/dev/vda1) is used by the guest as /boot and the second (/dev/vda2) is a LVM physical volume which holds VGDomU and all the logical volumes for the various filesystems on the guest.

The second disk (/dev/vdb) has a single 1TB partition used to store bacula volumes.

The guest can be probed for it's disk contents with virt-list-partitions or virt-filesystems. Note the device names will vary slightly. Sda versus Vda.

[root@sammy ~]# virt-list-partitions -lht f12-bacula
/dev/sda1 ext4 100.0M
/dev/sda2 pv 9.7G
/dev/sdb1 ext4 1024.0G
/dev/sda device 10.0G
/dev/sdb device 1024.0G

Using libvirt storage pools and volumes

From the perspective of the host, the object to be expanded is a logical volume (/dev/VGRAID/LV_f12-bacula). Libvirt abstracts that object into a pool (which could be a directory, a disk, a VG, or a NFS mount, or iSCSI target for example) and a volume (which could be a file, a parition or a logical volume for example) and provides tools to manipulate storage an in a portable way.

I have 3 storage pools defined on this host.

[root@sammy ~]# virsh pool-list
Name                 State      Autostart
-----------------------------------------
default              active     yes
donkey               active     yes
VGRAID               active     yes

The identified logical volume is in the LVM volume group VGRAID, which has been mapped to a libvirt storage pool of the same name. The virsh pool-info command will provide an overview of that pool.

[root@sammy ~]# virsh pool-info VGRAID
Name:           VGRAID
UUID:           42e5e7e9-edd3-7c6d-0364-8551ac3f04bd
State:          running
Persistent:     yes
Autostart:      yes
Capacity:       1.82 TB
Allocation:     1.17 TB
Available:      661.93 GB

The virsh vol-list command will list the volumes that have been provisioned in this storage pool.

[root@sammy ~]# virsh vol-list VGRAID
Name                 Path
-----------------------------------------
LV_bewley_home       /dev/VGRAID/LV_bewley_home
LV_bewley_media      /dev/VGRAID/LV_bewley_media
LV_f12-bacula        /dev/VGRAID/LV_f12-bacula
LV_f12-zimbra        /dev/VGRAID/LV_f12-zimbra
LV_f12-zimbra-old    /dev/VGRAID/LV_f12-zimbra-old
LV_f13-bewley        /dev/VGRAID/LV_f13-bewley
LV_SambaData         /dev/VGRAID/LV_SambaData
LVBackup             /dev/VGRAID/LVBackup
LVBackupStore        /dev/VGRAID/LVBackupStore
LVHome               /dev/VGRAID/LVHome
LVLogArchive         /dev/VGRAID/LVLogArchive
LVOpt                /dev/VGRAID/LVOpt
LVRoot               /dev/VGRAID/LVRoot
LVSwap               /dev/VGRAID/LVSwap
LVUsr                /dev/VGRAID/LVUsr
LVVar                /dev/VGRAID/LVVar
LVVirt               /dev/VGRAID/LVVirt
LVWww                /dev/VGRAID/LVWww
VM_Seitan-Virt       /dev/VGRAID/VM_Seitan-Virt
VM_Win7-Cheech       /dev/VGRAID/VM_Win7-Cheech
VM_Win7-Cheech-clone /dev/VGRAID/VM_Win7-Cheech-clone

Not only the volumes used by libvirt are listed, but also the logical volumes used as filesystems or anything else on the host are listed. In fact the storage pool "default" is a directory pool living in a filesysem on the logical volume LVVirt.

[root@sammy ~]# virsh pool-dumpxml default | xpath /pool/target/path
Found 1 nodes:
-- NODE --
/var/lib/libvirt/images
[root@sammy ~]# df -h /var/lib/libvirt/images
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VGRAID-LVVirt
                      193G  105G   79G  58% /var/lib/libvirt/images

And finally, how big is our identified volume? That could be found with lvs, but to stay in the libvirt paradigm use virsh vol-info.

[root@sammy ~]# virsh vol-info --pool VGRAID LV_f12-bacula
Name:           LV_f12-bacula
Type:           block
Capacity:       10.00 GB
Allocation:     10.00 GB

Create a larger storage volume with virsh

The virt-resize tool can be used to copy the contents of a image to another, and either grow or shrink partitions and filesystems on the fly. Virt-resize can not grow a disk image in place, so you must create a new volume first.

[root@sammy ~]# virsh vol-create-as VGRAID LV_f12-bacula-larger 15G
Vol LV_f12-bacula-larger created

Migrate to the larger volume with virt-resize

Now use virt-resize to copy the guest domain to it and grow to fill the new space

[root@sammy ~]# virt-resize --expand /dev/sda2 /dev/VGRAID/LV_f12-bacula /dev/VGRAID/LV_f12-bacula-larger
Summary of changes:
/dev/sda1: partition will be left alone
/dev/sda2: partition will be resized from 9.7G to 14.9G
/dev/sda2: content will be expanded using the 'pvresize' method
Copying /dev/sda1 ...
Copying /dev/sda2 ...
[############################################################################]
Expanding /dev/sda2 using the 'pvresize' method

Point guest configuration to larger volume

There is now a new 15G volume call LV_f12-bacula-larger, but the guest domain config still uses the old smaller 10G image. Edit the domain and change the source for disk vda to point the new volume.

[root@sammy ~]# virsh edit f12-bacula
[root@sammy ~]# virsh dumpxml f12-bacula | xpath /domain/devices/disk/source
Found 2 nodes:
-- NODE --
<source dev="/dev/VGRAID/LV_f12-bacula-larger" />
-- NODE --
<source dev="/dev/sdd" />

Double check to see that the disk has been grown inside the still shutdown guest.

[root@sammy ~]# virt-list-partitions -lht f12-bacula
/dev/sda1 ext4 100.0M
/dev/sda2 pv 14.9G
/dev/sdb1 ext4 1024.0G
/dev/sda device 15.0G
/dev/sdb device 1024.0G

Restore grub

At this point the guest may not boot up and hang at "Booting from Hard Disk...". There's a tip in the man page for virt-resize that solves this problem by reinstalling grub. How do you do that if the guest won't boot? With the amazing guestfish!

[root@sammy ~]# guestfish -i -a /dev/VGRAID/LV_f12-bacula-larger

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for a list of commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 12 (Constantine)
/dev/VGDomU/LVRoot mounted on /
/dev/vda1 mounted on /boot
/dev/VGDomU/LVHome mounted on /home
/dev/VGDomU/LVUsr mounted on /usr
/dev/VGDomU/LVVar mounted on /var
/dev/VGDomU/LVVarBackup mounted on /var/backup

><fs> cat /boot/grub/device.map
# this device map was generated by anaconda
(hd0)     /dev/vda

><fs> grub-install / /dev/vda
><fs> exit

Boot the guest and expand filesystems

Now the guest's disk has been grown, the partition holding the LVM has been expanded, and the PV on that partition has been expanded to fill it. Finally, the guest LVs filesystems can be expanded.

[root@bacula ~]# lvresize --size=+1G /dev/VGDomU/LVVar
  Extending logical volume LVVar to 5.00 GiB
  Logical volume LVVar successfully resized
[root@bacula ~]# resize2fs /dev/VGDomU/LVVar
resize2fs 1.41.9 (22-Aug-2009)
Filesystem at /dev/VGDomU/LVVar is mounted on /var; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/VGDomU/LVVar to 1310720 (4k) blocks.
The filesystem on /dev/VGDomU/LVVar is now 1310720 blocks long.

Explore the image with virt-rescue

The image could also be manipulated almost like you booted it from a rescue CD by using the virt-rescue tool.

virt-rescue /dev/VGRAID/LV_f12-bacula-larger
...
><rescue> sfdisk -l /dev/vda

Disk /dev/vda: 31207 cylinders, 16 heads, 63 sectors/track
Units = cylinders of 516096 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/vda1   *      0+    203-    204-    102400   83  Linux
                end: (c,h,s) expected (203,3,51) found (12,191,51)
/dev/vda2        203+  31203-  31001-  15624064   8e  Linux LVM
                start: (c,h,s) expected (203,3,52) found (12,191,52)
                end: (c,h,s) expected (1023,15,63) found (1023,254,63)
/dev/vda3          0       -       0          0    0  Empty
/dev/vda4          0       -       0          0    0  Empty
><rescue> fdisk -l /dev/vda

Disk /dev/vda: 16.1 GB, 16106127360 bytes
16 heads, 63 sectors/track, 31207 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0007b0d7

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           1         204      102400   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2             204       31204    15624064   8e  Linux LVM
Partition 2 does not end on cylinder boundary.
><rescue> pvs
  PV         VG     Fmt  Attr PSize  PFree
  /dev/vda2  VGDomU lvm2 a-   14.88g 6.03g
><rescue> vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  VGDomU   1   6   0 wz--n- 14.88g 6.03g
><rescue> lvs
  LV          VG     Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  LVHome      VGDomU -wi-a- 128.00m
  LVRoot      VGDomU -wi-a- 512.00m
  LVSwap      VGDomU -wi-a- 512.00m
  LVUsr       VGDomU -wi-a-   1.25g
  LVVar       VGDomU -wi-a-   4.00g
  LVVarBackup VGDomU -wi-a-   2.47g
><rescue> mount /dev/VGDomU/LVRoot /sysroot
[  352.049597] EXT4-fs (dm-0): mounted filesystem with ordered data mode
><rescue> cd sysroot
...

Cleanup

Once everything is tested and known working, the volume can be renamed to drop the "-larger". Shutdown the guest, remove the old volume, and rename the volume from LV_f12-bacula-larger to LV_f12-bacula, and finally re-point the guest configuration to the image with the original name.

P.S. Mac OS X tweaks needed for virt-manager

I was using a mac as a client and found that I could not release the cursor from the virt-viewer pane or type anything useful into it. The fix requires changes on the client and the server.

If you look at the details tab of a guest and examine the graphics device most likely the keymap is set to none.
Stop the guest and use 'virsh edit guest' to include a keymap in the graphics node. i.e.

                <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>

Place the following into ~/.Xmodmap on the client to tell the mac to send the proper keys

   clear Mod1
   keycode 66 = Alt_L
   keycode 69 = Alt_R
   add Mod1 = Alt_L
   add Mod1 = Alt_R

Then from the mac be sure to close the X server if it is open and type:

   xhost + kvmhost
   ssh -X root@kvmhost virt-manager

tofu.org a big wad of tofu

Navigation

User login

Recent comments

Expanding a guest disk image using libvirt and libguestfs