Log in

   Journal    Friends    Archive    Profile    Memories

HowTo: Shrink a Linux VM's VMDK File in ESXi 4.1 - zeuzeu

Jun. 26th, 2011 10:00 am HowTo: Shrink a Linux VM's VMDK File in ESXi 4.1

I have a Linux VM used for backups with VMDK file approx. 300GB.
I wanted to migrate it to another ESXi box, but its datastore didn't support files larger than 256GB.

Luckily, this VM had a very convenient partitioning:

[root@srv35 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
                      2.0G  642M  1.3G  35% /
                       30G  430M   28G   2% /var
                      4.9G  1.6G  3.1G  35% /usr
                      247G  186G   51G  79% /home
/dev/sda1             289M   24M  251M   9% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm
                      496M   20M  451M   5% /tmp
[root@srv35 ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 39162.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          38      305203+  83  Linux
/dev/sda2              39        9137    73087717+  8e  Linux LVM
/dev/sda3            9138       39162   241175812+  83  Linux

Command (m for help): q

As you can see, this VM uses LVM. I checked the volume groups with vgdisplay and confirmed that there was only 1 group VolGroup00 and it didn't have any free extents. THIS IS IMPORTANT.
Also note that the size of /home is little less than 256GB, so let's move it to a separate VMDK file first:

Moving /home to a separate virtual disk

1. Using your vSphere Client, add another drive to this VM, then reboot VM.
2. Add the new hard drive to the Volume Group and create a new filesystem for /home:
pvcreate /dev/sdb
vgextend VolGroup00 /dev/sdb
lvm lvcreate -l +100%FREE  -nHOME VolGroup00
mkfs.ext3 /dev/VolGroup00/HOME
mkdir /mnt/home
mount /dev/VolGroup00/HOME /mnt/home

Since VolGroup00 didn't have any free extens prior to adding /dev/sdb, the new home partition is physically located on the second drive only.
3. Copy the data over:
shut down all services that might use /home partition, then copy the data with rsync or cpio.
To find out what processes holding the locks on /home, execute:
     fuser -v -m /home
To copy the data with cpio, execute:
     cd /home ; find ./ -xdev -print0 | cpio -pa0V /mnt/home/
but for copying not system partitions, I like rsync better, so I did
rsync -avz -e ssh /home /mnt

4. Now mount your new home filesystem at /home and make the change permanent:
umount /home
umount /mnt/home
mount /dev/VolGroup00/HOME /home

Bring up all the services, check that everything works and make the change permanent by modifying /etc/fstab file.
We are now ready to shrink the 300GB VMDK file.

Freeing up some space

First, get rid of the old home filesystem:
[root@srv35 ~]# lvremove /dev/VolGroup00/LogVol05
Do you really want to remove active logical volume LogVol05? [y/n]: y
  Logical volume "LogVol05" successfully removed

This releases the space allocated for the old home partition back to the Volume Group.
Luckily, LogVol05 was physically located on /dev/sda3 only, filling it 100%, so /dev/sda3 is now completely unused and can be removed from the Volume Group.
How do I know? By looking at the number of "Total PE" and comparing it to the number of "Free PE":
[root@srv35 ~]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  12
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               554.66 GB
  PE Size               32.00 MB
  Total PE              17749
  Alloc PE / Size       9615 / 300.47 GB
  Free  PE / Size       8134 / 254.19 GB
  VG UUID               0NudeN-KPoC-fJah-myiN-Wrof-sD2m-EcHeIT
[root@srv35 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               VolGroup00
  PV Size               69.70 GB / not usable 14.72 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              2230
  Free PE               774
  Allocated PE          1456
  PV UUID               c00AU3-mxKW-fo0y-MC1G-bnRg-bLKj-9iXxfe

  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               VolGroup00
  PV Size               230.00 GB / not usable 3.25 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              7360
  Free PE               7360
  Allocated PE          0
  PV UUID               KVuWfS-39JR-szwJ-wXZa-Oct1-4RQe-zURsn1

  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               VolGroup00
  PV Size               255.00 GB / not usable 32.00 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              8159
  Free PE               0
  Allocated PE          8159
  PV UUID               AVNHKW-G3t3-dLjK-0ysf-rpIg-8xq9-LftBFN
In general case, though, a physical volume is used by multiple logical volumes, so you might need to migrate the data to another physical volume using pvmove.

The following command removes the physical volume /dev/sda3 from the volume group VolGroup00:
[root@srv35 ~]# vgreduce VolGroup00 /dev/sda3
  Removed "/dev/sda3" from volume group "VolGroup00"

Little VMDK magic

Shut down your VM,login into ESXi box and navigate to the VMDK file. Back it up then edit it as follows:
Find section that looks like
# Extent description
RW 629145600 VMFS "srv35-backup-flat.vmdk"

and change the value 629145600 (which corresponds to 629145600/2/1024/1024=300GB) to 157286400 (157286400/2/1024/1024=75GB).
If you take a closer look at fdisk output above, you will notice that /dev/sda1 and /dev/sda2 take up about 70GB, but I did not want to get into somewhat complicated calculations of partition boundaries, so I left more space than necessary. Now, clone the disk using vmkfstools tool:
/vmfs/volumes/4d628e64-e26b84be-5900-a4badb2d782d/srv35-backup # vmkfstools -i srv35-backup.vmdk -d thin srv35-backup-CLONE.vmdk
Destination disk format: VMFS thin-provisioned
Cloning disk 'srv35-backup.vmdk'...
Clone: 100% done.

In vSphere client, select this VM, click "Edit Settings" and set it up to use srv35-backup-CLONE.vmdk instead of srv35-backup.vmdk. Turn the VM on and make sure it works properly. You can migrate it now. After migration, delete /dev/sda3 and re-create it obeying the new virtual drive's boundaries.

Lesson learned:
While provisioning a new datastore, think twice before you use non-default block size.
As a rule, make sure all your datastores have the same blocksize.

How to move your filesystem to a new hard drive
Copying/resizing partitions for a dual boot system

LVM Administrator's Guide: Removing Physical Volumes from a Volume Group
LVM HOWTO: Removing physical volumes from a volume group
LVM Administrator's Guide: Online Data Relocation

How To Shrink a VMDK File in ESX 4 and Windows
SolsTech Stuff: Resize VM Disk File in VMWare ESXi Server

1 comment - Leave a comment Share Next Entry


Date:October 8th, 2013 06:09 pm (UTC)
Thanks for sharing your thoughts. I truly appreciate your efforts and I am waiting for your next post thank you once again.