Recovering a failed drive or volume

If you need to get the data off a LVM disk that isn’t booting. The following steps will make the data accessible without removing the drive from the server

Check in the BIOS and ensure the server can PXE boot (ideally from the internal network NIC)
Select ‘CentOS Rescue’from the PXE boot menu and pick all the defaults
Once this finished the rescue will have mounted the filesystems it found, if not try the following

lvm vgscan
lvm vgchange -ay
lvm lvs

The above should make all the volumes available and we can either use rsync or scp to move the files off to another server or another drive on the same server. Or even repair the filesystem if required.

Resizing a LVM

umount /dev/vg_blah/lv_blah
e2fsck -f /dev/vg_blah/lv_blah
resize2fs /dev/vg_blah/lv_blah 9G
lvreduce -L -1G /dev/vg_blah/lv_blah

vgreduce vg_blah /dev/sdxy

Warning: It is crucial that you use the exact amount of space. If you reduce volume more than the filesytem you truncate the logical volume and you risk data loss!

resize2fs /dev/vg_blag/lv_blah 100G
lvextend -l +100%FREE /dev/vg_blag_lv_blah (extends to the rest of the free space)

Mirroring a LVM volume

This example is going to add /dev/sdc as a mirror to the existing VolGrp00/LogVol02
(Volume group VolGrp00 logical volume LogVol02). This also assumes that you either have a spare drive, or have mounted one using iSCSI

pvcreate /dev/sdc
vgextend VolGrp00 /dev/sdc
lvconvert -m1 VolGrp00/LogVol02 –mirrorlog core

Removing the mirror one step
lvconvert -m0 VolGrp00/LogVol02
lvremove

Missing LVM devices

If getting the error
Couldn’t find device with uuid ‘VLwXTu-7Wdp-wU3h-pEld-S282-SKFv-fXkLYe’.
There are 1 physical volumes missing.

And, the logical volumes are all correct. Or if something was removed and the volume group needs to be updated, simply run

vgreduce –removemissing VolGrp00

This will clear out any missing devices from VolGrp00

Making a snapshot of a LVM

The snapshot doesn’t have to be any bigger than the amount of expected changes over the lifetime of the snapshot. But, there does have to be enough space to create it

lvcreate -L 500M -s -n RootSnap /dev/VolGroup00/LogVol02

(creates a snapshot called RootSnap of LogVol02 from VolGroup00)

If there isn’t enough room. Simply mount an additional iSCSI target, extend the volume group and then use that. For this example we assume /dev/sdd has been mounted via iSCSI

pvcreate /dev/sdd
vgextend VolGroup00 /dev/sdd

This will now give you enough room to create a snapshot. Once finished
lvremove /dev/VolGroup00/RootSnap
vgreduce VolGroup00

Using the Ubuntu Live CD and attaching a iSCSI device

If you need to boot the server off a rescue device and mount an iSCSI drive the following can be used. Boot off the Ubuntu Live CD. It should autoload everything and you’ll end up in the graphical desktop. Hit ALT and 1 and it will switch you to the text console

apt-get install open-iscsi
iscsiadm -m discovery -t st -p 10.1.1.119
10.1.1.119:3260,1 iqn.2011-10.com.centos:ds19.target01
10.1.1.119:3260,1 iqn.2011-10.com.centos:ds19.target02

iscsiadm -m node –targetname “iqn.2011-10.com.centos:target01” –portal “10.1.1.119” –login

Check dmesg and the drive should show up

Replacing a failed hard drive with a larger one

This example assumes a 80GB drive sda is failing
Its being replaced by a 300GB drive sdb

Use dd or dd_rescue to mirror the old drive to the new one
dd if=/dev/sda of=/dev/sdb

Once the copy is done, reboot into one of the CentOS rescue modes via PXE
We’ll assume there’s two partitions, boot and the lvm

Device Boot Start End Blocks Id System
/dev/sda1 1 486 3903763+ 83 Linux
/dev/sda2 487 60801 484480237+ 8e Linux LVM

Using fdisk, delete the second partition
Then, create it starting at the same point, and then using the rest of the disk (fdisk should do this automatically)
Change the ID to LVM (t, 2, 8e)

It may need a reboot to load the new partitions. Then

pvresize /dev/sda2

You can then pick which logical volume you want to extend

lvextend -L +10G –name lv01 /dev/VolGroup00/LogVol00
fsck -y -f /dev/VolGroup00/LogVol00
resize2fs /dev/VolGroup00/LogVol00
fsck -y -f /dev/VolGroup00/LogVol00

(do a forced check of all remaining filesystems on the drive)

Leave a Reply

Your email address will not be published. Required fields are marked *