I was putting a RedHat server onto a SAN and I could not find any clear instructions on how to grow a single mirrored LUN on the fly. Anyway, here are some notes on the process. First the setup: Two LUN’s mirrored across two SAN’s with LVM volume on the top of it. I could have easily just presented another set of mirrored LUN’s, add them to VG and go from there. I wanted to avoid that, as that kind of setup can quickly get out of hand as the number of presented LUN’s grows. If there is a more “sensible” and flexible setup, I would most definitely want to know about it.

For sake of completeness, here are steps to recreate the initial setup I had:

  1. Create a mirror from two LUN’s
  2. Use the mirror as PV
  3. Create a VG using the PV
  4. Create LV on the top of the VG
  5. Make ext3 filesystem on the top of LV and mount it

Here are the actual steps with some output:

[root@ultra /]#  mdadm --create /dev/md10 --level=1 --raid-devices=2 /dev/mapper/mpath4 /dev/mapper/mpath5  
mdadm: array /dev/md10 started.
[root@ultra /]#  pvcreate /dev/md10  
Physical volume "/dev/md10" successfully created
[root@ultra /]#  vgcreate testvg /dev/md10  
Volume group "testvg" successfully created
[root@ultra /]#  lvcreate -l+100%FREE -n testlv testvg  
Logical volume "testlv" created
[root@ultra /]#  mkfs -t ext3 /dev/testvg/testlv  
[root@ultra /]#  mount /dev/testvg/testlv /tmp/test

Now the resizing part. There might be a few steps but the upshot is that the filesystem can stay mounted and in use. High level overview of steps to take:

  1. Grow the two LUN’s using SAN management software
  2. Fail and remove one of the submirrors
  3. Force the kernel to see the size increase of the submirror
  4. Flush and recreate the multipath device map so multipathing sees the new size
  5. Re-add the submirror to the mirror and let it sync
  6. Repeat 2-4 for the second submirror
  7. Resize the PV
  8. Resize the LV
  9. Resize the filesystem

First, you fail and remove the submirror:

[root@ultra /]#  mdadm /dev/md10 -f /dev/mapper/mpath4 -r /dev/mapper/mpath4  
mdadm: set /dev/mapper/mpath4 faulty in /dev/md10
mdadm: hot removed /dev/mapper/mpath4

Now, note all paths to the LUN. Kernel sees a separate device at the end of each path to a LUN. In this case they are sdj, sdt, sdg and sdq.

[root@ultra /]#  multipath -ll mpath4  
mpath4 (3600508b400011c300000f000008d0000)
[size=12 GB][features="1 queue_if_no_path"][hwhandler="0"]
_ round-robin 0 [prio=100][active]
._ 1:0:3:1     sdj 8:144  [active][ready]
._ 2:0:3:1     sdt 65:48  [active][ready]
_ round-robin 0 [prio=20][enabled]
._ 1:0:2:1     sdg 8:96   [active][ready]
._ 2:0:2:1     sdq 65:0   [active][ready]

At this point the problem is to get the kernel to recognize the new size without reboot. After a lot of trying and sifting through man pages I found that blockdev command does the magic. Then I googled “blockdev resize” and I found this confirming my finding. So, the next step is to probe all logical paths to the LUN:

[root@ultra /]#  blockdev --rereadpt /dev/sdj  
[root@ultra /]#  blockdev --rereadpt /dev/sdt  
[root@ultra /]#  blockdev --rereadpt /dev/sdg  
[root@ultra /]#  blockdev --rereadpt /dev/sdq

You should see messages in /var/log/messages about kernel seeing new size on all paths. If you were to issue multipath -ll right now you would see that multipathing is still reporting old size. To fix that, flush the device map of the LUN and then recreate it:

[root@ultra /]#  multipath -f mpath4  
[root@ultra /]#  multipath -v2  
create: mpath4 (3600508b400011c300000f000008d0000)
[size=13 GB][features="0"][hwhandler="0"]
_ round-robin 0 [prio=100]
._ 1:0:3:1     sdj 8:144  [ready]
._ 2:0:3:1     sdt 65:48  [ready]
_ round-robin 0 [prio=20]
._ 1:0:2:1     sdg 8:96   [ready]
._ 2:0:2:1     sdq 65:0   [ready]

Multipathing should be reporting the new size. Now you are ready to put back the grown submirror and let the whole mirror sync:

[root@ultra /]#  mdadm /dev/md10 -a /dev/mapper/mpath4  
mdadm: hot added /dev/mapper/mpath4

When the mirror has synced up, repeat the above process for the second submirror and wait for the sync to finish. Time to grow the mirror device itself:

[root@ultra /]#  mdadm --grow /dev/md10 --size=max

After the completion /proc/mdstat should report increase in size of /dev/md10. Moving on you need to grow the PV that resides on_ /dev/md10_:

[root@ultra /]#  pvresize /dev/md10  
Physical volume "/dev/md10" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, you need to resize the LV:

[root@ultra /]#  lvresize -l+100%FREE testvg/testlv  
Extending logical volume testlv to 13.00 GB
Logical volume testlv successfully resized

Of course, don’t forget to grow the filesystem itself:

[root@ultra /]#  ext2online /dev/testvg/testlv  
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
[root@ultra /]#  df  
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-rootlv
                    132304280      5104976  120478588     5% /
/dev/md0              132134         32791      92521    27% /boot
none                 8202920             0    8202920     0% /dev/shm
/dev/mapper/testvg-testlv
                     13413488         63516   1266882     1% /tmp/test

That should be it. The sync time for huge volumes is going to be something to keep in mind. The whole setup is clean and neat without clutter. I could have opted to mirror using LVM, but there seems to be a strange requirement for third, log volume. It is possible to keep the log in memory, but that supposedly causes resync on boot.