How to NOT destroy a RAID array

by Stephen Fluin 2009.04.15

This afternoon I was attempting to increase my RAID 5 array from 3 disks to 4 disks. I keep my home directory on my RAID, so even after unmounting the array, some files were still in use on the drive. After repeated uses of lsof and mounting and unmounting the drive, I made a mistake, accidentally running sudo resize2fs /dev/md0 on a mounted drive, resulting in the following:

resize2fs 1.41.3 (12-Oct-2008)
Filesystem at /dev/md0 is mounted on /super; on-line resizing required
old desc_blocks = 88, new_desc_blocks = 131
Performing an on-line resize of /dev/md0 to 549428928 (4k) blocks.

This was frightening to me because fsck and resize2fs repeatedly gave me warnings when attempting to resize the unmounted drive. These warnings were about busy files, and high probability data loss. Given this fact, along with the fact that I had unintentionally started a filesystem resize on a mounted filesystem, I was almost certain I had lost data. I couldn't even kill the process as this would almost guarantee corruption.

Luckily, about 5 minutes later I noticed that the command had finished, and I checked df, and attempted to use the filesystem. Everything checked out without any problems at all. It was an mdadm miracle!