ZFS – Dealing with Failed Drives, using spares and new drives – and autoreplace

Links/Citations:

*http://www.firstalt.co.uk/sample/s10up.pdf
*http://www.bolsondigital.com/2013/06/01/replacing-an-hdd-in-a-zfs-pool-on-nas4free/
*http://www.packetmischief.ca/2013/06/10/how-i-replace-a-failedfailing-hdd-in-a-zfs-storage-pool/
*http://docs.oracle.com/cd/E19253-01/819-5461/gbcet/
*http://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gbbxi/index.html
*https://blogs.oracle.com/eschrock/entry/zfs_hot_spares

REPLACING FAILED DRIVE WITH SPARE

Spares usually auto fall a failed drive (this is not to be confused with autoreplace feature, which as you will see in a little bit, just replaces a bad drive with another drive it detects in its place).

The automatic process defined by Oracle blogger “eschrock” (see link above) happens automatically like so:

“There is now an FMA agent, zfs-retire, which subscribes to vdev failure faults and automatically initiates replacements if there are any hot spares available.”

To manually do it:

# zpool offline <pool> <old drive>

# zpool replace <pool> <old drive> <old bad drive>

The process for Spares to fall in place of a bad drive should be automatic, and depend on the fmd ZFS agent, which by default works.

NOTE: Drives <disk> have numerous different ways of appearing, here are some variations (from zdb output):
guid: 3116999365654966099
path: ‘/dev/dsk/c0t5000C5004EF4D025d0s0’ (s0 thats a slice/partition, which elapses full drive)
devid: ‘id1,sd@n5000c5004ef4d025/a’
phys_path: ‘/scsi_vhci/disk@g5000c5004ef4d025:a’
Also depending on your distro the path could be different like the simple: /dev/dsk/c2t4d0
Note when refering to a drive by path, you can ommit the /dev/dsk part and just say:
c2t4d0
or
c0t5000C5004EF4D025d0 (notice I remove the s0)

DISCONNECTING OLD DRIVE AND PLUGGING IN NEW DRIVE

When you add the new drive (either turn off the system, or hot add), here are the zfs commands (I dont know about commands for such things as disk discovery):

The steps should be similar to this (from: http://docs.oracle.com/cd/E19253-01/819-5461/gbcet/)

Before you take out bad drive:

it might be advisable to do this step now:

zpool detach <pool> <old bad drive>

Now lets find out the Sata name of the <old bad drive>

I will call <old bad drive> as simply <disk> here.

# cfgadm | grep <disk>
<sata>::dsk/<disk> disk connected configured ok

Example of what <sata> could say is: sata1/3

# cfgadm -c unconfigure <sata>

Press Y at the prompt to unconfigure the drive from its driver.

Confirm disk shows as unconfigured

# cfgadm | grep <sata>
sata1/3 disk connected unconfigured ok

Now replace the bad drive with the good drive physically on the unit (you can do it cold, unit is off, or hot, I recommend doing it hot, unit is on)

# cfgadm -c configure <Sata>

# cfgadm | grep <sata>
sata1/3::dsk/<disk> disk connected configured ok

Notice that <disk> is the same name as old bad drive. Now in some cases, if your disk names are more unique and perhaps use a hash of the GUID or SERIAL NUMBER. Or perhaps its just the GUID that your using for your disk names, then instead this name should be new.

REPLACING NEW DRIVE WITH SPARE

A QUICKNOTE ON AUTO REPLACE

The Autoreplace feature is supposed to automate this whole procedure (maybe even the unconfiguring and configuring seen above):

zpool autoreplace (Off by default) as defined by Oracle:
“Controls automatic device replacement. If set to off, device replacement must be initiated by using the zpool replace command. If set to on, any new device found in the same physical location as a device that previously belonged to the pool is automatically formatted and replaced. The property abbreviation is replace.”

So it only replaces drives with new drives in same physical location – not with spares. Spares automatically fall in to place of a dead drive, weather autoreplace is on or off.

Of course for this feature to work best, its probably best to set this on before any occurance of problems (not tested but if you turn this on during failure, you probably still have to do the manual commands)

To turn on autoreplace:

# zpool set autoreplace=on mpool

To turn off autoreplace:

# zpool set autoreplace=off mpool

To get current setting of autoreplace:

# zpool get autoreplace mpool
NAME PROPERTY VALUE SOURCE
mpool autoreplace on default

Another way to get current autoreplace status:

# zpool get all 
# zpool get all | egrep autoreplace

MANUAL REPLACE

Do this next step if you still see the old bad drive:

# zpool detach <pool> <old bad drive>

Continue here after:

# zpool replace <pool> <spare drive> <new drive>

After this is done the spare should go back to being a spare but if it doesnt:

# zpool add <pool> spare <spare drive>

Repeat last command for any other pools the <spare> was a spare for.

confirm:

# zpool status

You can use the GUID of the drive inplace of the drive name, to get all GUIDS:

# zdb

You will see a similar output to zpool status, except with alot more info in place of each line.

FINISHING MOVES HOUSE KEEPING AND GOOD GENERAL PRACTICE

To go over the data with a fine tooth comb and pick out any errors, and fix the obvious ones (checksum is wrong, but the other parities say it should be this, thus it can calculate what needs to go in place of data to fix check sum):

# zpool scrub <pool>

A scrub as defined by Oracle (see link above):

“This operation traverses all the data in the pool once and verifies that all blocks can be read. Scrubbing proceeds as fast as the devices allow, though the priority of any I/O remains below that of normal operations. This operation might negatively impact performance, though the file system should remain usable and nearly as responsive while the scrubbing occurs. “

To stop a scrub:

# zpool scrub -s <pool>

More from oracle:

“During a scrub performance may take a hit.

In most cases, a scrub operation, to ensure data integrity, should continue to completion. Stop a scrub at your own discretion if system performance is impacted by a scrub operation.

Performing routine scrubbing also guarantees continuous I/O to all disks on the system. Routine scrubbing has the side effect of preventing power management from placing idle disks in low-power mode. If the system is generally performing I/O all the time, or if power consumption is not a concern, then this issue can safely be ignored.”

To see all of your zfs/zpool commands

zpool history <pool>
history | grep "zfs\|zpool"