ZFS Failed Disk Replacement

Copy partitions from good disk sda to blank disk sdb

sgdisk /dev/sda -R /dev/sdb	# sgdisk  /dev/sda<From this disk> -R /dev/sdb<Replicate to this disk>
sgdisk -G /dev/sdb		# randomize the GUID on the new disk since it was copied from the other drive.

Using Parted to verify the partition table of /dev/sdb

(parted) select /dev/sdb
Using /dev/sdb

(parted) p
    Model: ATA WDC WD2000FYYZ-0 (scsi)
    Disk /dev/sdb: 2000398934016B
    Sector size (logical/physical): 512B/512B
    Partition Table: gpt
    Disk Flags:
    Number Start End Size File system Name Flags
    1 1048576B 2097151B 1048576B Grub-Boot-Partition bios_grub
    2 2097152B 136314879B 134217728B fat32 EFI-System-Partition boot, esp
    3 136314880B 2000397885439B 2000261570560B zfs PVE-ZFS-Partition

(Ok partitions copied)

Copy data from /dev/sda1 to /dev/sdb1

dd if=/dev/sda1 of=/dev/sdb1 bs=512 #This is the bios boot partition  
root@folkvang:~# dd if=/dev/sda1 of=/dev/sdb1 bs=512
2014+0 records in   
2014+0 records out  
1031168 bytes (1.0 MB) copied, 0.10164 s, 10.1 MB/s  

Replace the failed partition in the zpool

Find the ID of the failed block device

root@folkvang:~# zpool status
pool: rpool
    state: DEGRADED
    status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state.
    action: Replace the device using 'zpool replace'.
    see: http://zfsonlinux.org/msg/ZFS-8000-4J
    scan: scrub repaired 0 in 0h25m with 0 errors on Sun May  8 11:20:27 2016
    config
    NAME                    STATE     READ WRITE CKSUM
    rpool                   DEGRADED     0     0     0
      mirror-0              DEGRADED     0     0     0
        993077023721924477  FAULTED      0     0     0  was /dev/sdk2
        sdk2                ONLINE       0     0     0
    errors: No known data errors

Call zpool to replace the failed device

root@folkvang:~# zpool replace -f rpool 993077023721924477 /dev/sdl2

Make sure to wait until resilver is done before rebooting.

root@folkvang:~# zpool statuspool: rpool
    state: DEGRADED
    status: One or more devices is currently being resilvered.  The pool will continue to function, possibly in a degraded state.
    action: Wait for the resilver to complete.
    scan: resilver in progress since Fri Sep  2 16:45:53 2016
    13.2M scanned out of 8.83G at 902K/s, 2h50m to go
    12.9M resilvered, 0.15% done
    config:
    NAME                      STATE     READ WRITE CKSUM
    rpool                     DEGRADED     0     0     0
      mirror-0                DEGRADED     0     0     0
        replacing-0           UNAVAIL      0     0     0
          993077023721924477  FAULTED      0     0     0  was /dev/sdk2
          sdl2                ONLINE       0     0     0  (resilvering)
        sdk2                  ONLINE       0     0     0
    errors: No known data errors

After fixing the drive, we need to ensure that the boot sectors are configured.

grub-install /dev/sdk
grub-install /dev/sdl
update-grub