I have long contended that for small, home NAS systems, mirrors are the way to go. You don’t have to calculate how much parity you need, similarly, the backup size is also obvious. It costs less upfront, you only need two disks. And because you only need two disks, you can get the biggest disks you can afford, which usually have excellent GB/cent cost ratios.
Finally, you can grow mirrors! As long as your storage needs don’t outstrip storage space growth, you can grow your mirrors very economically. Case in point, I was running a 2TB mirror which was getting close to 80% full. I was able to buy 8TB disks very cheaply, and grow my mirror to 8TB.
First things first, don’t be an idiot, this guide is how to modify storage. Triple check everything, and make sure you have backups. This guide also assumes the disks are blank, i.e. unformatted.
Step 1: Install new drive
I had no spare bays/SATA connections, but if you did, you could potentially do this in a more safer way by adding the new disks to a mirror to create a 3 or 4-way mirror, and then pulling the old disks out.
I had to do it more riskily by degrading the mirror. First, I powered down the NAS. Then I replaced one of the old disks in the mirror with a new one and powered up again. First, let’s check the right disk was pulled:
$ zpool status pool pool: pool state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scan: scrub repaired 0 in 0 days 02:49:09 with 0 errors on Sun May 13 02:49:10 2018 config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410 gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 ONLINE 0 0 0 errors: No known data errors
As expected, the mirror is degraded. So, now to figure out which drive is the new drive:
$ ls -1 /dev/ada* /dev/ada0 /dev/ada1 /dev/ada1p1 /dev/ada1p2 /dev/ada2 /dev/ada2p1 /dev/ada3 /dev/ada3p1
What’s going on? Well, drives are mounted in /dev and receive a name, e.g. ada1. The partitions are also mounted as /dev/ada1p1 and /dev/ada1p2. So it’s already pretty obvious that the new drive is ada0.
Just to check, we’ll look at the labels and partitions. BSD labels drives with UUIDs, since the name assigned in /dev is basically random, and drives could get reassigned on start-up.
$ glabel status
Name Status Components
gptid/23edea4a-1742-11e8-b551-28924a390410 N/A da0p1
gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 N/A ada1p2
gptid/d98b522f-178c-11e8-9b2e-28924a390410 N/A ada1p1
$ gpart show
=> 40 15663024 da0 GPT (7.5G)
40 1024 1 bios-boot (512K)
1064 15661992 2 freebsd-zfs (7.5G)
15663056 8 - free - (4.0K)
=> 40 3907029088 ada1 GPT (1.8T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834688 2 freebsd-zfs (1.8T)
3907029120 8 - free - (4.0K)
$ glabel list | grep ada0
$
Ignore da0, it is the OS USB disk. The blank disk is ada0, it has no labels or partition map.
Step 2: Create GPT partition map
Create the partition map on the new disk:
$ sudo gpart create -s gpt ada0 ada0 created
Now re-create partitions similar to the other disk in the mirror. My distro FreeNAS allocates a bit of swap space on all drives, so we want to replicate that. Get the parameters for gpart add from gpart list, -t is the type, -b is the the starting offset, -i is the index, and -s is the size. I don’t know why FreeNAS sets a starting offset for the swap partition, but whatever.
$ gpart show ada0 => 40 15628053088 ada0 GPT (7.3T) 40 15628053088 - free - (7.3T) $ gpart show ada1 => 40 3907029088 ada1 GPT (1.8T) 40 88 - free - (44K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834688 2 freebsd-zfs (1.8T) 3907029120 8 - free - (4.0K) $ gpart list ada1 [...] scheme: GPT Providers: 1. Name: ada1p1 Mediasize: 2147483648 (2.0G) [...] type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: ada1p2 Mediasize: 1998251360256 (1.8T) [...] type: freebsd-zfs index: 2 end: 3907029119 start: 4194432 Consumers: 1. Name: ada1 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e3 $ sudo gpart add -t freebsd-swap -b 128 -i 1 -s 2G ada0 ada0p1 added $ gpart show ada0 => 40 15628053088 ada0 GPT (7.3T) 40 88 - free - (44K) 128 4194304 1 freebsd-swap (2.0G) 4194432 15623858696 - free - (7.3T) $ gpart show ada1 => 40 3907029088 ada1 GPT (1.8T) 40 88 - free - (44K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834688 2 freebsd-zfs (1.8T) 3907029120 8 - free - (4.0K) $ sudo gpart add -t freebsd-zfs -i 2 ada0 ada0p2 added $ gpart show ada0 => 40 15628053088 ada0 GPT (7.3T) 40 88 - free - (44K) 128 4194304 1 freebsd-swap (2.0G) 4194432 15623858688 2 freebsd-zfs (7.3T) 15628053120 8 - free - (4.0K) $ gpart show ada1 => 40 3907029088 ada1 GPT (1.8T) 40 88 - free - (44K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834688 2 freebsd-zfs (1.8T) 3907029120 8 - free - (4.0K)
Step 3: Add the new disk/partition to the mirror
Now that the disk has a label/UUID, it’s time to stop referring to it as ada0 where possible.
$ glabel status Name Status Components gptid/23edea4a-1742-11e8-b551-28924a390410 N/A da0p1 gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 N/A ada1p2 gptid/fa78eb2a-178f-11e8-99e9-28924a390410 N/A ada2p1 gptid/fb1b40fa-178f-11e8-99e9-28924a390410 N/A ada3p1 gptid/d98b522f-178c-11e8-9b2e-28924a390410 N/A ada1p1 gptid/e5ebdfcc-726e-11e8-a5a9-28924a390410 N/A ada0p1 gptid/03d44ba4-726f-11e8-a5a9-28924a390410 N/A ada0p2 $ gpart list ada0 | grep rawuuid rawuuid: e5ebdfcc-726e-11e8-a5a9-28924a390410 rawuuid: 03d44ba4-726f-11e8-a5a9-28924a390410 $ zpool status pool pool: pool state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scan: scrub repaired 0 in 0 days 02:49:09 with 0 errors on Sun May 13 02:49:10 2018 config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410 gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 ONLINE 0 0 0 errors: No known data errors
In this case, the label is 03d44ba4-726f-11e8-a5a9-28924a390410 for the second partition, and I want to resilver from the online disk, d9ae78ea-178c-11e8-9b2e-28924a390410:
$ sudo zpool attach pool \ /dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 \ /dev/gptid/03d44ba4-726f-11e8-a5a9-28924a390410 $ zpool status pool pool: pool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sun Jun 17 13:54:32 2018 862G scanned at 464M/s, 3.13G issued at 31.4M/s, 1.70T total 3.12G resilvered, 0.18% done, 0 days 15:43:42 to go config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410 gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 ONLINE 0 0 0 gptid/03d44ba4-726f-11e8-a5a9-28924a390410 ONLINE 0 0 0 (resilvering) errors: No known data errors
Step 4: Wait
Resilvering takes time.
Step 5: Repeat with the other drive in the mirror
And please make sure you pull the right drive!
$ zpool status pool pool: pool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sun Jun 17 18:10:33 2018 865G scanned at 183M/s, 331M issued at 4.67M/s, 1.70T total 323M resilvered, 0.02% done, no estimated completion time config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410 6780713986779558112 UNAVAIL 0 0 0 was /dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 gptid/03d44ba4-726f-11e8-a5a9-28924a390410 ONLINE 0 0 0 gptid/1d2d3246-7294-11e8-85fd-28924a390410 ONLINE 0 0 0 (resilvering) errors: No known data errors $ zpool status pool pool: pool state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scan: resilvered 1.70T in 0 days 02:58:23 with 0 errors on Sun Jun 17 21:08:56 2018 config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410 6780713986779558112 UNAVAIL 0 0 0 was /dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 gptid/03d44ba4-726f-11e8-a5a9-28924a390410 ONLINE 0 0 0 gptid/1d2d3246-7294-11e8-85fd-28924a390410 ONLINE 0 0 0 errors: No known data errors
Step 6: Detach old disks and watch the mirror grow
$ sudo zpool detach pool 11300205822825511459 $ zpool status pool | grep 'state:' state: DEGRADED $ sudo zpool detach pool 6780713986779558112 $ zpool status pool pool: pool state: ONLINE scan: resilvered 1.70T in 0 days 02:58:23 with 0 errors on Sun Jun 17 21:08:56 2018 config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/03d44ba4-726f-11e8-a5a9-28924a390410 ONLINE 0 0 0 gptid/1d2d3246-7294-11e8-85fd-28924a390410 ONLINE 0 0 0 errors: No known data errors $ zpool list -o name,size NAME SIZE pool 7.27T freenas-boot 7.44G
Job done.