I have long contended that for small, home NAS systems, mirrors are the way to go. You don’t have to calculate how much parity you need, similarly, the backup size is also obvious. It costs less upfront, you only need two disks. And because you only need two disks, you can get the biggest disks you can afford, which usually have excellent GB/cent cost ratios.
Finally, you can grow mirrors! As long as your storage needs don’t outstrip storage space growth, you can grow your mirrors very economically. Case in point, I was running a 2TB mirror which was getting close to 80% full. I was able to buy 8TB disks very cheaply, and grow my mirror to 8TB.
First things first, don’t be an idiot, this guide is how to modify storage. Triple check everything, and make sure you have backups. This guide also assumes the disks are blank, i.e. unformatted.
Step 1: Install new drive
I had no spare bays/SATA connections, but if you did, you could potentially do this in a more safer way by adding the new disks to a mirror to create a 3 or 4-way mirror, and then pulling the old disks out.
I had to do it more riskily by degrading the mirror. First, I powered down the NAS. Then I replaced one of the old disks in the mirror with a new one and powered up again. First, let’s check the right disk was pulled:
$ zpool status pool
pool: pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 0 days 02:49:09 with 0 errors on Sun May 13 02:49:10 2018
config:
NAME STATE READ WRITE CKSUM
pool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 ONLINE 0 0 0
errors: No known data errors
As expected, the mirror is degraded. So, now to figure out which drive is the new drive:
$ ls -1 /dev/ada*
/dev/ada0
/dev/ada1
/dev/ada1p1
/dev/ada1p2
/dev/ada2
/dev/ada2p1
/dev/ada3
/dev/ada3p1
What’s going on? Well, drives are mounted in /dev and receive a name, e.g. ada1. The partitions are also mounted as /dev/ada1p1 and /dev/ada1p2. So it’s already pretty obvious that the new drive is ada0.
Just to check, we’ll look at the labels and partitions. BSD labels drives with UUIDs, since the name assigned in /dev is basically random, and drives could get reassigned on start-up.
$ glabel status
Name Status Components
gptid/23edea4a-1742-11e8-b551-28924a390410 N/A da0p1
gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 N/A ada1p2
gptid/d98b522f-178c-11e8-9b2e-28924a390410 N/A ada1p1
$ gpart show
=> 40 15663024 da0 GPT (7.5G)
40 1024 1 bios-boot (512K)
1064 15661992 2 freebsd-zfs (7.5G)
15663056 8 - free - (4.0K)
=> 40 3907029088 ada1 GPT (1.8T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834688 2 freebsd-zfs (1.8T)
3907029120 8 - free - (4.0K)
$ glabel list | grep ada0
$
Ignore da0, it is the OS USB disk. The blank disk is ada0, it has no labels or partition map.
Step 2: Create GPT partition map
Create the partition map on the new disk:
$ sudo gpart create -s gpt ada0
ada0 created
Now re-create partitions similar to the other disk in the mirror. My distro FreeNAS allocates a bit of swap space on all drives, so we want to replicate that. Get the parameters for gpart add from gpart list, -t is the type, -b is the the starting offset, -i is the index, and -s is the size. I don’t know why FreeNAS sets a starting offset for the swap partition, but whatever.
$ gpart show ada0
=> 40 15628053088 ada0 GPT (7.3T)
40 15628053088 - free - (7.3T)
$ gpart show ada1
=> 40 3907029088 ada1 GPT (1.8T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834688 2 freebsd-zfs (1.8T)
3907029120 8 - free - (4.0K)
$ gpart list ada1
[...]
scheme: GPT
Providers:
1. Name: ada1p1
Mediasize: 2147483648 (2.0G)
[...]
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada1p2
Mediasize: 1998251360256 (1.8T)
[...]
type: freebsd-zfs
index: 2
end: 3907029119
start: 4194432
Consumers:
1. Name: ada1
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e3
$ sudo gpart add -t freebsd-swap -b 128 -i 1 -s 2G ada0
ada0p1 added
$ gpart show ada0
=> 40 15628053088 ada0 GPT (7.3T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 - free - (7.3T)
$ gpart show ada1
=> 40 3907029088 ada1 GPT (1.8T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834688 2 freebsd-zfs (1.8T)
3907029120 8 - free - (4.0K)
$ sudo gpart add -t freebsd-zfs -i 2 ada0
ada0p2 added
$ gpart show ada0
=> 40 15628053088 ada0 GPT (7.3T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858688 2 freebsd-zfs (7.3T)
15628053120 8 - free - (4.0K)
$ gpart show ada1
=> 40 3907029088 ada1 GPT (1.8T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834688 2 freebsd-zfs (1.8T)
3907029120 8 - free - (4.0K)
Step 3: Add the new disk/partition to the mirror
Now that the disk has a label/UUID, it’s time to stop referring to it as ada0 where possible.
$ glabel status
Name Status Components
gptid/23edea4a-1742-11e8-b551-28924a390410 N/A da0p1
gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 N/A ada1p2
gptid/fa78eb2a-178f-11e8-99e9-28924a390410 N/A ada2p1
gptid/fb1b40fa-178f-11e8-99e9-28924a390410 N/A ada3p1
gptid/d98b522f-178c-11e8-9b2e-28924a390410 N/A ada1p1
gptid/e5ebdfcc-726e-11e8-a5a9-28924a390410 N/A ada0p1
gptid/03d44ba4-726f-11e8-a5a9-28924a390410 N/A ada0p2
$ gpart list ada0 | grep rawuuid
rawuuid: e5ebdfcc-726e-11e8-a5a9-28924a390410
rawuuid: 03d44ba4-726f-11e8-a5a9-28924a390410
$ zpool status pool
pool: pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 0 days 02:49:09 with 0 errors on Sun May 13 02:49:10 2018
config:
NAME STATE READ WRITE CKSUM
pool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 ONLINE 0 0 0
errors: No known data errors
In this case, the label is 03d44ba4-726f-11e8-a5a9-28924a390410 for the second partition, and I want to resilver from the online disk, d9ae78ea-178c-11e8-9b2e-28924a390410:
$ sudo zpool attach pool \
/dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 \
/dev/gptid/03d44ba4-726f-11e8-a5a9-28924a390410
$ zpool status pool
pool: pool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Jun 17 13:54:32 2018
862G scanned at 464M/s, 3.13G issued at 31.4M/s, 1.70T total
3.12G resilvered, 0.18% done, 0 days 15:43:42 to go
config:
NAME STATE READ WRITE CKSUM
pool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 ONLINE 0 0 0
gptid/03d44ba4-726f-11e8-a5a9-28924a390410 ONLINE 0 0 0 (resilvering)
errors: No known data errors
Step 4: Wait
Resilvering takes time.
Step 5: Repeat with the other drive in the mirror
And please make sure you pull the right drive!
$ zpool status pool
pool: pool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Jun 17 18:10:33 2018
865G scanned at 183M/s, 331M issued at 4.67M/s, 1.70T total
323M resilvered, 0.02% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
pool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
6780713986779558112 UNAVAIL 0 0 0 was /dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410
gptid/03d44ba4-726f-11e8-a5a9-28924a390410 ONLINE 0 0 0
gptid/1d2d3246-7294-11e8-85fd-28924a390410 ONLINE 0 0 0 (resilvering)
errors: No known data errors
$ zpool status pool
pool: pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: resilvered 1.70T in 0 days 02:58:23 with 0 errors on Sun Jun 17 21:08:56 2018
config:
NAME STATE READ WRITE CKSUM
pool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
11300205822825511459 UNAVAIL 0 0 0 was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
6780713986779558112 UNAVAIL 0 0 0 was /dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410
gptid/03d44ba4-726f-11e8-a5a9-28924a390410 ONLINE 0 0 0
gptid/1d2d3246-7294-11e8-85fd-28924a390410 ONLINE 0 0 0
errors: No known data errors
Step 6: Detach old disks and watch the mirror grow
$ sudo zpool detach pool 11300205822825511459
$ zpool status pool | grep 'state:'
state: DEGRADED
$ sudo zpool detach pool 6780713986779558112
$ zpool status pool
pool: pool
state: ONLINE
scan: resilvered 1.70T in 0 days 02:58:23 with 0 errors on Sun Jun 17 21:08:56 2018
config:
NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/03d44ba4-726f-11e8-a5a9-28924a390410 ONLINE 0 0 0
gptid/1d2d3246-7294-11e8-85fd-28924a390410 ONLINE 0 0 0
errors: No known data errors
$ zpool list -o name,size
NAME SIZE
pool 7.27T
freenas-boot 7.44G
Job done.