Growing ZFS mirrors

30 Jun '18

I have long contended that for small, home NAS systems, mirrors are the way to go. You don’t have to calculate how much parity you need, similarly, the backup size is also obvious. It costs less upfront, you only need two disks. And because you only need two disks, you can get the biggest disks you can afford, which usually have excellent GB/cent cost ratios.

Finally, you can grow mirrors! As long as your storage needs don’t outstrip storage space growth, you can grow your mirrors very economically. Case in point, I was running a 2TB mirror which was getting close to 80% full. I was able to buy 8TB disks very cheaply, and grow my mirror to 8TB.

First things first, don’t be an idiot, this guide is how to modify storage. Triple check everything, and make sure you have backups. This guide also assumes the disks are blank, i.e. unformatted.

Step 1: Install new drive

I had no spare bays/SATA connections, but if you did, you could potentially do this in a more safer way by adding the new disks to a mirror to create a 3 or 4-way mirror, and then pulling the old disks out.

I had to do it more riskily by degrading the mirror. First, I powered down the NAS. Then I replaced one of the old disks in the mirror with a new one and powered up again. First, let’s check the right disk was pulled:

$ zpool status pool
  pool: pool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 0 days 02:49:09 with 0 errors on Sun May 13 02:49:10 2018
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool                                            DEGRADED     0     0     0
      mirror-0                                      DEGRADED     0     0     0
        11300205822825511459                        UNAVAIL      0     0     0  was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
        gptid/d9ae78ea-178c-11e8-9b2e-28924a390410  ONLINE       0     0     0

errors: No known data errors

As expected, the mirror is degraded. So, now to figure out which drive is the new drive:

$ ls -1 /dev/ada*
/dev/ada0
/dev/ada1
/dev/ada1p1
/dev/ada1p2
/dev/ada2
/dev/ada2p1
/dev/ada3
/dev/ada3p1

What’s going on? Well, drives are mounted in /dev and receive a name, e.g. ada1. The partitions are also mounted as /dev/ada1p1 and /dev/ada1p2. So it’s already pretty obvious that the new drive is ada0.

Just to check, we’ll look at the labels and partitions. BSD labels drives with UUIDs, since the name assigned in /dev is basically random, and drives could get reassigned on start-up.

$ glabel status
                                      Name  Status  Components
gptid/23edea4a-1742-11e8-b551-28924a390410     N/A  da0p1
gptid/d9ae78ea-178c-11e8-9b2e-28924a390410     N/A  ada1p2
gptid/d98b522f-178c-11e8-9b2e-28924a390410     N/A  ada1p1

$ gpart show
=>      40  15663024  da0  GPT  (7.5G)
        40      1024    1  bios-boot  (512K)
      1064  15661992    2  freebsd-zfs  (7.5G)
  15663056         8       - free -  (4.0K)

=>        40  3907029088  ada1  GPT  (1.8T)
          40          88        - free -  (44K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834688     2  freebsd-zfs  (1.8T)
  3907029120           8        - free -  (4.0K)

$ glabel list | grep ada0
$

Ignore da0, it is the OS USB disk. The blank disk is ada0, it has no labels or partition map.

Step 2: Create GPT partition map

Create the partition map on the new disk:

$ sudo gpart create -s gpt ada0
ada0 created

Now re-create partitions similar to the other disk in the mirror. My distro FreeNAS allocates a bit of swap space on all drives, so we want to replicate that. Get the parameters for gpart add from gpart list, -t is the type, -b is the the starting offset, -i is the index, and -s is the size. I don’t know why FreeNAS sets a starting offset for the swap partition, but whatever.

$ gpart show ada0
=>         40  15628053088  ada0  GPT  (7.3T)
           40  15628053088        - free -  (7.3T)

$ gpart show ada1
=>        40  3907029088  ada1  GPT  (1.8T)
          40          88        - free -  (44K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834688     2  freebsd-zfs  (1.8T)
  3907029120           8        - free -  (4.0K)

$ gpart list ada1
[...]
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   [...]
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 1998251360256 (1.8T)
   [...]
   type: freebsd-zfs
   index: 2
   end: 3907029119
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3

$ sudo gpart add -t freebsd-swap -b 128 -i 1 -s 2G ada0
ada0p1 added
$ gpart show ada0
=>         40  15628053088  ada0  GPT  (7.3T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  15623858696        - free -  (7.3T)

$ gpart show ada1
=>        40  3907029088  ada1  GPT  (1.8T)
          40          88        - free -  (44K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834688     2  freebsd-zfs  (1.8T)
  3907029120           8        - free -  (4.0K)

$ sudo gpart add -t freebsd-zfs -i 2 ada0
ada0p2 added
$ gpart show ada0
=>         40  15628053088  ada0  GPT  (7.3T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  15623858688     2  freebsd-zfs  (7.3T)
  15628053120            8        - free -  (4.0K)

$ gpart show ada1
=>        40  3907029088  ada1  GPT  (1.8T)
          40          88        - free -  (44K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834688     2  freebsd-zfs  (1.8T)
  3907029120           8        - free -  (4.0K)

Step 3: Add the new disk/partition to the mirror

Now that the disk has a label/UUID, it’s time to stop referring to it as ada0 where possible.

$ glabel status
                                      Name  Status  Components
gptid/23edea4a-1742-11e8-b551-28924a390410     N/A  da0p1
gptid/d9ae78ea-178c-11e8-9b2e-28924a390410     N/A  ada1p2
gptid/fa78eb2a-178f-11e8-99e9-28924a390410     N/A  ada2p1
gptid/fb1b40fa-178f-11e8-99e9-28924a390410     N/A  ada3p1
gptid/d98b522f-178c-11e8-9b2e-28924a390410     N/A  ada1p1
gptid/e5ebdfcc-726e-11e8-a5a9-28924a390410     N/A  ada0p1
gptid/03d44ba4-726f-11e8-a5a9-28924a390410     N/A  ada0p2

$ gpart list ada0 | grep rawuuid
   rawuuid: e5ebdfcc-726e-11e8-a5a9-28924a390410
   rawuuid: 03d44ba4-726f-11e8-a5a9-28924a390410

$ zpool status pool
  pool: pool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 0 days 02:49:09 with 0 errors on Sun May 13 02:49:10 2018
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool                                            DEGRADED     0     0     0
      mirror-0                                      DEGRADED     0     0     0
        11300205822825511459                        UNAVAIL      0     0     0  was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
        gptid/d9ae78ea-178c-11e8-9b2e-28924a390410  ONLINE       0     0     0

errors: No known data errors

In this case, the label is 03d44ba4-726f-11e8-a5a9-28924a390410 for the second partition, and I want to resilver from the online disk, d9ae78ea-178c-11e8-9b2e-28924a390410:

$ sudo zpool attach pool \
  /dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410 \
  /dev/gptid/03d44ba4-726f-11e8-a5a9-28924a390410
$ zpool status pool
  pool: pool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Jun 17 13:54:32 2018
    862G scanned at 464M/s, 3.13G issued at 31.4M/s, 1.70T total
    3.12G resilvered, 0.18% done, 0 days 15:43:42 to go
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool                                            DEGRADED     0     0     0
      mirror-0                                      DEGRADED     0     0     0
        11300205822825511459                        UNAVAIL      0     0     0  was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
        gptid/d9ae78ea-178c-11e8-9b2e-28924a390410  ONLINE       0     0     0
        gptid/03d44ba4-726f-11e8-a5a9-28924a390410  ONLINE       0     0     0  (resilvering)

errors: No known data errors

Step 4: Wait

Resilvering takes time.

Step 5: Repeat with the other drive in the mirror

And please make sure you pull the right drive!

$ zpool status pool
  pool: pool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Jun 17 18:10:33 2018
    865G scanned at 183M/s, 331M issued at 4.67M/s, 1.70T total
    323M resilvered, 0.02% done, no estimated completion time
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool                                            DEGRADED     0     0     0
      mirror-0                                      DEGRADED     0     0     0
        11300205822825511459                        UNAVAIL      0     0     0  was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
        6780713986779558112                         UNAVAIL      0     0     0  was /dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410
        gptid/03d44ba4-726f-11e8-a5a9-28924a390410  ONLINE       0     0     0
        gptid/1d2d3246-7294-11e8-85fd-28924a390410  ONLINE       0     0     0  (resilvering)

errors: No known data errors
$ zpool status pool
  pool: pool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 1.70T in 0 days 02:58:23 with 0 errors on Sun Jun 17 21:08:56 2018
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool                                            DEGRADED     0     0     0
      mirror-0                                      DEGRADED     0     0     0
        11300205822825511459                        UNAVAIL      0     0     0  was /dev/gptid/d8bea02b-178c-11e8-9b2e-28924a390410
        6780713986779558112                         UNAVAIL      0     0     0  was /dev/gptid/d9ae78ea-178c-11e8-9b2e-28924a390410
        gptid/03d44ba4-726f-11e8-a5a9-28924a390410  ONLINE       0     0     0
        gptid/1d2d3246-7294-11e8-85fd-28924a390410  ONLINE       0     0     0

errors: No known data errors

Step 6: Detach old disks and watch the mirror grow

$ sudo zpool detach pool 11300205822825511459
$ zpool status pool | grep 'state:'
 state: DEGRADED
$ sudo zpool detach pool 6780713986779558112
$ zpool status pool
  pool: pool
 state: ONLINE
  scan: resilvered 1.70T in 0 days 02:58:23 with 0 errors on Sun Jun 17 21:08:56 2018
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool                                            ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/03d44ba4-726f-11e8-a5a9-28924a390410  ONLINE       0     0     0
        gptid/1d2d3246-7294-11e8-85fd-28924a390410  ONLINE       0     0     0

errors: No known data errors
$ zpool list -o name,size
NAME           SIZE
pool          7.27T
freenas-boot  7.44G

Job done.

ZFS, FreeNAS, NAS, storage, BSD

Newer Older