FreeNAS, HP Microserver N40L, and some loose drives (part 2)

OK, first disk switch.  Power off the n40l, move the drives across one position, and add the new drive.

(In my case I've dropped the 250gb, and added a 1.5tb)

Disk layout is now

d0	d1	d2	d3
360	-	360	360
360	-	360	360
-	360	360	360
-	360	360	360

...

# 1500 (new)
gpart create -s gpt /dev/ada3 
gpart add -b 128 -t freebsd-swap -s 4194304 ada3                           
gpart add -t freebsd-zfs -s 720000000 -l ant_3 ada3
gpart add -t freebsd-zfs -s 720000000 -l bat_3 ada3                                                                          
gpart add -t freebsd-zfs -s 720000000 -l cat_3 ada3
gpart add -t freebsd-zfs -s 720000000 -l dog_3 ada3
gpart bootcode -b /boot/pmbr-datadisk /dev/ada3

# replace
...
zpool offline tank ada0p2
zpool replace tank ada0p3 gpt/cat_3
zpool replace tank ada0p4 gpt/dog_3
...
zzz

# change partitioning.
swapoff /dev/ada0p1
gpart delete -i 4 ada0
gpart delete -i 3 ada0
gpart delete -i 2 ada0
gpart delete -i 1 ada0
gpart destroy ada0
gpart create -s gpt /dev/ada0
gpart add -b 128 -t freebsd-swap -s 4194304 ada0
gpart add -t freebsd-zfs -s 720000000 -l ant_1 ada0
gpart add -t freebsd-zfs -s 720000000 -l bat_1 ada0
gpart bootcode -b /boot/pmbr-datadisk /dev/ada0

# replace
zpool replace ada0p2 ant_1
zpool replace ada1p2 bat_1

# change partitioning
gpart create -s gpt /dev/ada1
gpart add -b 128 -t freebsd-swap -s 4194304 ada1
gpart add -t freebsd-zfs -s 720000000 -l cat_1 ada1
gpart add -t freebsd-zfs -s 720000000 -l dog_1 ada1
gpart bootcode -b /boot/pmbr-datadisk /dev/ada1

# replace
zpool replace tank ada1p3 gpt/cat_1
zpool replace tank ada1p4 gpt/dog_1
zpool detach tank ada1p3
zpool detach tank ada1p4

...


# change partitioning
gpart create -s gpt /dev/ada2
gpart add -b 128 -t freebsd-swap -s 4194304 ada2
gpart add -t freebsd-zfs -s 720000000 -l ant_2 ada2
gpart add -t freebsd-zfs -s 720000000 -l bat_2 ada2                                                                          
gpart add -t freebsd-zfs -s 720000000 -l cat_2 ada2
gpart add -t freebsd-zfs -s 720000000 -l dog_2 ada2
gpart bootcode -b /boot/pmbr-datadisk /dev/ada2


# replace
zpool replace tank ada2p2 gpt/ant_2  # fails, have to use temp storage
zpool replace tank ada2p3 gpt/bat_2
zpool replace tank ada2p4 gpt/cat_2
zpool replace tank ada2p5 gpt/dog_2
zpool detach tank ada2p2
zpool detach tank ada2p3
zpool detach tank ada2p4
zpool detach tank ada2p5

# done

If reusing the same slice/partition you'll get a device in use failure.  You'll have to either replace to a temporary disk (I did it with a USB drive), or use a sparse file (dd if=/dev/zero of=/path/to/disk.img bs=1m seek=... count=1).

freenas# zpool list
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank  4.01T  1.45T  2.55T    36%  ONLINE  /mnt
freenas# zpool status
  pool: tank
 state: ONLINE
 scrub: scrub in progress for 0h30m, 15.96% done, 2h41m to go
config:

        NAME           STATE     READ WRITE CKSUM
        tank           ONLINE       0     0     0
          raidz1       ONLINE       0     0     0
            gpt/ant_3  ONLINE       0     0     0
            gpt/ant_1  ONLINE       0     0     0
            gpt/ant_2  ONLINE       0     0     0
          raidz1       ONLINE       0     0     0
            gpt/bat_3  ONLINE       0     0     0
            gpt/bat_1  ONLINE       0     0     0
            gpt/bat_2  ONLINE       0     0     0
          raidz1       ONLINE       0     0     0
            gpt/cat_3  ONLINE       0     0     0
            gpt/cat_1  ONLINE       0     0     0
            gpt/cat_2  ONLINE       0     0     0
          raidz1       ONLINE       0     0     0
            gpt/dog_3  ONLINE       0     0     0
            gpt/dog_1  ONLINE       0     0     0
            gpt/dog_2  ONLINE       0     0     0

errors: No known data errors
freenas# gpart show  -l | sed '/ada0/,$!d'
=>        34  1465149101  ada0  GPT  (699G)
          34          94        - free -  (47K)
         128     4194304     1  (null)  (2.0G)
     4194432   720000000     2  ant_1  (343G)
   724194432   720000000     3  bat_1  (343G)
  1444194432    20954703        - free -  (10G)

=>        34  1465149101  ada1  GPT  (699G)
          34          94        - free -  (47K)
         128     4194304     1  (null)  (2.0G)
     4194432   720000000     2  cat_1  (343G)
   724194432   720000000     3  dog_1  (343G)
  1444194432    20954703        - free -  (10G)

=>        34  2930277101  ada2  GPT  (1.4T)
          34          94        - free -  (47K)
         128     4194304     1  (null)  (2.0G)
     4194432   720000000     2  ant_2  (343G)
   724194432   720000000     3  bat_2  (343G)
  1444194432   720000000     4  cat_2  (343G)
  2164194432   720000000     5  dog_2  (343G)
  2884194432    46082703        - free -  (22G)

=>        34  2930277101  ada3  GPT  (1.4T)
          34          94        - free -  (47K)
         128     4194304     1  (null)  (2.0G)
     4194432   720000000     2  ant_3  (343G)
   724194432   720000000     3  bat_3  (343G)
  1444194432   720000000     4  cat_3  (343G)
  2164194432   720000000     5  dog_3  (343G)
  2884194432    46082703        - free -  (22G)

freenas#

Hope it's of use to someone!