Thursday, March 22, 2007

Gconcat on FreeBSD

The last time I wanted to combine two smaller drives into a single virtual drive on FreeBSD I used Gvinum. Ceri Davies posted a helpful comment indicating I should try using gconcat(8). I did that today and thanks to an insightful piece of advice from Robert Watson, I got it working.

This is what the drive looked like.

shuttle01# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ad5s1a 496M 36M 420M 8% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad5s1f 989M 22K 910M 0% /home
/dev/ad5s1h 54G 4.0K 50G 0% /nsm1
/dev/ad7s1d 361G 4.0K 332G 0% /nsm2
/dev/ad5s1g 989M 12K 910M 0% /tmp
/dev/ad5s1d 1.9G 531M 1.3G 29% /usr
/dev/ad5s1e 4.8G 1.6M 4.5G 0% /var

I want to create /dev/concat/nsm. However, if I try to do that while /nsm1 and /nsm2 are mounted I'll get errors like this:

shuttle01# gconcat label -v nsm ad5s1h ad7s1d
Can't store metadata on ad5s1h: Operation not permitted.

So, unmount /nsm1 and /nsm2 before continuing. Next:

shuttle01# umount /nsm1
shuttle01# umount /nsm2
shuttle01# gconcat label -v nsm ad5s1h ad7s1d
Metadata value stored on ad5s1h.
Metadata value stored on ad7s1d.
Done.
shuttle01# ls /dev/concat/nsm
/dev/concat/nsm
shuttle01# newfs /dev/concat/nsm
/dev/concat/nsm: 438631.6MB (898317520 sectors) block size 16384, fragment size 2048
using 2387 cylinder groups of 183.77MB, 11761 blks, 23552 inodes.
super-block backups (for fsck -b #) at:
160, 376512, 752864, 1129216, 1505568, 1881920, 2258272, 2634624, 3010976,
...truncated...

Now I can create /nsm and mount it.

shuttle01# mkdir /nsm
shuttle01# mount /dev/concat/nsm /nsm
shuttle01# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ad5s1a 496M 36M 420M 8% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad5s1f 989M 22K 910M 0% /home
/dev/ad5s1g 989M 12K 910M 0% /tmp
/dev/ad5s1d 1.9G 531M 1.3G 29% /usr
/dev/ad5s1e 4.8G 1.6M 4.5G 0% /var
/dev/concat/nsm 415G 4.0K 382G 0% /nsm
shuttle01# touch /nsm/test

Cool. If I don't want to use it again:

shuttle01# umount /nsm
shuttle01# gconcat stop nsm
shuttle01# gconcat unload
shuttle01# ls /dev/concat

To use it again:
shuttle01# gconcat load nsm
shuttle01# ls /dev/concat/nsm
/dev/concat/nsm
shuttle01# mount /dev/concat/nsm /nsm
shuttle01# ls /nsm
.snap test

Remember to enable gconcat in /boot/loader.conf:

shuttle01# cat /boot/loader.conf
geom_concat_load="YES"

Remember to edit /etc/fstab. The original looked like this:

# Device Mountpoint FStype Options Dump Pass#
/dev/ad5s1b none swap sw 0 0
/dev/ad5s1a / ufs rw 1 1
/dev/ad5s1f /home ufs rw 2 2
/dev/ad5s1h /nsm1 ufs rw 2 2
/dev/ad7s1d /nsm2 ufs rw 2 2
/dev/ad5s1g /tmp ufs rw 2 2
/dev/ad5s1d /usr ufs rw 2 2
/dev/ad5s1e /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0

The new one looks like this:

# Device Mountpoint FStype Options Dump Pass#
/dev/ad5s1b none swap sw 0 0
/dev/ad5s1a / ufs rw 1 1
/dev/ad5s1f /home ufs rw 2 2
#/dev/ad5s1h /nsm1 ufs rw 2 2
#/dev/ad7s1d /nsm2 ufs rw 2 2
/dev/ad5s1g /tmp ufs rw 2 2
/dev/ad5s1d /usr ufs rw 2 2
/dev/ad5s1e /var ufs rw 2 2
/dev/concat/nsm /nsm ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0

I'm deploying this system in a RAID 0 configuration because it's a Shuttle with two HDDs. I'm not worrying about recovering from a disaster with this box. It's going to help with the ShmooCon network. If I had three or more drives I'd consider using gstripe(8).

3 comments:

Anonymous said...

man richard, i am learning everyday from you. I guess you must be an OS expert.

http://hackathology.blogspot.com

Anonymous said...

Everytime I reboot the /dev/concat/storage has disappeared and gives errors upon boot. Any ideas?

Richard Bejtlich said...

Note to self: geom_concat doesn't seem to like adding /nsm1 and /nsm2 if /nsm2 occupies the entire second disk. Solution: create a small /tmp2 on the second disk and then /nsm2 on the second disk, eg:

# mount
/dev/ad2s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad2s1e on /home (ufs, local, soft-updates)
/dev/ad2s1h on /nsm1 (ufs, local, soft-updates)
/dev/ad3s1e on /nsm2 (ufs, local, soft-updates)
/dev/ad2s1f on /tmp (ufs, local, soft-updates)
/dev/ad3s1d on /tmp2 (ufs, local, soft-updates)
/dev/ad2s1d on /usr (ufs, local, soft-updates)
/dev/ad2s1g on /var (ufs, local, soft-updates)
# umount /nsm1
# umount /nsm2
# gconcat label -v nsm ad2s1h ad3s1e
Metadata value stored on ad2s1h.
Metadata value stored on ad3s1e
# newfs /dev/concat/nsm
# mkdir /nsm
# mount /dev/concat/nsm /nsm
# cat /boot/loader.conf
geom_concat_load="YES"
# cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/ad2s1b none swap sw 0 0
/dev/ad2s1a / ufs rw 1 1
/dev/ad2s1e /home ufs rw 2 2
#/dev/ad2s1h /nsm1 ufs rw 2 2
#/dev/ad3s1e /nsm2 ufs rw 2 2
/dev/concat/nsm /nsm ufs rw 2 2
/dev/ad2s1f /tmp ufs rw 2 2
/dev/ad3s1d /tmp2 ufs rw 2 2
/dev/ad2s1d /usr ufs rw 2 2
/dev/ad2s1g /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0