Friday, May 28, 2004

Building Kernel and World on One System, Installing on Another

I'd read Tracking for Multiple Machines in the FreeBSD Handbook, which gives hints on building the FreeBSD userland, or "world," and kernel on one system and installing them on another system. You might do this because the target system is slow and your build machine is fast, or because you prefer to let production machines serve users rather than use CPU cycles rebuilding the world and kernel.

Inspired by this post, I decided to try building the world and kernel on my package builder, "neely," and installing them on target systems. I chose my sensor platform "bourque," as a test system.

First I made sure that neely's world was up to date. I'm tracking the security release of FreeBSD 5.2 (which takes care of 5.2.1). My /usr/local/etc/release-supfile looks like this, with all of the commented lines removed:

*default host=cvsup16.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_5_2
*default delete use-rel-suffix

*default compress

src-all

I ran this script, update_source_release.sh, which takes care of neely's world and kernel:

#!/bin/sh
# Source updated by Richard Bejtlich
# 1215 07 Nov 03

SERVER=`fastest_cvsup -q -c us`

HOSTNAME=`/bin/hostname -s`
echo "Beginning to update $HOSTNAME."

echo "cvsup -g -L 2 -h $SERVER /usr/local/etc/release-supfile"
cvsup -g -L 2 /usr/local/etc/release-supfile

echo "cd /usr/src/"
cd /usr/src/
echo "/usr/src/usr.sbin/mergemaster/mergemaster.sh -p"
/usr/src/usr.sbin/mergemaster/mergemaster.sh -p

echo "cd /usr/obj"
cd /usr/obj
echo "chflags -R noschg *"
chflags -R noschg *
echo "rm -rf *"
rm -rf *
echo "cd /usr/src"
cd /usr/src
echo "make cleandir"
make cleandir
echo "make cleandir"
make cleandir
echo "make buildworld"
make buildworld

echo "cd /usr/src"
cd /usr/src
echo "make buildkernel KERNCONF=$HOSTNAME"
make buildkernel KERNCONF=$HOSTNAME
echo "make installkernel KERNCONF=$HOSTNAME"
make installkernel KERNCONF=$HOSTNAME

echo "make installworld"
make installworld
echo "/usr/src/usr.sbin/mergemaster/mergemaster.sh"
/usr/src/usr.sbin/mergemaster/mergemaster.sh

echo "Done updating $HOSTNAME"

Now that neely's world and kernel were up to date, I decided to rebuild the kernel for other machines.

I created a separate kernel configuration file for bourque in neely's /usr/src/sys/i386/conf directory. I named that file "bourque" and edited it to suit the bourque system's kernel needs. Next I created the following script to build a kernel for bourque:

neely:/usr/local/bin$ cat rebuild_kernel_bourque.sh
#!/bin/sh
# Kernel rebuilder by Richard Bejtlich
# 1550 26 May 04

HOSTNAME=bourque
echo "Beginning to rebuild $HOSTNAME kernel."

echo "cd /usr/src"
cd /usr/src
echo "make buildkernel KERNCONF=$HOSTNAME"
make buildkernel KERNCONF=$HOSTNAME

echo "Done rebuilding $HOSTNAME kernel."

When I ran the script, here is some of what I saw:

neely# rebuild_kernel_bourque.sh

Beginning to rebuild bourque kernel.
cd /usr/src
make buildkernel KERNCONF=bourque

--------------------------------------------------------------
>>> Kernel build for bourque started on Wed May 26 16:42:32 EDT 2004
--------------------------------------------------------------
===> bourque
mkdir -p /usr/obj/usr/src/sys
cd /usr/src/sys/i386/conf; PATH=/usr/obj/usr/src/i386/legacy/usr/sbin:/usr/obj/
usr/src/i386/legacy/usr/bin:/usr/obj/usr/src/i386/legacy/usr/games:/usr/obj/usr/
src/i386/usr/sbin:/usr/obj/usr/src/i386/usr/bin:/usr/obj/usr/src/i386/usr/games:
/sbin:/bin:/usr/sbin:/usr/bin config -d /usr/obj/usr/src/sys/bourque
/usr/src/sys/i386/conf/bourque
Kernel build directory is /usr/obj/usr/src/sys/bourque
Don't forget to do a ``make depend''
cd /usr/obj/usr/src/sys/bourque; MAKEOBJDIRPREFIX=/usr/obj
MACHINE_ARCH=i386 MACHINE=i386 CPUTYPE=
GROFF_BIN_PATH=/usr/obj/usr/src/i386/legacy/usr/bin
...truncated...

I ensured that neely was exporting its /usr directory via NFS with this /etc/exports file:

/usr -alldirs

Now I moved to bourque, where I created and ran this script:

#!/bin/sh
# Source updater via master builder by Richard Bejtlich
# 1614 26 May 04

HOSTNAME=`/bin/hostname -s`
echo "Beginning to update $HOSTNAME."

echo "Mounting /usr/src and /usr/obj on neely"
mount_nfs -T neely:/usr/src /usr/src
mount_nfs -T neely:/usr/obj /usr/obj
mount

echo "cd /usr/src"
cd /usr/src
echo "make installkernel KERNCONF=$HOSTNAME"
make installkernel KERNCONF=$HOSTNAME

echo "make installworld"
make installworld
echo "/usr/src/usr.sbin/mergemaster/mergemaster.sh"
/usr/src/usr.sbin/mergemaster/mergemaster.sh

echo "Done updating $HOSTNAME"

Here is some of what I saw when I ran the script:

bourque# update_source_release.sh ^M
Beginning to update bourque.
Mounting /usr/src and /usr/obj on neely
/dev/ad0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad0s1f on /home (ufs, local, soft-updates)
/dev/ad1s1d on /nsm (ufs, local, soft-updates)
/dev/ad0s1g on /tmp (ufs, local, soft-updates)
/dev/ad0s1d on /usr (ufs, local, soft-updates)
/dev/ad0s1e on /var (ufs, local, soft-updates)
/dev/ad3s1d on /var/db (ufs, local, soft-updates)
neely:/usr/src on /usr/src (nfs)
neely:/usr/obj on /usr/obj (nfs)
cd /usr/src
make installkernel KERNCONF=bourque
cd /usr/obj/usr/src/sys/bourque; MAKEOBJDIRPREFIX=/usr/obj MACHINE_ARCH=i386
MACHINE=i386 CPUTYPE= GROFF_BIN_PATH=/usr/obj/usr/src/i386/legacy/usr/bin GRO
FF_FONT_PATH=/usr/obj/usr/src/i386/legacy/usr/share/groff_font GROFF_TMAC_PATH=
/usr/obj/usr/src/i386/legacy/usr/share/tmac PATH=/usr/obj/usr/src/i386/legacy/us
r/sbin:/usr/obj/usr/src/i386/legacy/usr/bin:/usr/obj/usr/src/i386/legacy/usr/gam
es:/usr/obj/usr/src/i386/usr/sbin:/usr/obj/usr/src/i386/usr/bin:/usr/obj/usr/src
/i386/usr/games:/sbin:/bin:/usr/sbin:/usr/bin make KERNEL=kernel install
...truncated...

When I was done I rebooted bourque, and then checked uname output:

bourque# uname -a
FreeBSD bourque.taosecurity.com 5.2.1-RELEASE-p8 FreeBSD 5.2.1-RELEASE-p8 #1:
Wed May 26 16:42:32 EDT 2004 root@neely.taosecurity.com:
/usr/obj/usr/src/sys/bourque i386

I've started using this method on all of my FreeBSD machines with nonstandard kernels. Systems with GENERIC kernels can still track Colin Percival's freebsd-update.

Update: After speaking with Colin, I have another option that is a hybrid of building your own kernel and using Colin's updates of the userland.

Build the kernel you need for the remote system as shown above with the rebuild_kernel_bourque.sh script. This creates a kernel for the remote machine. On the remote machine, run freebsd-update. If you have a GENERIC kernel on the remote machine, freebsd-update will update the kernel as well as the userland. Now install your kernel using this modified script:

#!/bin/sh
# Source updater via master builder, kernel only by Richard Bejtlich
# 1445 4 Jun 04

HOSTNAME=`/bin/hostname -s`
echo "Beginning to update $HOSTNAME."

echo "Mounting /usr/src and /usr/obj on neely"
mount_nfs -T host_with_kernel:/usr/src /usr/src
mount_nfs -T host_with_kernel:/usr/obj /usr/obj
mount

echo "cd /usr/src"
cd /usr/src
echo "make installkernel KERNCONF=$HOSTNAME"
make installkernel KERNCONF=$HOSTNAME

#No need to installworld when using freebsd-update
#echo "make installworld"
#make installworld
echo "/usr/src/usr.sbin/mergemaster/mergemaster.sh"
/usr/src/usr.sbin/mergemaster/mergemaster.sh

echo "Done updating $HOSTNAME"

Essentially the only change is not running 'make installworld'. The next time you run freebsd-update on the remote machine, freebsd-update will update userland and leave the new kernel alone.

No comments: