Wednesday, December 28, 2005

Rough Sguil Installation Script

My last Sguil Installation Guide, for Sguil 0.5.3 was a mix of English description and command line statements. This did not help much when I needed to install a new Sguil deployment. I essentially followed my guide and typed everything by hand.

Today I decided that would be the end of that process. I am excited by the new InstantNSM project, and I intend to support it with respect to FreeBSD. But for today, I decided to just script as many Sguil installation commands as possible. For items that I couldn't easily script (due to my weak script-fu), I decided to edit the files manually and generate a patch for each one.

This post describes the end result, which you can download at www.bejtlich.net/sguil_install_v0.1.sh. I should warn you that this is not meant for public production use. However, someone trying to install Sguil might find it useful.

The purpose of this script is to automate, as much as possible, the creation of a Sguil sensor, server, and database on a FreeBSD 6.0/i386 platform. The platform is a VMware image whose hostname is gruden.taosecurity.com and whose management IP address is 192.168.2.121. I have stored several files at www.bejtlich.net to facilitate the installation. I will explain where that matters as I progress.


#!/bin/sh
#
# Sguil installation script by Richard Bejtlich (richard@taosecurity.com)
# v1-0 28 December 2005
#
# Tested on FreeBSD 6.0 RELEASE
#
# This script sets up all Sguil components on a single FreeBSD 6.0 system
# This is not intended for production use where separate sensor, server,
# and client boxes are recommended

echo "Sguil Installation Script"
echo
echo "By Richard Bejtlich"
echo
echo "This is mainly for personal use, but it documents how to build"
echo "a FreeBSD 6.0 system with Sguil sensor, server, and database"
echo "components. The Sguil client must be deployed separately."

First I update the time. I am running this in a VM and time can be problematic. With FreeBSD 6 as a guest OS on VMware Workstation, I create /boot/loader.conf with 'hint.apic.0.disabled=1' to mitigate time issues.

# Update date and time

ntpdate clock.isc.org

Next I set some environment variables. I designate my proxy server, which received heavy use as I tested this script. Note that using a proxy server means copies of patches and other files are cached. To clear the cache after changing a file and uploading it www.bejtlich.net, the process involves stopping Sguid, clearing the cache map with 'echo "" >> /usr/local/squid/cache/swap.state', and restarting Sguid.

# Set environment variable for package adds

# Use proxy server if you have it!

HTTP_PROXY=192.168.2.7:3128; export HTTP_PROXY

By default this script used FreeBSD 6 packages.

# Use the following for FreeBSD 5 packages

#PACKAGESITE=ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest/; export PACKAGESITE

# FreeBSD 6 packages

PACKAGESITE=ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-6-stable/Latest/; export PACKAGESITE

Here is where the sensor name is determined. In other places (like patch files) I use the sensor name, gruden, explicity.

# Determine sensor name

SENSOR=`hostname -s`

# Set Sguil version

SGUIL=sguil-0.6.0p1

# Set Snort major version

SNORTMV=2.4

Now I create directories used by Sguil components.

# Create directories

mkdir -p /nsm/$SENSOR/dailylogs
mkdir -p /nsm/$SENSOR/portscans
mkdir -p /nsm/$SENSOR/sancp
mkdir -p /nsm/rules/$SENSOR
mkdir -p /var/log/snort
mkdir -p /usr/local/etc/nsm
mkdir -p /usr/local/src
mkdir -p /nsm/archive
mkdir -p /nsm/rules/$SENSOR


chown -R sguil:sguil /nsm

chown -R sguil:sguil /var/log/snort

chown -R sguil:sguil /usr/local/etc/nsm

Now I start getting software packages and archives.

# Retrieve software

cd /usr/local/src
fetch http://internap.dl.sourceforge.net/sourceforge/sguil/$SGUIL.tar.gz
tar -xzf $SGUIL.tar.gz

# Install Snort

pkg_add -r snort

cd /nsm/rules/$SENSOR
fetch http://www.snort.org/pub-bin/downloads.cgi/Download/vrt_pr/snortrules-pr-$SNORTMV.tar.gz
tar -xzf snortrules-pr-$SNORTMV.tar.gz
mv /nsm/rules/$SENSOR/rules/* /nsm/rules/$SENSOR

chown -R sguil:sguil /usr/local/etc/snort

cd /root

# Install Tcl

pkg_add -r tcl84
mv /usr/local/bin/tclsh /usr/local/bin/tclsh.orig
ln -s /usr/local/bin/tclsh8.4 /usr/local/bin/tclsh

The installation of Barnyard uses a package I built, as described here, because the stock Barnyard package does not support Sguil 0.6.0p1.

# Install Barnyard

cd /tmp
fetch http://www.bejtlich.net/barnyard-0.2.0.tbz
pkg_add barnyard-0.2.0.tbz

# Install SANCP

pkg_add -r sancp

# Install MySQL

pkg_add -r mysql50-server
/usr/local/bin/mysql_install_db --user=mysql
/usr/local/bin/mysqld_safe --user=mysql &

# Install Tcltls

pkg_add -r tcltls

# Install Tcllib

pkg_add -r tcllib

# Install TclX

pkg_add -r tclX

I have to install my own version of MySQLTcl. This was not as complicated as Barnyard. The problem with the stock package is that it is compiled against MySQL 4.1.x, and I am using MySQL 5.0.x. Simply building my own package on sguilref, a FreeBSD 6 host with MySQL 5.0.16 installed, is enough to create the proper mysqltcl package.

# Install MySQLTcl from own version compiled for MySQL 5.x

fetch http://www.bejtlich.net/mysqltcl-3.01.tbz
pkg_add mysqltcl-3.01.tbz

# Install P0f

pkg_add -r p0f

# Install Tcpflow

pkg_add -r tcpflow

Now I copy some configuration files and set up the Sguil database.

# Copy configuration files

cp /usr/local/src/$SGUIL/sensor/sensor_agent.conf /usr/local/etc/nsm
cp /usr/local/src/$SGUIL/server/sguild.conf /usr/local/etc/nsm
cp /usr/local/etc/snort/snort.conf-sample /usr/local/etc/nsm/snort.conf
cp /usr/local/etc/barnyard.conf-sample /usr/local/etc/nsm/barnyard.conf
cp /usr/local/src/$SGUIL/sensor/sancp/sancp.conf /usr/local/etc/nsm
cp /usr/local/src/$SGUIL/sensor/log_packets.sh /usr/local/etc/nsm

# Set up database

/usr/local/bin/mysql -e "CREATE DATABASE sguildb"
/usr/local/bin/mysql -D sguildb < /usr/local/src/$SGUIL/server/sql_scripts/create_sguildb.sql
/usr/local/bin/mysql -e "GRANT ALL on sguildb.* to sguil@localhost"
/usr/local/bin/mysql -e "GRANT FILE on *.* to sguil@localhost"
/usr/local/bin/mysql -e "SET password for sguil@localhost=password('sguil')"
/usr/local/bin/mysql -e "SHOW TABLES" sguildb
/usr/local/bin/mysql -e "SET password for root@localhost=password('r00t')"
/usr/local/bin/mysql --password=r00t -e "FLUSH PRIVILEGES"

I couldn't think of an easy way to apply changes to the configuration files, so I edited them by hand to suit my needs and generated patches.

Here is my patch generation procedure for sensor_agent.conf.patch as an example.

First, make a copy that will contain the changes.

cp sensor_agent.conf sensor_agent.conf.diff

Now edit sensor_agent.conf.diff to include the desired changes. I use vi. Next create the patch.

diff -u sensor_agent.conf sensor_agent.conf.diff > sensor_agent.conf.patch

The sensor_agent.conf.patch looks like this:

--- sensor_agent.conf Wed Dec 28 14:57:30 2005
+++ sensor_agent.conf.diff Wed Dec 28 14:58:33 2005
@@ -13,7 +13,7 @@
set DAEMON 0

# Name of sguild server
-set SERVER_HOST 192.168.8.8
+set SERVER_HOST localhost
# Port sguild listens on for sensor connects
set SERVER_PORT 7736
# Port sensor_agent lisens on for barnyard connects
@@ -22,10 +22,10 @@
# Note: Sensors monitoring multiple interfaces need to use a unique 'hostname'
# for each interface. Make sure this name is the same in the respective
# log_packets.sh
-set HOSTNAME gateway
+set HOSTNAME gruden

# The root of your log dir for data like pcap, portscans, sessions, etc
-set LOG_DIR /snort_data
+set LOG_DIR /nsm

# Where to look for files created by modded spp_portscan
set PORTSCAN_DIR ${LOG_DIR}/${HOSTNAME}/portscans
@@ -49,7 +49,7 @@
# 2: sancp (http://www.metre.net/sancp.html)

#Enable Stream4 keep_stats (1=enable 0=disable)
-set S4_KEEP_STATS 1
+set S4_KEEP_STATS 0
# Where to look for ssn files created by modded spp_stream4
set SSN_DIR ${LOG_DIR}/${HOSTNAME}/ssn_logs

I do not think this is a bad way to handle the issue, although I welcome simpler suggestions. If you wanted to use my script, for example, you could copy the patches, edit them, and then have the script apply them as shown below. Note this is a place where sensor name and IP address can matter. Note in the above patch the sensor name, gruden, is explicitly mentioned.

# Fetch text file patches

cd /usr/local/etc/nsm

fetch http://www.bejtlich.net/sensor_agent.conf.patch
fetch http://www.bejtlich.net/sguild.conf.patch
fetch http://www.bejtlich.net/snort.conf.patch
fetch http://www.bejtlich.net/barnyard.conf.patch
fetch http://www.bejtlich.net/sancp.conf.patch
fetch http://www.bejtlich.net/log_packets.sh.patch
fetch http://www.bejtlich.net/log_packets.sh.crontab

# Apply patches

patch -p0 < sensor_agent.conf.patch
patch -p0 < sguild.conf.patch
patch -p0 < snort.conf.patch
patch -p0 < barnyard.conf.patch
patch -p0 < sancp.conf.patch
patch -p0 < log_packets.sh.patch
crontab -u root log_packets.sh.crontab

Next I put log_packets.sh where it belongs, move some Snort configuration files, and retrieve some simple startup scripts.

# Install log_packets.sh

cp /usr/local/etc/nsm/log_packets.sh /usr/local/bin

# Copy Snort conf files

cp /nsm/rules/$SENSOR/classification.config /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/gen-msg.map /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/reference.config /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/sid-msg.map /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/threshold.conf /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/unicode.map /usr/local/etc/nsm

# Get startup scripts

cd /home/sguil
fetch http://www.bejtlich.net/barnyard_start.sh
fetch http://www.bejtlich.net/sguild_start.sh
fetch http://www.bejtlich.net/sensor_agent_start.sh

chown sguil:sguil /home/sguil/*.sh
chmod +x /home/sguil/*.sh

cd /root
fetch http://www.bejtlich.net/snort_start.sh
fetch http://www.bejtlich.net/sancp_start.sh

chmod +x /root/*.sh

Now I modify /etc/rc.conf so MySQL will start at boot, but only listen on localhost. The sniffing interface on this system is lnc1, so I bring it up without the capability to arp.

# Modify /etc/rc.conf

echo "mysql_enable=YES" >> /etc/rc.conf
echo "mysql_args=--bind-address=127.0.0.1" >> /etc/rc.conf
echo "ifconfig_lnc1=-arp" >> /etc/rc.conf

Several of the Sguil components, like barnyard, sensor_agent.conf, and SANCP run as user sguil and need to write their PID files to /var/run. I decided to make /var/run mode 777 to let them write to the directory. This is not the best idea, so I might change it.

# Set up /var/run

chmod 777 /var/run

Finally I add the user 'sguil' with password 'sguil' so clients can access the Sguil server.

# Add Sguil client user

echo "Create a Sguil client user password when prompted."
cd /usr/local/src/$SGUIL/server
./sguild -c sguild.conf -u sguild.users -adduser sguil

In this last section I tell how to get all of the components running. By default all of them will run in the background. Each *start.sh script has an option for running in the foreground for debugging purposes, if you uncomment the foreground option and comment out the background option.

# Messages to users

echo "To start Sguil, execute the following."
echo
echo "As user sguil:"
echo
echo "/home/sguil/sguild_start.sh"
echo "/home/sguil/sensor_agent_start.sh"
echo "/home/sguil/barnyard_start.sh"
echo
echo "Next, as user root:"
echo
echo "/root/sancp_start.sh"
echo "/root/snort_start.sh"
echo "/usr/local/bin/log_packets.sh restart"
echo
echo "You will then be able to connect using the separate Sguil client."

Once you have this script installed on a suitable FreeBSD 6/i386 system, you can run it. Here is the partition layout I created, using only 1024 MB. I installed the "minimal" distribution, which is the smallest non-custom distro.

$ uname -a
FreeBSD gruden.taosecurity.com 6.0-RELEASE FreeBSD
6.0-RELEASE #0: Thu Nov 3 09:36:13 UTC 2005
root@x64.samsco.home:/usr/obj/usr/src/sys/GENERIC i386
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ad0s1a 124M 55M 59M 48% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad0s1g 62M 40K 57M 0% /home
/dev/ad0s1f 124M 4.0K 114M 0% /nsm
/dev/ad0s1h 62M 12K 57M 0% /tmp
/dev/ad0s1d 248M 100M 128M 44% /usr
/dev/ad0s1e 124M 206K 114M 0% /var

I added two users.

  • User analyst is a member of the wheel group and can therefore su - to root.

  • User sguil is not a member of the wheel group. However, I run as many parts of Sguil as possible using this user.


Here is how to invoke the script:

$ su -
Password:
gruden# fetch http://www.bejtlich.net/sguil_install_v0.1.sh
sguil_install_v1-0.sh 100% of 6023 B 83 kBps
gruden# chmod +x sguil_install_v0.1.sh
gruden# ./sguil_install_v0.1.sh

When the sguil client user password prompt appears, enter something like 'sguil'. This is the only pause in the script.

The end result of running this script inside a FreeBSD VM I created is a Sguil sensor, server, and database. I'll describe that in my next post.

7 comments:

Beau said...

One thing you forgot to mention (or if you did, I missed it): You have to manually add the sguil user account before running the script.

Richard Bejtlich said...

Beau,

Good point. I just added some info to this post.

Thank you!

Beau said...

I finally got this up and running right. THANKS!

It would be really great if the script would prompt for the sensor name and the interface and put that into the .patch files. Finding and changing those variables are the most difficult part of setting up a sguil sensor now!

Thanks for helping out the novices.

JD said...

RYN: Will try this script on my home machine on Friday the 20th. Thanks for pointing it out; this looks like it will make things easier for us newbies!

Anonymous said...

I'm getting an error when I try to run sguild. (I have not used the scripts because I am trying to get it running on my debian box) I searched and found that someone else got the same error http://pastebin.ca/126774 but have yet to find a solution. Any ideas?

-Grateful

Richard Bejtlich said...

Grateful,

Ask in #snort-gui on irc.freenode.net or via email to sguil-users [at] lists.sourceforge.net.

Richard Bejtlich said...

s/Sguid/Squid