Notes on Installing Sguil Using FreeBSD 7.1 Packages

It's been a while since I've looked at the Sguil ports for FreeBSD, so I decided to see how they work.

In this post I will talk about installing a Sguil sensor and server on a single FreeBSD 7.1 test VM using packages shipped with FreeBSD 7.1.

To start with the system had no packages installed.

After running pkg_add -vr sguil-sensor, I watched what was added to the system. I'm only going to document that which I found interesting.

The sguil-sensor-0.7.0_2 package installed the following into /usr/local.

x bin/sguil-sensor/log_packets.sh
x bin/sguil-sensor/example_agent.tcl
x bin/sguil-sensor/pcap_agent.tcl
x bin/sguil-sensor/snort_agent.tcl
x etc/sguil-sensor/example_agent.conf-sample
x etc/sguil-sensor/pcap_agent.conf-sample
x etc/sguil-sensor/snort_agent.conf-sample
x etc/sguil-sensor/log_packets.conf-sample
x share/doc/sguil-sensor <- multiple files, omitted here
x etc/rc.d/example_agent
x etc/rc.d/pcap_agent
x etc/rc.d/snort_agent

Note that you have to copy

pcap_agent.conf-sample
log_packets.conf-sample
snort_agent.conf-sample

to

pcap_agent.conf
log_packets.conf
snort_agent.conf

and edit each, prior to starting

pcap_agent.tcl
log_packets.sh
snort_agent.tcl

via

rc.d/pcap_agent
cron
rc.d/snort_agent

Also, as noted in the configuration options, PADS and SANCP are not installed by default, so the package doesn't include them:

===> The following configuration options are available for sguil-sensor-0.7.0_2:
     SANCP=off (default) "Include sancp sensor"
     PADS=off (default) "Include pads sensor"
===> Use 'make config' to modify these settings

The snort-2.8.2.1_1 package installed the following.

x man/man8/snort.8.gz
x bin/snort
x etc/snort/classification.config-sample
x etc/snort/gen-msg.map-sample
x etc/snort/reference.config-sample
x etc/snort/sid-msg.map-sample
x etc/snort/snort.conf-sample
x etc/snort/threshold.conf-sample
x etc/snort/unicode.map-sample
x src/snort_dynamicsrc/bitop.h
x src/snort_dynamicsrc/debug.h
x src/snort_dynamicsrc/pcap_pkthdr32.h
x src/snort_dynamicsrc/preprocids.h
x src/snort_dynamicsrc/profiler.h
x src/snort_dynamicsrc/sf_dynamic_common.h
x src/snort_dynamicsrc/sf_dynamic_meta.h
x src/snort_dynamicsrc/sf_dynamic_preproc_lib.c
x src/snort_dynamicsrc/sf_dynamic_preproc_lib.h
x src/snort_dynamicsrc/sf_dynamic_preprocessor.h
x src/snort_dynamicsrc/sf_snort_packet.h
x src/snort_dynamicsrc/sf_snort_plugin_api.h
x src/snort_dynamicsrc/sfghash.h
x src/snort_dynamicsrc/sfhashfcn.h
x src/snort_dynamicsrc/sfsnort_dynamic_detection_lib.c
x src/snort_dynamicsrc/sfsnort_dynamic_detection_lib.h
x src/snort_dynamicsrc/str_search.h
x src/snort_dynamicsrc/stream_api.h
x lib/snort/dynamicengine/libsf_engine.so
x lib/snort/dynamicengine/libsf_engine.so.0
x lib/snort/dynamicengine/libsf_engine.la
x lib/snort/dynamicengine/libsf_engine.a
x lib/snort/dynamicrules/lib_sfdynamic_example_rule.so
x lib/snort/dynamicrules/lib_sfdynamic_example_rule.so.0
x lib/snort/dynamicrules/lib_sfdynamic_example_rule.la
x lib/snort/dynamicrules/lib_sfdynamic_example_rule.a
x lib/snort/dynamicpreprocessor/lib_sfdynamic_preprocessor_example.a
x lib/snort/dynamicpreprocessor/lib_sfdynamic_preprocessor_example.la
x lib/snort/dynamicpreprocessor/lib_sfdynamic_preprocessor_example.so
x lib/snort/dynamicpreprocessor/lib_sfdynamic_preprocessor_example.so.0
x lib/snort/dynamicpreprocessor/libsf_dcerpc_preproc.a
x lib/snort/dynamicpreprocessor/libsf_dcerpc_preproc.la
x lib/snort/dynamicpreprocessor/libsf_dcerpc_preproc.so
x lib/snort/dynamicpreprocessor/libsf_dcerpc_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_dns_preproc.a
x lib/snort/dynamicpreprocessor/libsf_dns_preproc.la
x lib/snort/dynamicpreprocessor/libsf_dns_preproc.so
x lib/snort/dynamicpreprocessor/libsf_dns_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_ftptelnet_preproc.a
x lib/snort/dynamicpreprocessor/libsf_ftptelnet_preproc.la
x lib/snort/dynamicpreprocessor/libsf_ftptelnet_preproc.so
x lib/snort/dynamicpreprocessor/libsf_ftptelnet_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_smtp_preproc.a
x lib/snort/dynamicpreprocessor/libsf_smtp_preproc.la
x lib/snort/dynamicpreprocessor/libsf_smtp_preproc.so
x lib/snort/dynamicpreprocessor/libsf_smtp_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_ssh_preproc.a
x lib/snort/dynamicpreprocessor/libsf_ssh_preproc.la
x lib/snort/dynamicpreprocessor/libsf_ssh_preproc.so
x lib/snort/dynamicpreprocessor/libsf_ssh_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_ssl_preproc.a
x lib/snort/dynamicpreprocessor/libsf_ssl_preproc.la
x lib/snort/dynamicpreprocessor/libsf_ssl_preproc.so
x lib/snort/dynamicpreprocessor/libsf_ssl_preproc.so.0
x share/examples/snort/classification.config-sample <- copied to classification.config
x share/examples/snort/create_db2
x share/examples/snort/create_mssql
x share/examples/snort/create_mysql
x share/examples/snort/create_oracle.sql
x share/examples/snort/create_postgresql
x share/examples/snort/gen-msg.map-sample <- copied to gen-msg.map
x share/examples/snort/reference.config-sample <- copied to reference.config
x share/examples/snort/sid-msg.map-sample <- copied to sid-msg.map
x share/examples/snort/snort.conf-sample <- copied to snort.conf
x share/examples/snort/threshold.conf-sample <- copied to threshold.conf
x share/examples/snort/unicode.map-sample <- copied to unicode.map
x share/doc/snort <- multiple files, omitted here
x etc/rc.d/snort

These are the configuration options for Snort.

===> The following configuration options are available for snort-2.8.2.2_2:
     DYNAMIC=on (default) "Enable dynamic plugin support"
     FLEXRESP=off (default) "Flexible response to events"
     FLEXRESP2=off (default) "Flexible response to events (version 2)"
     MYSQL=off (default) "Enable MySQL support"
     ODBC=off (default) "Enable ODBC support"
     POSTGRESQL=off (default) "Enable PostgreSQL support"
     PRELUDE=off (default) "Enable Prelude NIDS integration"
     PERPROFILE=off (default) "Enable Performance Profiling"
     SNORTSAM=off (default) "Enable output plugin to SnortSam"
===> Use 'make config' to modify these settings

I'm glad dynamic plugin support is enabled, but disappointed to see performance profiling disabled. The --enable-timestats option isn't available via the port at all, apparently.

The FreeBSD port/package can't ship with rules, so you need to download your own rules from Sourcefire, along with any Emerging Threats rules you might want to enable. You then need to edit the snort.conf file to account for your HOME_NET and rule preferences.

The barnyard-sguil-0.2.0_5 package installed the following.

x bin/barnyard
x etc/barnyard.conf-sample <- copied to etc/barnyard.conf by the port
x share/doc/barnyard <- multiple files, omitted here
x etc/rc.d/barnyard

I noticed the barnyard.conf only contained

output sguil

Usually we need something like this:

output sguil: sensor_name sensornamegoeshere

When done the following packages are installed:

tao# pkg_info
barnyard-sguil-0.2.0_5 An output system for Snort (patched for sguil)
mysql-client-5.0.67_1 Multithreaded SQL database (client)
pcre-7.7_1          Perl Compatible Regular Expressions library
sguil-sensor-0.7.0_2 Sguil is a network security monitoring program
snort-2.8.2.1_1     Lightweight network intrusion detection system
tcl-8.4.19,1        Tool Command Language
tclX-8.4_1          Extended TCL
tcltls-1.6          SSL extensions for TCL; dynamicly loadable

Because I want this test system to host the Sguil server too, I decided to move to that phase of the testing.

Before add the sguil-server package, I need to install MySQL server 5.0. This is due to the configuration options:

===> The following configuration options are available for sguil-server-0.7.0_2:
     MYSQL50=off (default) "Install mysql50 server"
===> Use 'make config' to modify these settings

I assume this is the case because the port maintainer prefers running MySQL on one system and the Sguil server on another.

Therefore, I install MySQL server 5.0 using pkg_add -vr mysql50-server.

Next I stopped MySQL via /usr/local/etc/rc.d/mysql stop. This is critical for the next step in the process.

I installed sguil-server next via pkg_add -vr sguil-server.

The sguil-server-0.7.0_2 package installed the following.

x bin/archive_sguildb.tcl
x bin/incident_report.tcl
x bin/sguild
x etc/sguil-server/autocat.conf-sample
x etc/sguil-server/sguild.access-sample
x etc/sguil-server/sguild.conf-sample
x etc/sguil-server/sguild.email-sample
x etc/sguil-server/sguild.queries-sample
x etc/sguil-server/sguild.reports-sample
x etc/sguil-server/sguild.users-sample
x lib/sguil-server/SguildAccess.tcl
x lib/sguil-server/SguildAutoCat.tcl
x lib/sguil-server/SguildClientCmdRcvd.tcl
x lib/sguil-server/SguildConnect.tcl
x lib/sguil-server/SguildCreateDB.tcl
x lib/sguil-server/SguildEmailEvent.tcl
x lib/sguil-server/SguildEvent.tcl
x lib/sguil-server/SguildGenericDB.tcl
x lib/sguil-server/SguildGenericEvent.tcl
x lib/sguil-server/SguildHealthChecks.tcl
x lib/sguil-server/SguildLoaderd.tcl
x lib/sguil-server/SguildMysqlMerge.tcl
x lib/sguil-server/SguildPadsLib.tcl
x lib/sguil-server/SguildQueryd.tcl
x lib/sguil-server/SguildReportBuilder.tcl
x lib/sguil-server/SguildSendComms.tcl
x lib/sguil-server/SguildSensorAgentComms.tcl
x lib/sguil-server/SguildSensorCmdRcvd.tcl
x lib/sguil-server/SguildTranscript.tcl
x lib/sguil-server/SguildUtils.tcl
x share/sguil-server/create_ruledb.sql
x share/sguil-server/create_sguildb.sql
x share/sguil-server/migrate_event.tcl
x share/sguil-server/migrate_sancp.tcl
x share/sguil-server/sancp_cleanup.tcl
x share/sguil-server/update_0.7.tcl
x share/sguil-server/update_sguildb_v5-v6.sql
x share/sguil-server/update_sguildb_v6-v7.sql
x share/sguil-server/update_sguildb_v7-v8.sql
x share/sguil-server/update_sguildb_v8-v9.sql
x share/sguil-server/update_sguildb_v9-v10.sql
x share/sguil-server/update_sguildb_v10-v11.sql
x share/sguil-server/update_sguildb_v11-v12.sql
x share/doc/sguil-server/CHANGES
x share/doc/sguil-server/FAQ
x share/doc/sguil-server/INSTALL
x share/doc/sguil-server/INSTALL.openbsd
x share/doc/sguil-server/LICENSE.QPL
x share/doc/sguil-server/OPENSSL.README
x share/doc/sguil-server/TODO
x share/doc/sguil-server/UPGRADE
x share/doc/sguil-server/USAGE
x share/doc/sguil-server/sguildb.dia
x etc/rc.d/sguild

What came next was very interesting. The port maintainer created a script to help set up the server. I'll show relevant excerpts.

Running pre-install for sguil-server-0.7.0_2..
This sguild install script creates a "turnkey" install
of sguild, including configuing the database and conf files
and user accounts so that sguild can be started immediately.

You may have already done all this (especially if this is an upgrade)
and may not be interested in iterating through cert creation and
everything else that the script does.

This portion of the script creates user and group accounts named "sguil".
Would you like to opt out of this portion of the install script
n
==> Pre-installation configuration of sguil-server-0.7.0_2
User 'sguil' create successfully.
sguil:*:1002:1002::0:0:User &:/home/sguil:/usr/sbin/nologin
...edited...
Running post-install for sguil-server-0.7.0_2..
This sguild install script creates a "turnkey" install
of sguild, including configuing the database and conf files
and user accounts so that sguild can be started immediately.

You may have already done all this (especially if this is an upgrade)
and may not be interested in iterating through cert creation and
everything else that the script does.

Would you like to opt out of the entire install script
and configure sguild manually yourself?
n
There are a few things that need to be done to complete the install.
First, you need to create certs so that the ssl connections between server and
sensors will work, you need to create the database, the account to access it and
the tables for the database and you need to create the directories where all the
data will be stored. (You will also need to edit the conf files for your setup.)


If you haven't already done this, I can do it for you now.
Would you like to create certs now? (y for yes, n for no)
y
Creating /usr/local/etc/sguil-server/certs ....
First we need to create a password-protected CA cert.

(The Common Name should be the FQHN of your squil server.)
Generating a 1024 bit RSA private key
.....++++++
.......................................++++++
writing new private key to 'privkey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:VA
Locality Name (eg, city) []:M
Organization Name (eg, company) [Internet Widgits Pty Ltd]:T
Organizational Unit Name (eg, section) []:O
Common Name (eg, YOUR name) []:R
Email Address []:o

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:sguil
An optional company name []:
Now we need to create the actual certificate for your server.
Signature ok
subject=/C=US/ST=VA/L=M/O=T/OU=O/CN=R/emailAddress=o
Getting CA Private Key
Enter pass phrase for privkey.pem:
Finally, we need to move the certs to the '/usr/local/etc/sguil-server/certs}' directory
and clean up the port directory as well.
mv: rename /a/ports/security/sguil-server/sguild.key to /usr/local/etc/sguil-server/certs/sguild.key:
 No such file or directory
mv: rename /a/ports/security/sguil-server/sguild.pem to /usr/local/etc/sguil-server/certs/sguild.pem:
 No such file or directory
rm: /a/ports/security/sguil-server/CA.pem: No such file or directory
rm: /a/ports/security/sguil-server/privkey.pem: No such file or directory
rm: /a/ports/security/sguil-server/sguild.req: No such file or directory
rm: /a/ports/security/sguil-server/file.sr1: No such file or directory

Those errors happen because the script was written with the assumption that it would be run from a ports installation, not a package installation. I emailed the ports maintainer to see if the problem can be fixed.

Is the installation of mysql brand new and unaltered?
By default, when mysql is installed, it creates five accounts.
None of those accounts are protected by passwords.  That needs to be corrected.
The five accounts are:
        root@localhost
        root@127.0.0.1
        root@tao.taosecurity.com
            @localhost
            @tao.taosecurity.com
I can remove all of the accounts except root@localhost (highly recommended)
and I can set the password for the root@localhost account. (If you get an error
don't worry about it.  The account may not have been created to begin with.
Would you like me to do that now?
y
Enabling mysql in /etc/rc.conf and starting the server.....
It appears that mysql is already enabled!

The mysql pid is ....
Starting mysql.
Deleting users from mysql......
All done deleting.......
What would you like root@localhost's password to be?
root
Would you like to bind mysql to localhost so it only listens on that address?

y
The mysql pid is 1694.....
Stopping mysql.
Waiting for PIDS: 1694.
Starting mysql.
Would you like to create the database to store all nsm data?

y
NOTE: If you're upgrading, you do NOT want to do this!  You want to upgrade.
./+INSTALL: cannot open /work/a/ports/security/sguil-server/work/sguil-0.7.0/server/sql_scripts/create_sguildb.sql:
 No such file or directory

This error is similar to the previous error. I also emailed the port maintainer.

Would you like to create a user "sguild@localhost" for database access?

y
Please enter the password that you want to use for the sguild account.

sguil
Creating account for sguild with access to sguildb.....
Would you like to create the data directory and all its subdirectories?

y
What do you want the name of the main directory to be?
(Be sure to include the full path to the directory - e.g. /var/nsm)
/var/nsm
The main directory will be named '/var/nsm'.
Creating /var/nsm ....
Creating /var/nsm/archives ....
Creating /var/nsm/rules ....
Creating /var/nsm/load ....
Would you like to enable sguild in /etc/rc.conf?

y
iWriting to /etc/rc.conf....

If the sguild.conf file does not exist, I will create and edit it now.

Preparing to edit the sguild.conf file......
You still need to review all the conf files and configure sguil
per your desired setup before starting sguild.  Refer to the port docs in
/usr/local/share/doc/sguil-server before proceeding.

Right now, all the conf files except sguild.conf are set to the defaults.
...edited...

That ends the need for user input. The final step advises the user on other required changes.

         ***********************************
         * !!!!!!!!!!! WARNING !!!!!!!!!!! *
         ***********************************

PLEASE NOTE: If you are upgrading from a previous version,
read the UPGRADE doc (in /usr/local/share/doc/sguil-server) before proceeding!!!
Some noteworthy changes in version 0.7.0:
SSL is now required for server, sensor and client.
The sguild.conf and sguild.email files have changed.
You MUST run the upgrade_0.7.tcl script to clean up and
prepare the database before running the new version.  BE SURE
TO BACK UP YOUR DATABASE BEFORE PROCEEDING!!!

If you had existing config files in /usr/local/etc/sguil-server
they were not overwritten.  If this is a first time install, you
must copy the sample files to the corresponding conf file and
edit the various config files for your site.  See the INSTALL
doc in /usr/local/share/doc/sguil-server for details.  If this is an upgrade, replace
your existing conf file with the new one and edit accordingly.

The sql scripts for creating database tables were placed in
the /usr/local/share/sguil-server/ directory.  PLEASE
NOTE: LOG_DIR is not set by this install.  You MUST create the
correct LOG_DIRS and put a copy of the snort rules you use in
LOG_DIR/rules.

The sguild, archive_sguildb.tcl and incident_report.tcl scripts
were placed in /usr/local/bin/.  The incident_report.tcl
script is from the contrib section.  There is no documentation
and the script's variables must be edited before it is used.

A startup script, named sguild.sh was installed in
/usr/local/etc/rc.d/.  To enable it, edit /etc/rc.conf
per the instructions in the script.

NOTE:  Sguild now runs under the sguil user account not root!

At the end of the process the system had these packages installed.

tao# pkg_info
barnyard-sguil-0.2.0_5 An output system for Snort (patched for sguil)
mysql-client-5.0.67_1 Multithreaded SQL database (client)
mysql-server-5.0.67_1 Multithreaded SQL database (server)
mysqltcl-3.05       TCL module for accessing MySQL databases based on msqltcl
p0f-2.0.8           Passive OS fingerprinting tool
pcre-7.7_1          Perl Compatible Regular Expressions library
sguil-sensor-0.7.0_2 Sguil is a network security monitoring program
sguil-server-0.7.0_2 Sguil is a network security monitoring program
snort-2.8.2.1_1     Lightweight network intrusion detection system
tcl-8.4.19,1        Tool Command Language
tclX-8.4_1          Extended TCL
tcllib-1.10_1       A collection of utility modules for Tcl
tcltls-1.6          SSL extensions for TCL; dynamicly loadable
tcpflow-0.21_1      A tool for capturing data transmitted as part of TCP connec

If I wanted to go from here to actually run the Sguil server, I would have to manually create the database and certificates. Once the script is fixed I shouldn't have to do that.

The major configuration issue that remains is ensuring that data is being written to logical locations. This primarily means pcap data is stored in a partition that can accommodate it, and the database is located in a partition that can handle growing tables.

I think it should be clear at this point that the easiest way to try Sguil is to use NSMNow. I recommend that only for demo installations, although you can tweak the installation to put what you want in locations you like.

---

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.