Tuesday, February 03, 2009

Notes on Installing Sguil Using FreeBSD 7.1 Packages

It's been a while since I've looked at the Sguil ports for FreeBSD, so I decided to see how they work.

In this post I will talk about installing a Sguil sensor and server on a single FreeBSD 7.1 test VM using packages shipped with FreeBSD 7.1.

To start with the system had no packages installed.

After running pkg_add -vr sguil-sensor, I watched what was added to the system. I'm only going to document that which I found interesting.

The sguil-sensor-0.7.0_2 package installed the following into /usr/local.

x bin/sguil-sensor/log_packets.sh
x bin/sguil-sensor/example_agent.tcl
x bin/sguil-sensor/pcap_agent.tcl
x bin/sguil-sensor/snort_agent.tcl
x etc/sguil-sensor/example_agent.conf-sample
x etc/sguil-sensor/pcap_agent.conf-sample
x etc/sguil-sensor/snort_agent.conf-sample
x etc/sguil-sensor/log_packets.conf-sample
x share/doc/sguil-sensor <- multiple files, omitted here
x etc/rc.d/example_agent
x etc/rc.d/pcap_agent
x etc/rc.d/snort_agent

Note that you have to copy




and edit each, prior to starting




Also, as noted in the configuration options, PADS and SANCP are not installed by default, so the package doesn't include them:

===> The following configuration options are available for sguil-sensor-0.7.0_2:
SANCP=off (default) "Include sancp sensor"
PADS=off (default) "Include pads sensor"
===> Use 'make config' to modify these settings

The snort- package installed the following.

x man/man8/snort.8.gz
x bin/snort
x etc/snort/classification.config-sample
x etc/snort/gen-msg.map-sample
x etc/snort/reference.config-sample
x etc/snort/sid-msg.map-sample
x etc/snort/snort.conf-sample
x etc/snort/threshold.conf-sample
x etc/snort/unicode.map-sample
x src/snort_dynamicsrc/bitop.h
x src/snort_dynamicsrc/debug.h
x src/snort_dynamicsrc/pcap_pkthdr32.h
x src/snort_dynamicsrc/preprocids.h
x src/snort_dynamicsrc/profiler.h
x src/snort_dynamicsrc/sf_dynamic_common.h
x src/snort_dynamicsrc/sf_dynamic_meta.h
x src/snort_dynamicsrc/sf_dynamic_preproc_lib.c
x src/snort_dynamicsrc/sf_dynamic_preproc_lib.h
x src/snort_dynamicsrc/sf_dynamic_preprocessor.h
x src/snort_dynamicsrc/sf_snort_packet.h
x src/snort_dynamicsrc/sf_snort_plugin_api.h
x src/snort_dynamicsrc/sfghash.h
x src/snort_dynamicsrc/sfhashfcn.h
x src/snort_dynamicsrc/sfsnort_dynamic_detection_lib.c
x src/snort_dynamicsrc/sfsnort_dynamic_detection_lib.h
x src/snort_dynamicsrc/str_search.h
x src/snort_dynamicsrc/stream_api.h
x lib/snort/dynamicengine/libsf_engine.so
x lib/snort/dynamicengine/libsf_engine.so.0
x lib/snort/dynamicengine/libsf_engine.la
x lib/snort/dynamicengine/libsf_engine.a
x lib/snort/dynamicrules/lib_sfdynamic_example_rule.so
x lib/snort/dynamicrules/lib_sfdynamic_example_rule.so.0
x lib/snort/dynamicrules/lib_sfdynamic_example_rule.la
x lib/snort/dynamicrules/lib_sfdynamic_example_rule.a
x lib/snort/dynamicpreprocessor/lib_sfdynamic_preprocessor_example.a
x lib/snort/dynamicpreprocessor/lib_sfdynamic_preprocessor_example.la
x lib/snort/dynamicpreprocessor/lib_sfdynamic_preprocessor_example.so
x lib/snort/dynamicpreprocessor/lib_sfdynamic_preprocessor_example.so.0
x lib/snort/dynamicpreprocessor/libsf_dcerpc_preproc.a
x lib/snort/dynamicpreprocessor/libsf_dcerpc_preproc.la
x lib/snort/dynamicpreprocessor/libsf_dcerpc_preproc.so
x lib/snort/dynamicpreprocessor/libsf_dcerpc_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_dns_preproc.a
x lib/snort/dynamicpreprocessor/libsf_dns_preproc.la
x lib/snort/dynamicpreprocessor/libsf_dns_preproc.so
x lib/snort/dynamicpreprocessor/libsf_dns_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_ftptelnet_preproc.a
x lib/snort/dynamicpreprocessor/libsf_ftptelnet_preproc.la
x lib/snort/dynamicpreprocessor/libsf_ftptelnet_preproc.so
x lib/snort/dynamicpreprocessor/libsf_ftptelnet_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_smtp_preproc.a
x lib/snort/dynamicpreprocessor/libsf_smtp_preproc.la
x lib/snort/dynamicpreprocessor/libsf_smtp_preproc.so
x lib/snort/dynamicpreprocessor/libsf_smtp_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_ssh_preproc.a
x lib/snort/dynamicpreprocessor/libsf_ssh_preproc.la
x lib/snort/dynamicpreprocessor/libsf_ssh_preproc.so
x lib/snort/dynamicpreprocessor/libsf_ssh_preproc.so.0
x lib/snort/dynamicpreprocessor/libsf_ssl_preproc.a
x lib/snort/dynamicpreprocessor/libsf_ssl_preproc.la
x lib/snort/dynamicpreprocessor/libsf_ssl_preproc.so
x lib/snort/dynamicpreprocessor/libsf_ssl_preproc.so.0
x share/examples/snort/classification.config-sample <- copied to classification.config
x share/examples/snort/create_db2
x share/examples/snort/create_mssql
x share/examples/snort/create_mysql
x share/examples/snort/create_oracle.sql
x share/examples/snort/create_postgresql
x share/examples/snort/gen-msg.map-sample <- copied to gen-msg.map
x share/examples/snort/reference.config-sample <- copied to reference.config
x share/examples/snort/sid-msg.map-sample <- copied to sid-msg.map
x share/examples/snort/snort.conf-sample <- copied to snort.conf
x share/examples/snort/threshold.conf-sample <- copied to threshold.conf
x share/examples/snort/unicode.map-sample <- copied to unicode.map
x share/doc/snort <- multiple files, omitted here
x etc/rc.d/snort

These are the configuration options for Snort.

===> The following configuration options are available for snort-
DYNAMIC=on (default) "Enable dynamic plugin support"
FLEXRESP=off (default) "Flexible response to events"
FLEXRESP2=off (default) "Flexible response to events (version 2)"
MYSQL=off (default) "Enable MySQL support"
ODBC=off (default) "Enable ODBC support"
POSTGRESQL=off (default) "Enable PostgreSQL support"
PRELUDE=off (default) "Enable Prelude NIDS integration"
PERPROFILE=off (default) "Enable Performance Profiling"
SNORTSAM=off (default) "Enable output plugin to SnortSam"
===> Use 'make config' to modify these settings

I'm glad dynamic plugin support is enabled, but disappointed to see performance profiling disabled. The --enable-timestats option isn't available via the port at all, apparently.

The FreeBSD port/package can't ship with rules, so you need to download your own rules from Sourcefire, along with any Emerging Threats rules you might want to enable. You then need to edit the snort.conf file to account for your HOME_NET and rule preferences.

The barnyard-sguil-0.2.0_5 package installed the following.

x bin/barnyard
x etc/barnyard.conf-sample <- copied to etc/barnyard.conf by the port
x share/doc/barnyard <- multiple files, omitted here
x etc/rc.d/barnyard

I noticed the barnyard.conf only contained

output sguil

Usually we need something like this:

output sguil: sensor_name sensornamegoeshere

When done the following packages are installed:

tao# pkg_info
barnyard-sguil-0.2.0_5 An output system for Snort (patched for sguil)
mysql-client-5.0.67_1 Multithreaded SQL database (client)
pcre-7.7_1 Perl Compatible Regular Expressions library
sguil-sensor-0.7.0_2 Sguil is a network security monitoring program
snort- Lightweight network intrusion detection system
tcl-8.4.19,1 Tool Command Language
tclX-8.4_1 Extended TCL
tcltls-1.6 SSL extensions for TCL; dynamicly loadable

Because I want this test system to host the Sguil server too, I decided to move to that phase of the testing.

Before add the sguil-server package, I need to install MySQL server 5.0. This is due to the configuration options:

===> The following configuration options are available for sguil-server-0.7.0_2:
MYSQL50=off (default) "Install mysql50 server"
===> Use 'make config' to modify these settings

I assume this is the case because the port maintainer prefers running MySQL on one system and the Sguil server on another.

Therefore, I install MySQL server 5.0 using pkg_add -vr mysql50-server.

Next I stopped MySQL via /usr/local/etc/rc.d/mysql stop. This is critical for the next step in the process.

I installed sguil-server next via pkg_add -vr sguil-server.

The sguil-server-0.7.0_2 package installed the following.

x bin/archive_sguildb.tcl
x bin/incident_report.tcl
x bin/sguild
x etc/sguil-server/autocat.conf-sample
x etc/sguil-server/sguild.access-sample
x etc/sguil-server/sguild.conf-sample
x etc/sguil-server/sguild.email-sample
x etc/sguil-server/sguild.queries-sample
x etc/sguil-server/sguild.reports-sample
x etc/sguil-server/sguild.users-sample
x lib/sguil-server/SguildAccess.tcl
x lib/sguil-server/SguildAutoCat.tcl
x lib/sguil-server/SguildClientCmdRcvd.tcl
x lib/sguil-server/SguildConnect.tcl
x lib/sguil-server/SguildCreateDB.tcl
x lib/sguil-server/SguildEmailEvent.tcl
x lib/sguil-server/SguildEvent.tcl
x lib/sguil-server/SguildGenericDB.tcl
x lib/sguil-server/SguildGenericEvent.tcl
x lib/sguil-server/SguildHealthChecks.tcl
x lib/sguil-server/SguildLoaderd.tcl
x lib/sguil-server/SguildMysqlMerge.tcl
x lib/sguil-server/SguildPadsLib.tcl
x lib/sguil-server/SguildQueryd.tcl
x lib/sguil-server/SguildReportBuilder.tcl
x lib/sguil-server/SguildSendComms.tcl
x lib/sguil-server/SguildSensorAgentComms.tcl
x lib/sguil-server/SguildSensorCmdRcvd.tcl
x lib/sguil-server/SguildTranscript.tcl
x lib/sguil-server/SguildUtils.tcl
x share/sguil-server/create_ruledb.sql
x share/sguil-server/create_sguildb.sql
x share/sguil-server/migrate_event.tcl
x share/sguil-server/migrate_sancp.tcl
x share/sguil-server/sancp_cleanup.tcl
x share/sguil-server/update_0.7.tcl
x share/sguil-server/update_sguildb_v5-v6.sql
x share/sguil-server/update_sguildb_v6-v7.sql
x share/sguil-server/update_sguildb_v7-v8.sql
x share/sguil-server/update_sguildb_v8-v9.sql
x share/sguil-server/update_sguildb_v9-v10.sql
x share/sguil-server/update_sguildb_v10-v11.sql
x share/sguil-server/update_sguildb_v11-v12.sql
x share/doc/sguil-server/CHANGES
x share/doc/sguil-server/FAQ
x share/doc/sguil-server/INSTALL
x share/doc/sguil-server/INSTALL.openbsd
x share/doc/sguil-server/LICENSE.QPL
x share/doc/sguil-server/OPENSSL.README
x share/doc/sguil-server/TODO
x share/doc/sguil-server/UPGRADE
x share/doc/sguil-server/USAGE
x share/doc/sguil-server/sguildb.dia
x etc/rc.d/sguild

What came next was very interesting. The port maintainer created a script to help set up the server. I'll show relevant excerpts.

Running pre-install for sguil-server-0.7.0_2..
This sguild install script creates a "turnkey" install
of sguild, including configuing the database and conf files
and user accounts so that sguild can be started immediately.

You may have already done all this (especially if this is an upgrade)
and may not be interested in iterating through cert creation and
everything else that the script does.

This portion of the script creates user and group accounts named "sguil".
Would you like to opt out of this portion of the install script
==> Pre-installation configuration of sguil-server-0.7.0_2
User 'sguil' create successfully.
sguil:*:1002:1002::0:0:User &:/home/sguil:/usr/sbin/nologin
Running post-install for sguil-server-0.7.0_2..
This sguild install script creates a "turnkey" install
of sguild, including configuing the database and conf files
and user accounts so that sguild can be started immediately.

You may have already done all this (especially if this is an upgrade)
and may not be interested in iterating through cert creation and
everything else that the script does.

Would you like to opt out of the entire install script
and configure sguild manually yourself?
There are a few things that need to be done to complete the install.
First, you need to create certs so that the ssl connections between server and
sensors will work, you need to create the database, the account to access it and
the tables for the database and you need to create the directories where all the
data will be stored. (You will also need to edit the conf files for your setup.)

If you haven't already done this, I can do it for you now.
Would you like to create certs now? (y for yes, n for no)
Creating /usr/local/etc/sguil-server/certs ....
First we need to create a password-protected CA cert.

(The Common Name should be the FQHN of your squil server.)
Generating a 1024 bit RSA private key
writing new private key to 'privkey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:VA
Locality Name (eg, city) []:M
Organization Name (eg, company) [Internet Widgits Pty Ltd]:T
Organizational Unit Name (eg, section) []:O
Common Name (eg, YOUR name) []:R
Email Address []:o

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:sguil
An optional company name []:
Now we need to create the actual certificate for your server.
Signature ok
Getting CA Private Key
Enter pass phrase for privkey.pem:
Finally, we need to move the certs to the '/usr/local/etc/sguil-server/certs}' directory
and clean up the port directory as well.
mv: rename /a/ports/security/sguil-server/sguild.key to /usr/local/etc/sguil-server/certs/sguild.key:
No such file or directory
mv: rename /a/ports/security/sguil-server/sguild.pem to /usr/local/etc/sguil-server/certs/sguild.pem:
No such file or directory
rm: /a/ports/security/sguil-server/CA.pem: No such file or directory
rm: /a/ports/security/sguil-server/privkey.pem: No such file or directory
rm: /a/ports/security/sguil-server/sguild.req: No such file or directory
rm: /a/ports/security/sguil-server/file.sr1: No such file or directory

Those errors happen because the script was written with the assumption that it would be run from a ports installation, not a package installation. I emailed the ports maintainer to see if the problem can be fixed.

Is the installation of mysql brand new and unaltered?
By default, when mysql is installed, it creates five accounts.
None of those accounts are protected by passwords. That needs to be corrected.
The five accounts are:
I can remove all of the accounts except root@localhost (highly recommended)
and I can set the password for the root@localhost account. (If you get an error
don't worry about it. The account may not have been created to begin with.
Would you like me to do that now?
Enabling mysql in /etc/rc.conf and starting the server.....
It appears that mysql is already enabled!

The mysql pid is ....
Starting mysql.
Deleting users from mysql......
All done deleting.......
What would you like root@localhost's password to be?
Would you like to bind mysql to localhost so it only listens on that address?

The mysql pid is 1694.....
Stopping mysql.
Waiting for PIDS: 1694.
Starting mysql.
Would you like to create the database to store all nsm data?

NOTE: If you're upgrading, you do NOT want to do this! You want to upgrade.
./+INSTALL: cannot open /work/a/ports/security/sguil-server/work/sguil-0.7.0/server/sql_scripts/create_sguildb.sql:
No such file or directory

This error is similar to the previous error. I also emailed the port maintainer.

Would you like to create a user "sguild@localhost" for database access?

Please enter the password that you want to use for the sguild account.

Creating account for sguild with access to sguildb.....
Would you like to create the data directory and all its subdirectories?

What do you want the name of the main directory to be?
(Be sure to include the full path to the directory - e.g. /var/nsm)
The main directory will be named '/var/nsm'.
Creating /var/nsm ....
Creating /var/nsm/archives ....
Creating /var/nsm/rules ....
Creating /var/nsm/load ....
Would you like to enable sguild in /etc/rc.conf?

iWriting to /etc/rc.conf....

If the sguild.conf file does not exist, I will create and edit it now.

Preparing to edit the sguild.conf file......
You still need to review all the conf files and configure sguil
per your desired setup before starting sguild. Refer to the port docs in
/usr/local/share/doc/sguil-server before proceeding.

Right now, all the conf files except sguild.conf are set to the defaults.

That ends the need for user input. The final step advises the user on other required changes.

* !!!!!!!!!!! WARNING !!!!!!!!!!! *

PLEASE NOTE: If you are upgrading from a previous version,
read the UPGRADE doc (in /usr/local/share/doc/sguil-server) before proceeding!!!
Some noteworthy changes in version 0.7.0:
SSL is now required for server, sensor and client.
The sguild.conf and sguild.email files have changed.
You MUST run the upgrade_0.7.tcl script to clean up and
prepare the database before running the new version. BE SURE

If you had existing config files in /usr/local/etc/sguil-server
they were not overwritten. If this is a first time install, you
must copy the sample files to the corresponding conf file and
edit the various config files for your site. See the INSTALL
doc in /usr/local/share/doc/sguil-server for details. If this is an upgrade, replace
your existing conf file with the new one and edit accordingly.

The sql scripts for creating database tables were placed in
the /usr/local/share/sguil-server/ directory. PLEASE
NOTE: LOG_DIR is not set by this install. You MUST create the
correct LOG_DIRS and put a copy of the snort rules you use in

The sguild, archive_sguildb.tcl and incident_report.tcl scripts
were placed in /usr/local/bin/. The incident_report.tcl
script is from the contrib section. There is no documentation
and the script's variables must be edited before it is used.

A startup script, named sguild.sh was installed in
/usr/local/etc/rc.d/. To enable it, edit /etc/rc.conf
per the instructions in the script.

NOTE: Sguild now runs under the sguil user account not root!

At the end of the process the system had these packages installed.

tao# pkg_info
barnyard-sguil-0.2.0_5 An output system for Snort (patched for sguil)
mysql-client-5.0.67_1 Multithreaded SQL database (client)
mysql-server-5.0.67_1 Multithreaded SQL database (server)
mysqltcl-3.05 TCL module for accessing MySQL databases based on msqltcl
p0f-2.0.8 Passive OS fingerprinting tool
pcre-7.7_1 Perl Compatible Regular Expressions library
sguil-sensor-0.7.0_2 Sguil is a network security monitoring program
sguil-server-0.7.0_2 Sguil is a network security monitoring program
snort- Lightweight network intrusion detection system
tcl-8.4.19,1 Tool Command Language
tclX-8.4_1 Extended TCL
tcllib-1.10_1 A collection of utility modules for Tcl
tcltls-1.6 SSL extensions for TCL; dynamicly loadable
tcpflow-0.21_1 A tool for capturing data transmitted as part of TCP connec

If I wanted to go from here to actually run the Sguil server, I would have to manually create the database and certificates. Once the script is fixed I shouldn't have to do that.

The major configuration issue that remains is ensuring that data is being written to logical locations. This primarily means pcap data is stored in a partition that can accommodate it, and the database is located in a partition that can handle growing tables.

I think it should be clear at this point that the easiest way to try Sguil is to use NSMNow. I recommend that only for demo installations, although you can tweak the installation to put what you want in locations you like.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.


wxs said...

If the maintainer doesn't respond in a week or two feel free to ping me and I can address some of the problems you describe.

Richard Bejtlich said...

wxs, Paul emailed me, although he might want to talk to you anyway.

Scott said...

Mr. Bejtlich,
My new weekend hobby is working on making all the pieces-parts of sguil work on a FreeBSD-CURRENT (7.2) machine - preferably using packages since it's a slow machine that's laying around. This post was invaluable (although I don't have it running yet, feels like I'm getting close). Thanks for that.

A couple things that might really help the novice / hobbyist user: 1. Differentiating between what's actually required for sguil, and what's a "nice to have" (Are sancp, p0f and barnyard actually required for any install of sguil, or just interesting additions to an nsm?)
2. a diagram of how all the pieces communicate (in visio, powerpoint jpeg, or ascii) would be extraordinarily useful - I guess most users of snort understand somewhat how it works, but to understand all the additional tools' roles would be invaluable.
3. Your script repository on sourceforge was extremely helpful in understanding part of what's going on. I realize they're for older versions, but it might be really useful to link that here.

Back to it. Thanks again,

Richard Bejtlich said...

Hi Scott,

We consider SANCP and Barnyard to be required elements for NSM. P0f is optional.

You can see architectural diagrams here:


My latest scripts are available at:


Anonymous said...

Please, I need your help. I have installed sguil but when I restart my Ubuntu desktop I can not to run sguil.tk. I have received a message:
luis@luis-9-10:/var/log/nsm/server100$ cat sguild.log
Executing: sguild -c /etc/nsm/server100/sguild.conf -a /etc/nsm/server100/autocat.conf -u /etc/nsm/server100/sguild.users -g /etc/nsm/server100/sguild.queries -A /etc/nsm/server100/sguild.access -C /etc/nsm/server100/certs
pid(1945) Loading access list: /etc/nsm/server100/sguild.access
pid(1945) Sensor access list set to ALLOW ANY.
pid(1945) Client access list set to ALLOW ANY.
pid(1945) Connecting to localhost on 3306 as sguil
pid(1945) MySQL Version: version 5.1.37-1ubuntu5.1
pid(1945) SguilDB Version: 0.12


ERROR: You appear to be using an old version of the
sguil database schema that does not support the MERGE tables
Please use the migrate_event.tcl script and see the CHANGES
document for more information

. Table event returned status => event MRG_MYISAM 10 Dynamic 23 377 2972 0 0 0 {} {} {} {} latin1_swedish_ci {} {} {}

SGUILD: Exiting...

Please, I need your help.

Anonymous said...

Hello again,
I have another question. I am from Spain and I have learning thinks about NSM. Please, I need your recomendation about a training (formation) about NSM in Spain with LABS, etc.
Regards and thanks you

Richard Bejtlich said...

Hello anonymous,

Please post any Sguil questions to the sguil-users list at


I'm teaching NSM in Barcelona next week:


Anonymous said...

Hello Richard,
I am not form BCN (Barcelona) and I can not go to BCN the next week. Please, Could you possible to contact with you and try to explain what is my neccessary formation? (by e-mail o telephone number)

Richard Bejtlich said...

Anonymous, if you can't attend my class then I don't have any recommendations on NSM training. Sorry.

Anonymous said...

Hello Richard,
My city is 400 Km from BCN, sorry but I can not to go your training.
Please, Is it possible that you go to Madrid this year and impart the training.
Madrid is closer than BCN.

Richard Bejtlich said...

Sorry, I only teach in Washington, DC; this year in Barcelona; and in Las Vegas, NV -- all for Black Hat.

Anonymous said...

Hello Richard,
Ok, I will try the next year, in Barcelona or Madird.
Thanks you

Anonymous said...

Hello Richard Again,
I am seend your scheduled and I will wise assit the event but I supposse that it is very later.
I will try to inform me the next year.
Please, Is there any e-mail for information?