Monday, December 29, 2008

Installing Sguil Using NSMNow



In my post NSM-Friendly VMware Lab Setup I mentioned wanting to use NSMNow to install Sguil on Ubuntu 8.04 for student use in my next class. I had tried the Securix-NSM live CD but I had not tried installing Sguil using the same project's NSMNow scripts. I just did it:


root@twsu804:/usr/local/src# wget http://www.securixlive.com/download/nsmnow/NSMnow-1.1.1.tar.gz
--22:14:38-- http://www.securixlive.com/download/nsmnow/NSMnow-1.1.1.tar.gz
=> `NSMnow-1.1.1.tar.gz'
Resolving www.securixlive.com... 202.191.61.156
Connecting to www.securixlive.com|202.191.61.156|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 164,613 (161K) [application/x-gzip]

100%[====================================>] 164,613 53.85K/s

22:14:42 (53.80 KB/s) - `NSMnow-1.1.1.tar.gz' saved [164613/164613]

root@twsu804:/usr/local/src# tar -xzvf NSMnow-1.1.1.tar.gz
NSMnow-1.1.1/
NSMnow-1.1.1/NSMnow-core
NSMnow-1.1.1/RELEASE.NOTES
NSMnow-1.1.1/templates/
NSMnow-1.1.1/templates/lib/
NSMnow-1.1.1/templates/lib/lib-console-utils
NSMnow-1.1.1/templates/init/
NSMnow-1.1.1/templates/init/sancpd
NSMnow-1.1.1/templates/init/snortl-newday
NSMnow-1.1.1/templates/init/snortu
NSMnow-1.1.1/templates/init/pcap_agent
NSMnow-1.1.1/templates/init/barnyard2
NSMnow-1.1.1/templates/init/sguild
NSMnow-1.1.1/templates/init/snort_agent
NSMnow-1.1.1/templates/init/snortl
NSMnow-1.1.1/templates/init/sancp_agent
NSMnow-1.1.1/templates/rules/
NSMnow-1.1.1/templates/rules/pop3.rules
NSMnow-1.1.1/templates/rules/finger.rules
NSMnow-1.1.1/templates/rules/dos.rules
NSMnow-1.1.1/templates/rules/shellcode.rules
NSMnow-1.1.1/templates/rules/dns.rules
NSMnow-1.1.1/templates/rules/attack-responses.rules
NSMnow-1.1.1/templates/rules/local.rules
NSMnow-1.1.1/templates/rules/icmp-info.rules
NSMnow-1.1.1/templates/rules/policy.rules
NSMnow-1.1.1/templates/rules/web-cgi.rules
NSMnow-1.1.1/templates/rules/ddos.rules
NSMnow-1.1.1/templates/rules/mysql.rules
NSMnow-1.1.1/templates/rules/oracle.rules
NSMnow-1.1.1/templates/rules/other-ids.rules
NSMnow-1.1.1/templates/rules/icmp.rules
NSMnow-1.1.1/templates/rules/experimental.rules
NSMnow-1.1.1/templates/rules/chat.rules
NSMnow-1.1.1/templates/rules/info.rules
NSMnow-1.1.1/templates/rules/web-attacks.rules
NSMnow-1.1.1/templates/rules/nntp.rules
NSMnow-1.1.1/templates/rules/telnet.rules
NSMnow-1.1.1/templates/rules/scan.rules
NSMnow-1.1.1/templates/rules/rservices.rules
NSMnow-1.1.1/templates/rules/web-php.rules
NSMnow-1.1.1/templates/rules/bad-traffic.rules
NSMnow-1.1.1/templates/rules/snmp.rules
NSMnow-1.1.1/templates/rules/web-coldfusion.rules
NSMnow-1.1.1/templates/rules/tftp.rules
NSMnow-1.1.1/templates/rules/ftp.rules
NSMnow-1.1.1/templates/rules/misc.rules
NSMnow-1.1.1/templates/rules/multimedia.rules
NSMnow-1.1.1/templates/rules/web-frontpage.rules
NSMnow-1.1.1/templates/rules/imap.rules
NSMnow-1.1.1/templates/rules/porn.rules
NSMnow-1.1.1/templates/rules/web-client.rules
NSMnow-1.1.1/templates/rules/netbios.rules
NSMnow-1.1.1/templates/rules/p2p.rules
NSMnow-1.1.1/templates/rules/rpc.rules
NSMnow-1.1.1/templates/rules/web-misc.rules
NSMnow-1.1.1/templates/rules/backdoor.rules
NSMnow-1.1.1/templates/rules/pop2.rules
NSMnow-1.1.1/templates/rules/exploit.rules
NSMnow-1.1.1/templates/rules/sql.rules
NSMnow-1.1.1/templates/rules/virus.rules
NSMnow-1.1.1/templates/rules/x11.rules
NSMnow-1.1.1/templates/rules/smtp.rules
NSMnow-1.1.1/templates/rules/deleted.rules
NSMnow-1.1.1/templates/rules/web-iis.rules
NSMnow-1.1.1/LICENCE
NSMnow-1.1.1/NSMnow.conf
NSMnow-1.1.1/libs/
NSMnow-1.1.1/libs/barnyard2.pm
NSMnow-1.1.1/libs/utils.pm
NSMnow-1.1.1/libs/sguilsensor.pm
NSMnow-1.1.1/libs/sguilclient.pm
NSMnow-1.1.1/libs/utils.sh
NSMnow-1.1.1/libs/mysql.pm
NSMnow-1.1.1/libs/sguiltools.pm
NSMnow-1.1.1/libs/tcl.pm
NSMnow-1.1.1/libs/os.pm
NSMnow-1.1.1/libs/buildessential.pm
NSMnow-1.1.1/libs/sguilserver.pm
NSMnow-1.1.1/libs/os.sh
NSMnow-1.1.1/libs/snort.pm
NSMnow-1.1.1/libs/sancp.pm
NSMnow-1.1.1/README
NSMnow-1.1.1/INSTALL
NSMnow-1.1.1/NSMnow.log
NSMnow-1.1.1/run-init
NSMnow-1.1.1/NSMnow
NSMnow-1.1.1/README.apparmor
NSMnow-1.1.1/MANUAL

root@twsu804:/usr/local/src# cd NSMnow-1.1.1/

root@twsu804:/usr/local/src/NSMnow-1.1.1# ls
INSTALL MANUAL NSMnow-core README.apparmor templates
libs NSMnow NSMnow.log RELEASE.NOTES
LICENCE NSMnow.conf README run-init

root@twsu804:/usr/local/src/NSMnow-1.1.1# ./NSMnow -i

Allow pre-checks to install requisite packages [Y]:
[2008/12/29 22:18:05] #1 - Performing NSMnow pre-checks.
[2008/12/29 22:21:06] #1 - Pre-checks completed successully
[2008/12/29 22:21:06] #1 - Detected platform: UBUNTU
[2008/12/29 22:21:06] #1 - Action: Installing package(s).

Download Directory
Path where all downloaded files will be saved to
[./source]:

Source Directory
Path where all source tarballs will be extracted to
[./source]:

Sensor Name
A unique name given to deliniate sensors from one another
[sensor1]: twsu804a

Sensor Interface
Enter the interface that this sesnor will be monitoring
[eth0]: eth1

Configuration Path
Path to where all sensor related configuration files will be stored
[/etc/nsm]:

Sensor Data Path
Path to where all sensor captured information will be stored
[/nsm/sensor_data]:

Server Host
Hostname or IP of the server component that this sensor will connect to
[localhost]:

Server Name
A unique name given to deliniate servers from one another
[server1]:

Server Data Path
Path to where all server collected information will be stored
[/nsm/server_data]:

Server Database Name
Name of the sguil database which will store all sguil correlated information.
[sguildb]:

Server Database User
Name of the user who will have access rights to the sguil database.
[sguil]:

Server Database Password
Password of the user who will have access rights to the sguil database.
[password]: sguil

Client User
Name of the sguil client user who will have access the sguil server.
[sguil]:

Client Password
Password of the sguil client user who will have access to the sguil server.
[password]: sguil

Server Host
Hostname or IP of the server component that this client will connect to
[localhost]:
[2008/12/29 22:23:16] #1 - Installing package: mysql
[2008/12/29 22:23:16] #1 - Installing with: apt-get -y install mysql-server
[2008/12/29 22:28:22] #1 - Installing package: tcl
[2008/12/29 22:28:22] #1 - Installing with: apt-get -y install tcl8.3
itcl3 mysqltcl tcltls tcllib tcl8.3-dev iwidgets4 tclx8.4 itk3 tcl8.4 tk8.4
[2008/12/29 22:29:23] #1 - Installing package: buildessential
[2008/12/29 22:29:23] #1 - Installing with: apt-get -y install libpcre3-dev
libpcap0.8-dev build-essential
[2008/12/29 22:29:43] #1 - Installing package: snort
Download snort tarball? [Y]: y
[2008/12/29 22:39:37] #1 - Configuring with: ./configure --enable-perfprofiling
[2008/12/29 22:40:29] #1 - Compiling with: make
[2008/12/29 22:43:50] #1 - Installing with: make install
[2008/12/29 22:44:02] #1 - Installing package: barnyard2
Download barnyard2 tarball? [Y]: y
[2008/12/29 22:44:29] #1 - Configuring with: ./configure --with-tcl=/usr/lib/tcl8.3
[2008/12/29 22:44:54] #1 - Compiling with: make
[2008/12/29 22:45:10] #1 - Installing with: make install
[2008/12/29 22:45:10] #1 - Installing package: sancp
Download sancp tarball? [Y]: y
[2008/12/29 22:45:18] #1 - Compiling with: make linux
[2008/12/29 22:45:37] #1 - Installing with: cp sancp /usr/local/bin
[2008/12/29 22:45:37] #1 - Installing package: sguilsensor
Download sguil-sensor (sguil) package(s)? [Y]: y
[2008/12/29 22:46:09] #1 - Installing sguil-sensor binaries
[2008/12/29 22:46:10] #1 - Installing package: sguilclient
[2008/12/29 22:46:10] #1 - Installing sguil-client library files
[2008/12/29 22:46:10] #1 - Installing sguil-client binary
[2008/12/29 22:46:10] #1 - Installing package: sguilserver
[2008/12/29 22:46:10] #1 - Installing sguil-server library files
[2008/12/29 22:46:10] #1 - Installing sguil-server binary
[2008/12/29 22:46:10] #1 - Installing package: sguiltools
[2008/12/29 22:46:10] #1 - Installing with: apt-get -y install wireshark p0f tcpflow tcpdump
[2008/12/29 22:47:24] #1 - Configuring package: mysql
* Stopping MySQL database server mysqld [ OK ]
* Stopping MySQL database server mysqld [ OK ]
Reloading AppArmor profiles : done.
* Starting MySQL database server mysqld [ OK ]
* Checking for corrupt, not cleanly closed and upgrade needing tables.
[2008/12/29 22:48:07] #1 - Configuring package: tcl
[2008/12/29 22:48:10] #1 - Configuring package: buildessential
[2008/12/29 22:48:10] #1 - Configuring package: snort
[2008/12/29 22:48:10] #1 - Generating snort config file: /etc/nsm/twsu804a/snort.conf
[2008/12/29 22:48:11] #1 - Configuring package: barnyard2
[2008/12/29 22:48:11] #1 - Generating barnyard2 config file: /etc/nsm/twsu804a/barnyard2.conf
[2008/12/29 22:48:12] #1 - Configuring package: sancp
[2008/12/29 22:48:12] #1 - Generating sancp config file: /etc/nsm/twsu804a/sancp.conf
[2008/12/29 22:48:12] #1 - Configuring package: sguilsensor
[2008/12/29 22:48:12] #1 - Generating sensor agent config file(s)
[2008/12/29 22:48:12] #1 - Configuring package: sguilclient
[2008/12/29 22:48:12] #1 - Generating sguil-client config file: /etc/sguil/sguil.conf
[2008/12/29 22:48:12] #1 - Configuring package: sguilserver
[2008/12/29 22:48:12] #1 - Configuring AppArmor profile
[2008/12/29 22:48:12] #1 - Ensure you restart AppArmor to apply changes
[2008/12/29 22:48:12] #1 - Generating sguil-server config file: /etc/sguild/sguild.conf
[2008/12/29 22:48:13] #1 - Updating sguild init file: /etc/init.d/sguild
Copy default rules file(s)? [Y]: y
What Sensor name is to be associated with these rules [sensor1]: twsu804a
[2008/12/29 22:49:20] #1 - Creating the CA certificate
[2008/12/29 22:49:22] #1 - Creating certificate request for: server1
[2008/12/29 22:49:22] #1 - Signing server certificate for: server1
[2008/12/29 22:49:22] #1 - Adding client user "sguil" to sguil server ACL.
[2008/12/29 22:49:22] #1 - Creating database and initial user.

You will need the mysql root password.
Enter password:
[2008/12/29 22:49:29] #1 - Configuring package: sguiltools
[2008/12/29 22:49:29] #1 - Completed installing package(s) successfully.

NOTE: Snort can log in either UTC or the localtime, so firstly make sure that all machines
are synced together.Secondly, either set the timezone on all machines to UTC or set the
timezone on all machines to the same andremove the $UTC variable from the OPTIONS variable
in both /etc/init.d/snortu and /etc/init.d/snortl

I decided to comment out the $UTC variable in the two scripts. Then I started the programs.

root@twsu804:/usr/local/src/NSMnow-1.1.1# ./run-init start
Starting - sguil server (sguild) [ OK ]
Starting - sguil: sensor snort_agent (snort_agent) [ OK ]
Starting - sguil: sensor pcap_agent (pcap_agent) [ OK ]
Starting - sguil: sensor sancp_agent (sancp_agent) [ OK ]
Starting - snort: IDS mode, unified output (snort_unified) [ OK ]
* output in /nsm/sensor_data/twsu804a, /ssn_logs, /portscans
Starting - barnyard2 (barnyard2) [ OK ]
* created directory: /var/log/barnyard2
* created directory: /var/log/barnyard2/twsu804a
Starting - sancp: session logging (sancpd) [ OK ]
* output in /nsm/sensor_data/twsu804a/sancp
Starting - snort: logging mode (snort_packetlogging) [ OK ]
* output in /nsm/sensor_data/twsu804a/dailylogs/2008-12-30
* created directory: /nsm/sensor_data/twsu804a/dailylogs
* created directory: /nsm/sensor_data/twsu804a/dailylogs/2008-12-30
* disk space currently at 43%
root@twsu804:/usr/local/src/NSMnow-1.1.1#

At this point I could start a user terminal, type sguil.tk, and start using the Sguil console. The only real change I made was to alter the default fonts. I would probably consolidate the three panels into 1 as well.

Very impressive! Great work guys.


Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

9 comments:

Brad Haas said...

Richard,

I love the NSMNow script. It's the only way I've ever been able to get Sguil et al. to play nice. But I have noticed that the current version of NSMNow doesn't grab the latest version of Snort. To fix this, you have to edit NSMnow-1.1.1/libs/snort.pm , and add another entry to @SNORT_PARAMS to include the URL for the latest Snort release.

I like your books too, by the way.

Brad

Anonymous said...

Man the installation of Sguil sure has come a long way. I need to give this a try!

inuk-x said...

Richard,

Thanks for the heads up on this configuration method. I've just tested it on two systems tonight, the first no issues, the second moderate issues.

NSMNow is exactly what InstantNSM was supposed to be and will Sguil deployment so much easier.

Matteo said...

There are at least two errors installing NSMnow on Ubuntu 8.10: Snort compilation error and Sguild database creation. I have fixed these problems and uploaded two files "server_stats.c" for Snort and "create_sguildb.sql" for Sguil. Both files are linked in my post: http://www.bufferoverflow.it/2009/01/01/network-security-monitoring-server-con-ubuntu-810-e-nsmnow/

Have a good day

Anonymous said...

"Richard,

I love the NSMNow script. It's the only way I've ever been able to get Sguil et al. to play nice. But I have noticed that the current version of NSMNow doesn't grab the latest version of Snort. To fix this, you have to edit NSMnow-1.1.1/libs/snort.pm , and add another entry to @SNORT_PARAMS to include the URL for the latest Snort release."


HOW CAN I DO THAT????

lavi said...

Hi Richard,

It was very informative, thanx... well there are three hurdles i came across and being new to ubuntu, i admit they might be trivial but at the moment they're tricky enough for me... well first one is...
"either set the timezone on all machines to UTC or set the
timezone on all machines to the same andremove the $UTC variable from the OPTIONS variable
in both /etc/init.d/snortu and /etc/init.d/snortl"
both the files aren't available...

Secondly when i tried to type this command on the terminal window

root@twsu804:/usr/local/src/NSMnow-1.1.1# ./run-init startit says: lavi@ubuntu:/usr/local/src/NSMnow-1.4.0$ ./run-init start
bash: ./run-init: No such file or directory
Thirdly when i type: sguil.tk

i get this:

Couldn't determine where the sguil config file is
Looked for /home/lavi/sguil.conf and ./sguil.conf.
Error in startup script: invalid command name "DisplayUsage"
while executing
"DisplayUsage $argv0"
invoked from within
"if { ![info exists CONF_FILE] } {
# No conf file specified check the defaults
if { [file exists $env(HOME)/sguil.conf] } {
set CONF_FILE $env(..."
(file "/usr/local/bin/sguil.tk" line 2014)

lavi said...

Hi Richard, i would like to add that i'm using nsmnow-1.4.0....

Richard Bejtlich said...

Sorry, this is not a support forum for NSMNow. Please contact the developers with questions.

chami said...

hi mate, i got the same problem. but i found the error. as i remember u need to start nsm manually.
/etc/init.d/nsm start
then run sguil.tk. hope u got the answer.