Sunday, February 24, 2013

Recovering from Suricata Gone Wild

Recently I tried interacting with one of my lab Security Onion sensors running the Suricata IDS. I found the Sguil server was taking a really long time to offer services on port 7734 TCP. Since I hadn't worked with this lab system in a while, I guessed that there might be too many uncategorized events in the Sguil database. I dusted off an old blog post titled More Snort and Sguil Tuning from 2006 and took a look at the system.

First I stopped the NSM applications on the server.

sudo service nsm stop
Stopping: securityonion
  * stopping: sguil server                                [  OK  ]
Stopping: HIDS
  * stopping: ossec_agent (sguil)                         [  OK  ]
Stopping: Bro
stopping ds61so-eth1-1 ...
stopping proxy ...
stopping manager ...
Stopping: ds61so-eth1
  * stopping: netsniff-ng (full packet data)              [  OK  ]
  * stopping: pcap_agent (sguil)                          [  OK  ]
  * stopping: snort_agent (sguil)                         [  OK  ]
  * stopping: suricata (alert data)                       [  OK  ]
  * stopping: barnyard2 (spooler, unified2 format)        [  OK  ]
  * stopping: prads (sessions/assets)                     [  OK  ]
  * stopping: sancp_agent (sguil)                         [  OK  ]
  * stopping: pads_agent (sguil)                          [  OK  ]
  * stopping: argus                                       [  OK  ]
  * stopping: http_agent (sguil)                      
Next I ran a query to look for the top uncategorized events.
$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1639
Server version: 5.5.29-0ubuntu0.12.04.1 (Ubuntu)

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use securityonion_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
| count   | signature                                                                        |
| 2299160 | SURICATA STREAM Packet with invalid ack                                          |
| 2298505 | SURICATA STREAM ESTABLISHED invalid ack                                          |
| 1777530 | SURICATA STREAM ESTABLISHED packet out of window                                 |
|   38700 | SURICATA STREAM ESTABLISHED retransmission packet before last ack                |
|   24181 | SURICATA STREAM TIMEWAIT ACK with wrong seq                                      |
|    5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management |
|    3160 | SURICATA STREAM Last ACK with wrong seq                                          |
|     753 | ET POLICY Offsite File Backup in Use                                 |
|     637 | SURICATA HTTP unknown error                                                      |
|     626 | SURICATA STREAM SHUTDOWN RST invalid ack                                         |
|     505 | SURICATA STREAM FIN1 FIN with wrong seq                                          |
|     494 | SURICATA HTTP request field too long                                             |
|     448 | ET POLICY PE EXE or DLL Windows file download                                    |
|     315 | ET RBN Known Malvertiser IP (22)                                                 |
|     270 | ET POLICY iTunes User Agent                                                      |
|     266 | SURICATA STREAM CLOSEWAIT ACK out of window                                      |
|     237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                   |
|     219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                      |
|     217 | SURICATA STREAM 3way handshake with ack in wrong dir                             |
|     151 | SURICATA STREAM FIN2 FIN with wrong seq                                          |
20 rows in set (15.24 sec)
Wow, that's a lot of SURICATA STREAM events. I need to categorize them as non-issues to recover the Sguil server.

mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE 'SURICATA STREAM%';
Query OK, 6443375 rows affected, 65535 warnings (3 min 4.89 sec)
Rows matched: 6443375  Changed: 6443375  Warnings: 6443375
Let's see what the database thinks now.
mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
| cnt  | signature                                                                               |
| 5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management        |
|  753 | ET POLICY Offsite File Backup in Use                                        |
|  637 | SURICATA HTTP unknown error                                                             |
|  494 | SURICATA HTTP request field too long                                                    |
|  448 | ET POLICY PE EXE or DLL Windows file download                                           |
|  315 | ET RBN Known Malvertiser IP (22)                                                        |
|  270 | ET POLICY iTunes User Agent                                                             |
|  237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                          |
|  219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                             |
|  133 | ET INFO PDF Using CCITTFax Filter                                                       |
|  106 | ET POLICY Pandora Usage                                                                 |
|   97 | ET CHAT Facebook Chat (buddy list)                                                      |
|   93 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard via GET                     |
|   58 | ET POLICY Internal Host Retrieving External IP via - Possible Infection  |
|   41 | PADS New Asset - ssl TLS 1.0 Client Hello                                               |
|   39 | SURICATA HTTP response header invalid                                                   |
|   39 | ET CURRENT_EVENTS Exploit Kit Delivering Compressed Flash Content to Client             |
|   36 | ET POLICY Python-urllib/ Suspicious User Agent                                          |
|   36 | ET MALWARE Possible Windows executable sent when remote host claims to send a Text File |
|   28 | ET POLICY Http Client Body contains pw= in cleartext                                    |
20 rows in set (0.03 sec)
That's much better.

Before restarting the NSM services, I edit the autocat.conf file to add the following.

This will auto-categorize any SURICATA STREAM alerts as non-issues. I want to keep adding events to the database for testing purposes, but I don't want to see them in the console.

Now I restart the NSM services.

sudo service nsm start
Starting: securityonion
  * starting: sguil server                                                                [  OK  ]
Starting: HIDS
  * starting: ossec_agent (sguil)                                                         [  OK  ]
Starting: Bro
starting manager ...
starting proxy ...
starting ds61so-eth1-1 ...
Starting: ds61so-eth1
  * starting: netsniff-ng (full packet data)                                              [  OK  ]
  * starting: pcap_agent (sguil)                                                          [  OK  ]
  * starting: snort_agent (sguil)                                                         [  OK  ]
  * starting: suricata (alert data)                                                       [  OK  ]
  * starting: barnyard2 (spooler, unified2 format)                                        [  OK  ]
  * starting: prads (sessions/assets)                                                     [  OK  ]
  * starting: pads_agent (sguil)                                                          [  OK  ]
  * starting: sancp_agent (sguil)                                                         [  OK  ]
  * starting: argus                                                                       [  OK  ]
  * starting: http_agent (sguil)                                                          [  OK  ]
  * disk space currently at 22%
I check to see if port 7734 TCP is listening.
sudo netstat -natup | grep 7734
tcp        0      0  *               LISTEN      10729/tclsh
Now the Sguil server is listening. I can connect with a Sguil client, even the 64 bit Windows .exe that I just found this morning. Check it out at

Friday, February 22, 2013

Using Bro to Log SSL Certificates

I remember using an older version of Bro to log SSL certificates extracted from the wire. The version shipped with Security Onion is new and that functionality doesn't appear to be enabled by default. I asked Seth Hall about this capability, and he told me how to get Bro to log all SSL certs that it sees.

Edit /opt/bro/share/bro/site/local.bro to contain the changes as shown below.

diff -u /opt/bro/share/bro/site/local.bro.orig /opt/bro/share/bro/site/local.bro
--- /opt/bro/share/bro/site/local.bro.orig      2013-02-23 01:54:53.291457193 +0000
+++ /opt/bro/share/bro/site/local.bro   2013-02-23 01:55:16.151996423 +0000
@@ -56,6 +56,10 @@
 # This script enables SSL/TLS certificate validation.
 @load protocols/ssl/validate-certs

+# Log certs per Seth
+@load protocols/ssl/extract-certs-pem
+redef SSL::extract_certs_pem = ALL_HOSTS;
 # If you have libGeoIP support built in, do some geographic detections and
 # logging for SSH traffic.
 @load protocols/ssh/geo-data
Restart Bro.
~# broctl

Welcome to BroControl 1.1

Type "help" for help.

[BroControl] > install
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating standalone-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... done.
[BroControl] > status
Name       Type       Host       Status        Pid    Peers  Started            
bro        standalone localhost  running       3042   0      17 Feb 13:22:42
[BroControl] > restart
stopping ...
stopping bro ...
starting ...
starting bro ...
[BroControl] > exit

After restarting you will have a new log for all SSL certs:

ls -al certs-remote.pem
-rw-r--r-- 1 root root 31907 Feb 23 02:05 certs-remote.pem

New certs are appended to the file as Bro sees them. A cert looks like this:

OpenSSL can read them one at a time, e.g.:
openssl x509 -in certs-remote.pem -text -noout
        Version: 3 (0x2)
        Serial Number:
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=Terms of use at (c)06, CN=VeriSign Class 3 Extended Validation SSL CA
            Not Before: Feb 29 00:00:00 2012 GMT
            Not After : Feb 28 23:59:59 2013 GMT
        Subject: Organization/serialNumber=2927442, C=US/postalCode=60603, ST=Illinois, L=Chicago/street=135 S La Salle St, O=Bank of America Corporation, OU=Network Infrastructure,
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
            X509v3 Subject Key Identifier:
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 CRL Distribution Points:

                Full Name:

            X509v3 Certificate Policies:
                Policy: 2.16.840.1.113733.

            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Authority Key Identifier:

            Authority Information Access:
                OCSP - URI:
                CA Issuers - URI:

    Signature Algorithm: sha1WithRSAEncryption
Since each cert has a standard header and footer, I bet someone could write a parser to extract each cert from the certs-remote.pem file to separate files. Thanks a lot Seth!

Monday, February 11, 2013

Practical Network Security Monitoring Book on Schedule

First the good news: my new book Practical Network Security Monitoring is on track, and you can pre-order with a 30% discount using code NSM101.

I'm about 1/3 of the way through writing the book. Since I announced the project last month, I've submitted chapters 1, 2, and 3. They are in various stages of review by No Starch editors and my technical editors. I seem to be writing more than I expected, despite trying to keep the book at an introductory level. I find that I want to communicate the topic sufficiently to make my point, but I try to avoid going too deeply into related areas.

I'm also encountering situations where I have to promise to explain some concepts later, rather than explain everything immediately. I believe once I get the first chapter ironed out with the editor, the rest will be easier to digest. I'm taking a fairly methodical approach (imagine that), so once the foundation in chapter 1 is done the rest is more straightforward.

I'm keeping a fairly aggressive schedule. Basically I have to write a chapter each week, get it to my technical editors, and then spend additional time working with No Starch to get the text legible and ready for print. All of this is happening in parallel in order to have the books in print by Black Hat. That means the text must done by the first week in April. My family is helping me stay on track by giving me time and space to write, especially on the weekends. Thank you!

When working on the examples, I've been very pleased with the performance of VMWare Workstation 9. I have one copy installed on Windows 7, where I write with Word. I have a second copy installed on Ubuntu Server, where it acts like a "VMWare Server." I used to run a real ESXi server on server-class hardware. Now, to save electricity and to more tailor my computer power to my requirements, I run a Shuttle DS61 with a Core i5-3450S 2.80GHz CPU, 16 GB RAM, 750 GB HDD, and two onboard NICs. The two NICs are really awesome in a device this small -- 190(L) x 165(W) x 43(H) mm. With two NICs, I can devote one for management and one for network traffic collection and interpretation. I use a Net Optics Dual Port Aggregator Tap for access to the wire.

I use VMWare Workstation this way. I run a Linux VM on Workstation on my Windows 7 laptop. I connect via Workstation to the Workstation instance on Ubuntu on the DS61. Then I create whatever VMs I need on the DS61. For example, I created a Security Onion server and sensor to test that setup. With 16 GB RAM, I have plenty of RAM for both, plus another VM that I'm running as my "production" Security Onion sensor for the lab network.

Writing is going well, despite the fact that I last wrote a book in 2005. I promised my youngest daughter, who wasn't born until 2006, that this new book is for her. If you have any questions on the writing process, please post them here or ask me on Twitter.