Recovering from Suricata Gone Wild
Recently I tried interacting with one of my lab Security Onion sensors running the Suricata IDS. I found the Sguil server was taking a really long time to offer services on port 7734 TCP. Since I hadn't worked with this lab system in a while, I guessed that there might be too many uncategorized events in the Sguil database. I dusted off an old blog post titled More Snort and Sguil Tuning from 2006 and took a look at the system.
First I stopped the NSM applications on the server.
sudo service nsm stop Stopping: securityonion * stopping: sguil server [ OK ] Stopping: HIDS * stopping: ossec_agent (sguil) [ OK ] Stopping: Bro stopping ds61so-eth1-1 ... stopping proxy ... stopping manager ... Stopping: ds61so-eth1 * stopping: netsniff-ng (full packet data) [ OK ] * stopping: pcap_agent (sguil) [ OK ] * stopping: snort_agent (sguil) [ OK ] * stopping: suricata (alert data) [ OK ] * stopping: barnyard2 (spooler, unified2 format) [ OK ] * stopping: prads (sessions/assets) [ OK ] * stopping: sancp_agent (sguil) [ OK ] * stopping: pads_agent (sguil) [ OK ] * stopping: argus [ OK ] * stopping: http_agent (sguil)Next I ran a query to look for the top uncategorized events.
$ mysql -uroot Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1639 Server version: 5.5.29-0ubuntu0.12.04.1 (Ubuntu) Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> use securityonion_db; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20; +---------+----------------------------------------------------------------------------------+ | count | signature | +---------+----------------------------------------------------------------------------------+ | 2299160 | SURICATA STREAM Packet with invalid ack | | 2298505 | SURICATA STREAM ESTABLISHED invalid ack | | 1777530 | SURICATA STREAM ESTABLISHED packet out of window | | 38700 | SURICATA STREAM ESTABLISHED retransmission packet before last ack | | 24181 | SURICATA STREAM TIMEWAIT ACK with wrong seq | | 5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management | | 3160 | SURICATA STREAM Last ACK with wrong seq | | 753 | ET POLICY Dropbox.com Offsite File Backup in Use | | 637 | SURICATA HTTP unknown error | | 626 | SURICATA STREAM SHUTDOWN RST invalid ack | | 505 | SURICATA STREAM FIN1 FIN with wrong seq | | 494 | SURICATA HTTP request field too long | | 448 | ET POLICY PE EXE or DLL Windows file download | | 315 | ET RBN Known Malvertiser IP (22) | | 270 | ET POLICY iTunes User Agent | | 266 | SURICATA STREAM CLOSEWAIT ACK out of window | | 237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging) | | 219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard | | 217 | SURICATA STREAM 3way handshake with ack in wrong dir | | 151 | SURICATA STREAM FIN2 FIN with wrong seq | +---------+----------------------------------------------------------------------------------+ 20 rows in set (15.24 sec)Wow, that's a lot of SURICATA STREAM events. I need to categorize them as non-issues to recover the Sguil server.
mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE 'SURICATA STREAM%'; Query OK, 6443375 rows affected, 65535 warnings (3 min 4.89 sec) Rows matched: 6443375 Changed: 6443375 Warnings: 6443375Let's see what the database thinks now.
mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20; +------+-----------------------------------------------------------------------------------------+ | cnt | signature | +------+-----------------------------------------------------------------------------------------+ | 5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management | | 753 | ET POLICY Dropbox.com Offsite File Backup in Use | | 637 | SURICATA HTTP unknown error | | 494 | SURICATA HTTP request field too long | | 448 | ET POLICY PE EXE or DLL Windows file download | | 315 | ET RBN Known Malvertiser IP (22) | | 270 | ET POLICY iTunes User Agent | | 237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging) | | 219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard | | 133 | ET INFO PDF Using CCITTFax Filter | | 106 | ET POLICY Pandora Usage | | 97 | ET CHAT Facebook Chat (buddy list) | | 93 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard via GET | | 58 | ET POLICY Internal Host Retrieving External IP via whatismyip.com - Possible Infection | | 41 | PADS New Asset - ssl TLS 1.0 Client Hello | | 39 | SURICATA HTTP response header invalid | | 39 | ET CURRENT_EVENTS Exploit Kit Delivering Compressed Flash Content to Client | | 36 | ET POLICY Python-urllib/ Suspicious User Agent | | 36 | ET MALWARE Possible Windows executable sent when remote host claims to send a Text File | | 28 | ET POLICY Http Client Body contains pw= in cleartext | +------+-----------------------------------------------------------------------------------------+ 20 rows in set (0.03 sec)That's much better.
Before restarting the NSM services, I edit the autocat.conf file to add the following.
none||ANY||ANY||ANY||ANY||ANY||ANY||%%REGEXP%%^SURICATA STREAM||1This will auto-categorize any SURICATA STREAM alerts as non-issues. I want to keep adding events to the database for testing purposes, but I don't want to see them in the console.
Now I restart the NSM services.
sudo service nsm start Starting: securityonion * starting: sguil server [ OK ] Starting: HIDS * starting: ossec_agent (sguil) [ OK ] Starting: Bro starting manager ... starting proxy ... starting ds61so-eth1-1 ... Starting: ds61so-eth1 * starting: netsniff-ng (full packet data) [ OK ] * starting: pcap_agent (sguil) [ OK ] * starting: snort_agent (sguil) [ OK ] * starting: suricata (alert data) [ OK ] * starting: barnyard2 (spooler, unified2 format) [ OK ] * starting: prads (sessions/assets) [ OK ] * starting: pads_agent (sguil) [ OK ] * starting: sancp_agent (sguil) [ OK ] * starting: argus [ OK ] * starting: http_agent (sguil) [ OK ] * disk space currently at 22%I check to see if port 7734 TCP is listening.
sudo netstat -natup | grep 7734 tcp 0 0 0.0.0.0:7734 0.0.0.0:* LISTEN 10729/tclshNow the Sguil server is listening. I can connect with a Sguil client, even the 64 bit Windows .exe that I just found this morning. Check it out at sourceforge.net/projects/sguil/
Comments
IT engineers
part of this line is missing in your article:
mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature
could you pleas complete it?
thanks!
mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE 'SURICATA STREAM%';
Cheers!