More Snort and Sguil Tuning

Let's assume you built a new Sguil sensor and have tuned Snort using advice in my Tuning Snort article. What I like to do next is wait a day or so and then run the following query to look for problematic alert types.

mysql> select count(*) as total, event.signature from event where
 event.status=0 group by event.signature order by total desc;
+-------+------------------------------------------------------------------------+
| total | signature                                                              |
+-------+------------------------------------------------------------------------+
| 43416 | SHELLCODE x86 NOOP                                                     | 
|  4145 | POLICY SMTP relaying denied                                            | 
|  3394 | WEB-MISC PCT Client_Hello overflow attempt                             | 
|  2148 | WEB-CGI calendar access                                                | 
|  1645 | tag: Tagged Packet                                                     | 
|  1267 | SHELLCODE x86 stealth NOOP                                             | 
|  1031 | WEB-MISC weblogic/tomcat .jsp view source attempt                      | 
|   817 | http_inspect: OVERSIZE REQUEST-URI DIRECTORY                           | 
|   683 | INFO web bug 0x0 gif attempt                                           | 
|   491 | SHELLCODE x86 0x90 unicode NOOP                                        | 
|   175 | ftp_pp: Invalid FTP command                                            | 
|   128 | WEB-IIS view source via translate header                               | 
|   126 | WEB-MISC robots.txt access                                             | 
|    88 | Snort Alert [1:5856:0]                                                 | 
|    74 | SHELLCODE x86 inc ebx NOOP                                             | 
|    71 | portscan: Open Port                                                    | 
|    47 | WEB-MISC intranet access                                               |
|    28 | MISC MS Terminal server request                                        | 
|    28 | WEB-FRONTPAGE /_vti_bin/ access                                        | 
|    27 | portscan: TCP Portsweep                                                | 
|    21 | WEB-MISC SSLv2 Client_Hello with pad Challenge Length overflow attempt | 
|    19 | ATTACK-RESPONSES 403 Forbidden                                         | 
|    16 | http_inspect: BARE BYTE UNICODE ENCODING                               | 
|    13 | Snort Alert [1:6031:0]                                                 | 
|    12 | ATTACK-RESPONSES directory listing                                     | 
|    12 | SHELLCODE x86 0xEB0C NOOP                                              | 
|     5 | Snort Alert [1:6033:0]                                                 | 
|     4 | portscan: TCP Portscan                                                 |
|     4 | Snort Alert [1:5718:0]                                                 | 
|     3 | EXPLOIT gobbles SSH exploit attempt                                    | 
|     3 | MISC MS Terminal server request RDP                                    | 
|     2 | MS-SQL sp_start_job - program execution                                | 
|     2 | WEB-IIS WebDAV file lock attempt                                       | 
|     1 | WEB-FRONTPAGE posting                                                  | 
|     1 | ATTACK-RESPONSES Invalid URL                                           | 
|     1 | WEB-MISC Chunked-Encoding transfer attempt                             | 
|     1 | MS-SQL xp_reg* - registry access                                       | 
|     1 | WEB-FRONTPAGE rad fp30reg.dll access                                   | 
|     1 | WEB-MISC WebDAV search access                                          | 
|     1 | NEW POLICY RDP attempted Administrator connection request              | 
+-------+------------------------------------------------------------------------+
40 rows in set (2.40 sec)

Wow, that first line is popular. Let's break these out by source IP. (Thanks to VictorJ for catching a dumb error in my original syntax.)

mysql> select count(*) as total, event.signature, INET_NTOA(event.src_ip)
 from event where event.status=0 and event.signature='SHELLCODE x86 NOOP'
 group by event.signature, event.src_ip order by total desc;
+-------+--------------------+-------------------------+
| total | signature          | INET_NTOA(event.src_ip) |
+-------+--------------------+-------------------------+
| 43002 | SHELLCODE x86 NOOP | 10.20.30.14             | 
|   334 | SHELLCODE x86 NOOP |   6.16.237.132          | 
|    46 | SHELLCODE x86 NOOP |   6.16.237.140          | 
|    18 | SHELLCODE x86 NOOP | 192.168.0.14            | 
|    11 | SHELLCODE x86 NOOP |   3.87.180.190          | 
|     3 | SHELLCODE x86 NOOP |   6.16.237.138          | 
|     2 | SHELLCODE x86 NOOP | 192.168.1.197           | 
+-------+--------------------+-------------------------+
7 rows in set (0.64 sec)

10.20.30.14 is the big offending source IP. Let's see the destination IPs.

mysql> select count(*) as total, event.signature, INET_NTOA(event.src_ip), 
 INET_NTOA(event.dst_ip) from event where event.status=0 and 
 event.signature='SHELLCODE x86 NOOP' and event.src_ip=INET_ATON('10.20.30.14')
 group by event.signature, event.src_ip, event.dst_ip order by total desc;
+-------+--------------------+-------------------------+-------------------------+
| total | signature          | INET_NTOA(event.src_ip) | INET_NTOA(event.dst_ip) |
+-------+--------------------+-------------------------+-------------------------+
| 43002 | SHELLCODE x86 NOOP | 10.20.30.14             |   6.16.237.137          | 
+-------+--------------------+-------------------------+-------------------------+
1 row in set (0.94 sec)

It looks like there are 43002 SHELLCODE x86 NOOP from 10.20.30.14 to 6.16.237.137. Is that really true though? We can validate these findings by looking for alerts where 6.16.237.137 is not the destination IP.

mysql> select count(*) as total, event.signature, INET_NTOA(event.src_ip), 
 INET_NTOA(event.dst_ip) from event where event.status=0 and
 event.signature='SHELLCODE x86 NOOP' and event.src_ip=INET_ATON('10.20.30.14') and 
 event.dst_ip!=INET_ATON('6.16.237.137') group by event.signature, event.src_ip,
 event.dst_ip order by total desc;
Empty set (1.20 sec)

With an empty response, we confirm there are 43002 SHELLCODE x86 NOOP from 10.20.30.14 to 6.16.237.137.

I have two options here. First, I could simply delete all of these items from the database. I doubt 43002 SHELLCODE alerts from an internal IP address are malicious. On the other hand, I might want to look at them.

The problem with inspecting them is that they are currently in Sguil's inspection queue. If I fire up Sguil it's going to load 43002 alerts into the client. While they will be aggregated, it will take a long time for that to happen.

In cases like these I prefer to mark the events as "NA" and then inspect a sample later.

Before running the following, kill sguild. You should never update the database while Sguil is trying to make sense of it.

mysql> UPDATE event SET status=1, last_modified='2006-08-28 15:45:00', last_uid='sguil'
 where event.status=0 and event.signature='SHELLCODE x86 NOOP' and 
 event.src_ip=INET_ATON('10.20.30.14');
Query OK, 43002 rows affected, 43002 warnings (5.91 sec)
Rows matched: 43002  Changed: 43002  Warnings: 43002

You can repeat this process for the other large event count alerts, or just restart sguild and fire up the Sguil client to deal with them manually.

How can we prevent such a huge number of alerts from occurring again?

My original article mentioned "Thresholding and Suppression." The SHELLCODE x86 NOOP alert might be a good candidate for thresholding.

A look in sid-msg map shows two sids:

$ grep "SHELLCODE x86 NOOP" sid-msg.map 
648 || SHELLCODE x86 NOOP || arachnids,181
1394 || SHELLCODE x86 NOOP

A look in the .rules files shows two rules.

grep "SHELLCODE x86 NOOP" *.rules
shellcode.rules:alert ip $EXTERNAL_NET $SHELLCODE_PORTS -> $HOME_NET any
 (msg:"SHELLCODE x86 NOOP"; content:"|90 90 90 90 90 90 90 90 90 90 90 90 90 90|"; 
 reference:arachnids,181; classtype:shellcode-detect; sid:648; rev:8;)
shellcode.rules:alert ip $EXTERNAL_NET $SHELLCODE_PORTS -> $HOME_NET any 
 (msg:"SHELLCODE x86 NOOP"; content:"aaaaaaaaaaaaaaaaaaaaa";
 classtype:shellcode-detect; sid:1394; rev:5;)

I add the following to threshold.conf to limit Snort to generate one alert per 60 seconds for each source IP causing SHELLCODE x86 NOOP alerts.

threshold gen_id 1, sig_id 648, type limit, track by_src, count 1, seconds 60
threshold gen_id 1, sig_id 1394, type limit, track by_src, count 1, seconds 60

Remember to enable your local threshold.conf in snort.conf:

include /usr/local/etc/nsm/threshold.conf

After restarting Snort, I can expect a lower number of alerts should SHELLCODE x86 NOOP get out of hand again. When Snort starts you'll see the sig-id of your new threshold rule applied.

| gen-id=1      sig-id=648        type=Limit     tracking=src count=1   seconds=60
...edited...
| gen-id=1      sig-id=1394       type=Limit     tracking=src count=1   seconds=60

That's one item for tuning... expect others in the future.