Problems with Signature and Protocol Anomaly Detection Methods

I often have to describe what differentiates network security monitoring from traditional intrusion detection. Now that "intrusion prevention" is all the rage (when was preventing intrusions not popular?), I have to think in terms of blocking traffic that is potentially suspicious or malicious. Recently while performing network security monitoring, I received the following Snort alert, as reported by Sguil. I had never seen an alert like this before. Here is a text-based representation of what Sguil displayed in its GUI:

Count:1 Event#3.57183 2005-03-31 23:24:14
FTP invalid MODE
1.30.163.130 -> 2.35.23.101
IPVer=4 hlen=5 tos=0 dlen=48 ID=42411 flags=2
 offset=0 ttl=117 chksum=0
Protocol: 6 sport=9542 -> dport=21

Seq=3804195312 Ack=735521810 Off=5 Res=0
 Flags=***AP*** Win=63842 urp=23116 chksum=0
Payload:
4D 4F 44 45 20 5A 0D 0A                         MODE Z..

That looks odd. What is "MODE Z"? Here is the Snort rule that produced this alert:

alert tcp $EXTERNAL_NET any -> $HOME_NET 21
 (msg:"FTP invalid MODE"; flow:to_server,established;
 content:"MODE"; nocase; pcre:"/^MODE\s+[^ABSC]{1}/msi";
 classtype:protocol-command-decode; sid:1623; rev:6;)

I am no PCRE expert, but I can see that "MODE Z" will trigger this rule. The Snort rule documentation doesn't shed any real light on the subject. A Google search result reveals that "MODE Z" enables dynamic data compression via the zlib library.

If we were using a Web-based alert browser, this would be the end of the line. There would be no other data to review other than the packet that tripped the rule and the rule itself. How did the Web server respond? Did it respond at all? What FTP server is running? These and other questions remain unanswered when you use an IDS whose chief purpose in life is to generate alert data.

Fortunately, we're using Sguil. We recognize there's more to life than alert data; there's session, full content, and statistical data. We can generate a transcript of their entire conversation between the FTP client and server if we collected the full content data between both parties. Here is that transcript:

Sensor Name: bej-sensor-fl
Timestamp: 2005-03-31 23:24:14
Connection ID: .bej-sensor-fl_57183
Src IP:  1.30.163.130 (mail.example.com)
Dst IP:  2.35.23.101 (Unknown)
Src Port:  9542
Dst Port:  21
OS Fingerprint: 1.30.163.130:9542 - Windows XP Pro SP1, 2000 SP3 
OS Fingerprint:   -> 2.35.23.101:21 (distance 11, link: ethernet/modem)

DST: 220 Serv-U FTP Server v5.1 for WinSock ready...
DST: 
SRC: USER example
SRC: 
DST: 331 User name okay, need password.
DST: 
SRC: PASS example
SRC: 
DST: 230 User logged in, proceed.
DST: 
SRC: SYST
SRC: 
DST: 215 UNIX Type: L8
DST: 
SRC: FEAT
SRC: 
DST: 211-Extension supported
DST: 
DST: AUTH TLS
DST: SSCN
DST: PBSZ
DST: PROT
DST: CCC
DST: CLNT
DST: MDTM
DST: MDTM YYYYMMDDHHMMSS[+-TZ];filename
DST: SIZE
DST: SITE PSWD;EXEC;SET;INDEX;ZONE;CHMOD;MSG
DST: REST STREAM
DST: XCRC filename;start;end
DST: MODE Z
DST: 211 End
DST: 
SRC: CLNT SmartFTP 1.1.984
SRC: 
DST: 200 Noted.
DST: 
SRC: PWD
SRC: 
DST: 257 "/" is current directory.
DST: 
SRC: MODE Z
SRC: 
DST: 200 MODE Z ok.
DST: 
SRC: PASV
SRC: 
DST: 227 Entering Passive Mode (2,35,23,101,9,24)
DST: 
SRC: LIST -aLT
SRC: 
DST: 150 Opening ASCII mode data connection for /bin/ls.
DST: 
DST: 226 Transfer complete.
DST: 
SRC: TYPE I
SRC: 
DST: 200 Type set to I.
DST: 
SRC: PASV
SRC: 
DST: 227 Entering Passive Mode (2,35,23,101,9,25)
DST: 
SRC: STOR test.txt.asc
SRC: 
DST: 150 Opening BINARY mode data connection for test.txt.asc.
DST: 
DST: 226 Transfer complete.
SRC: QUIT
SRC: 
DST: 221 Goodbye!
DST: 

Based on the transcript, we see the Serv-U FTP Server v5.1 for WinSock offer MODE Z, and the client decides to use it before transferring several files. There is nothing suspicious or malicious about this exchange. It is completely within business norms.

This example highlights the use of a signature to implement protocol anomaly detection. The signature provided a "white list" of acceptable MODEs. These MODEs were believed to represent normal use of the FTP protocol. Deviations from these norms were thought to be anomalies that indicate intrusion.

This is a fine approach to detecting intrusions, but it highlights a problem that I constantly debate with security developers. I contend that the Internet and its protocols are too dynamic to code into highly reliable systems. Application developers will forever be ahead of security developers. Application developers will always be crafting new protocols or bending and breaking old protocols. Security developers will constantly try to keep redefining what is "normal" and "abnormal" to enforce access control, detect intrusions, and so on.

The consequences of firing an alert for the MODE Z case are minimal when an IDS is involved. If the IDS were replaced by an inline, so-called "IPS," it may have dropped the traffic. The layer 7 firewall -- sorry, "IPS" -- might believe MODE Z is an attack, thereby denying legitimate business functions. Sure, you can "tune it," but there's another application developer out there creating another twist on what the security developer considers normal. So, in addition to the never-ending battle with intruders, security staff face a never-ending battle with developers!

What is my answer? Block what you can, as smartly as you can, and then keep track of everything else via a network audit system. Content-neutral systems store session and full content data to provide network audit. If you ever forget the network audit piece of the IDS/"IPS" puzzle, and concentrate on alert data, you will lose.