One of the Holy Grails of network security monitoring is Port Independent Protocol Identification (PIPI -- lousy acronymn, but technically useful). PIPI allows inspection of protocols regardless of the port in use. PIPI has many security implications for discovery and (preferably) denial of covert channels, back doors, and other policy-violating channels. PIPI can also help network engineers better understand the legitimate use of protocols on their networks.
Some implementations exist. Last year after visiting Fidelis Security I mentioned their appliance uses port-neutral methods to identify protocols. Sourcefire's RNA also does PIPI. The Linux-only Application Layer Packet Classifier for Linux (L7-filter) and IPP2P projects use signatures to discover protocols on arbitrary ports. I'd like to hear of other approaches.
Today, thanks to geek00l, I read the paper Dynamic Application-Layer Protocol Analysis
for Network Intrusion Detection by an all-star team from Technische Universität München and Berkeley's ICSI Center for Internet Research. From the abstract:
In this paper, we discuss the design and implementation of a NIDS extension to perform dynamic application-layer protocol analysis. For each connection, the system first identifies potential protocols in use and then activates appropriate analyzers to verify the decision and extract higher-level semantics. We demonstrate the power of our enhancement with three examples: reliable detection of applications not using their standard ports, payload inspection of FTP data transfers, and detection of IRC-based botnet clients and servers.
Even better, their implementation is scheduled for integration in the next release of Bro, perhaps next month.
On a related PIPI note, in the future I expect we will not create firewall policies using port numbers as a major component. A security policy enforcement system might instead allow an administrator to implement a policy like "deny all outbound HTTP except [real] HTTP on port 80 and HTTPS on port 443." In other words, network (i.e., traffic-centric) security policy will be decoupled from ports and instead focus on applications and data.