Monday, August 28, 2006

Network Forensics with NetWitness

Ten days ago I had the privilege of attending a day of product training for NetWitness. NetWitness is a real network forensics tool produced by a company of the same name. Anyone who's read my books or attended my training knows I am a big fan of open source tools. NetWitness, however, is built to facilitate investigating network traffic.

It's important to differentiate between packet collectors, protocol analyzers, and network forensics tools. Dumpcap is the prototypical packet collector. Simpler than Tcpdump and much simpler than Tshark, all Dumpcap does is record packets. It's especially suited for this role, however, because it offers native trace rotation capabilities.

Tcpdump, Tshark, and Wireshark are all protocol analyzers. Yes, Tcpdump is a protocol analyzer, although it is not as robust as Tshark or Wireshark. Protocol analyzers are suited for packet-centric inspection. For example, I used Wireshark extensively while learning about 802.11 traffic. Protocol analyzers are also helpful for network troubleshooting, with varying degrees of automated analysis. Commercial protocol analyzers are especially robust in this regard. Protocol analyzers often feature tools for rebuilding TCP sessions, but that is usually the extent of those features. Protocol analyzers also permit searching traffic, but the analyst generally must have a good grasp of what he is looking for and how to get that idea across to the protocol analyzer.

Network forensics tools are not packet-centric; they are data centric. NetWitness, for example, cares less about the underlying packets and more about the data they contain. The partial screen capture (original here) hints at NetWitness' depiction of files, accounts, and email addresses recognized in a network trace.

Investigators don't (necessarily) look at packets when they use NetWitness for network forensics. Rather, they look for useful data. One investigation might require finding all information related to a specific username. That username (or a portion of it) would be searchable in email, instant messaging, Web logins, documents, and so forth. While you could do some manual searching with tools like Ngrep or Flowgrep, NetWitness is built around discovering information and is well-suited for this process of discovery.

When you find results, they are presented within the context of the session in which they were contained. NetWitness rebuilds the session and presents it in human-friendly format. If they subject viewed an email, you will see the email. If he visited a Web page, you will see the Web page. There are limitations to this model, such as a browser displaying cache graphics instead of requesting them on the wire. However, this sort of model works very well for forensic analysts.

This second partial screen capture (original here) demonstrates two other powerful NetWitness features. First, NetWitness represents traffic using a noun-verb-adjective-like language. When you see the mention of "GET", for example, it's not necessarily an HTTP get. GET actions include FTP retrievals and other actions where a subject acquires data. This meta-language simplifies investigations by letting the analyst look for actions and not for specific protocol activities.

Second, NetWitness performs port-agnostic protocol identification. When you see "HTTP" it doesn't just mean port 80. HTTP is identified by inspecting traffic and looking for the HTTP protocol. This is an important mechanism for finding back doors and covert channels. Obviously encryption will hamper this feature, but port-agnostic protocol identification is a must-have for forensics tools.

I could say quite a bit about NetWitness, but I hope you've gotten the idea that it's a powerful tool. In case you are wondering, I am not being compensated for this review. I did get to attend the training class for free. I am also not selling NetWitness to anyone. The purpose of this review is to share insights on this tool, and to keep those of us with ties to the open source world aware of applications outside of it.

I am open to hearing from NetWitness' main competitors, Niksun and Sandstorm, if they'd like to comment. Sandstorm's NetIntercept holds a special place in my heart, since it runs on FreeBSD. NetWitness is a Windows-based product. (In production I recommend capturing traffic with a program like Dumpcap or Tcpdump, and then analyzing it in NetWitness.)

9 comments:

Anonymous said...

Richard,

I believe Niksun's NetDetector/NetVCR runs on FreeBSD and I know it uses Snort for IDS capabilities. Your screenshots from NetWitness make it look more user-friendly and intuitive than Niksun's product, which I have used in the past.

nr

The Netlocksmith said...

I've used both Niksun NetDetector and Netwitness extensively, and I don't really think of them as competitors. They do different things and are thus each is better at something different.

The NetDetector acts in most ways like a conventional sniffer/protocol analyzer. What makes it a useful for forensics is that it's an appliance with huge storage space designed to record every byte flowing over the wire (or a subset of your choosing) over extended periods of time, and then allow searches of that dataset later. It's sort of the network equivalent of a disk forensics tool like Encase, providing a byte-for-byte identical replica of the original with customizable browsing, searching and parsing. The searches can be slow, however, if you have a lot of capture data to search through.

As you explained, Netwitness focuses on doing a quick extraction of a few useful data fields and presenting them in an easy-to-use (even by non-techies), at-a-glance format. It's VERY fast, crunching gigs of data in seconds. It's pretty useless for packet analysis, however; it's simply not designed for that.

If I wanted to investigate a hacker attack, I'd want to use the NetDetector, since so many exploits involve invalid or non-standard network data that couldn't be parsed properly by Netwitness. I'd need to see exactly what the hacker transmitted in the order they sent it, and if I wanted to use it as evidence in court, I'd need that byte-for-byte identical, forensically-sound record.

If, on the other hand, I had ordinary traffic consisting of valid data and standard protocols, and I just wanted to quickly review it to see "who used what when" (for example, to find a policy violation), Netwitness is more efficient.

I often used the two together, utlizing the search results from one to help me focus my searches/analysis in the other. When I have the budget for it, I prefer to have both on hand.

Anonymous said...

There are several forensics solutions available similar to netdetector or netwitness such as Infinistream, FlowRecorder, Intelica IP Inspect, etc. If you want long-term packet capture with snort based intrusion detection, reporting and packet analysis that is available as appliance based or software-only solution (linux) take a look at Intelica IP Inspect.

Anonymous said...

Have you tried NetworkMiner? It can sniff networks but also load PCAP files for off-line analyzis. One nice thing about it is that it is host centric, i.e. it focuses on displaying information about the hosts and their activities rather than being packet centric. It does also use databases from p0f and Ettercap to passively fingerprint the operating systems of hosts on the network.

Richard Bejtlich said...

NetMiner looks neat -- I will have to try it.

Anonymous said...

NetWitness will not stay in the business long.... Niksun will stay in business longer just for the mere fact that NetWitness is trying to take over part of Niksun's sales and they will not make it with the condescending mentality that the executives have at that company. You purchase a product or whatever from NetWitness, don't expect any support.

Finnish guy said...

One interesting option is Tia from Clarified Networks (a small startup from Finland). You can see videos of Tia at Youtube

Donald Dave said...

Niksun NetDetector and NetWitness Decoder are indeed very useful tool in network forensics and investigation.

E-Detective also comes with similar capabilities like NetWitness. It is capable to do online real-time decoding and reconstruction functions with various standrd protocols like Email (POP3, SMTP), Webmail (Yahoo Mail, Gmail, Windows Live Hotmail etc.), IM (MSN, Yahoo, ICQ, AOL, IRC etc.), Web Browsing, Telnet, FTP, P2P etc. It also comes with comprehensive reporting (Statistical Reports per IP - Account etc.).

Anonymous said...

RSA has really smart people; however no one in education services has ever worked with network forensics. The instructors none of whom have any background in forensics (except 1 from another continent), yes there are a few contract instructors are very sharp but again only 2 who know the subject. There is one brilliant instructor from down under who is one of the best; God only knows why he works as instructor. The manager of education services has NO background in technology the manager’s background is in HR… WTF over? So you have a great product with unqualified people teaching the subject unless you are lucky enough to get a contractor, someone who does not speak English or the one good RSA instructor. BTW the only training given to instructors is the same crappy training you will receive. No continuing education within education services, ever. Take someone who has never been an instructor and never worked in the field give them 13 days of the worst training available on the planet and send him/her out to teach others. Great company just don’t deal with education services. Request a GOOD support engineer to come out and work with you for a week or two it will be worth 100x what you will pay for training.