Monday, February 12, 2007

Earth to MARS

Disclaimer: I'm going to single out a book by Cisco employees that talks about a Cisco product. I have no personal feelings about Cisco. I have friends there. I've done work for Cisco. Since I think Cisco is eventually going to own all network security functions in their switches, I may even work for Cisco one day.

This post is for all product vendors who approach understanding and defending the network in the ways described here. Wherever you read "Cisco" feel free to add products that share the characteristics I outline below.

Once again I found myself hanging in the sky last week. Trips to and from the West Coast gave me the opportunity to read Security Threat Mitigation and Response: Understanding Cisco Security MARS by Dale Tesch and Greg Abelar. This is mainly another Cisco marketing book, like Self-Defending Networks: The Next Generation of Network Security by Duane DeCapite. While I have a few thoughts on the book, I would much rather address the underlying philosophy presented by the authors. I'm fairly sure they're only repeating Cisco market-speak, but I hear the same message from many vendors, consultants, and individuals.

In this post I'd like to take issue with that message. In short, almost nothing about the approach taken by Cisco MARS (and similar products) is new or will "solve" your security problems. Unless you augment tactics and technologies like MARS, you will find yourself wondering why you spent time and effort to end up as frustrated and confused as you were pre-MARS.

I'll refer to this book as STM, short for the Security Threat Mitigation term in the title. STM is supposed to be "beyond Security Information Management" (SIM). According to STM, the five core SIM functions (pp 6-7) are:

  • Collect event data

  • Store data

  • Correlate to show relationships (what the book calls "true power")

  • Present data

  • Report on, alarm on, and/or notify about data


STM starts to knock the value of SIMs by introducing the idea of "garbage in, garbage out," saying "information or events from several different sources can be 'garbage' unless they are put together in a useful way." I disagree -- garbage in always produces garbage out. The idea that a ton of garbage can be turned into something valuable (like gold from coal) is the fallacy of SIM/SEM technology.

The book tries to position "STM" as "beyond SIM" by granting STM the following attributes:

  • Data reduction

  • Timely attack mitigation

  • End-to-end network awareness

  • Integrated vulnerability assessment

  • Session correlation


You are probably think what I am thinking; how is that really different from a SIM/SEM? I bet all the SIM/SEM vendors are saying "we do that already."

Things start to get really weird when the authors talk about the "advantages of a proactive security framework" (pp 14-15) compared to what everyone else must be using. STM says:

The key to this framework is the network's capability to behave actively. This does not necessarily mean to take action itself, but to automatically collect data from numerous sources and come to a decision..." (emphasis added)

... so that a person can react like we've always done. It is intellectual dishonesty to claim this product (or any other) is acting "actively" when the end result is still waiting for a person to react. Mind you, I'm not knocking reaction. Too many people seems to think proactivity is king and reactivity is evil, but sometimes there's no other option. It bugs me that Cisco is packaging manual "shunning" in a new wrapping and calling it "self-defending" and "active" when the end result is a person having to make a reactive decision.

The real problem is far more insidious, however. As has always been the case with alert-centric products, there is not enough data available to make a decision. In other words, Cisco MARS, like other products, still cannot tell me if an attack was successful.

More absurdity appears in the section (p 15) on "false-positive reduction," where three methods are given for MARS to "determine the validity of event data." They are, basically:

  1. Network topology

  2. Vulnerability assessment via limited network scanning

  3. User determination


So, we have 1) could the attack reach the target? Guess what -- if it's a TCP connection it probably did. Next we have 2) scan to see if the IIS attack hit an Apache server; limited at best, worthless at other times. Finally -- and this kills me -- 3) let the user decide. How is this better than anything else again?

The section "Enhancing the Self-Defending Network" adds insult to injury by naming three "missing links" in the SDN addressed by STM:

  1. Automated log correlation

  2. Automated threat response

  3. Automated mitigration


We know SIM/SEM does #1. A product that accomplished 2 and 3 would be impressive, but guess again -- MARS does neither automatically:

Automated mitigation is not yet achievable because many security devices still put out false-positive alerts, but CS-MARS makes a recommendation for mitigation and offers security responders a single click to deploy commands on devices that will stop offending traffic after the responder has analyzed the attack data. (emphasis added)

Again, it all comes back to having the data necessary to make a decision, and then letting an analyst make the decision. There's nothing proactive or special here.

STM emphasizes the speed with which one can respond, but how can an analyst even know to start the escalation process? The end of chapter 3 presents a case that aims to show "ROI" for MARS based on shutting down a spreading virus faster than a non-MARS solution. Here's the core of the problem, however:

The virus starts trying to infect other hosts... [MARS sends] an email page... to a security responder 24 hours a day, 7 days a week... The security responder gets the page and logs into CS-MARS.

And that's faster or better how?

STM demonstrates real ignorance of the sorts of data an analyst needs to make decisions regarding security events. On MARS, "forensic analysis" is "visual tools for attack-path analysis and attack propagation" (p 82) or mapping NAT'd IPs to public IPs or IPs to MAC addresses. "Attack validation" (i.e., should I worry?) is knowing "if an attack actually reached the intended destination." Furthermore, p 89 says:

[U]pon detecting an anomalous behavior, CS-MARS starts to dynamically store the full NetFlow records for the anomalous activity. This intelligent collection system provides all the information that a security analyst needs. (emphasis added)

Wrong. What else does MARS offer? Page 127 says "retrieval of raw messages" is available as a download of a zipped text file! I bet that's easy to manipulate when trying to understand alerts. On pp 229-230 we see examples of the sort of "incident ID" data available to analysts -- almost all of which is unactionable and worthless. On MARS, the analysis process consists of looking at a rule MARS used to generate an alert and then guessing at the relationship between the rule and events listed in the "detailed data" section.

I think the real problem with this approach is demonstrated by the case studies in chapter 9, which I guess Cisco sees as a validation of their approach. We read about a Trojan found at a state agency via an IDS alert (no different from non-MARS). We see a .edu respond using MARS because phone calls prompted an investigation of traffic on port 25! We read about a hospital that used a spike in traffic to a Web server as a reason to investigate the Web server (wow). A financial firm investigates high ICMP traffic, and a small business sees odd DNS names in its weekly MARS reports to find another company using its wireless access point. Honestly, is this supposed to make me a believer?

If you want to know how I recommend dealing with this and similar situations, I've already written about it. In brief, analyst must have high-fidelity, original, content-neutral data sources to investigate. That is the key term here. If you do not have such data (as provided by NSM) to investigate, you are doing alert management. And you will lose.

So what is MARS (and similar products) good for? In short, there is value in centralizing and presenting events from disparate security products. There is value in having a single window into that world, with some sort of accountability, escalation, and case management. There is value in being able to contain network-centric exploitation via centralized device control. Just remember that making the decisions to protect the enterprise requires having the proper data. NSM is one way to get that data.

11 comments:

A Green Needle said...

Hi Richard,
another great article.
what about using something like MARS in parallel with sguil ?

We are looking at Checkpoint's Eventia analyser for this reason.
Im hoping to use Eventia to point to the problems and then use sguil to do the investigation.
does this sound like a valid approach ?

BTW, hope to see you at AUSCERT if I can convince the "powers that be" :)

Richard Bejtlich said...

A Green Needle,

I plan to support clients who have MARS by running Sguil collecting session and full content data. I may or may not use Snort with Bleeding Rules as part of Sguil, since the MARS deployments are supposed to be collecting alerts from existing IDS'.

Daniel Cid said...

I agree with what you said, except:

"garbage in always produces garbage out"

have you ever heard of reclycling? :)

Daniel

Anonymous said...

This just seems indicative of the Cisco IDS platform management as a whole. While I don't use MARS, I do use Cisco IDS sensors. The ability to manage multiple sensors is horrible and the alert info always leaves an analyst wanting. The descriptions of the IDS signature never give enough information to determine what actually caused the event to trigger. This usually leads to running a packet capture if the activity is ongoing, or having to consult firewall or application logs. I wish we ran Snort sensors instead.

Anonymous said...

I have used MARS when it still belonged to Protego Networks and do so today now that is is rebranded as Cisco. It is not just alert-centric, in fact I am using it to baseline network usage via Netflow, SNMP polling for resource usage, custom script agents feeding metrics and CLI output via traps/syslog (requries custom parser on MARS end which it has support for).

Knock the book all you want but besides full content data, MARS with a little scripting is working out well.

I do work closely with Cisco and I can tell you they are currently adding full support for Netflow, user-configurable SNMP OID polling like OpenView, TCL scripting for pulling data not accessible in syslog/snmp and others they are considering but not committed to yet.

My user-defined rules kick butt, and I am mitigating by pushing IOS IPS sig packs to the edge, rate-limiting on the routers - and to be supported soon - dynamic ACLs and shunning to the firewalls. I am working on getting MARS to black-hole as well.

Anonymous said...

If you want true profiling and baselining then look at Mazu Networks . MARS does 1% of what Mazu does in terms of profiling and visibility.

Anonymous said...

Richard, I'm including a comment here, simply because you stated that a product that truly performs automated mitigation would be impressive, which suggests you are not familiar with TriGeo Network Security...? At present, the only SEM that offers comprehensive automatic remediation...mostly Windows centric, but there are a handful of other useful responses, as well. If you're interested, check it out at: www.trigeo.com

P.S. Voted by SC Magazine readers at "Best Event Management" solution for 07

Richard Bejtlich said...

I'll take a look at any products in this space, if someone can get me a demo account with data to review.

Thanks!

Davi Ottenheimer said...

"have you ever heard of reclycling"

Recycling starts with the paradigm shift that not all things being thrown out are actually garbage.

Thus it does not disprove Richard's point but rather re-emphasizes it. Even in recycling the quality of stuff collected for processing impacts the quality of stuff that comes out.

Thanks for the excellent review, Richard. I can probably get you the data you need if you've got the time...

Anonymous said...

It's Coal In, Diamonds Out....jeeeez. seriously though, bash cisco all you want, i dont think they are losing market share on any one front or in general. if it were called iMARS and produced by some Different Thinking company would that make you like it?

Richard Bejtlich said...

Anonymous,

Based on what I hear, Cisco builds MARS market share by giving the product away. Hardly anyone buys it.

As far as Apple goes, I don't care for them so much either.

Besides, didn't you read my very first paragraph?

Disclaimer: I'm going to single out a book by Cisco employees that talks about a Cisco product. I have no personal feelings about Cisco. I have friends there. I've done work for Cisco. Since I think Cisco is eventually going to own all network security functions in their switches, I may even work for Cisco one day.