Friday, March 16, 2007

Incident Response Clarifications

Recently I posted Five Thoughts on Incident Response. Based on the comments and some blog responses I wanted to clarify what I originally posted. The first three items seemed to attract the most attention so I'll only address those.

  1. Anti-Virus is (or should not be) an incident response tool. The emphasis here is on response. I agree that AV is often an incident detection tool, and ideally an incident avoidance tool. However, if you think AV is going to help recover from a totally compromised system, you are probably going to be upset by the results.

  2. Your default incident recovery strategy should be to rebuild from scratch. The emphasis here is on recovery. I am not saying your default incident response strategy should be to rebuild from scratch. Your default response strategy should be to investigate to determine how the victim was compromised, what aspects of Confidentiality/Integrity/Availability were violated, and so on. I agree that any response which begins with re-imaging the victim is a recipe for failure.

  3. SPAN ports should not be the default traffic access option. I'm standing by this one. The only time SPAN ports are superior to taps is the situation where intra-switch traffic needs to be seen. Otherwise, spend a few dollars and get a product designed to grant reliable traffic access. I'm talking about professional ways to perform incident response here. Hardware is the easiest thing to gain budget for in any organization. It's easier to buy a piece of hardware than it is to send a person to training, or hire new help, or bring in outside consultants, or any other activity that could benefit a security shop.

I appreciate the other recommendations I've seen. These are only a few big thoughts which struck me based on recent engagements. I have over a dozen recommendations in my Network Security Operations class and I think I cover similar material in Extrusion Detection.


Tim said...

I believe taps are a better choice. I'll get that right out of the way. They got flexibility and performance. However – you reasoning seems … na├»ve almost.

Configuration issues exist with a tap or a switch. If your network team can't perform the basic step of configuring a SPAN port - than your organization has bigger problems. You are not designing a MAN here - you are making a 3 line config (e.g. cisco) or menu change (e.g. nortel, 3com). You work with them to set a standard and move on with life. This costs less than procuring additional hardware.

Professionalism. What complete arbitrary bunk. The fact that you wear a tie in your photo does not make you more professional. If it takes a reciprocating saw to install a sensor in the network rack - so be it. I do recommend taking the tie off. I had two “professional-grade” taps installed by “professionals” fail on me in 2006. When they fail – you better be prepared to explain why you weren’t using a SPAN port.

Each has their uses – each has their place.

Richard Bejtlich said...

Tim, I'll let your ad hominem attack speak to the professionalism you're demonstrating.

Richard Bejtlich said...


If you care to comment, were those taps fiber or copper?

David said...

Richard, thanks for addressing my post.

I think we agree on the heart of the matter - AV is part of a complete defense - and for many smaller organizations, is one of the few layers that they have.

I tend to work with system administrators who are the only IT person for an entire business, or who take on incident response as a tiny subset of their duties. Some of them still do believe that an AV scan will tell them if they are compromised - that's a painful wakeup call to have to make. Training them to go a bit deeper than "Antivirus removed foo.exploit" is definitely worthwhile - that's the incident detection you mentioned.

In either case, my point stands - have AV along in your toolkit - it might help identify at least part of the big nasty rootkit that got dropped on your system.

I think we're in agreement on the second point - in fact, I think that the response side of how the one man shop administrator deals with it is a very interesting problem, and I'm hoping to address that more in the future. Most know how to rebuild, but analysis, even on a basic level is more difficult.

I'll also stand by my take on SPAN ports. I agree - a tap is definitely the preferably tool - given a budget that allows it, or a dedicated security professional's toolkit, it is definitely preferable. My point was that if you are restricted - in an organization in which the IT budget is small, the choice may be between a tap and buying a replacement switch for a network segment. For those places, knowing how to set up a SPAN port, or having a workaround is useful.

I'll also note that Tim has a point - sometimes, being able to get traffic without interrupting a link is handy. I've seen at least one instance where a fiber tap install didn't go as cleanly as expected and had to be delayed. You can always drop back to the SPAN port as plan b. Yes, it would be great to have taps in place beforehand - in a lot of organizations, that won't be the case.

David - Devil's Advocate Security

Keydet89 said...


Training them to go a bit deeper than "Antivirus removed foo.exploit" is definitely worthwhile - that's the incident detection you mentioned.

I tend to agree wholeheartedly with this, but the issue is really much greater than just training, particularly in situations such as you describe. If the organization only has one IT person then depending upon the overall size of the organization, it may be clear that the issue is really with how serious senior management is about security. I mention this as I've worked with several organizations in which the IT staffs have been far too small and undermanned for the task of simply managing what they had. might help identify...

At the risk of beating a dead horse here, I'd like to reinforce the point about training. I've had several incidents over the past year in which the malware wasn't identified by the AV wasn't a "0 day", it just wasn't covered by the up-to-date (yes, this was verified) AV definitions. In another instance, *three* AV products all identified different portions of the issue and none had sufficient cleaning instructions to remove enough of the malware to keep it gone.

My point is that IT staffs need to get the training to understand their systems better, under the hood, from the ground up.


Anonymous said...


I am quite new to NSM.
what do you recommend in an environment with vlans? I think I will have a lot of problems with a tap. Personally I would prefer a tap but is it possible to use it in such an environment?


Richard Bejtlich said...


Most likely if you want to watch a specific VLAN you'll need to use a SPAN port. Having said that it really depends on your monitoring requirements.

Anonymous said...


no it isn´t a specific, what if I want to watch all vlans at a switch with my ids. I think the SPAN port will loose too many packets. Is it right?

Richard Bejtlich said...

Depending on the number of VLANs, watching all of them is probably not a good idea. You may oversubscribe your SPAN port. A better idea is to decide what you want your monitoring to accomplish and then devise an instrumentation strategy. With internal monitoring you can follow a threat-centric or asset-centric approach, since watching everything all the time is probably not cost-possible.