Saturday, April 18, 2009

Speaking of Incident Response

In my last post I mentioned I will be speaking at another SANS IR event this summer. I just noticed a post on the ISC site titled Incident Response vs. Incident Handling. It states:

Incident Response is all of the technical components required in order to analyze and contain an incident.

Incident Handling is the logistics, communications, coordination, and planning functions needed in order to resolve an incident in a calm and efficient manner.


That's not right, and never was. I tried pointing that out via a comment on the ISC post, but apparently the moderators aren't willing to accept contradictory comments.

Incident response and incident handling are synonyms. If you need to differentiate between the role that does technical work and one which does leadership work, you can use incident response/handling for the former and incident management for the latter.

Ten years ago I took a course at CERT called Advanced Computer Security Incident Handling for Technical Staff. The class covered technical methodologies for responding to and handling incidents. The successor to that class is Advanced Incident Handling. Notice that CERT also offers the CERT®-Certified Computer Security Incident Handler certification. To CERT, incident response and incident handling are synonyms. If anyone should understand incidents, it's CERT.

I think SANS is the organization that needs to examine how it uses the term incident handler or incident handling. The GIAC Certified Incident Handler (GCIH) designation is 83% inappropriate. How do I arrive at that figure? If you review the day-by-day course overview you'll see that only one day, the first, involves Incident Handling Step-by-Step and Computer Crime Investigation. The next four days are Computer and Network Hacker Exploits, with the sixth day being an open lab. So, 5/6 of the class has little to nothing to do with incident response/handling.

This is a problem for three reasons. First, I have met people and heard of others who think they know how to "handle incidents" because they have the GCIH certification. "I'm certified," they say. This is dangerous. Second, respondents to the latest SANS 2008 Salary Survey considered their GCIH certification to be their most important certification. If you hold the GCIH and think it's important because you know how to "handle incidents," that is also dangerous. Third, SANS offers courses with far more IR relevance that that associated with GCIH, namely courses designed by Rob Lee. It's an historical oddity that keeps the name GCIH in play; it really should be retired, but there's too much "brand recognition" associated with it at this point. If you want to learn IR from SANS, see Rob.

To be fair, the title for the course which prepares students for the GCIH is Hacker Techniques, Exploits & Incident Handling. Putting IH at the end does list the subject in the proper context. I will also not deny that one should understand hacker techniques and exploits in order to do incident response/handling, but that knowledge should be its own material -- something to know in addition to the skills required for IR. Also, track 504 is really good; I remember it fondly, before it had that label. The material is kept fresh and the instructors are excellent.

The bottom line is that incident handling and response are synonyms, and those who think they are certified to do incident handling and response via GCIH are kidding themselves.


Richard Bejtlich is teaching new classes in Las Vegas in 2009. Early Las Vegas registration ends 1 May.

16 comments:

Russ McRee said...

Richard,
As one who has taken both the GCIH track & Rob's GCFA track, and lives, eats, breathes incident response, I fervently agree with you on all fronts:
1) Incident response & handling are indeed one and the same.
2) While one of my favorite tracks, GCIH is a hacker tools class under the shroud of incident handling. I believe SANS should split incident handling and the hacking techniques into distinct tracks.
3) Rob's GCFA course is, in fact, a much closer match to best-of-breed incident response methodology.
4) I'm disappointed that the ISC has chosen to not post your comment, and hope they'll reconsider.
Well stated, Richard.

Zeth said...

I agree with both Richard and Russ. Rob's course and CERT/CC courses are the best, but why aren't there any courses for incident coordination? I mean if you have an incident affecting several different business areas (or subsidairies) or even your branch offices in different countries, where can I find a descent course that tackle this - focusing on gathering information, assessment, prioritization, correlation, cooperation, coordination, communications, calling in people and get them to do the thing, etc?
For now the real life is my training and (lucky me) there are several cases each year.

Any ideas anyone?

Keydet89 said...

Zeth,

I would suggest that what you're looking for falls under what Richard is referring to as incident management. In a great many cases, what I've found during CSIRP development is that internal political issues, etc., will obviate courseware.

The short story is, simply take your incident response material and determine whether you're going to decentralize it with a small trained, knowledgeable staff at each location, or if you going to do it remotely...or some combination thereof. The political stuff is going to come into play for the calling in people and get them to do the thing stuff.

Anapologetos said...

I will be taking my GCIH exam this Thursday. Just because I spent the last couple months taking SANS SEC 504 OnDemand, I don't believe that I am an expert at IH--I would say that I feel like I have a good understanding of the process--

My question for you guys that have done IH in real life / classes other than SANS, what am I missing out on if I just took GCIH, and learned the 5-step IH process SANS teaches?

Why "If you hold the GCIH and think it's important because you know how to "handle incidents," that is also dangerous." is it dangerous?

Josh

Michael Cloppert said...

Richard, et. al.,

I'm glad to see concurrence here. I started typing a comment to this ISC blog entry myself but ran out of patience. I have a great frustration, as you heard last fall at the first SANS Forensics Summit, on the current state of thought on incident response. For the sake of brevity, I'll leave it at "IR processes are more fundamentally broken than terminology can fix," and third your view that these words are synonyms.

Keep an eye on the SANS Forensics & IR Blog. I'm planning a post soon elaborating on my thoughts, once I can articulate them clearly and concisely.

Armorguy said...

I think the point is well made and well taken...

I am not a fan of the "incident handling" term... Adding new terms to the lexicon that are merely synonyms doesn't do much for the profession, does it?

Quick answer for Josh - what's dangerous about the GCIH? It's dangerous if someone who has a GCIH thinks that, purely based on the training they've received, is fully qualified to run a incident response. Fact is, you aren't. It takes deeper training and/or much more experience to run a incident response.

Doesn't mean you won't ever be... It means that the SANS training for GCIH doesn't cut it, IMHO. (Full dislosure: I send my engineers to other SANS training that is fantastic and worthwhile.)

Mark Stingley said...

Richard;

I'm sorry, but this is the worst post you've ever made, and the first time that I simply disagree.

First of all, the world of computer 'incident handling' is in its infancy compared to the military history of such things as Navy damage control, aircraft mishap, and so on. Then, there are the well honed incident response practices of the emergency services outside the military. Sadly, the civilian and government world regarding computers have chosen to reinvent the wheel.

I propose that 'incident response' is a subset of 'incident handling'. They are not synonyms, no more than putting out a fire is a synonym for an arson investigation.

Some definitions:

incident; an occurrence

response; the act of responding, as to a call for help

handle; manage

So, to my way of thinking, any course that concentrates more on the administrative form of incident handling is a management course. Training that concentrates more on the skills needed by a responder is a tactical one.

My impression of the GCIH thus far is it gives the basic IH skills to a responder, while offering a wealth of hacker techniques training so that the responder can accurately assess the threat and risk. I appreciate that, since network security analysis is by far the weakest skill set of much of this modern 'non-sysadmin' trained information security profession.

I haven't had the CERT training. They don't seem to offer mentoring or on-demand training, or the considerable discount SANS gave my rather large organization. But, I do think the GCIH bashing is juvenile.

In short, anyone who takes any course from anyone who then thinks that they 'know' what to do is 'dangerous'. Walking out of any course on this planet, the attendee is a 'rookie', plain and simple. The course is only a map of a course to steer. The certification simply means that the user was able to pass a test on the content of the course. It then takes much experience to make them 'expert'.

I personally think that the GCIH could always be improved. But, I would much rather have first responders to a server compromise know as much about cyberwarfare tactics as they can in order to contain the threat as soon as possible without going overboard.

Lastly, I am totally confused by the preference of forensics over attack recognition. On one hand, a course that concentrates on forensics is great. But, the 504 that favors hacker tactics and attacks is bad? One is favored at the beginning of an incident response, the other later.

So, all of you GCIH bashers.. thanks much for trashing the money and time I've spent on the course. It reminds me of the Windows bashers in the Linux forums, and the Linux bashers in the Windows forums.

BTW... I'm a GCIA and GPEN, and I've been doing this since 1994. And, I'll still be proud of my GCIH in a few weeks.

Russ McRee said...

@Mark: I don't think anyone is bashing the GCIH, I loved that track. Yet, in all fairness, the class is 1/6th incident handling, and 5/6th hacker tools.
I loved the GPEN too, and the GCFA track. I'll always be proud of them all.

@Zeth: The GCFA is a great track to follow up your GCIH with to gain further applicable knowledge in the IH/IR realm.
Beyond that, even if IR/IH is not part of your current job duties, you can do a lot to teach yourself in a virtual environment. Build yourself a response VM and victim VM and practice scenarios. It really can pay off. I can offer a lot more in the way of suggestions via email, if you wish.

Mark Stingley said...

Zeth; I appreciate the advice, but I've been doing 'incident handling' since the 70's in my first US Navy tour. I ran a legal office, a duty office, and helped investigate many mishaps. I've been doing computer incident handling since the early 90's. In my current (over 10K hosts) environment, the average time from detection to containment is much less than an hour. And, we discover almost all of our own compromises.

I personally believe that the InfoSec industry is primarily staffed with (1) auditors, (2) appliance operators, and (3) compliance people. In a such an environment, an 'IR' course such as the GCIH is doing a service by concentrating 5/6 of the course on hacker tactics. Too many so-called InfoSec 'analysts' desperately need that skill set, along with GCIA skills. And, too many InfoSec managers are incredibly weak with 'hard' skills.

In addition, GCIH is typical of a SANS course, wherein one of the course books is usually worth two to three of the alternative. Book 1, day 1 of the GCIH is a concentrated guide, chock full of critical references and checklists.

Perhaps, there wouldn't be a problem if the course were entitled "Giac Certified Incident Responder".

I get hot when respected, industry leading figures bash something that is actually a good thing. Such irresponsible actions do damage to people.

Richard Bejtlich said...

Mark, I wonder if you would find it interesting if I published the names of people who privately agree with this post but don't want SANS to see it? There's more than one "respected industry leading figure" who shares my point of view.

Stephen Northcutt said...

Well for what it is worth, if I am not the guilty party, I am "a" guilty party. It was a decision Ed Skoudis and I made jointly in London, sorry can't remember the year.

Background, and hope this is not too much info. My first book was Incident Handling, some of you may still have the .pdf version.

The hottest two selling courses in 2001 were both hacking, one was Hacking Exposed ( perhaps you remember the all black costumes, nehru collars and dry ice smoke) and the other a hacking course put together by Ed Skoudis and Eric Cole. Skoudis/Cole was outscoring the other by about 1/10th of a point and we were more comfortable with it.

We felt there were two directions we could take the course if we expanded it, one down the response path, the other down the pen test path. We chose response and did not add the pen test course until 2008.

I feel that incident response is largely a process that can be taught in a day, but that to be effective you need a number of skills. A large number of those skills involve malware and exploits, because success with those tools is often why you are responding.

I also feel incident responders are akin to EMTs "first response" and this is why forensics should be a separate discipline. We have an obligation not to make it hard for digital forensics examiners to do their job by mucking up the evidence, but those are separate skills in my view.

I would be happy to open a dialog as to what we should do or say, but don't expect radical changes. There are about 4k certified folks and many more have taken the course and seem to enjoy it. It is fine to tell us we are wrong, but we have spent years tuning both the course and the concept. Still, always happy to be schooled, feel free to drop me a note stephen@sans.edu and when we get the messaging tuned, I am happy to forward to Ed for consideration.

I am sorry that your comment did not get posted, but in all fairness you have said some pretty mean things over time and if it is the ISC blog you are talking about they did just win the RSA Social Security technical blog award so they must be doing something right. I look forward to an open dialog with any or all, but please use the email I posted, I only check gmail once every quarter or so *grin*

Richard Bejtlich said...

In case anyone is wondering, here's the comment I tried to post to ISC that was censored.

You're introducing an unnecessary distinction by trying to assign different definitions to the terms incident handler and incident responder. What does SANS mean by GCIH, then? Is that supposed to be GCIH or really GCIR? You can avoid this confusion by using the term Incident Manager for the person who assumes the leadership role you describe. Incident handler and responder are defined to be the same role by most respectable IR organizations -- check out CERT's description of Certified Computer Security Incident Handler at http://www.cert.org/certification/ .

Joel Esler said...

I wonder if it was auto censored because it has a link in it. The system may have thought it was comment spam. Try putting your comment without the link.

Richard Bejtlich said...

Joel, somehow I doubt it. The two comments by "Josh" included a link.

J said...

Richard,

Bravo for saying something about GCIH! I mostly agree that the certification and course topics are off balance. As a very active incident responder - I was on the road about 80% of last year responding to a variety of incidents for a number of organizations - I can say that the content of the GCIH program would likely not have helped me. That's not to say the content is bad material - I just believe it's not the correct focus for IR. What I would like to see is a class that can show me how to find evil and solve crime, not lecture on how to run John the Ripper and nmap.

Adriendb said...

http://isc.sans.org/diary.html?storyid=6313

Cheers,
Adrien