Tuesday, August 01, 2006

Gadi Evron on Botnet Command and Control

Renowned botnet hunter Gadi Evron posted a message titled mitigating botnet C&Cs has become useless to several newsgroups. His post is a little tough to follow, but it seems his main point is it's too easy for intruders to establish new command and control networks. The good guys can't shut down the C&C networks fast enough to make a difference.

Paul Vixie extended this argument in 2004 in his Superbugs story. He said "Stomping a botnet is actually a bad thing to do. Read that again. Please." Vixie argues that shutting down simple C&C networks forces intruders to elevate their game.

I'm not sure what I should think about this issue. Paul Vixie, and definitely Gadi Evron, know far more about botnets than I do. However, I'm not sure that I can accept their argument about slowing down the digital arms race. I agree that confronting the intruders as directly as possible, though law enforcement, is the best course of action. On the other hand, if I worked for an ISP, I would not tolerate botnet command and control networks on my links just so intruders wouldn't learn to innovate.

12 comments:

Iain Wilkinson said...

Almost the whole history of Information Security has been a race between the good guys and the bad guys.

When a small number of botnet-herders come up with a good idea that bypasses current security, doing nothing does not mean that other bad guys will not start to use the same technique. In fact doing nothing will guarantee that ALL the bad guy will start to use this clever new technique – not doing so will put them at a commercial disadvantage in the Cybercrime world!

If the bad guys are elevating their game, it means we need to elevate ours.

Keydet89 said...

Iain has some excellent points, as does Gadi. However, I'm still left wondering...okay, so we know what the problem is, but what's the solution?

The Age of Nintendo Forensics is drawing to a close, my friends. As Iain said, infosec and CF in particular has always been an arms race of sorts, with the good guys struggling to keep up. And we, as a community, haven't kept up very well in the past several years...in fact, I would say that based on my experience as an incident responder, the gap has actually widened. This is with regards to IR/CF in general, not just hunting down botnets.

One doesn't have to look far. We've seen in the media recently how an organization had one breach that went unnoticed for over a year, and another that was notified of a breach by the FBI, *after* they'd gone through and declared themselves "clean".

In the Marine Corps, we had something called the "OODA loop", for Observe, Orient, Decide, Act. The idea was to process through the loop faster than your adversary. Sometimes, looking at the community of good guys as a whole, it seems like we don't even *have* an OODA loop, while the bad guys are cycling through theirs at Warp factor 4.

In a more general sense, all I can recommend at this time is training. The focus of the training and education needs to shift. Yes, I know it's out there, but not enough folks are attending it, and not enough of the good guys are taking away what they need.

In the specific sense of botnets, though...what's the answer? Do controllers really need to innovate? Shut down one C&C server, and the heads of Hydra raise twice more. So what's the answer? Perhaps the admins of the servers need to raise their game accordingly...tighten up their OODA loop, as it were.

Harlan Carvey
http://windowsir.blogspot.com

Anonymous said...

Hello Mr. Bejtlich. How is your last name pronounced?

Thanks,
Rod

Richard Bejtlich said...

"bate-lik"

LonerVamp said...

I like to think about botnets as an infestation of ants (roaches make me squirm too much). Yes, you can attack them and yes maybe they will adjust and work around your countermeasures to their advancements.

But the bottomline is that they are unwanted and harmful. That won't change whether you drive them off or they adjust their tactics. If you give up trying to bring down their networks, that won't magically make them tolerable or ok. The act of trying to isolate their networks, blacklist them, or ignore them is itself an offensive and they will work to move around that. Maybe you can't kill the ant colony, but by god you can put a wall up around them, right? Nope...

Basically, this arms race can be slowed down if we want, but that's just a false front for changing the tactics. It's not like we're suddenly just going to say, "you know, botnets aren't that bad, let's keep them around and hold hands." (Although, with p2p, botnets, and progress, I would take up this devil's advocate position over dinner just to have some good discussion.)

So really, I'm not sure what Gadi (or Paul) is trying to get at. Maybe our fundamental protections are just not the right approach. But then again, what is a correct approach? Taking your food and putting it up off the counters and out of easy reach of the ants so that they hopefully move on to easier houses? Moving our systems up the food chain a bit so they're not easy prey?

--shifting gears down a bit--

Unless you're a government or a large corporation with something to lose (customer data/image/trust) and money to spend, then security is a cost, not a return. For everyone else at home or in smaller businesses, security is something that needs to be a flip of a few switches one time and that's it, or something that is a benefit of hiring a kickass admin. Sadly, we all know that kind of security lasts all of a few months before something new pops up, or is defeated by users simply using the computers.

OODA works for people who have to do that. But for many administrators (myself included) security is something that is done on our own time or when everything else supporting the busines is done (and I mean everything else) or when a C-level reads the latest buzzwords and FUD from C-level mags. Otherwise security is only done on the fly, in between the daily tasks where we can slip it in. If we're on the firewall console already, then poke at it a few times to make sure things are still tightened, and move on.

That all being said, I don't see the fight getting any easier no matter what the approach. But then again, this helps ensure our jobs for quite some time, no? Lots of people say Microsoft keeps us employed (read into that as you will) but I say all the curious hackers and malicious criminals and experimental coders do this far more effectively than MS ever will...

I foresee ISPs being put more and more under regluations and guns to do the protecting, especially for home users who (and I hear this from friends and family a lot) are sick of the Internet because of the work needed to keep things running and smooth and safe. I foresee people and gov't moving to make ISPs the police of most networks, and then huge ISP corporations taking over for the local shops, creating what we always love in cable and telephone operations...wow, what a digression! I'll stop now. :)


(In re-reading this, I seem a bit doom-and-gloom and hopeless, but I'm not really. There are just some days I can't surround my cube with images of puppies, sunshine, and rainbows. :) )

Paul Schmehl said...

I think the answer is client mitigation. It's nigh unto impossible to stop the botnet herders from infecting machines (you'd have to fix every box that wasn't patched - continuously), but it's a lot easier to identify an infected machine.

If you quarantine the infected machines, you create a strong incentive for the owner to fix the problems and not allow them to reoccur. And you remove it from the botnet at the same time.

OC Computer Kid said...

Unfortunately, there is no "silver bullet" for defending networks and computer systems. Most of us in the industry are poignantly aware of that fact. And while I tend to agree that I would be unhappy about a botnot on my network (if I were an ISP), and would likely take action, there is a time and place to just "leave it alone". There is also (obviously) a time and place to take action. The "bad guys" are going to innovate and "elevate their game" irregardless of whether we, as defenders, elevate ours.

Mr. Carvey recommends training. I agree that education is very important, but there are a number of issues with education. "Education Vendors" exploit the needs of the industry and charge astronimcal amounts for training. And, all too often, you end up with "paper cert" administrators that have all the training in the world, but no experience and/or means to put that education into practice.

What's the solution? I don't know. I don't know that anyone does. Defense in depth? Mr. Carvey's OODA loop? Community information sharing? All of these are important, as are many more possibilities. We won't get a leg up on the bad guys until we can achieve the level of cooperation, innovation, and organization as a community of defenders that equals, or exceeds, that of the bad guys.

LonerVamp said...

Being relatively new to security, I am painfully aware of the paper certs problem that is kinda lurking in everyone's mind. I've known plenty of boneheads in IT already and others who make me wonder how they ever got their cert(s). The best you can say for them is to keep improving (if they have potential) or keep beating them with the clue bat until they start doing things right.

But let's face it, education and training is a long-term goal. It is easy to change a few of the easy practices that insecure people do, but to really change behavior, even online behavior, will take a long time. Look how long it has taken to move from an open network in the 80's? Some institutions still have pain over security vs open.

I am hoping that in the next couple years I can be working fully in a security role and start making a difference for my small sliver of control. Been a sysadmin for 4 years, IT for 5, and am looking to move for a bit into a networking role...but hopefully I can start really contributing and not find myself a paper-holder when I finally do start getting certs...

Michael Murr said...

I'm not quite sure I could tell people to stop trying to shutdown botnets, that's sort of like saying "don't install a security alarm or burglars will learn how to circumvent them" or "don't patch your systems or attackers will look for new exploits". I believe the evolution in knowledge is going to proceed forward regardless, although circumstance (e.g. C&C botnets getting shut down) can act as a catalyst.

I'm going to have to disagree with Harlan a little bit on this one. I don't think that the gap between the attackers and incident responders has widened, I think what we're seeing is the realization that we didn't know how big the gap was already. I do like "tightening the OODA loop", but it doesn't necessarily have a deterrent effect. My guess is that (a fair amount of) actual prosecution of the bot-herders/owners/etc. is going to have the strongest impact. No more "oh my botnet got shutdown/found out/etc. I don't care, I'll just use a new server and exploit". Looking at the news section of cybercrime.gov, there are 3 news releases surrounding bot related activities so far in 2006, things are going forward (albeit slowly). I can just imagine the red tape law enforcement would have to go through to prosecute a bot case, in addition to needing the technical expertise.

http://forensiccomputing.blogspot.com

Keydet89 said...

Michael,

Great comments! And great blog, too.

I don't see Gadi's article as advocating that we stop trying to shutdown botnets, as much as he's pointing out the futility (via quantifiable measures) of shutting down C&C servers. I do, however, feel that there is a lot of room for discussion regarding what should be done instead.

With regards to the gap I mentioned, please keep my perspective in mind. I'm an incident responder, currently as a consultant, but I've also held FTE positions. I don't get called in by people who know what they're doing.

For me, the typical conversation goes something like this:

Them: We want you to find out how the server got infected, and what the bad guy was doing while he had the C&C server running.

Me: Okay, but keep in mind that you ran a destructive A/V scan, took the system off-line, reformatted the hard drive, and reinstalled the OS from clean media *BEFORE* you called me...

Maybe what needs to be addressed is not simply shutting the C&C servers down, but having a process/methodology that can be followed so that ultimately, the botmasters are found and convicted.

With regards to the topic of training...I understand about unscrupulous education vendors preying on the community. However, it's simply a case of caveat emptor. I've provided training in the past in IR, and in one case in particular, an attendee went back to his desk during lunch and used the skills we learned that morning to resolve an issue. Yes, there are places out there that charge a lot of money and leave the attendee with nothing they can actually use...but it's up to the attendee to watch out for that.

LonerVamp said...

Something as "simple" as botnet issues driving forward questions of overall incident handling and training in IT... :) Yup!

Does anyone know of any IT forums or mailing lists that hold discussions like these? I hate to bog down Richard's comments with stuff, especially since I respect and enjoy all the comments thus far. :)

Jim Lippard said...

I don't think the arguments for *not* stomping botnet C&Cs are sound. As others have pointed out, this is a continuing arms race. Vixie's suggestion that law enforcement is the solution is, I think, naive. Better law enforcement is needed, but it is unlikely to ever be the primary solution or address more than a small percentage of high-profile cases--look at securities regulation enforcement for an idea of how a comprehensive system of regulation of a complex subject still lets a lot of violators get away with it.

On the other hand, I agree that we need to develop new methods to go after bots themselves (as well as continuing to go after botnet controllers)--and these need to be network-based methods deployed by the consumer ISPs where 99+% of bots reside. (BTW, the majority of botnet controllers reside at a fairly small number of webhosting companies--I'd estimate that 75% or more are hosted by about 20 entities).

I've commented a bit more at my blog.