Monday, January 25, 2010

Look Beyond the Exploit

The post One Exploit Should Not Ruin Your Day by Dino Dai Zovi made me think:

Finally, the larger problem is that it only took one exploit to compromise these organizations. One exploit should never ruin you day. [sic]

No, that is wrong. The larger problem is not that it "only took one exploit to compromise these organizations." I see this mindset in many shops who aren't defending enterprises on a daily basis. This point of view incorrectly focuses on exploitation as a point-in-time, "skirmish" event, disconnected from the larger battle or the ultimate campaign.

The real "larger problem" is that the exploit is only part of a campaign, where the intruder never gives up. In other words, comprehensive threat removal is the problem. There is no "cleaning," or "disinfecting," or "recovery" at the battle or campaign level. You might restore individual assets to a semi-trustworthy state, but the advanced persistent threat only cares that they can maintain long-term access to the environment.

If the problem were simply defending against a compromised asset, we would not still be talking about this issue. Rather, the problem is that it is exceptionally difficult, if not impossible, to remove this threat. Individual exploits add to the problem but they are only skirmishes.

16 comments:

javatard said...

As a "grunt" in this war (help desk/network support) it can become very difficult to try to explain to people higher up the command structure, that the Symantec removal instructions do NOT clean the compromised machine.
Compound this with a lack of structured rules for infected machines (ie the office will use their best judgment)and it makes for tough times.
Management doesn't want to wipe a server or laptop since the user may not have backed up their data. Symantec's paltry removal instructions ( Dump system restore, update definitions, scan in safe mode and maybe remove a Reg key...then ALL BETTER) do not do the complete job. The looks I get when I try to tell management that there are persistent back-doors that I can see, and god knows what else, mainly just get me the reputation of being negative and paranoid, and wanting to make more work that is needed. And forget about getting promoted with a reputation like that.

Matthew Wollenweber said...

I'd argue that Dino is on the right track. You're right when you state that the exploit is only the initial skirmish. However, you're fighting a losing battle if you're focusing on "cleaning", "disinfecting", and "recovery".

Everyone gets compromised. There is simply no way around it. Playing a game designed to not get compromised is destined to lose and spending to clean up will burn any amount of money.

The trick is to minimize the frequency and impact of compromises. That's accomplished by thoughtful system design, defense in depth, redundancy, and compartmentalization. If you do the heavy lifting before the compromise your response should be minimal. It also helps ensuring that "one exploit should never ruin your day" when the inevitable compromises happen.

There's no doubt that moving an enterprise toward a resilient system is an epic battle, but if you can make the move, I think it pays off.

javatard said...

If there is not "buy in" at the highest level, then I would have to say that the war was lost before it started.
If an Enterprise fosters an attitude of "clean and forget" or "everyone is an admin" then they are doomed to fail. Most places seem to just want to be able to check the little box on their SOX report stating that the virus was deleted and cleaned. Feeling this is following the procedures.
As a grunt, working their way up to a better place in the security industry, it can be hard to follow the procedures (which are full pf platatudes and broad generalized statements) knowing that their is so much more that needs to be done.
Management and Network admins don't want to look at Hex code, or wireshark captures or netsat outputs that clearly show open ports. In their minds, the Anti-virus says we are clean, there isn't a report waiting to be filled out. So shut up and don't rock the boat.
And let me tell you, the job market for someone with 6+years experience and no Masters degree or BS, is not a very good one.

PaulM said...

I appreciate Dino's point about defense in depth as a countermeasure to 0-day exploits, but the proposed remedy is somewhat naive. It's not as if organizations have exposed assets that 1) don't need to be exposed or 2) have users that don't need access to important data. But I agree with you, defense in depth isn't the issue here. Inability to remove attackers from the equation is.

Persistent threats as they pertain to computer crime aren't a new problem. In fact, it's the problem that defines computer crime. It's the reason that we have regulatory compliance standards for data security in most business verticals now. If we could successfully investigate and prosecute data theft and identity crime, these rules wouldn't be necessary.

Which leads me to what I feel is the obvious answer here. APT has been a fact of life and will remain so until attackers have something to lose. Today, they're on offense and we're on defense, and until that equation can change, settle in.

Dino A. Dai Zovi said...

Hi Richard,

With all due respect, iDefense first reported that the target of this campaign was intellectual property, primarily source code repositories. That does not require long-term persistence and at the time of threat removal, the code has left the building and the organization has already suffered that irreparable damage. For this reason, organizations where IP is their primary asset (and the disclosure thereof would cause considerable damage) must focus on the front lines of defense.

For the more general business enterprise, the threat is oriented toward maintaining persistent access and reacting to the overall campaign may be more reasonable. However, proactive steps like restricting local admin access and deploying DEP=OptOut dramatically reduces the cost of incident response. I consider it similar to the challenges of public health policy versus the health care industry: regular exercise, eating well, and healthy habits considerably reduce health care costs overall but would generate less profits in the health care industry.

Cheers,

-Dino

Richard Bejtlich said...

Hi Dino,

Thanks for responding. However, that's still "point-in-time" thinking. If you steal IP on day 1, does that mean the victim doesn't develop something useful the following week? If you steal email froma target on day 1, does that mean the victim doesn't say something useful the following week? These are battles and campaigns waged over years, not just days or even months.

Dino A. Dai Zovi said...

I will argue that IP is rarely "point-in-time". One commit to the repo is "point-in-time", but a "point-in-time" copy of the entire codebase encapsulates decades of work. Subsequent development follows from that point in time. If the attacker doesn't know how to write a single line of code, then they are dependent on maintaining access, otherwise they'll have a pretty idea of where things are headed. Case in point: OpenSSH forked from SSH.com's previously open source code. They certainly didn't have a problem continuing development of a competing (and better, in my opinion) product.

Of course, this is not only applicable to software. A point-in-time drop of the R&D behind a pharmaceutical, microprocessor, or any technology (use your imagination) is enough for someone with basic understanding of it to replicate it going forward. Point-in-time access to any technology related intellectual property is damaging to any organization that considers their exclusivity of that information a trade secret or competitive advantage.

Maintaining access is necessary for monitoring communications and predicting strategy, but not for IP theft.

Marcin said...

Richard,

I think the issue is more of "the attacker completed their goal; gameover". Maintaining access to these systems is not required, especially since that source code tree just bought you an unlimited number of potentially exploitable entry points. To me, it wouldn't matter what the target is developing while I was away... I'd find out soon enough.

Richard Bejtlich said...

Hi Dino,

I totally agree that "point-in-time" along is a disaster. However, these adversaries aren't looking to just cause disasters; they are leeches. One of their goals (see a recent post for others) is continious collection against victims. For one of their objectives, they are essentially outsourcing R&D at zero cost (beyond that required to pay for their offensive team) to the victim organizations.

Anonymous said...

Perhaps this a semantic difference?

Since the data (IP or trade secret or what not) is persistent there is no need for a persistent presence (of the threat) in a network. A campaign may exist outside of victim networks but the adversary can drop by time to time to cull what they need. Seems more prudent approach than having a persistent presence. (Of course, this works for time insensitive intelligence/data gathering and for real-time gathering you need to be persistent.)

Defenders may have to think and strategize to defend against the campaign, no doubt. But, mostly can only respond to persistent presence in their networks.

Or, are we seeing defender's beginning to respond to campaigns themselves (I would say that's what Google is trying to do)?

Anonymous said...

Have you heard of virtualization ? It's all about the browser, stupid. At the end of the day, all they want is your data so they'll use any means to send it out. If you virtualize your web browsing with some degree of common sense, then all these problems would go away. Convincing management is another matter...
Shiffting from "protecting the assets from the threat" to "isolating the threat from the assets"

Metajunkie said...

Richard,

I'm not sure I'm really following you on this one. Are you suggesting that the 'point in time' doesn't matter?

I generally find your 'out of the box' thinking refreshing (and often inspiring); but, I think I'm missing your point. Or, perhaps I'm just not agreeing with you.

I can agree that we are facing 'on-going' campaigns of cyber-threats in many arenas, and that we need to plan with the big picture in mind. But even in a physical campaign of war; while we must have high level strategy that leads battlefield level tactics, we must win the individual 'point in time' conflicts (at least the key ones) in order to win the war. Wouldn't you agree?

How does IT Security, or if you will allow the term cyber-warfare, differ? I have spent quite a bit of time converting Sun Tzu's The Art of War into IT Security wisdom. To me - his warfare consulting applies in cyberspace as well as physical terrain.

While Sun Tzu does advocate that the war is won or lost in the planning stage, before the enemy is even physically engaged; in the end, the best planning won't amount to a hill of beans if the boys in the trenches can't overcome their foes. That is IMHO the Zen aspect of IT Security - you have to be 'in the moment'.

From a Sun Tzu point of view, I believe that the lesson of his which most American companies that I've worked with are failing to heed is the "Know the Enemy, Know yourself." And of those two suggestions - it is actually the "know yourself" which is hurting the most. I could probably go on at the length of a book on that one... so I'll quit here ;)

Richard Bejtlich said...

Metajunkie, what I'm trying to emphasize here is that stopping exploitation, while obviously important, should not be even near the top of the list when it comes to ranking APT challenges. Sure, I would like to know that we could stop or even severely limit future exploitation using certain countermeasures. Unfortunately, most everyone fighting APT is well beyond the stage of worrying about future exploitation. In other words, planning and resisting are far in the past. Prevention already failed, probably a few years ago. At this point detection and response should be the priorities.

Keydet89 said...

I have to agree with javatard here, in that, as stated, "If there is not "buy in" at the highest level, then I would have to say that the war was lost before it started."

Regardless of your philosophy about all this, or what you call it (i.e., APT, etc.), or even who's responsible...it's all meaningless in the face of an organization where the "grunts" have to explain it management, rather than management mandating it to the "grunts".

Metajunkie said...

Richard,

Thank you for the clarifying words. I understand your position better now, and I agree.

I have been in a situation that I think exemplifies your position. I was doing some pentest work for a large custodian of private information as an InfoSec Consultant. They weren't ready for a pentest, but upper management was lacking valid situational awareness information and had contracted the company I was working for at the time for a "no holds barred" engagement. The pentest was a huge success (from the attacker's point of view).

While I was onsite delivering the final report, they became aware of an internal malware incident. I'm not going to give any details, because I really need to maintain anonymity for this org. Suffice it to say that the threat had a high risk of already dropping back-doors on production systems.

I advised they clean up the outbreak and conduct a thorough investigation to determine just how far the attackers were able to advance. I was actually in the process of setting up Squil for them when they pulled the plug on my engagement. They disagreed with me, and just wanted to patch the systems that I had ethically hacked into during the pentest.

Somehow I couldn't convince them that they were in all likelihood already compromised in a persistent fashion. The CIO's head went right back into the sand.

Did they need to patch all of the vulnerable systems? Yes. But, in their case, the intruders were already in their camp - and they chose to remain blissfully unaware.

Metajunkie

aka

Ken

Richard Bejtlich said...

Ken, we should talk -- can you email richard at taosecurity dot com?