Defining the Win
In March I posted Ten Themes From Recent Conferences, which included the following:Permanent compromise is the norm, so accept it. I used to think digital defense was a cycle involving resist -> detect -> respond -> recover. Between recover and the next attack there would be a period where the enterprise could be considered "clean." I've learned now that all enterprises remain "dirty" to some degree, unless massive and cost-prohibitive resources are directed at the problem.
We can not stop intruders, only raise their costs. Enterprises stay dirty because we can not stop intruders, but we can make their lives more difficult. I've heard of some organizations trying to raise the $ per MB that the adversary must spend in order to exfiltrate/degrade/deny information. (emphasis added)
Since then I've grappled with this idea of how to define the win. If you used to define the win as detecting and ejecting all intruders from your enterprise, you are going to be perpetually disappointed (unless your enterprise is sufficiently small). Are there are alternative ways to define the win if you have to accept permanent compromise as the norm? The following are a few ideas, credited where applicable.
The first two come from my post Intellectual Property: Develop or Steal, but I repost them here for easy reference.
- Information assurance (IA) is winning, in a broad sense, when the cost of stealing intellectual property via any means is more expensive than developing that intellectual property independently. Nice idea, but probably too difficult to measure.
- IA is winning, in a narrow sense, when the cost of stealing intellectual property via digital means is more expensive than stealing that data via nontechnical means (such as human agents placed inside the organization). Still difficult to measure, but might be estimated using red teaming/adversary simulation/penetration testing.
- IA is winning when detection operations can see the adversary's actions. This relates to Bruce Schneier's classic advice to Monitor First. The more mature answer is next.
- IA is winning when incident responders can anticipate the adversary's next target. I credit Kevin Mandia with this idea. I like it because it shows that complex enterprises will always have vulnerabilities and will always be targeted, but a sufficiently mature detection and response operation will at least be able to guess the intruder's next move. You can even test this by keeping a track record.
- IA is winning when the time to detect and remediate has been reduced to B. Insert your own value there. You can track your progress from time A to time B.
- IA is winning when your enterprise security integrity assessments show less than D percent of your assets are compromised. You can track progress from C percent to D percent over time. This leads to the more mature version which follows.
- IA is winning when your enterprise intrusion debt is reduced to F. You can measure intrusion debt as you like and take steps to reduce it from E to F.
Does anyone else have ideas on how to define the win?
Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.
Comments
IAF is winning when the business it protects has durable competitive advantage.
So, the best wining is: Max(E - F - IAcost)
I think #5 and #6 are measurable...perhaps could wrap a measurement around #3. But I don't see a good way to measure the rest of them reliably. I don't have a suggestion yet for alternatives, but am giving it some thought.
While certain companies have specific IP or data that may be desireable for a bad actor to steal, most crime, and I would argue cybercrime is no different, is more like the situation described above. IA doesn't win in the long term if it's easier just to try someone else down the block. Just like the shopping mall where those two stores are located doesn't win.
Low Bar Win - You can at least reliably detect when your owned.
Medium Bar Win - You are doing everything within your control and more importantly budget to secure your network and data. And of course documenting it :-)
High Bar Win - Nobody has a foothold on your network, your data is safe, and you can continuously prove it.
If so, you're encouraging the thieves to move on down the street and (yes, unfortunately) break into a neighbor's home. It would be admirable of you to start a neighborhood watch, join the local crime-stoppers association, etc. But I bet you're not doing nightly patrols of your neighborhood on foot, or helping your neighbors fund an alarm system for their homes. Bottom line: you're looking out for your own home, and if doing so causes the burglar to rob your neighbors instead, then you'll be satisfied that you've done your job, right? Why is the cyber world any different?
I'm sure you'd like all crime in your city to cease, and I'm sure the above commenter would like all cybercrime to go away. We live in the real world though, and winning is relative, not absolute.
Anyhow, I agree with your statement that "IA is winning at your company when the cost of stealing IP via digital means is more expensive than stealing someone else's."
3 & 4 are information counterintellingence not assurance. Information counterintelligence can help provide metrics for use in determining if you are winning or losing.
IA is winning when you are not losing.
IA has lost when protected information is released at the rate or cumulative magnitude above acceptable limits and used against the organisation.
IA stays lost until the released protected information no longer has an impact on the organisation.
This definition of winning illustrates some points.
- Some loss is expected.
- You need to know what the value of your information is
- You need to know the value of protecting your information from compromise
- If information is compromised but never used then the actual loss is minimal
- If you only have one secret then you need to protect it well. If you are continuously generating secrets then a single loss can be sustained better
- Effective response and recovery is required to stop losing
I think we're more in agreement than not. Initially I was making the point that most crimes are crimes of opportunity and that if we've got the locks on the doors but our neighbor doesn't, that's not good for the whole neighborhood. On re-reading everyone's posts on this discussion, I think we're all grappling with Richard's original problem of 'defining the win'. I originally took IA to mean the industry in general. On re-reading the OP I realize that Richard seemed more focused on an individual org's IA. He's a little more tree-focused, I'm a little more forest-focused. OK.
One of the chief things we're dealing with in security is trust. We are in the business of gauranteeing to the best of our abilities our business's customers' trust. That's it. While our individual company may be worthy of that trust, what happens when an industry as a whole is viewed as a "bad neighborhood", not worthy of trust? I ignore all emails, legitimate or not, from Ebay, Wal*Mart, Bank of America, and several other companies. Is that what those companies want? I doubt it. What happens to a company when they get lumped in with spammers, TJ Maxx, etc.? How do you prevent the customer lumping you in with those guys in the first place? I think that's a pretty big deal and it's a hard question to answer. But if you ignore what's going on down the block, you may end up with the most amazingly secure business that no one goes to anymore. That means no more paycheck. That's bad.
I still see things in black and white, after all these years.
mjr.
Lots of those "wins" are short-term rearguard actions and are nothing resemblin "winning" in a long-term context. And, in case nobody noticed, this is a long-term problem. Caving in (which is the usual eventual position of security) often results in a negotiated position that somewhat reduces risk in the short term but in the long term probability says that sooner or later, there will be a failure. That's the fallacy of risk reduction approaches: they're just an attempt to paper over the problem until someone else takes it over and hopefully it's their disaster to deal with later.
I could give countless examples of how this short/long-term dynamic has played itself out in security - in fact, the entire security industry is intellectually twisted around trying to apply short-term fixes to failed short-term fixes (antivirus is a good example) with full knowledge that the long-term endgame is 100% unadulterated fail.
mjr.
I do also like #4, but that, to me, means you have lots of info being thrown at really good staffers. They could be overwhelmed but still able to anticipate. However, I love the idea of this as a desired state for your operations.
I like a sort of combination of #4 (which incorporates #3), #5, #6, and #7. But it still seems complicated to tell your CSO that this means "win."
I think you have a point, but at the risk of mistaking your message, I'd love to read those examples before getting on the wagon or throwing eggs. And no, I don't want examples just to pick them apart, but rather to just help understanding. :) (Dare I say that analogies always fail in the long term?)
One thing that comes to mind is how quickly technology changes, making our long-term gameplans largely moot, when you get beyond overarcing policies.
1. You meet legal and/or regulatory compliance standards (ex: PCI DSS para 12.9 requires that you have a CSIRP, and para 12.9.6 requires that you test it annually).
2. You're no longer being notified of data breaches to your infrastructure by external third parties weeks (or months) after they happen.
Sometimes, the small steps make the biggest wins...