Defining the Win

In March I posted Ten Themes From Recent Conferences, which included the following:

Permanent compromise is the norm, so accept it. I used to think digital defense was a cycle involving resist -> detect -> respond -> recover. Between recover and the next attack there would be a period where the enterprise could be considered "clean." I've learned now that all enterprises remain "dirty" to some degree, unless massive and cost-prohibitive resources are directed at the problem.

We can not stop intruders, only raise their costs. Enterprises stay dirty because we can not stop intruders, but we can make their lives more difficult. I've heard of some organizations trying to raise the $ per MB that the adversary must spend in order to exfiltrate/degrade/deny information.‏
(emphasis added)

Since then I've grappled with this idea of how to define the win. If you used to define the win as detecting and ejecting all intruders from your enterprise, you are going to be perpetually disappointed (unless your enterprise is sufficiently small). Are there are alternative ways to define the win if you have to accept permanent compromise as the norm? The following are a few ideas, credited where applicable.

The first two come from my post Intellectual Property: Develop or Steal, but I repost them here for easy reference.

  1. Information assurance (IA) is winning, in a broad sense, when the cost of stealing intellectual property via any means is more expensive than developing that intellectual property independently. Nice idea, but probably too difficult to measure.

  2. IA is winning, in a narrow sense, when the cost of stealing intellectual property via digital means is more expensive than stealing that data via nontechnical means (such as human agents placed inside the organization). Still difficult to measure, but might be estimated using red teaming/adversary simulation/penetration testing.

  3. IA is winning when detection operations can see the adversary's actions. This relates to Bruce Schneier's classic advice to Monitor First. The more mature answer is next.

  4. IA is winning when incident responders can anticipate the adversary's next target. I credit Kevin Mandia with this idea. I like it because it shows that complex enterprises will always have vulnerabilities and will always be targeted, but a sufficiently mature detection and response operation will at least be able to guess the intruder's next move. You can even test this by keeping a track record.

  5. IA is winning when the time to detect and remediate has been reduced to B. Insert your own value there. You can track your progress from time A to time B.

  6. IA is winning when your enterprise security integrity assessments show less than D percent of your assets are compromised. You can track progress from C percent to D percent over time. This leads to the more mature version which follows.

  7. IA is winning when your enterprise intrusion debt is reduced to F. You can measure intrusion debt as you like and take steps to reduce it from E to F.

Does anyone else have ideas on how to define the win?

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.


Anonymous said…
"Does anyone else have ideas on how to define the win?"

IAF is winning when the business it protects has durable competitive advantage.
This is similar to your option 7, IA is winning when: E - F > IAcost
So, the best wining is: Max(E - F - IAcost)
Anonymous said…
In Six Sigma speak (they're still doing that at GE, right?!) you're describing "critical to quality characteristics" (CTQs) of a winning security strategy. However, for a CTQ to be useful, it must be be measurable (where the measurement is repeatable, cheap, and precise).

I think #5 and #6 are measurable...perhaps could wrap a measurement around #3. But I don't see a good way to measure the rest of them reliably. I don't have a suggestion yet for alternatives, but am giving it some thought.
John said…
IA is winning at your company when the cost of stealing IP via digital means is more expensive than stealing someone else's. Real world equivalent-a thief checks the locks on a store's doors at 3:00 in the morning, finds that it's locked, moves to the store next door, their door isn't locked. Which store gets broken into?

While certain companies have specific IP or data that may be desireable for a bad actor to steal, most crime, and I would argue cybercrime is no different, is more like the situation described above. IA doesn't win in the long term if it's easier just to try someone else down the block. Just like the shopping mall where those two stores are located doesn't win.
Anonymous said…
My 2 cents at a high level

Low Bar Win - You can at least reliably detect when your owned.

Medium Bar Win - You are doing everything within your control and more importantly budget to secure your network and data. And of course documenting it :-)

High Bar Win - Nobody has a foothold on your network, your data is safe, and you can continuously prove it.
Anonymous said…
John, you sound so noble. But do you have exterior lights on your house at night? Do you lock your doors at home when you leave? Have you bought a home alarm system?

If so, you're encouraging the thieves to move on down the street and (yes, unfortunately) break into a neighbor's home. It would be admirable of you to start a neighborhood watch, join the local crime-stoppers association, etc. But I bet you're not doing nightly patrols of your neighborhood on foot, or helping your neighbors fund an alarm system for their homes. Bottom line: you're looking out for your own home, and if doing so causes the burglar to rob your neighbors instead, then you'll be satisfied that you've done your job, right? Why is the cyber world any different?

I'm sure you'd like all crime in your city to cease, and I'm sure the above commenter would like all cybercrime to go away. We live in the real world though, and winning is relative, not absolute.
Anonymous said…
John - upon re-reading your post above, maybe we're actually in violent agreement... I'm still not sure if you're saying IA *is* winning when it's easier to steal someone else's, or if you're disagreeing with that perspective (in the last paragraph, you say "IA doesn't win...if it's easier to try someone else down the block," so I'm not sure).

Anyhow, I agree with your statement that "IA is winning at your company when the cost of stealing IP via digital means is more expensive than stealing someone else's."
KS Lee said…
I think that the most challenging part here is the quantitative measure of security in any network-the concept of "security metric". It will be useful but may be difficult to define all the times. e.g. for web application security.
Anonymous said…
IA has lost its focus if the definition of winning does not use information and assurance in the metrics.

3 & 4 are information counterintellingence not assurance. Information counterintelligence can help provide metrics for use in determining if you are winning or losing.

IA is winning when you are not losing.

IA has lost when protected information is released at the rate or cumulative magnitude above acceptable limits and used against the organisation.

IA stays lost until the released protected information no longer has an impact on the organisation.

This definition of winning illustrates some points.
- Some loss is expected.
- You need to know what the value of your information is
- You need to know the value of protecting your information from compromise
- If information is compromised but never used then the actual loss is minimal
- If you only have one secret then you need to protect it well. If you are continuously generating secrets then a single loss can be sustained better
- Effective response and recovery is required to stop losing
John said…
I think we're more in agreement than not. Initially I was making the point that most crimes are crimes of opportunity and that if we've got the locks on the doors but our neighbor doesn't, that's not good for the whole neighborhood. On re-reading everyone's posts on this discussion, I think we're all grappling with Richard's original problem of 'defining the win'. I originally took IA to mean the industry in general. On re-reading the OP I realize that Richard seemed more focused on an individual org's IA. He's a little more tree-focused, I'm a little more forest-focused. OK.

One of the chief things we're dealing with in security is trust. We are in the business of gauranteeing to the best of our abilities our business's customers' trust. That's it. While our individual company may be worthy of that trust, what happens when an industry as a whole is viewed as a "bad neighborhood", not worthy of trust? I ignore all emails, legitimate or not, from Ebay, Wal*Mart, Bank of America, and several other companies. Is that what those companies want? I doubt it. What happens to a company when they get lumped in with spammers, TJ Maxx, etc.? How do you prevent the customer lumping you in with those guys in the first place? I think that's a pretty big deal and it's a hard question to answer. But if you ignore what's going on down the block, you may end up with the most amazingly secure business that no one goes to anymore. That means no more paycheck. That's bad.
Anonymous said…
It sounds like everyone's defining "the win" as "we lost but don't want to admit it."

I still see things in black and white, after all these years.

Anonymous said…
OK, I need to elaborate on my previous comment.

Lots of those "wins" are short-term rearguard actions and are nothing resemblin "winning" in a long-term context. And, in case nobody noticed, this is a long-term problem. Caving in (which is the usual eventual position of security) often results in a negotiated position that somewhat reduces risk in the short term but in the long term probability says that sooner or later, there will be a failure. That's the fallacy of risk reduction approaches: they're just an attempt to paper over the problem until someone else takes it over and hopefully it's their disaster to deal with later.

I could give countless examples of how this short/long-term dynamic has played itself out in security - in fact, the entire security industry is intellectually twisted around trying to apply short-term fixes to failed short-term fixes (antivirus is a good example) with full knowledge that the long-term endgame is 100% unadulterated fail.

Unknown said…
Quite the doozy of a post, and a dilemma trying to define the win.

I do also like #4, but that, to me, means you have lots of info being thrown at really good staffers. They could be overwhelmed but still able to anticipate. However, I love the idea of this as a desired state for your operations.

I like a sort of combination of #4 (which incorporates #3), #5, #6, and #7. But it still seems complicated to tell your CSO that this means "win."
Unknown said…
I think you have a point, but at the risk of mistaking your message, I'd love to read those examples before getting on the wagon or throwing eggs. And no, I don't want examples just to pick them apart, but rather to just help understanding. :) (Dare I say that analogies always fail in the long term?)

One thing that comes to mind is how quickly technology changes, making our long-term gameplans largely moot, when you get beyond overarcing policies.
H. Carvey said…
At this point in the game, I'd settle for defining a "win" as:

1. You meet legal and/or regulatory compliance standards (ex: PCI DSS para 12.9 requires that you have a CSIRP, and para 12.9.6 requires that you test it annually).

2. You're no longer being notified of data breaches to your infrastructure by external third parties weeks (or months) after they happen.

Sometimes, the small steps make the biggest wins...

Popular posts from this blog

Zeek in Action Videos

MITRE ATT&CK Tactics Are Not Tactics

New Book! The Best of TaoSecurity Blog, Volume 4