Monday, May 24, 2010

Forget Pre-Incident Cost, How Much Did Your Last Incident Cost?

I just read this great post by Rich Mogull titled FireStarter: The Only Value/Loss Metric That Matters.

His basic argument, or at least the idea that I derived from it, is the following (all in my own words).

So-called "risk managers" spend a lot of time imagining they can determine "annualized loss expectancy" by predicting how much an incident will cost. Forget all that nonsense. Before imaging what a future incident will cost, figure out how much your last incident cost.

This is brilliant because it is so simple yet drives straight at the heart of the problem. We work incidents all the time and I can't tell you how much they cost. Think about all the factors to consider:

  • Value of professional time of everyone who detected and responded to the incident

  • Value of computing resources affected by the incident

  • Value of data affected by the incident, whether disclosed, degraded, or denied

  • Value of brand, reputation, and other "goodwill" items

  • What else can you imagine?


So, think about answering these questions for a really good recent interest before wasting time imagining costs of future incidents. I think what you will find is that this can be a really difficult exercise. However, if you can derive some general guidelines, it's worth it.

13 comments:

The Ubiquitous Mr. Lovegroove said...

And what about those enterprises that are yet to experience a note worthy incident?

itinsecurity said...

Well, sounds like a great idea. However, what makes you think it's easier to estimate actual loss, than to predict it?

In particular I am thinking about the last three items on your list:
Value of data, Value of brand, and whatever else you can imagine.

While it conceptually makes more sense, I don't quite see how it is going to be easier or better, given the "intangible unknowns" of the equation.

Richard Bejtlich said...

itinsecurity, that's basically my point. If you can't figure out a way to estimate post-incident cost, you need another way to measure incidents altogether.

Martin said...

My org has had a fair amount of success justifying security by adding another point which is very straightforward to measure: the amount of employee time lost while the asset is remediated. In other words, treat all incidents as having a DoS component for the asset involved. Even re-imaging machines takes an hour or two to get the employee back to work. Multiply that times the billable consultant rate for your employees, and you have something concrete to start with.

Martin said...

@Lovegroove Your org either has extremely draconian web policies, is extremely small, or you're just like Google, Adobe, etc. who didn't realize they'd been hacked for years. I suppose it's also possible that your definition of "note-worthy" is very different than most.

gunnar said...

Your list is a good start, the first three are internal metrics which are good. The fourth "brand, reputation, .." is external but harder to measure, however its worth pursuing. Some other external metrics that are worth gathering - how many customers did you lose (which in some cases is hard to tell), how much $ loss did your customer suffer (which in some cases you can get a very good idea of)

Anonymous said...

Richard,

Having done a fair bit of research into some of the bigger breaches - TJX and Heartland, for example - one of the things that becomes clear is the the cost of a breach unfolds over a period of years.

If you look at the SEC 10-K filings from TJX for 2007, 2008, and 2009, you will see that the while actual costs are recorded as such, there is also a reserve created that goes up and down in value as things are settled, estimates of future liability are resolved, etc.

The same seems to be true of Heartland.

I am not sure what the statute of limitations is with regard to a data breach - maybe a lawyer could respond, if one is following this discussion.

Patrick Florer
Dallas, Texas

Anonymous said...

All -

Something I forgot to mention -

For some breaches, fines and legal settlements can be significant, expecially when PCI contracts apply -

It was just announced in the press that Heartland and MasterCard had reached a tentative settlement for $41M over that breach.

With regard to brand, reputation, competitive advantage, stock price - I would forget about trying to quantify these things - it's too difficult, and such data as there are do not support any particular conclusion.

Patrick Florer
Dallas, Texas

Russell Thomas said...

While it may feel comforting to only measure historical costs of incidents (if you can get the data), it is simply inconsistent with rational economic decision-making.


ALL economic decisions are based on estimates of future cashflows, discounted for the time-value of money. If you can't estimate the cash flows, at least within some range, then you can't make rational economic decisions. Period.

Historical costs are only meaningful as input to future cost estimates.

By "rational economic decisions", I mean choices between security spending and other corporate spending, choices between spending now vs. spending later, choices between alternative security practices or policies or investments.

Now, you may be suggesting exactly this -- that information security is beyond the reach of economic rationality, and therefore we should fall back on other methods such as guidelines, heuristics, etc. that we can only hope will lead to better decisions and better results than simple-minded methods (hunch, impluse, fad-following, blind-leading-the-blind, etc.).

Lastly, I'd like to point out that historical costs associated with "brand, reputation, and other 'goodwill' items" play out over time, often many years. Thus the full costs won't be recognized if the incident is recent. This is the challenge that the Ponemon survey faces. They "solve" it by using some rule-of-thumb formulas for number of customers lost, cost to replace a customer, etc. but that is simply a way of forecasting future costs.

For more on this, you might look at the blog post I wrote recently about the general problem of incorporating the time dimension in security metrics: http://newschoolsecurity.com/2010/05/getting-the-time-dimension-right/

Richard Bejtlich said...

RT,

I thought it was obvious that the purpose of determining historical cost would be as an input for future decision making? The point I was trying to emphasize is that too many managers start backwards in my opinion, creating fantasies about future costs but never having measured past costs.

Russell Thomas said...

I want to offer an approach that might bridge the approaches.

I'm a big fan of measuring/estimating TOTAL costs of information security, including costs of prevention, detection, forensics, backup/recovery, incident response, help desk, security management, etc.

But I think this needs to be projected forward. The best way to do this is to look at BUDGETED costs, especially outside the security department. It takes some effort to tease out costs related to security (using cost drivers) but it's possible.

But this only works for the frequent and expected costs. TJX and Heartland do not budget in the future for all the costs that they incurred in their past (infamous) breaches.

There's no way around it -- you need to have some way of roughly estimating low probability, high impact costs, and your "willingness to pay" to avoid them.

Such a method is outlined in this presentation: http://meritology.com/resources/Total%20Cost%20of%20Cyber%20(In)security.ppt

But let me be clear -- this method is not yet ready for mainstream use. There are serious unsolved research problems. But I think they can be solved, at least to an acceptable level.

Charles Skamser said...

Brilliant Topic!

eDiscovery Solutions Group (www.ediscoverysolutionsgroup.com) actually has an incident management system that can track cost and therefore address this requirement.

The Ubiquitous Mr. Lovegroove said...

@Martin,
My slightly provocative question was aimed at the perception that security incidents will always be identified and have measurable cost.

Lost productivity is the only measure I can imagine which can be reliably tracked.