Salvaging Poorly Worded Statistics
Today I joined a panel held at FOSE chaired by Mischel Kwon and featuring Amit Yoran. One of the attendees asked the following:
At another session I heard that "80% of all breaches are preventable." What do you think about that?
My brief answer explained why that statement isn't very useful. In this post I'll explain why.
The first problem is the "80%." 80% of what? What is the sample set? Are the victims in the retail and hospitality sectors or the telecommunications and aerospace industries? Speaking in general terms, different sorts of organizations are at different levels of maturity, capability, and resourcefulness when it comes to digital security.
In the spirit of salvaging this poorly worded statistic, let's assume (rightly or wrongly) that the sample set involves the retail and hospitality sectors.
The second problem is the term "breach." What is a breach? Is it the compromise of a single computer? (What's compromise? Does it mean executing malicious code, or login via stolen credentials, or...?) What is the duration of the incident? There are dozens of questions that could be asked here.
To salvage this part, let's assume "breach" means "an incident involving execution of unauthorized code by an unauthorized intruder" on a single computer.
The third problem is the word "preventable." "Prevention" as a concept is becoming less useful by the second. Think about how an intruder might try to execute malicious code against a victim. Imagine a fully automated attack that happens when a victim visits a malicious Web site. An exploit kit could throw a dozen or more exploits against a browser and applications until one works. Are they all non-zero day, or are some zero day? Again, many questions beckon.
To salvage the end of the original statement, let's translate "preventable" into "exploitation of a vulnerability for which a patch had been publicly available for at least seven days."
Our new statement now reads something like "In the retail and hospitality sectors, 80% of the incidents where an unauthorized intruder successfully executed unauthorized code on a single computer exploited a vulnerability for which a patch had been publicly available for at least seven days."
Isn't that catchy! That's why we heard shortcuts like the original statement, which are basically worthless. Unfortunately, they end up driving listeners into poor conceptual and operational models.
The wordy but accurate statement says nothing about preventability, which is key. The reason is that a determined adversary, when confronted by a fully patched target, may decide to escalate to using a zero-day or other technique for which patches are irrelevant.
Comments
Great post! In a lot of ways, Twitter (and other abbreviated communications mediums, to include the option for simply clicking "Like" or "+1") have really done a lot to decimate the already poor communications skills of many.
One thing I keep in mind when I hear folks quoting any of the annual reports is that each of the organizations providing those reports has a different customer base, different focus, etc. Readers have to keep in mind which statistics are based on a minimalist approach to response, and which may be based on a more in-depth approach to intelligence gathering.
Finally, what is "preventable"? I've worked engagements where a patch would've "prevented" the incident that we uncovered, although we have no idea if the attacker would've walked away if the attempt had failed, or if they would have tried something else.
In trying to encapsulate something like the final statement in your post into 140 characters for easier transmission, or to be "the first" to get it out, we're not distilling the issue down to it's essential components, as much as we're just throwing away accuracy.