Sunday, January 08, 2012

Telling a Security Story with Charts

The image at left appeared in the 31 December 2011 edition of The Economist magazine in the article Economics focus -- How to get a date: The year when the Chinese economy will truly eclipse America’s is in sight. It depicts 15 measurements of the US and Chinese economies, with historical and projected data. There is a version available at this page with more statistics comparing the two nations.

The Economist presents these charts for the following reason:

In the spring of 2011 the Pew Global Attitudes Survey asked thousands of people worldwide which country they thought was the leading economic power. Half of the Chinese polled reckoned that America remains number one, twice as many as said “China”. Americans are no longer sure: 43% of US respondents answered “China”; only 38% thought America was still the top dog. The answer depends on which measure you pick. (emphasis added)

The reason I like these charts is that they remind me of how many security practitioners think about "being secure." Managers likely often ask security staff "Are we secure?" The truth is there is no single number, so anyone selling you a "risk" number is wasting your time (and probably your money). However, it would be much more useful to display a chart like that created by the Economist. The security staff could choose a dozen or more simple metrics to paint a picture, and let the viewer interpret the answer using his or her own emphasis and bias.

Another reason I like the Economist chart is that the magazine built it using specified assumptions of future activity, listed in the article. If you disagree with these assumptions you can visit the second link I posted to devise your own charts. Although not shown here, what would be even more useful is showing these charts as a time series, with snapshots for January, then February, and so on. This "small multiples" approach (promoted by Tufte) capitalizes on the skill of the human eye and brain to observe and observe differences in similar objects.

If you had to pick a dozen or so indicators of security for a chart, what would you depict? The two I consider non-negotiable are 1) incidents per unit time and 2) time to containment for incidents.

5 comments:

Patton said...

The Economist's chart is very nice (though it is silly to be hung up over "supremacy," especially considering the vast differences in population and resources between the U.S. and China); like any good intelligence assessment, it is necessary to have the data behind the numbers ready to share. Busy people often need a binary answer, so I don't think executives will always want to see charts like this; but top managers should, and good security analysts should be well trained in collecting relevant data and communicating its relevance in formats like this.

Although you kind of dismissed the management question, "Are we secure," picking a dozen or so indicators of security seems to serve exactly that question. But I understand what you mean, and ultimately "Are we secure?" is what a high-level manager needs to know. Analysis merely serves the information requirements of the organization's leadership (and helps leadership understand what it needs to know).

In addition to observed threat activity, reflected in your selections (and defining "incident" as "successful unauthorized access to proprietary information systems" or something similar to that), the measure of an organization's security depends on the accessibility of its information and systems to unauthorized parties. So, I would include measures that reflect basic security requirements: patch deployment speed, the institutional penetration rate of secure computing awareness, the network presence of legacy and high-risk systems and devices, etc.

Still, any assessment must include the caveat that as good as the organization's security practices and record may appear on paper, there is always the chance that an intruder will get in and do damage or steal valuable information. A positive security assessment should not permit decision makers to relax.

A. Thulin said...

It's a matter of how the term 'security' is defined. My favourite definition (though currently impratical) is 'Security is a measure of how well a system withstands the effects of unwanted events' -- the impracticality being that there is no accepted way to measure it.

However, if there is a definition, it's probably found in the insurance world, and if so it almost certainly contains the factors cost and time at the very least.

For that reason, I would suggest that important indicators must help provide estimates for cost and time. Once we have those, estimates of efficiency may be appropriate additions.

Thus, the only non-negotionable indicator I can imagine would be cost per incident over some suitable period of time.

Anonymous said...

assuming your company measures such things!

John Ward said...

Any good chart or report should tell a story that supports the question at hand. The chart depicted tells a high level story of in what year China did/will overtake the US in the given categories. To make it even better would assume that there is drill down into those categories with details to support that story, or that this was part of a dashboard.

Personally, I think asking "how secure are we" is a bad question. In terms of security, how do you define secure. There is the quantifiable metrics that can tell the story of "How have we fared so far?". This brings me back to the old BATC days where we had those god awful reports we had to do, but they supported several different stories, such as "How visible are we?", "How many external probes did we encounter?", "How many incidents were due to internal policy violations?", and "How many external intrusion attempts were successful?". With these metrics, you could easily tell good stories, such as "The security improvements we implemented have reduced external intrusions", and "our corporate policy changes and enforcement have reduced policy violations on the network by this much". Asking "how secure are we" based on these metrics is a bad idea however, because you can say "oh, we're very secure, because we have reduced external intrusions to almost nothing, haven't had policy violations in over a year, and we aren't getting probed as often due to new firewall rules", then BAM you get hit by some 0-day exploit, ruining all credibility in that statement with management.

Digital Forensics said...

Fantastic graphic. Thank you. I'll be printing a copy out for sure.

Although not relevant to reg analysis, one suggestion would be to include the setupapi.dev.log to show the first time a device was connected. It just be nice to see everything in one graphic.