Tuesday, July 17, 2007

NORAD-Inspired Security Metrics

When I was a second degree cadet at USAFA (so long ago that, of my entire class, only myself and three friends had 486 PCs with Ethernet NICs) I visited NORAD. I remember thinking the War Games set was cooler, but I didn't give much thought to the security aspects of their mission.

Today I remembered NORAD and considered their mission with respect to my post last year titled Control-Compliant vs Field-Assessed Security. In case you can't tell from the pithy title, the central idea was that it's more effective to measure security by assessing outcomes instead of inputs. For example, who cares if 100% of your systems have Windows XP SP2 if they are all 0wned by a custom exploit written just for your company? Your security has failed. Inputs are important, but my experience with various organizations is that they tend to be the primary means of "measuring" security, regardless of how well they actually preserve the CIA triad.

Let's put this in terms of NORAD, whose front page states:

The North American Aerospace Defense Command (NORAD) is a bi-national United States and Canadian organization charged with the missions of aerospace warning and aerospace control for North America. Aerospace warning includes the monitoring of man-made objects in space, and the detection, validation, and warning of attack against North America whether by aircraft, missiles, or space vehicles, through mutual support arrangements with other commands. Aerospace control includes ensuring air sovereignty and air defense of the airspace of Canada and the United States...

To accomplish the aerospace warning mission, the commander of NORAD provides an integrated tactical warning and attack assessment to the governments of Canada and the United States. To accomplish the aerospace control mission, NORAD uses a network of satellites, ground-based radar, airborne radar and fighters to detect, intercept and, if necessary, engage any air-breathing threat to North America.


What are some control-compliant or input metrics for NORAD?

  • Number of planes at the ready for intercepting rogue aircraft

  • Average pilot rating (i.e., some sort of assessment of pilot skill)

  • Radar uptime

  • Radar coverage (e.g., percentage of North American territory monitored)


These are all interesting metrics. You might see some comparisons to metrics you might track, like percentage of hosts with anti-virus.

Now consider: do any of those metrics tell you if NORAD is accomplishing its mission? In other words, what is the outcome of all those inputs? What is the score of this game?

Here are some field-assessed or outcome-based metrics.

  • Number of rogue aircraft penetrating North American territory (indicates a failure to deter activity)

  • Number of aircraft not detected by NORAD but discovered via other means to have penetrated North American territory (perhaps via intel sources; indicates a failure to detect activity)

  • Number of aircraft not repelled by interceptors (hopefully this would never happen!)

  • Time from first indication of rogue aircraft to launching interceptors (indicates effectives of pilot-to-plane-to-air process)


These metrics address the critical concern: accomplishing the mission.

Keep these in mind when you are devising metrics for your digital security program.

10 comments:

Anonymous said...
This comment has been removed by a blog administrator.
Kyle said...

This reminds me of the old saying "don't tell me how hard you worked, tell me what you got done."

syferium said...
This comment has been removed by a blog administrator.
PaulM said...

Comparing your statement about XP-SP2 being owned by a custom exploit to the NORAD analogy, what if the enemy has a stealth plane that we cannot detect via radar, satellite, wind-speed variance, or any other deployed means? And what if your intel doesn't tell us that such a vehicle exists? Then we have potentially millions of airspace breaches every year and our outcome metrics are not helping.

I'm not disagreeing with you that outcome metrics are ideally better data than compliance metrics. However, outcome metrics are difficult to identify and collect data on, and it can be difficult to discern how accurate your metrics actually are.

At least with compliance metrics, we can determine how good we are at doing what it is we say that we do. It has little relevance to operational security, but it's easy and the auditors seem to like it.

sovrevage said...

The problem with outcome metrics is that they are very difficult to measure. And might be impossible to measure if a breach has not yet occurred. Most people can't afford to do a "lets observe the next failure and then rethink our strategy." based approach.

I might be that I've misunderstood your proposal but from my experience it's already hard enough to be compliant within time and budget constraints.

Richard Bejtlich said...

PaulM,

I posted an entire reply here.

Thanks for your comment!

Anonymous said...

Not to complicate things (but I'm going to anyway :) - I would submit that outcome-based metrics work best when expressed as a ratio rather than as a stand-alone number. Example: Instead of measuring the success of an anti-spam filter by number of spams that get through (the lower the better), measure the number of spams that get through as a percentage of total spams received (number caught by the filter + number that get through.) This will prevent the metric from being skewed by a sharp rise or drop in the total number of spams received. Thoughts?

G said...

I agree with sovrevage, that it's often too expensive to measure outcome-based metrics. If the outcome is devastating to the entity measuring it (e.g., an airspace breach by hostile aircraft for NORAD, or a sensitive backup tape lost by the enterprise), then there appears to be little gain by using an outcome-based metric. Six Sigma would say that you measure the "little Xs" that drive the "Big Y." You need to identify what the Xs are first of course (and that's the hard part), but then you measure what you can control, knowing that these factors are what influence your outcomes.

Richard Bejtlich said...

G, the problem is no one is measuring the Y because all the attention is on the little Xs. I am advocating measuring the Y. If we can truly identify Xs that influence Y, I'm all for that. Unfortunately I believe many of the Xs currently being measured have little effect on Y, with the additional problem that no one knows what Y is anyway. Therefore you can measure all the Xs you want and no one holds you accountable because the effect on the outcome is unknown.

http://www.architectsban.webs.com said...
This comment has been removed by a blog administrator.