Wednesday, October 31, 2007

A Plea to the Worthies

You may have seen stories like Cybersecurity Experts Collaborate with subtitles like A think tank has tapped several heavyweight security experts to staff a commission that will advise the president. That story continues:

The Center for Strategic and International Studies (CSIS) wants the commission to come up with a list of recommendations that the new president who takes office in January 2009 "can pick up and run with right away," said James Lewis, director of the CSIS Technology and Public Policy Program. The commission, made up of 32 cybersecurity experts, plans to finish its work by the end of 2008. I am fairly confident that nothing of value will come from this group, but there is one task which could completely reverse my opinion. Rather than wasting time on recommendations that will probably be ignored, how about taking a step in a direction that will have real impact: security metrics. That's right. Spend the first day (or two, if you are a slow reader or can't sit still for long periods) reading Andy Jaquith's book. Next, and this is the crucial part:

Figure out how to play and score the game before you pretend to think you can improve the score.

What does this mean? Just a few ideas include:

  • Propose definitions for security, risk, threat, vulnerability, inside threat, external threat, and all the other words we use yet upon which we never agree. Hold hearings and invite real security people (not just digital security people) to express their views.

  • Propose some metrics and see how other operations define success. Hold hearings on the results of that process.

  • Apply metrics to some real organizations and gain a baseline set of numbers. Repeat the process at determined time intervals. Try to identify correlations and if possible causations. Be anonymous if necessary, but use a real methodology and not the self-selection applied by CSI/FBI and others.

Do you see where I am going here? At the end of the process we could have a framework for seeing just what is happening. I defy anyone to tell me just how bad or good our digital security situation is right now. Some say the sky is falling, others say we're happy! happy!, others say we're just as secure as we need to be to continue limping along. It is a proper role for a panel of worthies to help figure out how the game is played and then what the score is. It is a waste of time to make recommendations before those basic steps have been taken.


Siraj said...

I agree, we certainly need "definitions for security, risk, threat, vulnerability, inside threat, external threat, and all the other words we use yet upon which we never agree." But I wonder, and it is a general point here, whether any such framework would help? Because I think definitions and concepts generally are defined and clarified by their use over a period of time, by a variety of different people (academics, practitioners, critics, etc), after many deliberations and discussions. A framework like such may help a particular Govt or a corporation - but would it find wider practical and academic acceptance I wonder. And what would we need to do for such a framework to do so?

But then - perhaps paradoxically - what did I mean when I say "we certainly need definitions..." at the start of this comment? Every time I come across a definition, I try to be careful to appreciate the context in which its used, who is it used by and so on.

Ted said...

I read another article about this yesterday, here:

My reaction was very similar to yours. The probability that meaningful contributions will be manifested from this committee is slim. Optimistic scenario: work that is already done is repeated. Pessimistic scenario: more (potentially conflicting) compliance requirements and legislation. :(

This quote,

"Langevin said. 'I expect the recommendations will be a solid document that we can rely on to better secure our networks.'"

What is does "better" mean? Certainly we know how to "do more security", typically with limited real impact.

I think that we are defined by what we don't know. It is like the "badness meter" that MJR refers to.

We might be bad/good, and we know we want to get "better" until we are "good enough." (Whatever all that means).

Your post, here:

Makes more sense than is likely to come from this committee.

rybolov said...

Hi Richard

I was writing a response, but it was so long I made it into a full blog post. =)

Kevin said...

I heard a talk from this from a PhD candidate named Ross Goeres this week on why IT security metrics are so horrible. And it comes down to the statistical methods are wrong and the math doesn't work. Some of the numbers ended up, if you put units on them, of being something like dollars^2/feet^3. And why and how CVSS is so typically nearly useless. Better yet, he has some very interesting ideas on how to fix them that I think people will hear a lot more about.

VooDoo said...

Sour grapes? This is a commission that will result in a recommended cybersecurity AGENDA for the next President. C'mon Richard...think before you blog.

Richard Bejtlich said...

VooDoo, take your own advice. I've since learned this post has been seen by at least one person on this commission. That is why I wrote it.

Sektör said...
This comment has been removed by a blog administrator.
design directory said...

Thank you.