tag:blogger.com,1999:blog-4088979.post4109722452555152051..comments2023-10-16T06:06:25.012-04:00Comments on TaoSecurity Blog: NORAD-Inspired Security MetricsRichard Bejtlichhttp://www.blogger.com/profile/13512184196416665417noreply@blogger.comBlogger10125tag:blogger.com,1999:blog-4088979.post-12586754697192117912009-06-16T06:39:51.598-04:002009-06-16T06:39:51.598-04:00This comment has been removed by a blog administrator.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-4088979.post-90719493012123307282007-07-19T22:44:00.000-04:002007-07-19T22:44:00.000-04:00G, the problem is no one is measuring the Y becaus...G, the problem is no one is measuring the Y because all the attention is on the little Xs. I am advocating measuring the Y. If we can truly identify Xs that influence Y, I'm all for that. Unfortunately I believe many of the Xs currently being measured have little effect on Y, with the additional problem that no one knows what Y is anyway. Therefore you can measure all the Xs you want and no one holds you accountable because the effect on the outcome is unknown.Richard Bejtlichhttps://www.blogger.com/profile/13512184196416665417noreply@blogger.comtag:blogger.com,1999:blog-4088979.post-65675314364973895222007-07-19T22:34:00.000-04:002007-07-19T22:34:00.000-04:00I agree with sovrevage, that it's often too expens...I agree with sovrevage, that it's often too expensive to measure outcome-based metrics. If the outcome is devastating to the entity measuring it (e.g., an airspace breach by hostile aircraft for NORAD, or a sensitive backup tape lost by the enterprise), then there appears to be little gain by using an outcome-based metric. Six Sigma would say that you measure the "little Xs" that drive the "Big Y." You need to identify what the Xs are first of course (and that's the hard part), but then you measure what you can control, knowing that these factors are what influence your outcomes.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-4088979.post-71013425346308927162007-07-19T14:00:00.000-04:002007-07-19T14:00:00.000-04:00Not to complicate things (but I'm going to anyway ...Not to complicate things (but I'm going to anyway :) - I would submit that outcome-based metrics work best when expressed as a ratio rather than as a stand-alone number. Example: Instead of measuring the success of an anti-spam filter by number of spams that get through (the lower the better), measure the number of spams that get through as a percentage of total spams received (number caught by the filter + number that get through.) This will prevent the metric from being skewed by a sharp rise or drop in the total number of spams received. Thoughts?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-4088979.post-11745899965267147182007-07-18T20:07:00.000-04:002007-07-18T20:07:00.000-04:00PaulM,I posted an entire reply here.Thanks for you...PaulM,<BR/><BR/>I posted an entire reply <A HREF="http://taosecurity.blogspot.com/2007/07/no-undetectable-breaches.html" REL="nofollow">here</A>.<BR/><BR/>Thanks for your comment!Richard Bejtlichhttps://www.blogger.com/profile/13512184196416665417noreply@blogger.comtag:blogger.com,1999:blog-4088979.post-50357134192399345612007-07-18T19:49:00.000-04:002007-07-18T19:49:00.000-04:00The problem with outcome metrics is that they are ...The problem with outcome metrics is that they are very difficult to measure. And might be impossible to measure if a breach has not yet occurred. Most people can't afford to do a "lets observe the next failure and then rethink our strategy." based approach.<BR/><BR/>I might be that I've misunderstood your proposal but from my experience it's already hard enough to be compliant within time and budget constraints.Unknownhttps://www.blogger.com/profile/10221381132926044944noreply@blogger.comtag:blogger.com,1999:blog-4088979.post-58672964475073083582007-07-18T16:37:00.000-04:002007-07-18T16:37:00.000-04:00Comparing your statement about XP-SP2 being owned ...Comparing your statement about XP-SP2 being owned by a custom exploit to the NORAD analogy, what if the enemy has a stealth plane that we cannot detect via radar, satellite, wind-speed variance, or any other deployed means? And what if your intel doesn't tell us that such a vehicle exists? Then we have potentially millions of airspace breaches every year and our outcome metrics are not helping.<BR/><BR/>I'm not disagreeing with you that outcome metrics are ideally better data than compliance metrics. However, outcome metrics are difficult to identify and collect data on, and it can be difficult to discern how accurate your metrics actually are.<BR/><BR/>At least with compliance metrics, we can determine how good we are at doing what it is we say that we do. It has little relevance to operational security, but it's easy and the auditors seem to like it.PaulMhttps://www.blogger.com/profile/02530533566781746778noreply@blogger.comtag:blogger.com,1999:blog-4088979.post-37319442232442774492007-07-18T14:33:00.000-04:002007-07-18T14:33:00.000-04:00This comment has been removed by a blog administrator.syferiumhttps://www.blogger.com/profile/12775831796453259222noreply@blogger.comtag:blogger.com,1999:blog-4088979.post-62202145074654638182007-07-18T12:04:00.000-04:002007-07-18T12:04:00.000-04:00This reminds me of the old saying "don't tell me h...This reminds me of the old saying "don't tell me how hard you worked, tell me what you got done."Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-4088979.post-41727778967686474102007-07-17T23:24:00.000-04:002007-07-17T23:24:00.000-04:00This comment has been removed by a blog administrator.Anonymousnoreply@blogger.com