Wednesday, February 10, 2010

Thor vs Clown

It started with this post by M.D.Mufambisi to the pen-list list:

Im designing an SMS baking application but i need to research on the security risks involved first... What are the risks around this application? How are such applications normally subverted? Are there any case studies someone can point me to?

After a few responses, Craig Wright chimed in:

The solution needs to be based on risk.

Where a system uses an SMS response with a separate system (such as a web page), the probability that the banking user is compromised and a fraud is committed, P(Compromise), can be calculated as:

P(Compromise) = P(C.SMS) x P(C.PIN)

Where: P(C.SMS) is the probability of compromising the SMS function and P(C.PIN) is the compromise of the user authentication method


Craig followed up with a blog post:

Many people feel that it is not feasible to model risk quantitatively. This of course is blatantly false. In the past, many of the calculations have been computationally infeasible at worst and economically costly at best. This has changed. The large volumes of computational power that is available coupled with novel stochastic methods has resulted in an efficiently viable means of calculating risk quantitatively with a high degree of accuracy. This can be measured as a function of time (as survival time), finance (or monetary value) or any number of other processes...

Tim Mullen, a guy who I first met in 2002 teaching at Black Hat, responded on full-disclosure:

I'm looping in the FD list because often my replies don't make it to Pen-Test, and this has hit a nerve with me.

I've looked over your post...

Once I was able to get past the overwhelming egoism and self-substantiating claims of your contributions to the industry, I arrived at the conclusion that the only portion of the aforementioned page that is not complete drivel and even laughable to anyone who has actually worked towards ascertaining actual risk in production environments, is where you describe your own words as "ravings..."

I'm fine with you sitting back and gloating about the Security Hero award you got from Northcutt, but when I see that you are actually contributing to ANY level of Critical Infrastructure Protection, it makes me fear for anyone who might be counting on your presumed skillset to actually make intelligent decisions about risk where human safety is at stake.

Your "risk formula" is ridiculous. What number would your formula have yielded 2 weeks before SQL Slammer was released? Where is the variable for unpatched systems? What number do we plug in for malicious employee factorization? More importantly, where is the calculation for self absorbed snake-oil selling academics with no real experience using their calculator to come up with magic numbers that represent the risk of a nuclear power plant being hacked?

Since you are (self-described) as "currently the only GIAC GSE (Compliance) holder globally and the most highly accredited Global Information Security Professional" and thus (presumably, if only in your mind) the greatest security mind in the world, how about accepting a challenge to an open debate on the subject at Defcon? People like you are dangerous and need to be exposed before someone in a position of power actually believes that you know what you are talking about. Bring your abacus.


Craig then responded with some sort of monetary challenge, and Tim and Craig are now debating how to arrange that.

If you want history on why I consider model = clown, please check out the posts on my clown tag.

When I read

"In the past, many of the calculations have been computationally infeasible at worst and economically costly at best. This has changed. The large volumes of computational power that is available coupled with novel stochastic methods has resulted in an efficiently viable means of calculating risk quantitatively with a high degree of accuracy."

it is clear to me Craig is pretty well disconnected from reality. Did we not just suffer a global recession exacerbated by clowns who thought they could model risk "with a high degree of accuracy"?

33 comments:

Anonymous said...

As a former employee of Craig Wright, this blog reminds me of the days when he ran his network security company using unsecured WiFi, reachable from a car park across the road.

Anonymous said...

Its never good when one starts to believe their own hype. Sadly, it appears that Wright has become religious to the point of Jihad about his.

Mark said...

I haven't read all of the discussion so I'm talking of /orifice/alt here, BUTT...

Given a large enough set of empirical data I don't think that quantitative measures of risk are impossible to achieve. Insurance companies use quantitative measures of risk related to cybercrime. Are they accurate? I'm skeptical.

Craig Wright is a smart guy and I am interested in hearing how he calculates risk. Skepticism is merited given the lack of successful quantitative risk methodologies that exist. But I don't think the personal venomous attacks are merited until he has a chance to present his case.

jrodenbiker said...

Insurance works not so much because of their risk models, but because they do everything they can to increase their premiums and/or add customers, and then do everything they can to not pay out benefits.

higB said...

I don't know about these fancy equations or this chest pounding.. but I do know that we >>really<< hack phones and we >>really<< hack applications on phones that use SMS directed messages as control mechanisms. We eat mobile phones/apps and shit out hacks weekly.

Richard Bejtlich said...

higB: man that is funny.

Anonymous said...

There are two things I would like to point out:
1. Those who have never modelled anything critizing models is just as funny as those who have never actually done any real security work thinking models solve problems.

2. Don't blame the models. Your everyday things, like the car you use, bridge you cross, the building you work at, are modelled first then built. The right person with the appropriate knowledge can model things well enough to avoid trial-and-error.

3. The recession wasn't caused by models. It was caused by those ego and greed. They never factored those into the formula. A tool is only as good as it's user. People still make money modeling markets a ton of money. When it comes down to it these are in fact simple trend models of past, present, and future. That's why they need high frequency trading.

Mark said...

"1. Those who have never modelled anything critizing models is just as funny as those who have never actually done any real security work thinking models solve problems."

I didn't have to be a 1890's aeronautical engineer to recognize that none of the models worked and many of them merited criticism. (http://www.youtube.com/watch?v=kEdtvct6Tf0) So I think criticism of models is fair if models have failed to work effectively in the past.

Maybe the next Wright brothers first name is Craig. We will see.

Kyle Maxwell said...

Craig Wright probably wouldn't run into this sort of 'personal venomous attacks' if he didn't write in such a bombastic and impenetrable fashion. In communications, style counts. If you don't think so, imagine sending such a response in total 1337-speak and see how seriously you get taken.

At this point, the debate is largely about his self-importance and blowing smoke with misleadingly precise calculations, rather than the core concept of quantitative risk estimation. That's a shame, because the questions surrounding that core concept deserve further research.

Richard Bejtlich said...

Anonymous said "Those who have never modelled anything critizing models is just as funny as those who have never actually done any real security work thinking models solve problems."

Funny, I built models in undergrad, during a summer program during undergrad, and in graduate school. We did silly exercises like "how many non-white South Africans need to graduate from school each year in order to change the racial mix of the government's civil service by 2000, 2010, etc." or "how many Marines did it take to overwhelm Japanese positions in the South Pacific," and so on. Those and related exercises taught me the futility of applying those approaches to information security "risk."

Raymond said...

2. Don't blame the models. Your everyday things, like the car you use, bridge you cross, the building you work at, are modelled first then built. The right person with the appropriate knowledge can model things well enough to avoid trial-and-error.

I think it's important to point out that the things you reference are modeled against the laws of physics, aerodynamics, etc... Given buildings for example, you could model for weather variances using statistical averages over long periods to allow for things like snow loads. I'm fairly confident that the same model does not make allowances for the risk of someone driving a truck full of explosives into the same building.

I for one would argue that attempting to model risk in search of an absolute, quantitative value is an exercise in futility, bordering on delusional.

Dr Anton Chuvakin said...

>P(Compromise) = P(C.SMS) x P(C.PIN)

Guys, I am sure he meant it in jest. Nobody will suggest such hilarity serious, now, will they?

You should check the tag, maybe his post is also labeled "humor."

If not....well... most asylums would admit him, I am sure. For others, he just might be too insane.

gunnar said...

Did we not just suffer a global recession exacerbated by clowns who thought they could model risk "with a high degree of accuracy"?

Mostly agree, but I think the blame does not all go to models themselves. Everyone who deals with complexity has a model. The problem as you imply is the assumption that the model contains a "high degree of accuracy", also called "false precision" The idea that the model is reality is the dangerous one and the gaps in assurance and security are what we live with when this is the case, so you get the problems that swamp your whole model, as Charlie Munger says - "when you mix raisins and turds, you still have turds"

Or as Warren Buffett said (before the aforementioned false precision induced financial crisis) - "advocates of efficient market theory assume that the market is always efficient instead of mostly efficient. The difference between these two is night and day."

Michael Cloppert said...

I don't want to comment specifically on any individuals involved in what from a cursory glance looks like a flame war. The following is merely an observation I've made during my career.

I feel the information security domain often finds itself falling victim to physics envy, modeling solutions with equations not logically derived from valid and peer-reviewed theories, which obscure underlying subjective measures. For an example of this as applied to economics, as Richard suggests in his post, I recommend you read this Business Week article and consider for yourself the parallels to our field.

Anonymous said...

Flame war detector is going off .. but I feel it needs to be said...

I agree with the poster above about style being important. Myself and another friend in the infosec community have run into Craig on mailing lists before. He comes across as someone who likes to chest beat about how amazing he is, and seems more interested in "wowing" people with his amazing experience than genuinely helping people who pose questions.

It's a small community. He doesn't seem to be making many friends.

Alex said...

@Michael Cloppert is has made the smartest comment in this whole thread. All apologies to Gunnar.

@Richard - agreed that modeling risk is dangerous. It doesn't take a long look at GRC to cause me to weep softly.

@Raymond - I would argue that modeling risk to precision is stupid, However, I would also argue that ignoring risk modeling is impossible. We're all going to get up in the morning tomorrow, go to our jobs, and model risk. The only question is with what rigor and bias.

For me, the last 5 years has yielded the following - neither "side" of the flame war has a leg to stand. Current IRM models are impractical, but yet suggesting that you *really* hack *real* mobile phones is quite a bore to management who sees no history of incidents and cannot create for themselves an adequate picture of impact. Their little Bayesian-like mental processes fold in upon themselves because they have no reference.

Like the APT, risk is real. Like how Richard (or I) feel when we see people denying the reality of APT, I think people who deny the reality of risk-based decisions to be plain silly. And like much of the noise around APT, the noise around "risk management" detracts us from finding real solutions to the real problem.

Anonymous said...

To quote Marcus Ranum: "Any number multiplied by bullshit equals bullshit."

Dave Bowman said...

It's OK, Something Wonderful is about to happen to InfoSec.

Something Wonderful....

J. Oquendo said...

@Michael you state: "denying the reality of APT" I ask the following:

1) What's so "advanced" about it. Techniques used weren't all that high tech now, they exploited extremely old software (IE6). Nothing advanced about that considering if you're still using IE6, you seriously deserved to be compromised.

2) Aren't all attacks persistent to a degree.

Shouldn't all "malicious hacker attacks" be viewed as "advanced persistent threats", they certainly don't go away and they're certainly more advanced than the admins of those systems. Otherwise, the systems would have never gotten compromised.

Maybe Richard could re-do Extrusion Detection volume 2 and expound on the use of Squid and Snort to the nth degree.

Most of what I read doesn't make me cringe at the thought of anything uber coming out of China. In fact the opposite, makes me think of the stupidity in "security managers" and their bosses who base risk on insanely kooky models. "Still using IE6? WTH?!" How STUPID are we in the US, not how smart or motivated INSERT_GOV_SPONSORED_HACKERS_ARE_HERE

From all I've seen in writing out of "Operation Aurora" doesn't make me think of anything via way of "advanced" more like "obscure." The attacks and the wording used by certain companies describing the attacks, lead me to also believe something doesn't smell kosher and that something isn't coming from the attackers but from those in the security industry spreading half-truths in order to gain market share. Hence the introduction of "Dynamically Unique Metrics Based Analysis for Secure Systems" (DUMBASS)

http://www.theaeonsolution.com/security/?p=231

Hey whatever works for companies. I still don't see how some managers don't get it when it comes to security. AV * EF = JO/KE (no - seriously). I'm just glad I can fall back on my own metrics. RMBSS - Without a shadow of doubt!

http://infiltrated.net/rmbss.html

Richard Bejtlich said...

J. Oquendo: You might learn something if you paid attention. Why don't you run back to your "Ninja Chimp Strike Force" while people with real responsibilities do our work?

J. Oquendo said...

@Richard: paid attention? To what exactly. And I mean this as respectfully as possible. Paid attention to all that has been offered regarding Aurora, APT? APT is nothing new. None of the attacks used in APT are "awe inspiring"

Paid attention to... Thor vs. Craig? I believe Craig was spot on so please school... Pay attention to what?

Richard Bejtlich said...

http://taosecurity.blogspot.com/search/label/apt

Bill Lamoreaux said...

Am I the only one that is humored by the fact someone wants to make a "SMS baking application". Has Holly Hobby finally made it to the 21st century?

I should be trolling elsewhere, but Verizon blocked 4Chan... ;)

Christian said...

@J.Oquendo - Fixating on one descibed attack vector (IE6) misses the point.
It has been detailed that the attacks seen as performed by APT's often start with lowest hanging fruit, and more basic malware. Until detection - at which point they alter their methods or malware to more 'advanced' ones.
The 'persistance' is not about the attacks, but about the methodolgy used to remain in the compromised network - different to snatch and grab, or a regular trojan.

Anonymous said...

I agree that basing all risk on models is not a "good" idea. The problem is that usually the entire company is not IT there are other departments; namely accounting and executive management. You just can't go to them and say "hey they are shitting out hacks give me an unlimited budget for an unknown ROI/Risk Assessment" (they were business majors in college FFS). Best method would be some form of hybrid; such as "Here is the what the model says but by the way blah-blah is being focused on these days so we should give it more weight because of Blah-Blah".

I think the bigger problem is we ALL have seem to have to give names and methods for everything also making every one think it's the best thing since sliced bread...Models, etc. and things such as APT...

APT, AFAICT, is what was called a "targeted attack" in the 70-80s when hackers would infiltrate a network/system and not immediately "show" their hand. Actually have knowledgeable (no script kiddies/metasploit auto hacks)attack vectors in mind before going in to a network because they (or individual)were skilled and had a clear agenda. Not really sure why someone would have to come up with a new catch phrase like APT (unless they want to create buzz for a class they taught to get more students to sign up ;-) ). But to be fair it happens all the time in IT (and maybe in other fields who knows); Cloud Computing vs Clusters? (yes I understand there are slight differences eg. one uses web services/over internet... but really did we HAVE to have another name for it???)
...just my $.02 No flame intended for anyone :-)

mike said...

Reminds me of a great quip by Steve Martin.... "How to make a million dollars: First, get a million dollars"... Perhaps the infosec equiv... "I can tell you how to calculate risk. First, add up all the risk exposure you have"

Anonymous said...

I'm seriously confused. Some troll said Craig was "spot on" but to what? That we have "volumes of computational power coupled with novel stochastic methods"? Does he even know what stochastic means? It mean "random conjecture" people. Oquendo, you are saying that you agree with risk models derived from combining massive amounts of computing power with random conjecture? How can you possibly support this? I agree with Dr. Chuvakin - this is humor at best.

Anonymous said...

Looks like you need to refer to wikipedia regarding mathematical theory (probability theory) and AI. Sheesh...if you're gonna try and bash someone, at least get the information correct.

http://en.wikipedia.org/wiki/Stochastic

"In mathematics, specifically in probability theory, the field of stochastic processes has been a major area of research."

Anonymous said...

What I don't get, is all the energy expended trying to discredit, denigrate, or deny APT. Seriously, to anyone making these uninformed claims, I have to immediately question your background and/or intelligence. The only thing I can think of, is somehow these persons think they are coming off as experts by labeling ATP as FUD, but really they just look like @sshats.

Anonymous said...

The really funny thing is that the original post was most likely a homework problem someone was trying to get someone else to answer.

Anonymous said...

If we take a look at the probability formula, the probability of user authentication being compromised somehow is always 1.

The probability of a remote exploit affecting SMS at some time is very close to 1.world,,

In other words, its virtually certain that in the real, a system compromise will occur at some point in a reasonable time period - say 1-5 years.

On this basis, protecting the assets and having compromise mitigation in place reduce the impact of compromise.

Anonymous said...

Basically, it depends more on "who" created it, than what it is really!

Dr Craig S Wright GSE said...

Aversion to technology and change is nothing new.Complexity doesn't mean that we cannot calculate risk.

We forget that physics is a statistical estimate and not the precise set of equations that we seem to come to from high school. That statistics can provide good models as long as the underlying foundations are sound.

It seems to be overlooked that legislation made it a requirement that banks and others offer loans to those who could not be expected to repay them. Banks of course jumped on this as they had a government guarantee.

http://gse-compliance.blogspot.com/2011/10/preamble-into-aligning-systems.html