Alternatives to "Expert Opinions"
If you read The Doomsday Clock you probably recognize I have a dim opinion of "expert opinion," especially by committee. At the risk of making a political statement, I rank expert opinion alongside central planning as some of the worst ways to make decisions -- at least where a large amount of complexity must be accommodated.What is my alternative? I believe free markets are the best way to synthesize competing data points to produce an assessment. Does this sound familiar? If yes, you may be thinking of this 2003 story: The Case for Terrorism Futures:
Critics blasted policy-makers Tuesday for dropping a controversial plan to create a futures market to help predict terrorist strikes...
[S]upporters of the project point out that gathering intelligence is often a messy business, with payoffs to unsavory characters and the elimination of potential adversaries. The futures market, ugly as it may sound, doesn't involve any of those moral compromises, said Robin Hanson, one of the earlier promoters of the concept of trading floors for ideas and a PAM [Policy Analysis Market] project contributor. It's just a way of capturing people's collective wisdom...
Projects similar to PAM, like the Iowa Electronic Markets, which speculate on election results, have been surprisingly reliable indicators of what's going to happen next...
The price of orange juice futures has even been shown to accurately predict the weather...
Traders on the Hollywood Stock Exchange last year correctly picked 35 of the 40 Oscar nominees in the eight biggest categories, according to The New Yorker magazine...
Market mechanisms are more accurate than asking people their opinions because they're putting their money or reputation on the line," said Ken Killitz of the Foresight Exchange, which speculates on everything from the future of human cloning to the possibility that Roman Catholic priests will be allowed to marry. "It gives people an incentive to reveal what they know..."
[E]xchanges "tend to predict events really well when no one person knows the answer -- when information is distributed among many people with different knowledge bases," said Joyce Berg, a University of Iowa professor who helped organize the political trading floors...
Markets also bring together people with information about a particular subject in a way blue-ribbon panels of experts can't, added Hanson.
"You get people that know things about a subject, but don't have the credentials to say so," he said. "You get people who live in these areas (of the Middle East)."
There's also "less of an ability to spin" in markets than in policy debates, Hanson noted. "So you get what people actually think, not what they say."
I love this idea. The fact that intellectual pygmies in the Senate defeated it is a real shame.
I found many interesting articles on this subject by Robin D. Hanson from George Mason University and Oxford's Future of Humanity Institute; the latter offers a Global Catastrophic Risks program that is probably more interesting (but less marketing-savvy) than the Doomsday Clock.
If you're sufficiently motivated to start arguing against this idea, I will probably just point back into the literature (especially Hanson's) countering these complaints.
If you're wondering why I mention this at all, it ties into my mention of security breach derivatives in my post Excerpts from Ross Anderson / Tyler Moore Paper.
Comments
In what ways could a terrorism or information security market fail, and what would the consequences be? What happens when a party gets a monopoly on terrorism in a market sector? What happens when parties begin insider trading on information security markets?
Oops; another risk assessment. It's a vicious spiral.
After all, a prediction market is just a means to motivate people over a long period of time to reveal their true risk beliefs (opinions). You still have experts (those that assign value), making judgment calls (the value itself) - you're just doing a whole bunch of it. One might say that it is a "human" means of running Monte Carlo.
FWIW- Robin Hanson, the man who invented prediction markets runs a heavily Bayesian blog at http://www.overcomingias.com. It's an excellent read.
My problem with our use of prediction markets is that they indicate a form of risk (deviance from expected return) for a form of risk (probable frequency and probable impact of failure in a complex system). They aren't risk assessment in the way IRM professionals need to express risk. They are a means to account for uncertainty concerning IRM risk. This can be confusing.
In a sense what Richard's suggesting here is creation of a belief statement about individual belief statements.
I would label those who consistently have good information "experts".
A futures market seems like a kludge, and a difficult to implement one in this context too (feel like putting your company strategy out there for general comment, a bit facetious, but there is a confidentiality concern).
A sufficiently large and competent consulting house who are brought in for their 'opinion' and not implementation used in conjunction with in-house expertise sound like a good way to get close to the mark. It won't be perfect, but if good evaluation processes are followed in selecting the group it should bring your within tolerable limits.
Go on, prove me wrong ;)
Personally after seeing the experts resume, listening to him, I think he really knew his stuff. During the cross-examination however, the lawyer was able to make it seem like he was unqualified for the work. It was interesting to say the least. She actually didn't attack his work, more the fact that he was not a expert in two other fields that they felt were required. It was actually a valid argument, but I felt a little bad for the guy.
It opened my eyes if I am ever called to be an expert that's for sure.
-mike