Sunday, April 18, 2010

Measurement Over Models

Most blog readers know I strongly prefer measurement over models. In digital security, I think too many practitioners prefer to substitute their own opinions for data, i.e., "defense by belief" instead of "defense by fact." I found an example of a conflict between the two mindsets in Test flights raise hope for European air traffic:

Dutch airline KLM said inspection of an airliner after a test flight showed no damage to engines or evidence of dangerous ash concentrations. Germany's Lufthansa also reported problem-free test flights...

"We hung up filters in the engines to filter the air. We checked whether there was ash in them and all looked good," said a KLM spokeswoman. "We've also checked whether there was deposit on the plane, such as the wings. Yesterday's plane was all well..."

German airline Air Berlin was quoted as expressing irritation at the way the shutdown was decided.

"We are amazed that the results of the test flights done by Lufthansa and Air Berlin have not had any bearing on the decision-making of the air safety authorities," Chief Executive Joachim Hunold told the mass circulation Bild am Sonntag paper.

"The closure of the air space happened purely because of the data of a computer simulation at the Volcanic Ash Advisory Center in London."


I understand that safety officials need to make decisions based on the best information available at the time the decision needs to be made. However, when that information changes, the decision maker should re-evaluate his or her position. This reminds me of the silly policies mandated by various rule-makers regarding password complexity and frequency of change. They are basically completely disconnected with the modern attack and exploitation environment. That thinking recalls a time when guessing credentials or brute-forcing passwords took weeks instead of near-real-time, and was the prevalent way to compromise a system.

Returning to the volcano cloud -- I'm sure safety officials think they are acting in the best interests of passengers, but I don't see the airlines about to take actions that jeopardize their customers. Furthermore, customers who would be wary about flying through or near the ash cloud could decide not to do so. The problem is that safety officials bear none of the cost of their decisions while airlines and customers do.

10 comments:

Jack Daniel said...

The commercial airlines' results contradict the Finnish Air Force's test flights with a pair of F-18 Hornets. Engine damage on the Hornets was substantial enough that it has led Finnish aviation authorities to keep planes on the ground. Last I checked, F-18s were designed to fly in uglier conditions than commercial cattle-planes. Since the result of miscalculation is arguably high (others say death is overhyped), I can see erring on the side of caution.

Richard Bejtlich said...

Hi Jack,

So that's neat -- I just read the original article. At least now we have examples of data vs data instead of models vs data.

Dan O'Donnell said...

There are also two old data points the civil aviation authorities have relied on. In the 70s two airliners (including a British Airways 747) flew through volcanic ash clouds while at altitude and had flameouts in all four engines. Both were able to do restarts on their way down and successfully executed emergency landings. I assume that nobody wants a repeat of that, given the risk of a possible unsuccessful restart.

Anonymous said...

"The problem is that safety officials bear none of the cost of their decisions while airlines and customers do. " But they are the ones responsible when accidents due to this lead to deaths (and not the airlines). At least in Germany.

Anonymous said...

Any measurement is meaningless without a suitable model.

It is a false dichotomy.

Anonymous said...

http://godplaysdice.blogspot.com/2010/04/ash-clouds-and-probability.html

"Critics said the agency used a scientific model based on 'probability' rather than fact to forecast the spread of the ash cloud." See the Telegraph as well.

What else are they supposed to do? The agency here -- the Met Office, which is the national weather service of the UK -- doesn't know what the ash cloud is going to do. If they waited to see what the cloud does, the planes would already be in the air. It would be too late."

Anonymous said...

The airlines might be right. Or they might be starting their effort to build public support for a really big bailout. One or the other.

The Ubiquitous Mr. Lovegroove said...

I do security consulting and government is a big client. Government in CEE region is not afraid of montary losses (good luck suing!). Government officials are afraid only of loosing their chair. OTOH airlines can only loose money (and somebody can go to jail for gross negligence, of course, but that won't be the top guy).
So, if an airline has estimated that even 2% of all flights could result in emergency landings (remember, crash possibility is near 0 based on previous evidence) it makes more business sense to fly passengers in these conditions, even if only to minimize losses.
For the government official, having one plane crash-land will mean loosing his chair and maybe being criminally prosecuted.

In a risk-reward game, government has nothing to gain from enabling riskier flights while airlines have a lot.

Lawrence Munro said...

"customers who would be wary about flying through or near the ash cloud could decide not to do so."

I was stuck on an Island for a week... where there are only sailing routes to other Islands.

I would rather wait a few day than attempt a possibly unsafe flight or helish and expensive journey by bos or coach.

Planes fly in straight lines and the ash cloud covered hundreds of thousands of square miles in 3 dimensions. Due to wind and air currents the ash is distributed unevenly in the upper atmostphere. The only way to be 100% sure by applying a fly-by method would be to fly millions of test flights on all the vectors. Satelite maps and predictions are used for just this reason. In security, if in doubt... do a couple of tests then open it all up? Or close it down until you're sure about the threat?

John Reedaw said...

"Furthermore, customers who would be wary about flying through or near the ash cloud could decide not to do so.

I think, Richard, that while saying this you clearly forgot about the possibility of a plane falling down over a city or a group of buildings. What possible choice would these people have given (flying or not flying) under the possible bad conditions that the ash might have done to the safety of the planes?

Officials wouldn't only be responsible for all deaths occured in a plane if an accident would happen, but also all other deaths caused by this accident.