Sunday, February 05, 2006

Another Engineering Disaster

Does the following sound like any security project you may have worked?

  1. Executives decide to pursue a project with a timetable that is too aggressive, given the nature of the task.

  2. They appoint a manager with no technical or engineering experience to "lead" the project. He is a finance major who can neither create nor understand design documents. (This sounds like the news of MBAs being in vogue, as I reported earlier.)

  3. The project is hastily implemented using shoddy techniques and lowest-cost components.

  4. No serious testing is done. The only "testing" even tried does not stress the solution in any meaningful way -- it only "checks a box."

  5. Shortly after implementation, the solution shows signs of trouble. The project manager literally patches the holes and misdirects attention without addressing the underlying flaws.

  6. Catastrophe eventually ensues.

What I've just described is the Boston Molasses Flood of 1919, best described by the Boston Society of Civil Engineers in their newsletter (.pdf). I learned about this event by watching another episode of Engineering Disasters on Modern Marvels. Here's what happened.

  1. In 1915, United States Industrial Alcohol needed to build a tank in Boston to support World War I munitions production. They decide to place it in an immigrant-dominated portion of the city; Italians live there.

  2. USIA puts Arthur Jell in charge. He is a finance major with no technical or engineering experience or training. He can't even read blueprints, yet he designs a five-story, 90' diameter tank capable of holding over 2 million gallons of molasses, in the middle of a populated area.

  3. The tank is built by contractors who use thin steel and too few rivets. No one supervises their work. They hurry to complete the tank 2 days before it is filled.

  4. Prior to being filled, the tank is "tested" by holding between 4 and 8 inches of water!

  5. The tank stands three years, although apparently it was never filled to capacity until shortly before its collapse. During those three years, molasses leak from the tank on a daily basis. Jell orders the leaks plugged and has the tank painted brown to divert attention from the leaks.

  6. In 1918, with WWI ending and prohibition approaching, USIA decides to switch production from industrial alcohol to drinking alcohol. They want to cash out as fast as possible by supporting customers who want to "stock up" before prohibition begins. They accept a shipment of molasses from Cuba in January 1919, which fills the tank to capacity. Three days later, on January 15, the tank ruptures, killing 21 people and injuring 150.

USIA claimed Italian anarchists had destroyed the tank, but the evidence showed otherwise. USIA was subjected to the first ever class action lawsuit in the US, which the company lost. Safety regulations were enacted which required supervision of construction, real testing, and stamps of approval of blueprints by architects and engineers.

I foresee a similar event, with similar consequences, for the digital security industry. Hopefully not as much death and destruction will occur, but the remedies will be the same.


Anonymous said...

I can definitely see an analogy with some digital security practices and the solution to the Molasses Tank - if it leaks, paint it brown so no one notices the leaks any more! There - problem "fixed"

Hopefully this time round, they won't be looking back in 100 years and writing about the same kind of disaster happening again, but this time due to insufficient or inappropriate digital security...

Anonymous said...

How many "software engineers" have actually attended engineering school?
The answer is very few. Computer Science, which is probably the most common background for college-level programmers, is not an engineering discipline. Software development for the last 20 years has focused on rapid development and features, not on stability or simplification.

Complexity and rapidity are always enemies of good security. Until the day comes when the same level of expertise required to build bridges is also required of software developers, any software project can be a molasses tank waiting to explode.

Richard Bejtlich said...

Last anonymous -- please see my post today for further thoughts on this subject.