How Not to Administer Important Systems

Courtesy of SANS NewsBites (link will work shortly) I learned of a router or switch crash that shut down the San Francisco Bay Area Rapid Transit District (BART). According to the article:

BART officials promised Thursday to thoroughly investigate why technicians risked working on computers that control trains while the transit system was running, work that crashed BART's main computer, stalled 50 to 60 trains, and stranded 35,000 passengers for more than an hour at the peak of the Wednesday evening commute.

"The bottom line is we shouldn't have worked on it (during service hours)," BART spokesman Linton Johnson said.

No kidding. They're lucky the trains stopped running rather than keep running -- into each other.

"The network switch was not supposed to get overloaded,'' Johnson said. "It is not supposed to crash. But we shouldn't have been working on (the computer system) while trains were running."

Johnson described the technicians who caused the crash as conscientious workers who were frustrated by problems caused by the installation of new software on Monday and Tuesday. The software upgrade is intended to be more reliable and secure and to allow BART to limit problems instead of having them affect the entire system.

"We had some folks who have a long record of installing (software) components correctly and are proud of having very few problems," Johnson said. "When they had two, they wanted to get them fixed as soon as possible. It was a rush to do the right thing."

"Rush" is an ingredient in a recipe for disaster, despite the desire "to do the right thing." This is why frameworks like IT Infrastructure Library (ITIL) emphasize Service Management over cowboy administrative practices.

Comments

Anonymous said…
For every admin who rushes a project or a fix, there is likely a manager above him making the suggestions and demands.

I've seen far too often in business that despite pleas from the technical crews about doing things properly, management moves forward with over-aggressive timelines and demands that end up causing the rush that in turn causes so many problems and wasted money...whether those problems manifest right away like the BART issue, or remain latent in a configuration flaw that allows an attacker to exploit a year from now.

-LonerVamp
2:02 PM
Post a Comment

Popular posts from this blog

Zeek in Action Videos

Image
This is a quick note to point blog readers to my Zeek in Action YouTube video series for the Zeek network security monitoring project .  Each video addresses a topic that I think might be of interest to people trying to understand their network using Zeek and adjacent tools and approaches, like Suricata, Wireshark, and so on.  I am especially pleased with Video 6 on monitoring wireless networks . It took me several weeks to research material for this video. I had to buy new hardware and experiment with a Linux distro that I had not used before -- Parrot .  Please like and subscribe, and let me know if there is a topic you think might make a good video.
Read more

MITRE ATT&CK Tactics Are Not Tactics

Image
Just what are "tactics"? Introduction MITRE ATT&CK  is a great resource, but something about it has bothered me since I first heard about it several years ago. It's a minor point, but I wanted to document it in case it confuses anyone else. The MITRE ATT&CK Design and Philosophy document from March 2020 says the following: At a high-level, ATT&CK is a behavioral model that consists of the following core components: • Tactics, denoting short-term, tactical adversary goals during an attack; • Techniques, describing the means by which adversaries achieve tactical goals; • Sub-techniques, describing more specific means by which adversaries achieve tactical goals at a lower level than techniques; and • Documented adversary usage of techniques, their procedures, and other metadata. My concern is with MITRE's definition of "tactics" as "short-term, tactical adversary goals during an attack," which is oddly recursive. The key word in the tacti...
Read more

New Book! The Best of TaoSecurity Blog, Volume 4

Image
  I've completed the TaoSecurity Blog book series . The new book is  The Best of TaoSecurity Blog, Volume 4: Beyond the Blog with Articles, Testimony, and Scholarship .  It's available now for Kindle , and I'm working on the print edition.  I'm running a 50% off promo on Volumes 1-3 on Kindle through midnight 20 April. Take advantage before the prices go back up. I described the new title thus: Go beyond TaoSecurity Blog with this new volume from author Richard Bejtlich. In the first three volumes of the series, Mr. Bejtlich selected and republished the very best entries from 18 years of writing and over 18 million blog views, along with commentaries and additional material.  In this title, Mr. Bejtlich collects material that has not been published elsewhere, including articles that are no longer available or are stored in assorted digital or physical archives. Volume 4 offers early white papers that Mr. Bejtlich wrote as a network defender, either for technica...
Read more