Friday, April 13, 2007

FISMA Dogfights

My favorite show on The History Channel is Dogfights. Although I wore the US Air Force uniform for 11 years I was not a pilot. I did get "incentive" rides in T-37, F-16D, and F-15E jets as a USAFA cadet. Those experiences made me appreciate the rigor of being a fighter pilot. After watching Dogfights and learning from pilots who fought MiGs over North Vietnam, one on six, I have a new appreciation for their line of work.

All that matters in a dogfight is winning, which means shooting down your opponent or making him exit the fight. A draw happens when both adversaries decide to fight another day. If you lose a dogfight you die or end up as a prisoner of war. If you're lucky you survive ejection and somehow escape capture.

Winning a dogfight is not all about pilot skill vs pilot skill. Many of the dogfights I watched involved American pilots who learned enemy tactics and intentions from earlier combat. Some of the pilots also knew the capabilities of enemy aircraft, like the fact that the MiG 17 was inferior to the F-8 in turns below 450 MPH. Intelligence on enemy aircraft was derived by acquiring planes and flying them. In some cases the enemy reverse engineered American weapons, as happened with the K-13/AA-2 Atoll -- a copy of the Sidewinder.

All of this relates to FISMA. Imagine if FISMA was the operational theme guiding air combat. Consultants would spend a lot of time and money documenting American aircraft capabilities and equipment. We'd have a count of every rivet on every plane, annotated with someone's idea that fifty rivets per leading edge is better than forty rivets per leading edge. Every plane, every spare part, and every pilot would be nicely documented after a four to six month effort costing millions of dollars. Every year a report card would provide grades on fighter squadrons FISMA reports.

What would happen to these planes when they entered combat? The FISMA crowd would not care. American aircraft could be dropping from the sky and it would not matter to FISMA. All of the FISMA effort creates a theoretical, paper-based dream of how a "system" should perform in an environment. When that system -- say, a jet fighter -- operates under real life combat conditions, it may perform nothing like what the planners envisioned. Planners range from generals setting requirements for a new plane, engineers desiging the plane, and tacticians imagining how to use the plane in combat.

Perhaps the guns jam in high-G turns. Perhaps the missiles never acquire lock and always miss their targets. Maybe the enemy has stolen plans for the aircraft (or an actual aircraft!) and know that the jet cannot perform as well as the enemy plane doing vertical rolling scissors.

Furthermore, the enemy may not act like the planners imagined. This is absolutely crucial. The enemy may have different equipment or tactics, completely overpowering friendly capabilities.

Maybe FISMA would address these issues in three years, the next time a FISMA report is due. Meanwhile, the US has losts all its pilots and aircraft, along with control of its airspace.

Maybe this analogy will help explain the problems I have with FISMA. I already tried an American football analogy in my post Control-compliant vs Field-Assessed Security. My bottom line is that FISMA involves control compliance. That is a prerequisite for security, since no one should field a system known to be full of holes. However, effective, operational security involves field assessment. That means evaluating how a system performs in the real world, not in the mind of a consultant. Field-assessed security is absolutely missing in FISMA. Don't tell me the tests done prior to C&A count. They're static, controlled, and do not reflect the changing environment found on real networks attacked by real intruders.

Incidentally, I also really liked the BBC series Battlefield Britain and I may check out the other History Channel series Shootout!.

3 comments:

Alex said...

I really enjoyed the F-8 Crusader one that was on recently. I'd like to see a p-38 dogfight soon.

Or a Russian-German one.

Anonymous said...

I saw the F8 episode. It takes real balls to dogfight someone with no more weapons (the gun jammed in a high-G turn).

C&A itself has never made any system more secure, now it may have reveal certain insecurities at one point in time, but not 6 months afterwards. C&A is picture at one moment in time, albeit depending on who did the C&A it maybe a 10k foot view and very fuzzy.

I never have understood how pentesting consisted of running a Nessus|ISS|Retina scan against a box, considering that the application is web based.

Thomas

Chris said...

I agree to some extent.

I would say the Certification and Accreditation is necessary, and not evil. When properly applied it is exceedingly helpful. The problem that I see is that the Certification team is brought in too late in the Development lifecycle.

We perform the certification task, produce findings and it is like the five stages of grieving.

First the developers cry false positive, then they say it is too costly to fix, and lastly they say they don't have the time to fix it now that they got the money.

If proper Risk Assessments were applied and if System Security Plans were used as a development tool for the system, developers would be able to put the security into the system design.

Richard appears to like analogies, and I always compare system development to baking a cake. Security is the sugar of the cake. So go ahead and bake your cake, leave out the sugar. Then ice your cake to within an inch of its life and wonder why it tastes so bad.

I think the government has a huge issue in that accreditation authorities are willing to accept risk based on inconsistent methodologies. If Mitre came in and did an assessment, then Booz came in and assessed the same system. You will always get different results, even after the 800-53A is final.

Another issue is that continuous monitoring isn't always implemented or funded. So the system becomes more vulnerable over time.