FISMA Redux

Late last year I mentioned I planned to read and review FISMA Certification & Accreditation Handbook by Laura Taylor. You know if I read a book on Cisco MARS on one leg of my last trip, I probably read a different book on the return leg. FISMA was that book. These comments are going to apply most directly to FISMA itself, based on what I learned reading Ms. Taylor's book. I'll save comments on the book itself for a later date.

Last year I wrote FISMA is a joke.. I was wrong, and I've decided to revise my opinion. Based on my understanding of FISMA as presented in this book,

FISMA is a jobs program for so-called security companies without the technical skills to operationally defend systems. This doesn't mean that if you happen to conduct FISMA work, you're definitelTy without technical skills. I guarantee my friends at ClearNet Security are solid guys, just based on their ability to detect the C&A project they joined was worthless. Anyway, I guess it's time to put on a flame-retardant suit.

Let's start with p xxiii in FISMA to understand the thought process of those who believe in it. Foreword author Sunil James, "Former Staff Director of the FDIC," writes that the FISMA "process has been proven to reduce risk to federal information systems." I think he means that FISMA reduces the risk of unemployment for those who perform C&A on federal information systems.

Without saying anything else, I think I know the problem with the FISMA-supporting crowd. I bet they do not do any other work, especially not technical work and certainly not incident response work for the agencies they "certify" and "accredit." If they did have operational security responsibilities for their C&A clients they'd wonder "why are these systems repeatedly being 0wned if their C&A packages are up-to-date?" Those that keep one foot in the C&A world and another in the operational world realize their FISMA feet are wet and smelly and not doing anyone any good, period.

Another sad truth about FISMA, despite Mr. Porter's unsubstantiated claim, is that there is zero connection between high FISMA scores and lower impact or number of intrusions. If you don't believe me, keep your eyes open for the next FISMA report card from the House Committee on Oversight and Government Reform. A look at the 2006 scores is interesting. Figure out who has good scores, and then see how they fared in this staff report on Federal breaches since January 2003. Hint: everyone's been 0wned, is 0wned, and will continue to be 0wned while money is spent paying consultants to write FISMA C&A packages.

Let's see what this FISMA book has to say about C&A packages. Page 34 claims a good ISSO can maintain C&A packages for 6 systems, and that writing each package can take 3-6 months. They need to be updated every 3 years. The C&A handbook guiding the writing of packages is usally 200+ pages and needs to be kept current. The packages themselves are usually 500+ (!) pages, and require 2-4 weeks to be read by the accreditors.

On p 34 we come to a root of the problem:

Since once C&A package could easily take a year for a well-versed security expert to prepare, it is considered standard and acceptable for ISSOs to hire consultants from outside the agency to prepare the Certification Package.

Page 38 continues:

Since C&A, if done properly, is usually a much bigger job than most people realize, I cannot emphasize enough the value in using outside consultants. Putting together a Certification Package is a full-time job.

I do not have a problem with consultants. Heck, I am a consultant. However, the vast majority of my work does not revolve around writing 500 page reports based on self-assessments every three years.

Laura Taylor writes on pp 8-9:

C&A is essentially a documentation and paperwork nightmare... prepare yourself to write, write, and write some more. If you detest writing, you're in the wrong business.

Basically, preparing a Certification Package is writing about security -- extensive writing about security. When you are preparing a Certification Package, you usually don't perform any sort of hands-on security. You review the existing security design and architecture documents, interview various IT support and development folks familiar with the infrastructure, and document your findings. (emphasis added)

I am not making this up. The really sad part is this: the author then says

...why C&A exists -- it is a process that enables authorizing officials to discover the security truths about their infrastructure so that informed decisions can be made.

Security truths? What are they based on?

In chapter 8 we read about "security self-assessments." Maybe those are helpful? Hmm, probably not: "A security self-assessment is a survey-based audit that is essentially a long list of questions." What's worse, page 115 says:

[I]n September 2003 a report put out by the Office of Inspector General at the Environmental Protection Agency found that 36 percent of the responses to security self-assessments contained inaccurate information.

Ms. Taylor's recommendation?

Tip: Encourage self-assessment respondents to answer questions truthfully.

Maybe some other aspect of C&A and FISMA shows merit. Chapter 12, discussing Security Tests and Evaluation (ST&E), begins with this:

A Security Test & Evaluation, known among security experts as an ST&E, is a document that demonstrates that an agency has performed due diligence in testing security requirements and evaluation the outcome of the tests.

The ST&E is a C&A document that tends to give agencies lots of trouble. It's not clear to many agencies what tests they should be doing, who should be doing them, and what the analysis of the tests should consist of.

That's another winner in my book.

FISMA fans out there are going to cite the vulnerability scans which are usually part of ST&E as a sign that something technical happens during C&A. Believe me, I am sparing the author of this book and her "technical editors" by not reproducing their recommendations for assessments. (One word: Strobe.)

We could also look at the Privacy Impact Assessment, but guess what -- it's another self-survey.

The bottom line is that FISMA doesn't mention C&A at all, but the author thinks that's ok because C&A fulfill's FISMA's goals. The reality is far different. According to the act itself, the first "purpose" is to:

provide a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support Federal operations and assets. (emphasis added)

FISMA is failing miserably. It's ironic that this FISMA book begins each chapter with a quote, and this begins chapter 2:

It is common sense to take a method and try it. If it fails, admit it frankly and try another. But above all, try something. -- FDR

It's time for the FISMA fans to admit this five year FISMA experiment has been a waste of taxpayer money and agency resources.

Returning to my football analogy, C&A is an expensive, extensive practice session controlled by the players and overseen by agents who get paid the longer the team is on the field. Success is measured not by the score of the next game but by the number of worthless statistics written about the players prior to the first snap. Once the team takes to the field they are annihilated by the opposition, but the agents don't care because they're spending their money elsewhere.

If you are a C&A Package-writing company, I strongly recommend you gain some operational capabilities or look for a new line of business. I am committed to eliminating your position in the Federal government. Laura Taylor writes with some apparent regret that "most private and public companies don't put nearly as much time, effort, and resources into documenting their security as government agencies do." Let's keep it that way. At least it frees up resources for work that has a chance of stopping an intruder.

Do I have anything "nice" to say? Yes -- if you are so cursed as to be responsible for a FISMA C&A project, I do recommend reading this book. Forget the technical aspects and concentrate on understanding the FISMA maze. I thought Laura Taylor wrote a clear and well-organized book with lots of practical advice and good templates. I would much rather see her not have to write about this subject again, though!