Late last year I mentioned I planned to read and review FISMA Certification & Accreditation Handbook by Laura Taylor. You know if I read a book on Cisco MARS on one leg of my last trip, I probably read a different book on the return leg. FISMA was that book. These comments are going to apply most directly to FISMA itself, based on what I learned reading Ms. Taylor's book. I'll save comments on the book itself for a later date.

Last year I wrote FISMA is a joke.. I was wrong, and I've decided to revise my opinion. Based on my understanding of FISMA as presented in this book,

FISMA is a jobs program for so-called security companies without the technical skills to operationally defend systems. This doesn't mean that if you happen to conduct FISMA work, you're definitelTy without technical skills. I guarantee my friends at ClearNet Security are solid guys, just based on their ability to detect the C&A project they joined was worthless. Anyway, I guess it's time to put on a flame-retardant suit.

Let's start with p xxiii in FISMA to understand the thought process of those who believe in it. Foreword author Sunil James, "Former Staff Director of the FDIC," writes that the FISMA "process has been proven to reduce risk to federal information systems." I think he means that FISMA reduces the risk of unemployment for those who perform C&A on federal information systems.

Without saying anything else, I think I know the problem with the FISMA-supporting crowd. I bet they do not do any other work, especially not technical work and certainly not incident response work for the agencies they "certify" and "accredit." If they did have operational security responsibilities for their C&A clients they'd wonder "why are these systems repeatedly being 0wned if their C&A packages are up-to-date?" Those that keep one foot in the C&A world and another in the operational world realize their FISMA feet are wet and smelly and not doing anyone any good, period.

Another sad truth about FISMA, despite Mr. Porter's unsubstantiated claim, is that there is zero connection between high FISMA scores and lower impact or number of intrusions. If you don't believe me, keep your eyes open for the next FISMA report card from the House Committee on Oversight and Government Reform. A look at the 2006 scores is interesting. Figure out who has good scores, and then see how they fared in this staff report on Federal breaches since January 2003. Hint: everyone's been 0wned, is 0wned, and will continue to be 0wned while money is spent paying consultants to write FISMA C&A packages.

Let's see what this FISMA book has to say about C&A packages. Page 34 claims a good ISSO can maintain C&A packages for 6 systems, and that writing each package can take 3-6 months. They need to be updated every 3 years. The C&A handbook guiding the writing of packages is usally 200+ pages and needs to be kept current. The packages themselves are usually 500+ (!) pages, and require 2-4 weeks to be read by the accreditors.

On p 34 we come to a root of the problem:

Since once C&A package could easily take a year for a well-versed security expert to prepare, it is considered standard and acceptable for ISSOs to hire consultants from outside the agency to prepare the Certification Package.

Page 38 continues:

Since C&A, if done properly, is usually a much bigger job than most people realize, I cannot emphasize enough the value in using outside consultants. Putting together a Certification Package is a full-time job.

I do not have a problem with consultants. Heck, I am a consultant. However, the vast majority of my work does not revolve around writing 500 page reports based on self-assessments every three years.

Laura Taylor writes on pp 8-9:

C&A is essentially a documentation and paperwork nightmare... prepare yourself to write, write, and write some more. If you detest writing, you're in the wrong business.

Basically, preparing a Certification Package is writing about security -- extensive writing about security. When you are preparing a Certification Package, you usually don't perform any sort of hands-on security. You review the existing security design and architecture documents, interview various IT support and development folks familiar with the infrastructure, and document your findings.
(emphasis added)

I am not making this up. The really sad part is this: the author then says

...why C&A exists -- it is a process that enables authorizing officials to discover the security truths about their infrastructure so that informed decisions can be made.

Security truths? What are they based on?

In chapter 8 we read about "security self-assessments." Maybe those are helpful? Hmm, probably not: "A security self-assessment is a survey-based audit that is essentially a long list of questions." What's worse, page 115 says:

[I]n September 2003 a report put out by the Office of Inspector General at the Environmental Protection Agency found that 36 percent of the responses to security self-assessments contained inaccurate information.

Ms. Taylor's recommendation?

Tip: Encourage self-assessment respondents to answer questions truthfully.

Maybe some other aspect of C&A and FISMA shows merit. Chapter 12, discussing Security Tests and Evaluation (ST&E), begins with this:

A Security Test & Evaluation, known among security experts as an ST&E, is a document that demonstrates that an agency has performed due diligence in testing security requirements and evaluation the outcome of the tests.

The ST&E is a C&A document that tends to give agencies lots of trouble. It's not clear to many agencies what tests they should be doing, who should be doing them, and what the analysis of the tests should consist of.

That's another winner in my book.

FISMA fans out there are going to cite the vulnerability scans which are usually part of ST&E as a sign that something technical happens during C&A. Believe me, I am sparing the author of this book and her "technical editors" by not reproducing their recommendations for assessments. (One word: Strobe.)

We could also look at the Privacy Impact Assessment, but guess what -- it's another self-survey.

The bottom line is that FISMA doesn't mention C&A at all, but the author thinks that's ok because C&A fulfill's FISMA's goals. The reality is far different. According to the act itself, the first "purpose" is to:

provide a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support Federal operations and assets. (emphasis added)

FISMA is failing miserably. It's ironic that this FISMA book begins each chapter with a quote, and this begins chapter 2:

It is common sense to take a method and try it. If it fails, admit it frankly and try another. But above all, try something. -- FDR

It's time for the FISMA fans to admit this five year FISMA experiment has been a waste of taxpayer money and agency resources.

Returning to my football analogy, C&A is an expensive, extensive practice session controlled by the players and overseen by agents who get paid the longer the team is on the field. Success is measured not by the score of the next game but by the number of worthless statistics written about the players prior to the first snap. Once the team takes to the field they are annihilated by the opposition, but the agents don't care because they're spending their money elsewhere.

If you are a C&A Package-writing company, I strongly recommend you gain some operational capabilities or look for a new line of business. I am committed to eliminating your position in the Federal government. Laura Taylor writes with some apparent regret that "most private and public companies don't put nearly as much time, effort, and resources into documenting their security as government agencies do." Let's keep it that way. At least it frees up resources for work that has a chance of stopping an intruder.

Do I have anything "nice" to say? Yes -- if you are so cursed as to be responsible for a FISMA C&A project, I do recommend reading this book. Forget the technical aspects and concentrate on understanding the FISMA maze. I thought Laura Taylor wrote a clear and well-organized book with lots of practical advice and good templates. I would much rather see her not have to write about this subject again, though!


Shirkdog said…
A system is flawed when they are ways to subvert it. An example, a DAA can approve FTP, when the control clearly states "STOP USING FTP and INSECURE PROTOCOLS"

16.2.2 Are insecure protocols (FTP,UDP) disabled?

Don't forget the price tag of the 3rd Party Assessment. :-)
Roman said…
Shirkdog, you are assuming of course, that anyone told the DAA about it in the first place. More likely, they just used it, and if asked, they lie and say they don't.
Unknown said…
Documentation has it's place, but when it replaces actually *doing* anything...that's when weeble-wobbles cry.
Anonymous said…
Having been involved in C&A before, I couldn't agree more. It's a worthless paperwork exercise, producing countless documents that are outdated before they are even complete. It's no wonder that people dread the C&A process, and try to avoid it at every possible opportunity. Unfortunately, those of us with some security knowledge are often forced into taking on C&A responsibilities, since it's perceived as a security function. In reality, it is a task better suited for a technical writer.
Anonymous said…
You know those of us that refuse to do .gov work are all laughing. We laugh because we all know what a joke doing sec work in most of the government is.

A lot of government's inefficiencies are kept internal to the government for the most part. However when the government directly interacts with the public: hurricane response, filing taxes, etc. we all get to experience just how bad it is. With the internet, attackers to get to directly exploit all the inefficiencies.

A more accessible, inefficient government is REALLY accessible ;)

Maybe someday when I want to give up on learning or caring I'll do gov work.
Anonymous said…
DO you feel the same way about ISO 17799?
The last item I wrote on ISO 17799 is here. I have no direct exposure to it.
Anonymous said…
I'm not so sure that any standard will help .gov sites. The people implementing it will screw it up and the people auditing it will use low-skilled billable scanner jocks that will miss any non-signature based risk.

Sure there are exceptions to this. However why is it that most of the highly technical consultants stay away from regulatory/standards/gov work? If you are good, you'll go work in the private sector.
Anonymous said…
It's pretty easy for packet surgeons and the like to poke holes in FISMA and the attendant C&A process.

But, one good thing to come out of FISMA is the guidance from NIST on Information Security. As a result of FISMA, they have created a compendium of security, best practices and methodologies rooted in a common lexicon. The IA community needed this standardization and I think most of NIST's guidance is very valid. Specifically, the 800-53 Recommended Security Controls for Federal Information Systems.
Anonymous said…
I actually have to deal with the FISMA regulations. I was hired for "network security", and apparently, if our more political security officers had their way that means "write lots of .doc's on the state of affairs, THEN after those .doc's are done, go try to fix any findings, updating the .doc's as you go."

We started using NIST 800-26, and we had to redo a major section of our paperwork to move to NIST 800-53. I'm a programmer/system admin, and I've been yelling at these people for 3 years about the futility of the FISMA regulations and how if they spent half the money they're wasting on writing 500+ page C&A packages on intelligent System Admins and Network Analysts, they'd actually be improving security.

I like the concept of organizing security standards, but the auditing and C&A processes do not address security at all. They address document management and "BS Production". If you want a real laugh, check out NIST's "Central Logging Standards". I thought there might be interesting information, but it ended up being 64 pages of saying nothing.

Richard, thanks for summing up my frustrations with the system. I sent the article to our "Information Systems Security Officers", and no one replied :).

Your work is appreciated by my entire organization who have been railing against the system since it's inception.
Anonymous said…
FISMA itself is good... a law requiring security planning and budgeting. Hey, that's right-on with what it needs to be. Unfortunately, all this other junk gets strapped on to it, and nobody knows what they are doing.

It's like when I go to the Smithsonian and look at the exhibits on Africa. I realize that the more I learn about Africa, the more I realize I don't know jack about Africa.

There are some problems with the implementation of C&A in particular. Too many rules and everybody has a serious case of NIH. It's not supposed to be this much work.

I probably said it best here:
Blog Entry

The way to look at C&A from a technical standpoint is that it's an excuse to have people come to you and ask your opinion. Let them know what it is--right, wrong, or indifferent.
Anonymous said…
The FISMA book seems okay to me.
Anonymous said…
Your characterization of contracting companies and their relationship to FISMA packages shows a lack of understanding about FISMA, the role of contractors, and the role of agency IT security personnel.

A technical vulnerability test is part of the ST&E that comprises a C&A package. It is very much hands on engineering by professionals who must maintain knowledge of the latest infrastructure devices and security testing tools. They must know how to use the tools and interpret the results. Also, it is the government and those in IT security that require the 500 page packages. This is based on the requirements of IG auditors who know little about IT security but can discern if a document matches NIST template guidelines number for number.
I, too, believe there is a lot of waste in the FISMA compliance process. If you studied the subject rather than relying on one book, perhaps you would understand why.
Anonymous said…
I don't understand your comment on Amazon about the IP address. It would seem that not using IP addresses that are outside standard octets to be a good idea. I still have a copy of strobe. It seems to work.

How can you possibly defend looking for an "IP address" that technically CANNOT exist?
For those of you who haven't read my review, the IP address comment is:

The "Suspicious Events That Are Worth Auditing" chart on p 348 really made me laugh. Item "SE 6" says "Invalid IP addresses that are not in the range of acceptable octets, for example:" Are they SERIOUS?
Anonymous said…
I don't know for sure, but I'm guessing that the point that was trying to be made was that if you see an invalid IP address in a log file that obviously something is awry and you will know that something has been inappropriately modified. It is pretty easy to modify log files to make them say anything.
Anonymous said…
Only the destination IP address has to be valid when launching an attack.
Anonymous said…
there are programs that can generate dword ip addresses or octal ip addresses that may look non-standard. trying doing 'ping 0100.0351.0273.0143' which is an octal ip address. spammers sometimes use octal or dword ip address for obfuscation. try doing 'ping 1089059683' also try using hex ip addresses.
little G, please tell me how you would render 295 using the 8 bits available in the first octet of an IPv4 address?

In all my years doing incident response I have never seen anyone alter a log file to change an octet of an IP address to be a value outside of the range valid for IPv4. Maybe on Mars where I hear they use 36 bits (9 bits per "octet") for an IP address you can have 295 as the first value.

As for wii wii's comment, that is clearly not what the FISMA book is discussing.
Anonymous said…
To wii wii et al - The IP address is an error/typo that I should have caught. As I recall the thought was to discuss egress filtering and looking for invalid/spoofed IP addresses coming from inside one's own network (an IP address that is invalid on your own network). Thank you to Richard for his keen eye and pointing this out.
Anonymous said…
I would like to introduce one website for your visitors which I recently discovered a very good regulatory compliance website which provides all the useful information regarding FISMA and also provides good information about other regulatory compliance authorities such as HIPAA, ISO 17799, OSHA, etc.
Anonymous said…
Thanks for the nice post!
Anonymous said…
Enforcement of compliance regulation is must for many organizations but implementing, establishing and maintaining of same is a tough task due to complexity and cost. website provides a wonderful and valuable template suite which any organization, small or big, can use to meet their compliance requirements for HIPAA, Sarbanes Oxley (SOX), FISMA, ISO 17799 or any other regulation/standards requiring business impact analysis, risk assessment, disaster recovery planning (DRP), business continuity plan (BCP) and Testing & Revision of Plan.

Popular posts from this blog

Zeek in Action Videos

MITRE ATT&CK Tactics Are Not Tactics

New Book! The Best of TaoSecurity Blog, Volume 4