FISMA 2006 Scores
There are FISMA scores for 2006, along with 2005, 2004, and 2003 -- some of which I discussed previously. What I wrote earlier still stands:Notice that these grades do not reflect the effectiveness of any of these security measurements. An agency could be completely 0wn3d (compromised in manager-speak) and it could still receive high scores. I imagine it is difficult to grade effectiveness until a common set of security metrics is developed, including ways to count and assess incidents.
I still believe FISMA is a joke and a jobs program for so-called security companies without the technical skills to operationally defend systems.
The only benefit I've seen from FISMA is that low-scoring agencies are being embarrassed into doing more certification and accreditation. C&A is a waste of time and money. However, if security staff can redirect some of that time and money into technical security work that really makes a difference, then FISMA is indirectly helping agencies with poor scores. Agencies with high scores are no more secure than agencies with low scores. High-scoring agencies just write good reports, because FISMA is a giant paperwork exercise that makes no difference on the security playing field.
If you believe otherwise you're welcome to your opinion. You're also welcome to the lack of a future job when the FISMA consulting boondoggle ends and report jockeys are left without any marketable technical skills. If you want to know more about this, reading my old FISMA posts is sufficient. I don't need to restate my arguments when they're archived.
If I sound bitter, it's because I've seen my taxpaying dollars wasted for the past five years while various unauthorized parties have their way with these agencies. FISMA is not working.
Comments
Second question, what do you suggest as a replacement to FISMA? Remember, it has to scale across the breadth and diversity of the U.S. Government.
Dan
Asking me to solve the government's security problems in a blog comment is a nice trick because there's no way for me to win. That tactic is similar to the dozens of people who ask me consulting questions via email and expect free yet detailed and customized responses. For a hint of what I would do, please read Security Operations Fundamentals.
I'm not going answer your last email because I prefer not to carry some kind of back-channel conversation while addressing this topic publicly. If you can demonstrate how FISMA works, please do. Like I said in FISMA Redux:
The bottom line is that FISMA doesn't mention C&A at all, but the author [of a FISMA book] thinks that's ok because C&A fulfill's FISMA's goals. The reality is far different. According to the act itself, the first "purpose" is to:
provide a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support Federal operations and assets. (emphasis added)
Clearly FISMA is failing. How can anyone think it's succeeding, unless you're a FISMA contractor making money off it?
It looks like FISMA criteria does not consider the benefits MSSPs provide governement customers. Shouldn't these audits take into consideration the value-added services that governement agencies outsource because they don't have the in-house expertise to implement.
Thanks
The approach for grading the agencies doesn't take into account any kind of outsourcing, nor should it.
Basically, the government needs something done (in this instance, it's security), and the answer, just like for anything else, is always build, buy, or outsource. The question should be this: is the need being satisfied effectively?
The problem is that the grading system is based on audit findings and some level of "compliance" (read my blog to see how I feel about compliance).
So just like any school experiences you've had, how do you determine which grading criteria to use there, just how subjective is the grading system, and do you even need to give grades in the first place?
Then you have people who know how to game the grading system. We have too many of those people around, and just like in the school analogy, they're hated.
I'm one of the horde making money off FISMA, testing Systems for C&A... although I'd rather be working with the technical side like we used to do before 800-53.
The problem as I see it is that the government appears to be historically good at two things: producing paper and planning for long-term budgets. There appears to be little-no oversight in the short-term, and there is nothing sexy about getting in the trenches or finding the source of real problem.
Come on everyone, think like a manager: the spirit of security never even enters the mental picture. Think paper and planning... Think CYA... ONE OF US! ONE OF US!
As Bruce Potter once said: "No one cares about security, except security folks."
-sorry for the anon post... I'll be better next time.
You are so right on documentation versus doing. I was CTO of EPA during the F-A+ climb, so I have some experience here. My auditors kept me at low level during the time I did the implementation and only let me climb once I did the documents. so what - I did what was needed and took the lumps because it was right thing to do. Auditors have a problem in that no one in private or public sector has a good scorecard for the implmentation part so while FISMA is not perfect its a hell of lot better than nothing - it does force someone to look and forces somethings that are needed even if they are not sufficent.
I am still interested in your thoughts on what a better solution would look like. I've been reading through FISMA and the NIST SP-800 series documents plus I have a little practical experience in its implementation now to add to my previous experiences of federal security audits prior to FISMA. The differences are considerable. FISMA is unwieldy in implementation and has many areas in which to mature and improve. I could write volumes about what could be improved just from what little I have seen of it so far. Despite that it is far superior to what proceeded it. Namely a lack of motivation to ensure IT security (outside of not wanting to end up above the fold on the Washington Post front page) and a dearth of technical guidance.
As for the meaning of the report cards I think there's a simple logic to the problem with those report cards. It involves the learning curve. The people who do the C&A are learning to do it better each time they go through the process. As their ability to judge security matters improves they become increasingly aware of deficits in current security implementations. So they find more problems where before none were detected or reported. This filters up the food chain until they become failing grades in reports of questionable utility, but great visibility, which force action by management. But in the short term the feedback loop reports that there are problems, and that's what feedback is all about.
Speaking of FISMA, have you looked at the NIST SP-800-94 guide covering IDS and IPS? You have some expertise in the topic and it would be interesting to hear what you think of it.