Wednesday, June 10, 2015

My Federal Government Security Crash Program

In the wake of recent intrusions into government systems, multiple parties have been asking for my recommended courses of action.

In 2007, following public reporting on the 2006 State Department breach, I blogged When FISMA BitesInitial Thoughts on Digital Security Hearing. and What Should the Feds Do. These posts captured my thoughts on the government's response to the State Department intrusion.

The situation then mirrors the current one well: outrage over an intrusion affecting government systems, China suspected as the culprit, and questions regarding why the government's approach to security does not seem to be working.

Following that breach, the State Department hired a new CISO who pioneered the "continuous monitoring" program, now called "Continuous Diagnostic Monitoring" (CDM). That CISO eventually left State for DHS, and brought CDM to the rest of the Federal government. He is now retired from Federal service, but CDM remains. Years later we're reading about another breach at the State Department, plus the recent OPM intrusions. CDM is not working.

My last post, Continuous Diagnostic Monitoring Does Not Detect Hackers, explained that although CDM is a necessary part of a security program, it should not be the priority. CDM is at heart a "Find and Fix Flaws Faster" program. We should not prioritize closing and locking doors and windows while there are intruders in the house. Accordingly, I recommend a "Detect and Respond" strategy first and foremost.

To implement that strategy, I recommend the following, three-phase approach. All phases can run concurrently.

Phase 1: Compromise Assessment: Assuming the Federal government can muster the motivation, resources, and authority, the Office of Management and Budget (OMB), or another agency such as DHS, should implement a government-wide compromise assessment. The compromise assessment involves deploying teams across government networks to perform point-in-time "hunting" missions to find, and if possible, remove, intruders. I suspect the "remove" part will be more than these teams can handle, given the scope of what I expect they will find. Nevertheless, simply finding all of the intruders, or a decent sample, should inspire additional defensive activities, and give authorities a true "score of the game."

Phase 2: Improve Network Visibility: The following five points include actions to gain enhanced, enduring, network-centric visibility on Federal networks. While network-centric approaches are not a panacea, they represent one of the best balances between cost, effectiveness, and minimized disruption to business operations.

1. Accelerate the deployment of Einstein 3A, to instrument all Federal network gateways. Einstein is not the platform to solve the Federal government's network visibility problem, but given the current situation, some visibility is better than no visibility. If the inline, "intrusion prevention system" (IPS) nature of Einstein 3A is being used as an excuse for slowly deploying the platform, then the IPS capability should be disabled and the "intrusion detection system" (IDS) mode should be the default. Waiting until the end of 2016 is not acceptable. Equivalent technology should have been deployed in the late 1990s.

2. Ensure DHS and US-CERT have the authority to provide centralizing monitoring of all deployed Einstein sensors. I imagine bureaucratic turf battles may have slowed Einstein deployment. "Who can see the data" is probably foremost among agency worries. DHS and US-CERT should be the home for centralized analysis of Einstein data. Monitored agencies should also be given access to the data, and DHS, US-CERT, and agencies should begin a dialogue on whom should have ultimately responsibility for acting on Einstein discoveries.

3. Ensure DHS and US-CERT are appropriately staffed to operate and utilize Einstein. Collected security data is of marginal value if no one is able to analyze, escalate, and respond to the data. DHS and US-CERT should set expectations for the amount of time that should elapse from the time of collection to the time of analysis, and staff the IR team to meet those requirements.

4. Conduct hunting operations to identify and remove threat actors already present in Federal networks. Now we arrive at the heart of the counter-intrusion operation. The purpose of improving network visibility with Einstein (for lack of an alternative at the moment) is to find intruders and eliminate them. This operation should be conducted in a coordinated manner, not in a whack-a-mole fashion that facilitates adversary persistence. This should be coordinated with the "hunt" mission in Phase 1.

5. Collect metrics on the nature of the counter-intrusion campaign and devise follow-on actions based on lessons learned. This operation will teach Federal network owners lessons about adversary campaigns and the unfortunate realities of the state of their enterprise. They must learn how to improve the speed, accuracy, and effectiveness of their defensive campaign, and how to prioritize countermeasures that have the greatest impact on the opponent. I expect they would begin considering additional detection and response technologies and processes, such as enterprise log management, host-based sweeping, modern inspection platforms with virtual execution and detonation chambers, and related approaches.

Phase 3. Continuous Diagnostic Monitoring, and Related Ongoing Efforts: You may be surprised to see that I am not calling for an end to CDM. Rather, CDM should not be the focus of Federal security measures. It is important to improve Federal security through CDM practices, such that it becomes more difficult for adversaries to gain access to government computers. I am also a fan of the Trusted Internet Connection program, whereby the government is consolidating the number of gateways to the Internet.

Note: I recommend anyone interested in details on this matter see my latest book, The Practice of Network Security Monitoring, especially chapter 9. In that chapter I describe how to run a network security monitoring operation, based on my experiences since the late 1990s.


Jim Voorhees said...

This is a worthwhile program, certainly. I fear, however, that it is utterly impractical in the current environment.

Above all, the politics of the program, as you suggest in the description of Phase 1, will prevent much of it from being put into place. Would that it were otherwise. Despite the breaches at OMB, State, the IRS, and elsewhere, not enough people have suffered. Importantly, not enough heads have rolled.

Also, I have to wonder whether there are enough people skilled enough to find and remove the adversary. The OPM IG report suggests that it is hard enough to find people who can do compliance, which is less difficult technically. Finding enough competent people to scour all Federal networks for IOCs may be close to impossible.

These questions aside, however, thanks for laying out this program. It is, at the least, a useful model for those serious about the task.

CyberSITREP said...

Agree on the "hunter" mission to, as you stated, "Detect and Respond" - and this should be performed DoD-wide. And to utilize CDM as a Phase of this strategy, not the only phase. Unfortunately, staffing of skilled cyber professionals remains an issue and will probably be the last hurdle addressed - after useful tools and tactics are finally in place. Hopefully, someone in a position to bring change to the current way tasking/performing computer network defense within the government reads, and listens to your post.

Anonymous said...

There's only one disagreement I have with this, which is that you say that DHS should do this. Even staffing them up (and given the shortage of security professionals available in the market, that's no easy task), they're mindset and structure is suboptimal at best for owning this; they will never be the experts on this, even if it is technically their job thanks to their poorly structured charter. Security of the .gov domain should always have been an NSA function, given the technical skills and security roles they already have.

Mike Coomes said...

This blog lays out the basics to get started, there could be much more but this is a very good start. However the current culture in the government; executive, and legislative, looks for quick fixes - and we get whack a mole) or attempts to invent new programs. Until there is a major culture shift in how security is viewed, unfortunately, no one will act on this minimum list. Currently Agency Heads do not "own" their cyber risk; they should and should be terminated no matter how competent in other areas for these types of breaches just like the Target CEO/CISO. Until we look at cyber security as a business practice enables operation and is "owned" by the Agency head and deputy then we'll not make progress. While I don't advocate current heads to roll, after the next breach they should and this needs to be made clear by the White House to all the Departments/Agencies. Only then will a culture shift start to occur. Harsh but necessary. The other component is CISOs need to speak in terms of risk to the Department/Agency not in technical terms. That is required for the leadership to understand what is at risk.
To instill a culture of security is more than just technology, it is policy and it's enforcement, training for all users on a continual basis, and regular reviews of cyber security operation in light of new threats. This will ensure the basic program outlined in this blog will endure. We need the cyber equivalent of the WWII "loose lips sink ships" idea that everyone must be aware and on guard.

Anonymous said...

Security operations to identify and remove threat actors already present...

Any documentation on how to go about this?

Richard Bejtlich said...

Re: Anonymous:

Bill Voisine said...

This is a great guidance. I couldn't agree more about CDM not working.

Being a former Fed and hearing similar concerns from other former Feds; there is a large problem with focus on FISMA scores, since FISMA scores drive funding, as opposed to focusing first on security application from a strategic/tactical angle which would still eventually lead to compliance.

Frank Johnson said...

Hi, just curious -- the DHS says that CDM stands for "Continuous Diagnostics and Mitigation," but in this post and your previous one, you refer to the program as "Continuous Diagnostic Monitoring." Is this because you believe there's no true "mitigation" going on with the CDM program?

Richard Bejtlich said...

Thanks for the comment Frank. "Continuous Diagnostics and Mitigation" is the third name for this program. First it was "Continuous Monitoring." Then it became "Continuous Diagnostic Monitoring." Now it's "Continuous Diagnostics and Mitigation," as you mentioned. I have been using the 2nd term recently. I should have adopted the third, although the programs are the same.

Anonymous said...

Can anyone tell me if there are any resources out there for
procedures and strategies to conduct "Hunting" operations
for the private sector? I know that the NSA has special
training for that mission skill set, but any recommendations
for outside of that venue?

Anonymous said...

Position: Supervisory IT Specialist (INFOSEC) – Director, Cybersecurity Division, OCIO, GS-2210-15
Open Period: Wednesday, June 17, 2015 to Wednesday, June 24, 2015

Incumbent serves as Director, Cybersecurity Division for the Security Operations Center (SOC) and has responsibility for the overall leadership of the Division, cybersecurity Executive Office of the President (EOP)-wide, and management of the SOC. The Director will be responsible for oversight of cybersecurity through all branches within the Division; advising the CIO and other EOP stakeholders of cybersecurity concepts and outcomes; identifying and executing standardized IT solutions; assessing the impact on security of Federal regulation/policy compliance and ensuring that the business impact of implementing the best practices within the EOP are understood. - Supervisory IT Specialist (INFOSEC) – Delegated Examining

Anonymous said...

Was there any answer to Anon's question about resources for procedures and strategies for ops of this sort? VUlturing this blog entry for such, thanks.