Comments on SANS CDX Briefing
One of the benefits of paying for this week's SANS Log Management Summit was attending a briefing last week on the latest Cyber Defense Exercise conducted by the NSA. SANS organized a panel with a USAFA cadet, a USNA midshipman, a USMA-grad Army 2LT, and several NSA or ex-NSA representatives, along with their boss, Tony Sager. Although I've known of CDX for several years, this was my first real insight to how these exercises are conducted.
The NSA organizer, or "white cell leader," is Bruce Rogers. He explained that competitions can be conducted either as capture-the-flag style events or purely defensive affairs. CDX is purely defensive. When I asked Mr. Rogers if he had spoken to any organizers of other cyber competitions, like those of Def Con or ShmooCon, he said no. Mr. Rogers has 20 white controllers overseeing the exercise, which includes 6 targets (the six defending teams -- USAFA, USNA, USMA, USMMA, AFIT, and NPS).
The attackers are split into two groups. The first group consists of "tainters". These are 13 NSA personnel, which included one high school-age intern and one college-age intern. The tainters spent about 80 man-hours building and then misconfiguring, rootkitting, and otherwise tampering with virtual machines delivered to the CDX participants. The participants had two weeks to analyze these VMs for vulnerabilities and exploitation, after which they had to activate and then defend them. These compromised servers are supposed to be similar to "host nation machines" that military personnel might find themselves operating.
I was initially shocked by this news. Who in their right mind would trust host nation equipment for sensitive operations? Wouldn't it be best to rip everything out and start fresh with clean, trusted media? After some thought, I decided that this tainting phase was more realistic than I initially believed. Unless one joins a very small company, no new security or IT employee is ever allowed to begin work at a new job by rebuilding all infrastructure. When you join a new company, you're stuck with all of the garbage they give you.
The second group of attackers are the traditional red team. This group consists of real red teams from across the services, such as the USAF 92nd Information Warfare Aggressor Squadron. The red team hammered the 6 target networks for 4 straight days. The target networks were hosted on live network links at the respective schools and were connected back to NSA via VPN. No simulated non-malicious traffic was carried to or from the target networks. Everything on the wire was considered malicious since the red team was creating it. This is highly unrealistic, but partially driven by the bandwidth available to some of the teams. At least one hosted their target network on an ISDN line.
Each of the military participants said a few words about their teams and experiences. Three themes stood out. First, the team size varied widely. USAFA's team had 9 people. USMA's team had 35-40. (USAFA won.) Second, most of the teams admitted having little or no security training. I was amazed. Who signs up for a hack-fest without having security experience? Third, the networks designed by each of the teams varied widely. USAFA emphasized simplicity. USNA concentrated upon prevention, and never regained control once their servers were compromised. ("Prevention eventually fails." -- Tao) USMA's network was exceedingly complex, but they tried to watch outbound traffic for signs of compromise (e.g., extrusion detection). No team was allowed to block traffic from malicious IPs.
All of the target networks ended up being 0wn3d. USAFA didn't notice a rogue Apache module that resulted in a Web site defacement. USMA missed a default password on their router and lost control of it. The red team said that the best team only found 15% of the vulnerabilities created by the "tainters." Wow. By the way, the tainters did not tell the red team what they did to the VMs. The tainters dropped some clues as the exercise progressed, but the red team mostly used standard penetration techniques.
These were the lessons learned from the 2006 CDX.
Top 9 Exploited Vulnerabilities
Student Best Practices
Finally, two of the panel members (I remember USAFA cadet Michael Tanner told this story) participated in CDX and the National Collegiate Cyber Defense Competition. Rob Lemos wrote about it for SecurityFocus. Cadet Tanner said the national CDX was completely different from the military CDX. The military CDX allowed participants to protect and host their VMs using a variety of technologies. USAFA used mainly OpenBSD. AFIT used all Windows. Other groups uses other technologies. At the national CDX, participants were given a ton of commercial equipment (all from sponsors, no doubt) and then found themselves hacked to pieces five minutes into the exercise. Apparently they were given no opportunity to do anything with the equipment prior to the exercise starting?
Overall, I found the session to be extremely informative. I'd like to thank Alan Paller from SANS for organizing this event and I appreciate the participants sharing their experiences. If you want more details, I found some papers on both exercises posted at the The Colloquium for Information Systems Security Education.
The NSA organizer, or "white cell leader," is Bruce Rogers. He explained that competitions can be conducted either as capture-the-flag style events or purely defensive affairs. CDX is purely defensive. When I asked Mr. Rogers if he had spoken to any organizers of other cyber competitions, like those of Def Con or ShmooCon, he said no. Mr. Rogers has 20 white controllers overseeing the exercise, which includes 6 targets (the six defending teams -- USAFA, USNA, USMA, USMMA, AFIT, and NPS).
The attackers are split into two groups. The first group consists of "tainters". These are 13 NSA personnel, which included one high school-age intern and one college-age intern. The tainters spent about 80 man-hours building and then misconfiguring, rootkitting, and otherwise tampering with virtual machines delivered to the CDX participants. The participants had two weeks to analyze these VMs for vulnerabilities and exploitation, after which they had to activate and then defend them. These compromised servers are supposed to be similar to "host nation machines" that military personnel might find themselves operating.
I was initially shocked by this news. Who in their right mind would trust host nation equipment for sensitive operations? Wouldn't it be best to rip everything out and start fresh with clean, trusted media? After some thought, I decided that this tainting phase was more realistic than I initially believed. Unless one joins a very small company, no new security or IT employee is ever allowed to begin work at a new job by rebuilding all infrastructure. When you join a new company, you're stuck with all of the garbage they give you.
The second group of attackers are the traditional red team. This group consists of real red teams from across the services, such as the USAF 92nd Information Warfare Aggressor Squadron. The red team hammered the 6 target networks for 4 straight days. The target networks were hosted on live network links at the respective schools and were connected back to NSA via VPN. No simulated non-malicious traffic was carried to or from the target networks. Everything on the wire was considered malicious since the red team was creating it. This is highly unrealistic, but partially driven by the bandwidth available to some of the teams. At least one hosted their target network on an ISDN line.
Each of the military participants said a few words about their teams and experiences. Three themes stood out. First, the team size varied widely. USAFA's team had 9 people. USMA's team had 35-40. (USAFA won.) Second, most of the teams admitted having little or no security training. I was amazed. Who signs up for a hack-fest without having security experience? Third, the networks designed by each of the teams varied widely. USAFA emphasized simplicity. USNA concentrated upon prevention, and never regained control once their servers were compromised. ("Prevention eventually fails." -- Tao) USMA's network was exceedingly complex, but they tried to watch outbound traffic for signs of compromise (e.g., extrusion detection). No team was allowed to block traffic from malicious IPs.
All of the target networks ended up being 0wn3d. USAFA didn't notice a rogue Apache module that resulted in a Web site defacement. USMA missed a default password on their router and lost control of it. The red team said that the best team only found 15% of the vulnerabilities created by the "tainters." Wow. By the way, the tainters did not tell the red team what they did to the VMs. The tainters dropped some clues as the exercise progressed, but the red team mostly used standard penetration techniques.
These were the lessons learned from the 2006 CDX.
Top 9 Exploited Vulnerabilities
- Microsoft Windows LSASS Buffer Overflow Vulnerability
- Microsoft DCOM
- LM Hash versus NTLM Authentication Protocol
- Use of Weak Passwords
- Use of the same password on Multiple Systems
- Microsoft Windows Default Administrative Shares
- Rich Text Format / HTML Email
- Access to System Executables
- Use of Unnecessary Services / Accounts
Student Best Practices
- Know the Network and Keep it Simple: Each additional device is another avenue of attack. The entire team must understand the network. Troubleshooting is easier with a simple design.
- Deny by Default Policy: Only allow what is absolutely necessary. It's easier than blocking known bads.
- Remove Unnecessary Services, Software, and User Accounts: What is the role of the computer? Remove unnecessary software completely.
- Plan for Contingencies: All networks will eventually have a problem.
Finally, two of the panel members (I remember USAFA cadet Michael Tanner told this story) participated in CDX and the National Collegiate Cyber Defense Competition. Rob Lemos wrote about it for SecurityFocus. Cadet Tanner said the national CDX was completely different from the military CDX. The military CDX allowed participants to protect and host their VMs using a variety of technologies. USAFA used mainly OpenBSD. AFIT used all Windows. Other groups uses other technologies. At the national CDX, participants were given a ton of commercial equipment (all from sponsors, no doubt) and then found themselves hacked to pieces five minutes into the exercise. Apparently they were given no opportunity to do anything with the equipment prior to the exercise starting?
Overall, I found the session to be extremely informative. I'd like to thank Alan Paller from SANS for organizing this event and I appreciate the participants sharing their experiences. If you want more details, I found some papers on both exercises posted at the The Colloquium for Information Systems Security Education.
Comments
What I find a bit disturbing, and I am kinda hoping I misread your post, but the top 9 exploited vulnerabilities were those that the red team most often exploited?
This is disturbing because LSASS and DCOM are patchable. LM Hashes are a registry setting. Weak passwords are a no-brainer. Use of the same password om multiple systems is, sadly, a fact of life in networks, but for an event like this, I would take some care to make them all different, same with email settings (I really wish I could make everyone who loves Outlook use plain text, but Marketing would throw a fit due to their fancy emails). Etc, etc.
Simplify, simplify. Not only are simple networks easier to defend and troubleshoot, but the simple things in security are the best things to do. Get back to basics and leave the complex, fancy defenses and mitigations for a rainy day.
But you are right, this is all very much an ideal. If I could rebuild networks I have control over, they would be much better than the networks that have simply grown in the past years.
We have a long way to go in security...
-LonerVamp
LSASS and DCOM were the most often used routes to get into the target machine. Most of the pre-tainted exploits/rootkits/... were un-usable because the schools were doing proper firewalling.
Another thing of note: For the first three days, the pre-tainted stuff is not available. Hard/fresh attacks are all that is available. Day four, the pre-taint becomes fair game during the first half of the day. The last half of the day, all bets are off, and anything that you can possibly do is fair game. (including social engineering)
The thing that frustrated Red Team the most is that the schools were not following the ground rules with regards to the VM. Red Team would get in, be on the box for 30 seconds to 10 minutes, then the schools would discover it, and simply revert the VM image snapshot instead of actually finding and fixing the problem.
I was wondering what you guys thought on the honeypots that the second and third group implemented? Was this an overkill?
-VonayotsV