Get the Divers Out of the Water
I'm wondering if this story resonates with anyone.Imagine a group of undersea divers. They are swimming in the ocean doing some sort of productive activity, maybe retrieving treasure, or doing research, or something else. The divers receive instructions from managers in a boat.
Suddenly one of the divers is attacked by a shark. It tears right through his diving suit. There's blood in the water. The managers see the blood but tell the divers to keep doing their work. The injured diver attracts other sharks. Now the other divers are being attacked. The managers tell the divers to keep working.
It's a disaster. Divers are severely injured, and some are dying. In the boat some generalist first responders see the blood, and recommend putting the divers in protective cages. They aren't sure exactly what is happening so they fall back on the standard operating procedures.
A few of the divers seek shelter in the cages. Now the managers are howling that the divers aren't doing their work. They want the divers sent back out.
The generalist first responders don't know what to do. They ask if anyone else in the boat can help. Some specialist responders lower a camera into the water and see sharks eating divers. They tell the managers to pull the divers out.
The managers concede that the sharks are a problem but they want some sort of customized response for each injury. Can't we assess each diver, identify the damage, apply some bandages, and keep the work going?
This debate rages for hours, far too long in the opinion of everyone involved. More and more divers are hurt, the sharks continue to swarm, and no one is happy.
Let's explain this story.
- The divers are computers.
- The sharks are intruders, possibly even malware.
- Dying divers are computers whose data is being denied, degraded, or stolen by intruders.
- The managers are managers, or asset owners.
- The generalist first responders operate anti-malware software.
- The diving suit is anti-malware software in a default configuration.
- The cage is anti-malware software operated in a more aggressive configuration.
- Getting divers out of the water means isolating a compromised computer from the network.
- The specialist first responders are the incident response team.
- The camera lowered into the water is an investigation of the malware by the IR team.
My question is: how should this scenario have played out? I have a few recommendations:
- If you're going to swim in shark-infested waters, be resistant to shark attack, not ignorant of shark attack. Realize sharks are everywhere and prepare your defenses appropriately.
- If you're attacked by sharks, and your defenses fail, your first priority is to try to save the first victim.
- The second priority is to protect the rest of the divers so they can continue their mission.
- The priority should not be to keep everyone performing their mission, because it ignores the risk of the first diver dying (data loss, etc.) and the risk of exposing the other divers to attack (propagation of the malware).
- The fastest way to accomplish both priorities is to have a pre-approved incident response plan, with provisions for getting divers out of the water. This can involve an approval process where managers are told the situation and asked for approval to disconnect the victim. The difference between this process and what happened in the story is that the debate centers on whether or not to implement containment, not what should be done in general.
- Managers have to realize that they can't put vulnerable divers in the water and expect no negative consequences when they are attacked. Either spend resources up front to better protect the assets, or act quickly and decisively once trouble happens. Trying to plough on whatever the situation descends into lengthy and costly chaos.
I'm curious if anyone else has thoughts on this. I am interested in cases where the threat is fairly common (i.e., not advanced threats), so there is little to be gained by trying to learn more by observing the adversary.
Comments
at a very large organization, the sheer volume of "common" infections would preclude this analysis in every case; so maybe on, say, one out of twenty you take the time for detailed forensic analysis.
i think in most cases, though, if you've discovered the infection through some signature-based detection (IDS/IPS, AV, etc), someone's already enough analysis on the threat itself. if a quality writeup exists, you're golden. if it's a pervasive threat in your environment, you might benefit from further study of a sample.
in fact, our IR team is considering study of a a "known" threat that's been rather annoying for the past few days. at this point i think it's more out of spite for the threat than anything else :)
Yes, I agree -- I listed the "camera in the water" as the example of getting more information on the situation. There is a definitely a balance here but I think we would agree that you could contain the first injured diver sufficiently to identify the problem and apply the lessons to saving the others?
Maybe I misunderstand but diving has a number of 'safety first' principals that would stop any production effort to save lives including the 'dive buddy' concept. Really, the people in the water get to make the safety decision that is supported up top.
Can the principals of diver safety and dive buddy concept be added, does the analogy still work?
I like the "safety first" addition to the analogy. There are any number of IPS tools that can respond to an identified threat by taking the system off-line (or blocking its network communication) to prevent the spread of the malware.
The sharks in our case are not just sharks but sharks embedded with experience of a human adversary.
Instead, all of the divers would be killed, and all the sharks gone...maybe some other species of fish lower on the food chain would come in...and some other fishermen would come out weeks later to tell the guys in the boat that all of their divers are gone. The guys in the boat would be in complete disbelief, even though no one had noticed, or thought to question why, that none of the divers had returned to the surface, etc.
I believe managers should have well qualified divers to begin with. Divers, who although are aware of processes, procedures and rules should be trusted enough and capable enough of making critical decisions. I take this one step further and ask you in return... The divers themselves are the ones on the battlefield seeing the sharks in real-time. The managers are often looking from above potentially and highly likely to be unaware of ALL of the dangers. "What if the diver only has minutes left or his respirator is acting up, although the manager may see things from one perspective, the diver will always see another.
Having a "pre-approved" incident response plan sounds fine on paper but any documentation or plan is usually going to be based on experiences, what has been seen or known to have occurred. You cannot and never will be able to account for chaos. Many of the "written”, theoretical and “pre-approved” methods have failed time and time again. Check out: "The Byzantine Generals Problem" ACM Transactions on Programming Languages and Systems, Vol. 4, No. 3, July 1982: (http://research.microsoft.com/en-us/um/people/lamport/pubs/byz.pdf) In all examples used in the article they failed to take into account normal chaos, the inability to predict - after all I think we’d all be wealthy, war-free if anyone could predict outcomes of anything. We can continue tricking ourselves into believing this, but personally I think I’d be a fool solely relying on anything pre-approved. Here is a little thought that completely throws the Byzantine Generals Problem into chaos:
Commander --> Attack Lieutenant 1 --> Lieutenant 1 has a heart attack unable to progress
Lieutenant 2 --> Well I never received the go ahead from Lieutenant 1 therefore let’s stay put
Commander --> Lieutenant 2 --> We’re getting slaughtered why haven’t you moved!
Same issues can come into play in the diver analogy. What about deception, where would you place this? (http://all.net/journal/deception/Framework/Framework.html) Perhaps the manager thought about the sharks beforehand and had a tub of raw meat and blood capable of being thrown in the water to draw the sharks away from the divers. In this instance, the divers may not need to be taken out of the water. The one injured diver can be removed while work goes along. The divers were properly trained and were given a little bit of freedom to think for themselves. After all, the divers are in the dangerous water and will ultimately have to make a critical decision when all else fails... The manager, he is always going to be high and dry scrapping up sometimes useless plans which sound good on paper, but don’t pan out in the real world.
We are missing a lot of things to begin with.
For instance, in a true Network Security Monitoring situation, we'll have a firewall and router ACLs. That implies that there will be some sort of barrier around the divers that limits what they can access and what can access them. Likely, there will already be "cameras" setup watching everything that goes in and out. While not generally useful in real time, for Incident Response it can be invaluable as you will see in a moment. We also should have some Intrusion Detection and Prevention deployed to look closer at what is allowed in and out through the barrier.
Lets say we are protected against all known predators in the waters where the divers(computers), are working, but a diver gets attacked anyway. Putting a camera down at this point may help, but what you are looking for has already passed and is now inside the barrier. you don't know what it is, how many there are, or what the threat really is. By reviewing the footage on the cameras you already had as a part of NSM, you see that the diver was attacked by a previously unknown species of shark. Your defenses weren't aware and therefore it got through. You also pick up what the diver was doing that attracted the shark in the first place. Another quick search tells you how many got through and what other divers are in peril. The footage is then quickly handed to a shark expert (malware analyst) who does a surface and runtime scan of the footage to determine what the characteristics are and what other intent there may be.
It is determined that the shark does not radiate a signal, there is only one, and it is not capable of attacking the other divers. The diver and shark are isolated, the shark is removed, the defenses are adjusted for the new predator, and the rest of the divers can continue.
If it were some sort of spore that infected the diver and reproduced, then you may have to isolate a group of divers to prevent the spread and work on disinfecting them.
If several of the sharks came in disguised as a combination of angel fish, parrot fish, trunk fish, or groupers, then your NSM will help you track which ones were fakes determine if the divers they were heading for fell for the ruse, and then you could determine which needed to be isolated.
We know that there are unlimited variants of threats and that our protection only recognizes a very small portion of those threats. As long as we rely entirely on a product to protect us, we will be blind or worse to new threats emerging every day. The way you back up that protection is with NSM and a solid incident response program. It has been proven on a near weekly basis that if you have to wait for some large commercial security company to update their already overly large signature database, then it is entirely possible that your enterprise could be experiencing compromises you would have no way of detecting, and you would have very little response capability even if you were somehow made aware of it.
-LonerVamp
The scene: The Fremen have just captured a band of smugglers led by Gurney Halleck, Paul's old tutor and protector. Paul is inquiring about the public opinion Rabban, the Harkonen ruler (in relation to the opinion of him and the Fremen).
"What's the talk of Rabban in the Sinks and Villages?" Paul asked.
"They say they've fortified the graben villages to the point where you cannot harm them. They say the need only sit inside their defenses while you wear yourselves out in futile attack."
"In a word," Paul said, "they're immobilized."
"While you can go where you will," Gurney said.
"It's a tactic I learned from you," Paul said. "They've lost the initiative, which means they've lost the war."
Does any of this sound familiar? Are we Rabban, or Muad'Dib? Here's a hint. If your posture is more like Rabban's, you're doing something wrong.
-LonerVamp
if you have a flexible response to invasion, you can deflect, evade, or trap it much easier.
http://www.youtube.com/watch?v=eL5o4PFuxTY
You'll have to watch all the way to the end to get your answer, but it will be worth it.
(I'll sum up: Don't play by your enemies' rules.)