Network Security Monitoring Case Study
I received the following email from a friend. He agreed to share his story in exchange for commentary from me and fellow blog readers. I've added comments inline.
I'm now responsible for cleaning up a mid sized company perimeter defences... To be honest, at first glance the task is a daunting one, thousands of users, dozens of dis-separate systems and gigabits of network traffic plus as part of the enterprise support team, I have other projects and duties to deliver on.
I managed to get time with the systems architect and run through a number of questions stolen from your books and other smart folks on the history and state of affairs of my new domain. My questions were answered or put in to context except for one.
"Why don't you have any monitoring tools in place on the edge and perimeter systems?"
The answer I received wasn't what I expected. He simple stated that no-one had the time or energy to take on this massive task. He was very pro about having monitoring, but bluntly realistic on the amount of time monitoring and response took up. The business has other priorities for the IT teams.
It's not that the company isn't security aware, they have good policies, skilled staff, harden systems and layer defences, but no monitoring.
I was stuck with the thought "If I don't understand what's happening on the network, how can I know what's right or wrong, what's normal or abnormal?"
This is exactly right. My friend's co-workers are probably practicing management by belief instead of management by fact. If you do not know for sure what is happening inside your company, how can you make informed decisions? Failure will occur when your luck runs out because you are not spending the time and resources necessary to acquire and maintain digital situational awareness.
I gathered all the documentation, policies and firewall rules together and tried to make sense of them. I re-read Extrusion Detection on my commute in to help me frame a plan of attack. I got all the basic system data of every system, check time and date stamps were all in sync and cleaned up the network maps.
Then, as silly as this seems, the obvious answer came to me on where to start: at the internal firewall internal interface.
I started to review the drop logs of the firewall for that interface, reasoning that if there is a problem on the network, it should more apparent in those logs than anywhere else.
This is also exactly right. There is no need to start buying or building new tools if you are not leveraging your existing data sources or enabling data sources not yet activated.
Those reviews provided some startling results, fortunately for us, the problems were only mis-configured systems attempting outbound access. Mind you, they were generating considerable amounts of traffic and some over already congested WAN links.
Then I laid out the ingress and egress rule sets to find out the motives and owners of those rules in the firewalls and started to peel back those unused or unneeded rule sets, while monitoring the drop logs. All with management approval and sign off, of course!
This is another great move -- a stride towards simplification.
I'm now moving in to the next stage of working with other teams to clean up their problem systems. Some are going to be harder than others as I've found VAX's, AIX, Sun systems and others bouncing off the rules. I haven't seen these types of systems in a long time and as a simple Windows administrator, I have to find time and the right people to plug these holes.
I have a very long way to go, only focused on a very small section of the entire network and no certain path forward on how to apply a reasonable monitoring system - yet :-)
It's a challenge I'm looking forward to and one I'm sure I'll learn from.
The only comment haunting me is the not having any real time allotted to doing monitoring work. I'm going to talk with management and a surprisingly security literate CIO on if this can be addressed.
I know that the conversations have to be based on preventing/mitigating business risk, but believe it's going to be a hard sell with the very long list of other projects being pushed on the team as priority.
If you have any words of advice or useful pointers on convincing them the time is well spent, I'd be grateful.
So this is the major question. How do you convince management or other functional areas that monitoring is important? It sounds to me like my friend has already scored some wins by freeing bandwidth used by misconfigured systems, simplifying firewall rules, and examining individual problematic hosts.
It's important to remember that there is no return on security investment. Security is a cost center that exists to prevent or reduce loss. It is not financially correct to believe you are "earning" a "return" by spending time and money to avoid a loss.
If I need to spend $1000 to hire a guard to protect my $10000 taxi, I am not earning a return on my investment -- I am preventing the theft of my taxi. If I invest that $1000 in a ticketing and GPS system that makes me more productive ferrying passengers (perhaps increasing my dollars per hour worked), then I have enjoyed a ROI once my $1000 expense is covered.
This is not to say that money should not be spent on security. Rather, time and money should be balanced against the perceived risk. The same taxi driver will spend money on insurance, and may indeed need to spend $1000 on a guard if the protection delivered is seen to be necessary. However, the guard does not make the taxi driver more productive. Security is a "business enabler" here but it does not deliver "ROSI."
I'm not a big fan of FUD but something may be happening to executive perception of digital security. I am starting to hear questions like "How do I know that asset X is safe?" or "How do I know that asset Y is not being attacked?" (Answers: "safe" is relative, and Y is being attacked if it's accessible to anyone.)
I don't have any quick answers for my friend, which is why I'm posting this story. What are you doing to convince management that monitoring is necessary? I personally plan to do exactly what my friend did, namely starting with existing assets and showing quick wins to build momentum for more extensive visibility initiatives. You?
I'm now responsible for cleaning up a mid sized company perimeter defences... To be honest, at first glance the task is a daunting one, thousands of users, dozens of dis-separate systems and gigabits of network traffic plus as part of the enterprise support team, I have other projects and duties to deliver on.
I managed to get time with the systems architect and run through a number of questions stolen from your books and other smart folks on the history and state of affairs of my new domain. My questions were answered or put in to context except for one.
"Why don't you have any monitoring tools in place on the edge and perimeter systems?"
The answer I received wasn't what I expected. He simple stated that no-one had the time or energy to take on this massive task. He was very pro about having monitoring, but bluntly realistic on the amount of time monitoring and response took up. The business has other priorities for the IT teams.
It's not that the company isn't security aware, they have good policies, skilled staff, harden systems and layer defences, but no monitoring.
I was stuck with the thought "If I don't understand what's happening on the network, how can I know what's right or wrong, what's normal or abnormal?"
This is exactly right. My friend's co-workers are probably practicing management by belief instead of management by fact. If you do not know for sure what is happening inside your company, how can you make informed decisions? Failure will occur when your luck runs out because you are not spending the time and resources necessary to acquire and maintain digital situational awareness.
I gathered all the documentation, policies and firewall rules together and tried to make sense of them. I re-read Extrusion Detection on my commute in to help me frame a plan of attack. I got all the basic system data of every system, check time and date stamps were all in sync and cleaned up the network maps.Then, as silly as this seems, the obvious answer came to me on where to start: at the internal firewall internal interface.
I started to review the drop logs of the firewall for that interface, reasoning that if there is a problem on the network, it should more apparent in those logs than anywhere else.
This is also exactly right. There is no need to start buying or building new tools if you are not leveraging your existing data sources or enabling data sources not yet activated.
Those reviews provided some startling results, fortunately for us, the problems were only mis-configured systems attempting outbound access. Mind you, they were generating considerable amounts of traffic and some over already congested WAN links.
Then I laid out the ingress and egress rule sets to find out the motives and owners of those rules in the firewalls and started to peel back those unused or unneeded rule sets, while monitoring the drop logs. All with management approval and sign off, of course!
This is another great move -- a stride towards simplification.
I'm now moving in to the next stage of working with other teams to clean up their problem systems. Some are going to be harder than others as I've found VAX's, AIX, Sun systems and others bouncing off the rules. I haven't seen these types of systems in a long time and as a simple Windows administrator, I have to find time and the right people to plug these holes.
I have a very long way to go, only focused on a very small section of the entire network and no certain path forward on how to apply a reasonable monitoring system - yet :-)
It's a challenge I'm looking forward to and one I'm sure I'll learn from.
The only comment haunting me is the not having any real time allotted to doing monitoring work. I'm going to talk with management and a surprisingly security literate CIO on if this can be addressed.
I know that the conversations have to be based on preventing/mitigating business risk, but believe it's going to be a hard sell with the very long list of other projects being pushed on the team as priority.
If you have any words of advice or useful pointers on convincing them the time is well spent, I'd be grateful.
So this is the major question. How do you convince management or other functional areas that monitoring is important? It sounds to me like my friend has already scored some wins by freeing bandwidth used by misconfigured systems, simplifying firewall rules, and examining individual problematic hosts.
It's important to remember that there is no return on security investment. Security is a cost center that exists to prevent or reduce loss. It is not financially correct to believe you are "earning" a "return" by spending time and money to avoid a loss.
If I need to spend $1000 to hire a guard to protect my $10000 taxi, I am not earning a return on my investment -- I am preventing the theft of my taxi. If I invest that $1000 in a ticketing and GPS system that makes me more productive ferrying passengers (perhaps increasing my dollars per hour worked), then I have enjoyed a ROI once my $1000 expense is covered.
This is not to say that money should not be spent on security. Rather, time and money should be balanced against the perceived risk. The same taxi driver will spend money on insurance, and may indeed need to spend $1000 on a guard if the protection delivered is seen to be necessary. However, the guard does not make the taxi driver more productive. Security is a "business enabler" here but it does not deliver "ROSI."
I'm not a big fan of FUD but something may be happening to executive perception of digital security. I am starting to hear questions like "How do I know that asset X is safe?" or "How do I know that asset Y is not being attacked?" (Answers: "safe" is relative, and Y is being attacked if it's accessible to anyone.)
I don't have any quick answers for my friend, which is why I'm posting this story. What are you doing to convince management that monitoring is necessary? I personally plan to do exactly what my friend did, namely starting with existing assets and showing quick wins to build momentum for more extensive visibility initiatives. You?
Comments
Normally I would receive a "do whatever it takes to plug the hole, and bring me an expense report" response because I was lucky enough to come to the table with enough information to show the whole train of attack, from external to internal.
I think if you go to management with enough information, in language they can understand, it's going to gain traction than more of a general 'the sky is falling, give me money' attitude. With the advent of open source tools, you can also bolster your position by saying how little it is going to cost the company.
1. Complexity - Computer networks, particularly the Internet is probably the largest and most complex thing man has made. The more complex something is, the less secure.
2. Connectivity - Every time you connect to the Internet you are usually just milliseconds away from every adversary around the world and it's just a matter of minutes before you start getting attacked. You also have to deal with the defenders dilemma where the bad guys only have to find and exploit one vulnerability, yet you have to find and fix them all.
3. People - The end users responsible for the security of the network aren't aware of the threats they face, making it impossible for them to defend against them.
Most of the people, products, and procedures in security focus on prevention, but prevention eventually fails so it's important to have something to fall back on. The saying "prevention is idea, detection is a must" is true. Incidents are a lot like fires in that the quicker you detect and respond to an incident, the less damage there will be, so early detection can actually be prevention.
Hope that helps.
- Adam
That's true and is definitely the wrong way to determine the value of risk mitigation.
The proper way is to compare replacement cost to prevention cost and probability of loss.
The best way to convince management, especially executive management, to do anything related to risk mitigation is with a risk assessment. The risk assessment process defines the value of the assets in question, the risk factor posed to the assets (that is, the probability of loss), and the cost of mitigation.
I've come to this knowledge via my experience consulting to community banks in the Upper Great Plains of the USA.
The executive management of these firms are notoriously stingy and conservative. Most are still old men who don't "get" technology. The younger ones who "get" technology don't necessarily "get" that new risk goes hand-in-hand with new services.
It was a chore convincing these people that there is value in securing the companies assets beyond simply being in compliance with Federal regulations.
The tool I have found that works best is the risk assessment.
For those not in the know, the essence of a risk assessment:
1. Inventory the assets within a given scope.
2. Assign a priority/value rating to the assets.
3. Identify the risks (threats and vulnerabilities) posed to those assets.
4. Assign a risk rating to the risks.
5. For each asset and risk pair, determine the combined, unmitigated risk (as if you were doing nothing to prevent the risk). This is a function of asset rating and risk rating.
6. Now inventory the actual mitigating factors you have in place for each asset and risk pair.
7. Finally, for each asset and risk pair assign a final risk rating that is a function of the unmitigated risk rating and the mitigating controls currently in place.
At the end of this process you have a tool that will help you plan what you need to do in your organization as well as a tool that can convince management that action needs to be taken.
I'll leave applying the risk assessment process to the example as an exercise for the reader since this comment is long enough already.
Two final things I want to address:
1. "I personally plan to do exactly what my friend did, namely starting with existing assets and showing quick wins to build momentum for more extensive visibility initiatives."
The "quick wins" strategy is a great short-term strategy when you've just started a new job. It can strengthen your reputation and credibility.
I just started a new job in February where there previously wasn't a strong centralized security focus. By quickly cleaning up outstanding issues from an independent pen test, encrypting all laptop hard drives, and reviewing and updating infosec policies I've established credibility to start getting more invasive and expensive in the next budget cycle.
It isn't a good long-term strategy because it usually turns into management by belief if you aren't very careful to establish the risk assessment process as the basis and justification for your efforts.
2. "I used convincing by fear."
This is a terrible strategy because eventually management will get sick and tired of it. They may just stop listening, which makes you very ineffective, or they may replace the squeaky wheel, which makes you unemployed.
Anyone who is currently using this strategy (and it is very, very commonly amongst infosec professionals) needs to start using the risk assessment process ASAP. It provides the tool necessary to give you visibility of what needs to be done and to convince management.
Feel free to contact me to learn more about the risk assessment process.
--
John Rodenbiker, CISA
jrodenbiker@rodenbiker.net
In my last full time gig there were two pain points:
1. The company was DOSing itself from internally infected machines causing resource outages which disrupted manufacturing which cost the company money.
2. Employee infractions (porn) in one of our foreign manufacturing plants causing issues with local law enforcement.
After putting in the basic infrastructure to resolve those two issues I was able to gather additional metrics and show management the benefits of having the tools in place.
Gather allies and get them to spread the same message.
And lastly - keep a journal. When it comes to renew the support contract you can show your management what your infrastructure has done for the company.
well to answer you question, i did what you proposed. i delivered quick wins, closed holes, burried some mistakes of other ppl,too... but i guess what got me the farthest was extended paperwork.
once i realized our management didnt understand the technical problems and their implications, i played it a way they understood: every time i wanted something i delivered a whole bunch of alternatives (sometimes a real phallanx), that way they realized they could rely on my judgment -cause i oviously thought it through.
cost me much time i d liked to ve spent on more productive things, but i get more and more feedback saying i did good in talking their language.
...oh and one side effect was also very nice: since i emphasized the elaboration of alternatives that much, my colleagues learnt to not bother me with ill-conceived stuff but are recollecting on efficient work
good luck
Cisco referencing QRadars statistical and behavioural based anomaly-detection. Cisco seriously lacks presence in this space.
John Rodenbiker’s comment is worth attention, Risk Management is important especially when you speaking with ‘C’ level. Most of Middle-size companies always looking for IT Security Management (which includes monitoring and reviewing) as un-necessary in the current state but in the future they may consider it.
To convince them, talking from their business prospective and preparing answers of their expected questions.
I agree that IT Security is business enabler but also that is lead to be ROI. Why we implement Security system? We do it for CIA trialing. Which means increase you business process availability which lead to indirect ROI.
Kind Regards,
Ayman M. Galal
Jason