Tuesday, July 24, 2007

Enterprise Visibility Architect

Last month in Security Application Instrumentation I wrote:

Right now you're [developers] being taught (hopefully) "secure coding." I would like to see the next innovation be security application instrumentation, where you devise your application to report not only performance and fault logging, but also security and compliance logging.

This is a forward-looking plea. In the meantime, we are stuck with numerous platforms, operating systems, applications, and data (POAD) for which we have zero visibility.

I suggest that enterprises consider hiring or assigning a new role -- Enterprise Visibility Architect. The role of the EVA is to identify visibility deficiencies in existing and future POAD and design solutions to instrument these resources.

What does this mean in real life? Here is an example. Let's say a company operates a FTP proxy for business use. Consider various stakeholders involved with that server and the sorts of visibility they might want:

  • Data center managers want physical accountability for the server. They also want to know how much power is consumed and how much heat is output.

  • Network administrators want to know how much bandwidth the server uses for OS and application updates. They also want to know how much bandwidth is used by data transfers and backup processes.

  • System administraors want to know if the asset is performing properly.

  • Users and asset owners want to know how much data is transferred (in the vent they are billed for this service).

  • Human resources administrators and legal want to know what files are transferred, potentially to identify fraud, waste, and abuse.

  • Auditors want to validate and asses secure configurations.

  • Security analysts want to resist, detect, and respond to incidents.

  • Forensic investigators want to know the state of the asset and the files transferred to investigate incidents.


These are all requirements that should be included in the design of the server before it is deployed. However, no one does all of this, and only a few organizations accomplish a few of these items. The role of the EVA is to ensure all of these requirements are built into the design.

When these requirements are not built into the design (as is the case with 95+% of all infrastructure, I would wager) it's the job of the EVA to work with concerned parties to introduce visibility through instrumentation. For example, how could the investigators' concern be met?

  • If the proxy supports logging, enable it. (This is usually not done because of "performance concerns." If the resource were appropriately sized prior to deployment to account for visibility, this would not be a problem.)

  • Add a passive device to monitor traffic to and from the proxy server. Application-aware monitoring tools like Bro's FTP Analyzer can record FTP control channel activities. (The resistance to this technique involves not wanting to configure a SPAN port, or lack of a SPAN port. I prefer taps, but inserting the tap requires scheduling downtime -- another sticking point.)

  • If the investigator only wants IP addresses for the endpoints, then NetFlow could be enabled on a router through which traffic to the FTP server passes. Note that NetFlow cannot be configured to only provide flow data for a specific port (like 21 TCP) so filtering would have to happen on the NetFlow collector. Using NetFlow effectively requires building a NetFlow collector. (Other concerns include loading the router, which could have been accounted for when the design for this business system was created.)

  • If the investigator only wants IP addresses for the endpoints, then a logging ACL could be enabled on a router through which traffic to the FTP server passes. Hits on this ACL could be exported via Syslog. Using Syslog requires building a Syslog server. (This option will also add load to the router.)

  • Depending on the architecture, intervening firewalls could also be configured to log connection details in the same manner that NetFlow or router ACLs do.


I believe that logging integrated into the application (i.e., the FTP process) is the best option when one is designing a new resource. When visibility is introduced after the asset is deployed, instrumenting it becomes more difficult.

If you hadn't guessed, I am becoming the de facto EVA in my job as director of incident response because I need data to detect and respond to incidents. However, all of the stakeholders are natural allies because they want to know more about various assets.

Thanks to the I want to believe generator for the image above.

7 comments:

dre said...

They're called "NMS Tools Developers" and they usually require experience with Cisco MIB's, the parts of Cisco configurations that have nothing to do with routing/switching/voice, Nagios, SEC, RRDTool, XHTML, Javascript, at least one scripting language (usually Perl or Python), MySQL, Syslog, and SNMP.

They used to be called "Openview" or "Tivoli" or "Netcool" consultants, also known by their better sounding name, `system integrators'.

Speaking of which, Opsware just got bought by HP.

Richard Bejtlich said...

dre,

That's traditional NMS, and I'm aware of that -- but it's only part of my concern. Traditional NMS is basically performance and fault management, which is definitely important but not the same as what I'm describing. It's also not system integration because just about all system integrators completely ignore what I'm discussing, aside from being able to "ping the box," i.e., "I can ping it -- it's good!" Ouch.

Christofer Hoff said...

I thought this "oversight" fell within the jurisdiction and domain of the Enterprise Architect. Building "visibility" in is the same thing as building "security" in...

Why do we need yet another TOAD fragmenting the view of the POAD? ;)

The acronym soup associated with creating new titles to backfill for people who are just not doing the job seems to me to be treating the symptom and not the problem.

My $0.02...

/Hoff

Richard Bejtlich said...

Hi Hoff,

I thought of that too. You could say there should be no security people because security should be integrated. It's certainly not the case for legacy services and it's still not the case for future services. Furthermore, most people think security == prevention, so they think nothing bad will happen to their app. That is why you will never hear me say "prevention" again -- I say "resistance" because prevention eventually fails.

Incidentally I would be happy if all of this were integrated but it's clearly not. :(

Christofer Hoff said...

Fair enough.

It's a sad indictment as to the state of affairs, but just look at what this fragmentation is doing in the security & "risk management" space. We have:

CSO - Chief Security Officer
CISO - Chief InfoSec Officer
CPO - Chief Privacy Officer
CRO - Chief Risk Officer
CRO - Chief Reputation Officer
CSS - Chief Security Strategist
CSA - Chief Security Architect
CSE - Chief Security Evangelist

...dem's a lot of Chiefs! How many of these functions overlap to the point of diminishing value where the number of eyes on the problem yields negligible return since accountability becomes so distracted.

I suppose we're really stating the same problem but from different perspectives; I think we should hold existing Enterprise Architects responsible and you think we should create a new position.

'round and 'round we go?

Either way, I agree, the problem needs to be solved.

/Hoff

P.S. See you in a couple of days @ Weapons School in Vegas...

geek00L said...

Blame no clear definition of each role ;)

http://www.architectsban.webs.com said...
This comment has been removed by a blog administrator.