Wednesday, January 24, 2007

Monitor Your Routers

Today I read this new Cisco advisory containing these words:

Cisco routers and switches running Cisco IOS® or Cisco IOS XR software may be vulnerable to a remotely exploitable crafted IP option Denial of Service (DoS) attack. Exploitation of the vulnerability may potentially allow for arbitrary code execution. The vulnerability may be exploited after processing an Internet Control Message Protocol (ICMP) packet, Protocol Independent Multicast version 2 (PIMv2) packet, Pragmatic General Multicast (PGM) packet, or URL Rendezvous Directory (URD) packet containing a specific crafted IP option in the packet's IP header...

A crafted packet addressed directly to a vulnerable device running Cisco IOS software may result in the device reloading or may allow execution of arbitrary code.


This is the sort of "magic packet" that's an attacker's silver bullet. Send an ICMP echo with the right IP option to a router interface and whammo -- you could 0wn the router. Who would notice? Most people consider their border router an appliance that connects to an ISP, and only begin watching traffic behind the router. I've been worried about this scenario since I posted Ethernet to your ISP in 2005 and read Hacking Exposed: Cisco Networks in 2006.

A router, like a printer, is just a computer in a different box. We should design architectures such that all nodes within our control can be independently and passively observed by a trusted platform. We should not rely on a potential target to reliably report its security status without some form of external validation, such as traffic collected and analyzed by network security monitoring processes. This Cisco advisory reminds me of a case I encountered at Foundstone. Shortly after joining the company in early 2002 the Apache chunked encoding vulnerability became known in certain circles. As the new forensics guy on the incident response team, I was asked how I could tell if Foundstone's Apache Web server was hacked. I replied that I would first check session data collected by a trusted sensor looking for unusual patterns to and from the Web server, as far back as the data and my time would allow.

Foundstone, being an assessment company, didn't share my opinions on NSM. No such data was available, so the host integrity assessment descended into the untrustworthy and often fruitless attempt to find artifacts of compromise on the Web server itself. Of course the Web server could not be taken down for forensic imaging, so we performed a live response and looked for anything unusual.

This case was one of the easier ones in the sense that all that was at stake was a single server. We didn't have a farm of Web servers, any of which could have been hacked.

Whenever I am trying to scope an incident I always turn to network data first and host data second. The network data helps decide which hosts merit additional individual scrutiny at the host-centric level. It makes no sense and is not time-efficient to deeply or even quickly analyze a decent-sized group of potential victims.

Because most security shops do not bother to collect NSM data, their first response activities usually involve a system administrator perusing the victim file system, manually checking potentially altered or erased logs on the target, and inspecting potentially obfuscated processes in memory controlled by a root kit. While I believe it can be important to collect live response data and even image and inspect a hard drive, I never start an incident response with those activities.

Remember: network first. I don't care if the intruder uses an encrypted channel. As long as he has utilized the network in a mildly suspicious manner, I will have some validation that the system is not behaving as expected. This presumes system owners know what is normal, preferably by comparing a history of network data against more recent, potentially suspect, activity.


Copyright 2007 Richard Bejtlich

6 comments:

Rob Lewis said...

" We should design architectures such that all nodes within our control can be independently and passively observed by a trusted platform."

Why not make nodes like routers trusted themselves? What makes the box observing the nodes trusted?

I am tempted to say that if you did so, you would then check host data first, but you would not have incidents to begin with, if the node in question was trusted to begin with, because you then have prevention rather than remediation.

Richard Bejtlich said...

Anyone can make a router "trusted". The question is whether it is worthy of trust, i.e., "trustworthy." No one can do that. I increase my level of trust in a device as the opportunities for other parties to interact with it decrease. I thought I'd discussed this before but I cannot find a reference now. The bottom line is that 100% prevention is impossible, and if you don't believe that then you aren't in touch with reality.

Richard Bejtlich said...

Rob -- in my last comment I didn't mean to imply "you" as "Rob." I was referring to anyone who believes in 100% prevention.

Jon hart said...

When the 3 Cisco advisories came out the other day, I (like everyone else), jumped all over them. Fortunately the version(s) that we happen to be running are not vulnerable.

The Cisco advisory is a bit unclear -- yes , the packet must be sent to a physical/virtual IPv4 address on the device, but what if ACLs exist? I'd like to think that if an ACL dropped the packet, that would prevent this vulnerability from being exploitable.

With that thinking in mind, the threat is considerably narrowed. Sure, there are some situations where you cannot simply block all inbound traffic to your routers addresses as there are legitimate reasons to, for example, allow in ICMP pings and PIM. But, I'm fairly certain that most of the time you could restrict who needs access to those services, and in the end the only way to get traffic past an ACL and to the vulnerable device is to be "trusted" in the first place. There is still a threat, but not an "OMG drop everything and upgrade" sort of threat.

Vivek Rajan said...

jon,

The ACLs suggested as a workaround are too cumbersome to implement. If you have a IP address based ACL, it wont help because the killer packet can be spoofed.

Richard,

When you say "trusted platform", do you mean something that is completely external to the network elements ? If the untrusted network and the trusted platform are able to communicate (eg via routes that exist between the two), would it compromise the trust worthiness of the platform ?
I would love to hear your views on this.

jimmythegeek said...

I've been thinking about ways to enumerate the legitimate communication partners for my router, and blocking all else. (Naturally, not blocking transit traffic.) Spoofing would be a problem as Vivek mentions.

There's a beautiful anti-spoofing technique I am looking into. It's only useful for routers, for the most part (maybe in a SAN as well). Drop everything with TTL < 255.

You only do BGP with neighbors, right? You might collect SNMP from distant boxes, but that's an easy exception to put in, and a harder one for the attacker to discover or guess. A simple IP based ACL will fail when an attacker runs a traceroute and discovers who your neighbors are. But they can't spoof them AND ensure their packets show up with a TTL of 255. Unless the neighboring router is owned, of course. But the TTL limit sure narrows the list of suspects even in that case.

I heard about it at ISOI-2 last week.