NSM-Friendly VMWare Lab Setup

I'm working on labs for my all-new TCP/IP Weapons School 2.0 class (early registration ends Wednesday). Almost the whole class is labs; I'll have between 10 and 12 scenarios for students to investigate.

As you might imagine, network traffic will play a key role. I wanted to set up a VM running Ubuntu that could watch traffic involving other VMs. (Why not FreeBSD? Ubuntu is easier for students to use, and NSMnow makes it easy to get Sguil running. FreeBSD has also never seemed to run well in VMs due to some weird timing issues that have never been resolved.)

The problem, as I noted in Using VMware for Network Security Monitoring last year, is that modern versions of VMware Server (I run 1.0.8 now) act as switches and not hubs. That means each VM is connected to a virtual switch, effectively sheltered from other traffic. This is good for performance but bad for my monitoring needs.

Monitoring on the VMware server itself is not an option. Although it can see the traffic, I want to distribute a VM to the students that was running and capturing the traffic using Sguil and other tools as necessary.

Incidentally, here are two options for sniffing on the VMware server itself, for reference. VMware mentions its vmnet-sniffer, which is a console-output application with basically no features used only for troubleshooting:


richard@neely:~$ sudo vmnet-sniffer -e /dev/vmnet1

len   98 src 00:0c:29:7f:d6:a1 dst 00:0c:29:0a:0f:c1 IP src 10.1.1.3       
 dst 10.1.1.4        ICMP ping request - len=64 type=8
  00:0c:29:7f:d6:a1 08 00 88 e6 c0 17 00 01 ae 85 59 49 b5 2e 07 00 08
  09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 
  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36
  37 

len   98 src 00:0c:29:0a:0f:c1 dst 00:0c:29:7f:d6:a1 IP src 10.1.1.4       
 dst 10.1.1.3        ICMP ping reply

You could just as easily run Tcpdump or any other sniffer of your choice:


richard@neely:~$ sudo tcpdump -n -i vmnet1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmnet1, link-type EN10MB (Ethernet), capture size 96 bytes
20:41:51.272555 IP 10.1.1.3 > 10.1.1.4: ICMP echo request, id 49175, seq 1, length 64
20:41:51.273469 IP 10.1.1.4 > 10.1.1.3: ICMP echo reply, id 49175, seq 1, length 64

One note: vmnet-sniff can watch /dev/vmnet0 even though vmnet0 is not listed by ifconfig. vmnet0 is the bridged interface so you just watch it directly (e.g., eth0, etc.) with Tcpdump.

What to do? I decided that I could deploy the NSM sensor VM as a gateway, and put any hosts which I want to monitor as legs on that gateway. Consider this three-host scenario:

NSM sensor VM / gateway with 1) eth0 as 172.16.99.3, default gateway is 172.16.99.2, the VMware NAT /dev/vmnet8 gateway; 2) eth1 as 192.168.230.3, on a random subnet; and 3) eth2 as 10.1.1.3, on another random subnet

Windows victim with interface 192.168.230.4, default gateway 192.168.230.3

Linux attacker with interface 10.1.1.4, default gateway 10.1.1.3

I configured the NSM sensor to be a gateway, and told it to NAT connections outbound to the VMware server NAT interface:


root@tws-u804:~# echo "1" > /proc/sys/net/ipv4/ip_forward
root@tws-u804:~# iptables -t nat -A POSTROUTING -s 192.168.230.0/24 -o eth0 -j MASQUERADE
root@tws-u804:~# iptables -t nat -A POSTROUTING -s 10.1.1.0/24 -o eth0 -j MASQUERADE

Why this "second" level of NAT via MASQUERADE? It turns out that if you send traffic from, say, 10.1.1.4 through a gateway that doesn't NAT, when that gateway sends the traffic with source IP 10.1.1.4 to the NAT interface on the VMware server, the VMWare server doesn't know how to handle the replies. I saw the traffic exit properly (i.e., it was NATed out), but when the reply arrived the VMware server didn't know how to return it to 10.1.1.4. With this "second" NAT on the NSM sensor / gateway, the VMware server thinks the gateway is originating all traffic, so the hosts can reach the Internet.

With this setup I can now monitor traffic from 10.1.1.4 to 192.168.230.4, because the traffic is routed through the NSM sensor / gateway.

This seems kludgy, and I wish there were a way to just configure VMware Server to act like a hub and have all hosts see all traffic. If anyone knows how to do that, please let me know.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Comments

Crazy Computer Dad said…

Richard,
You might consider having the VMs use a single NIC that connects to a physical hub, and have your NSM VM use a USB NIC and connect it's sensor port to the hub. USB NICs are fairly cheap and you may find that using that method can help solve other problems too, such as separating host traffic from the VM traffic.

9:20 PM

Richard Bejtlich said…

Hey CCD,

That's an idea, but I don't want to force my students to buy extra hardware.

9:21 PM

dre said…

why not just go to vSphere (when it comes out) or use VI3 or ESX 3i?

11:33 PM

Are either free to distribute in a class?

7:57 AM

Looking at those other VMware offerings it seems like they are not intended for individuals to be running on laptops (Windows, Linux, Mac, etc.) brought to a class. I don't see students installing a hypervisor onto the laptops they bring to class.

8:05 AM

Anonymous said…

Why not just use a better VM, VBOX is is killin vmware, vmware is proprietary garbage. Host based networking, all networking I find with backtrack 3 to be so much more efficient on vbox. Why are you using vmware in security class anyway, cause it is more widely used? So is windows, but you don't use that for penetration testing and network monitoring.

9:31 AM

Anonymous, how does VirtualBox handle network traffic? Is it a switch or hub?

9:42 AM

You have a choice with either host based networking and just use the actual same physical nic, and bind virtual nic with physical with brctl command. You can set up NAT to where it is integrated with everything else on the lan just not the physical box you are on. Like I said though when binding virtual nic with physical with brctl you can see your own computer and be on same subnet as the rest of your lan, or set up a totally new subnet and be binded with your physical nic, you are just plumbing the nic virtually with brctl. I don't know though, is that what you were talking bout? If not there is still other option of internal network, and port forwarding method as explained in VirtualBox manual.

3:55 PM

The internal network type is not available from the GUI, and is reset by the GUI if any of the VM settings are changed. So this has to be performed from the command line.

4:01 PM

test said…

Richard,

I have replaced VMWare with VirtualBox OSE and can confirm that the "Internal Network" operates as a hub.

1:40 AM

You can always download an older version of VMWare, one that acts like a hub.

9:34 AM

Inuk-x, thanks for the info!

Anonymous, I can't rely on all the students running the same old version of a VMware product.

9:37 AM

THis is other anonymous, not one that said use older version of vmware. SO someone has confirmed that virtualbox ose will act as a hub, so there you go.

12:18 PM

I've used a similar setup before with VMWare Server 1.0.4 & 2.0, and have gotten it to work without the 2nd level of NAT. It requires adding a route statement on your physical router. In your example, the DG of 172.16.99.2 would need a statement pointing 10.1.1.0/24 to 172.16.99.3. I know that doesn't solve your original problem, but should work and might save a small amount of setup.

1:01 PM