Monday, May 09, 2005

Tap on a PCI Card

Those of you who've read my first book know I like to use taps built by Net Optics to access wired traffic. The device pictured at left is a port aggregator tap. It combines the TX side of whatever's plugged into port A with the TX side of port B into a single output on port C, using buffering if the aggregrate throughput exceeds 100 Mbps.

Today I got a chance to test the device pictured at left. It's a Net Optics PCI Port Aggregator tap. You plug this device into a 32 bit PCI slot on your monitoring station, and you effectively have the normal port aggregator tap I showed earlier sitting within your sensor. Let me show you what I mean in pictures.

This is the inside of my preferred monitoring platform, a Dell Poweredge 750. I've removed the dual Gigabit Ethernet NIC I usually order with these systems. That NIC is a PCI-X device recognized as em under FreeBSD.

In this next picture you see the Net Optics PCI tap at the top, and the dual Gigabit Ethernet NIC below it.

Here is the Net Optics PCI tap installed. It takes up almost all of the space available, but it fits nicely. You can barely see the three tap ports in this picture. The PCI tap does not appear active to the operating system. All the PCI tap needs from the PCI interface is power. There is a plug for a back-up power supply on the tap, but I did not connect it here. If power to the tap fails (i.e., the PC is off), the tap still passes traffic.

In this final picture, you see my test setup. This isn't how I would deploy it in real life. In a real deployment, say on a site perimeter, one tap interface would go to the router and the other tap interface would go to the firewall. The third tap interface connects to the sensor monitoring traffic passing between the router and firewall.

In this test deployment, I am essentially watching traffic to and from the server em0 interface by sending copies of that traffic through the tap and to the server em1 interface.

In the picture below, the tap ports are occupied by blue, yellow, and black cables. The yellow cable is connected to a switch with Internet access. The black cable next to the yellow line is connected to the live (IP addressed) interface on the server, em0. The blue cable connects the monitoring interface of the tap to the monitoring interface of the server, em1.

Do you see where I am going with this? What I can do now (for test purposes) is send and receive traffic on em0 and watch that traffic on em1. First I set up em1 to listen to what the tap sends it:

sensor# ifconfig em1 -arp up
sensor# tcpdump -n -i em1
tcpdump: WARNING: em1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes

Now I send traffic out em0:

$ ping -c 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=55 time=15.240 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.240/15.240/15.240/0.000 ms

Here is what em1 sees courtesy of the tap:

09:43:33.263664 IP >
27655+ A? (37)
09:43:33.276900 IP >
27655 1/2/2 A (131)
09:43:33.277149 IP > icmp 64: echo request seq 0
09:43:33.296513 IP > icmp 64: echo reply seq 0
4 packets captured
4 packets received by filter
0 packets dropped by kernel

I think that is cool. What's the practical application? You could ship a monitoring appliance to a customer, with built-in tap. The customer connects her lines to the tap and the sensor listens via normal cable connection from tap monitor port to sensor monitor port. No external tap is needed. If the server fails completely, the tap will keep passing packets. I like it. If you would like to know more, email me and I will hook you up with the people I know at Net Optics.

If you'll be near Sunnyvale, CA next week, stop by for the Net Optics Think Tank on 18 May. I'll be speaking there around lunch time.


Axel Eble said...

They seem to be quite nice things, those taps. However, they also come at a hefty list price of almost 1400 € (both the passive and the active response version).

Martin McKeay said...

The other big advantage I see to these is the recaptured rack space. In a crowded wiring closet finding room for the sensor can be hard enough, without the added 1 or 2U for the taps. And given that each of the external taps have 2 power bricks, the PCI solution can give you back quite a few power outlets in the right situation.

One questions: Do the taps pass traffic even when unconfigured and/or when the power is out to the computer? The external taps continue to pass traffic even when powered down, which to me is an important feature.

Martin McKeay said...

Dohhh! Reread the article and missed the line about passing traffic that I missed on the first run through it.

I hate it when that happens

Martin McKeay said...

Just having one of those weeks where nothing I type comes out right.

I meant to say:

Reread the article and found the line about passing traffic that I missed on the first run through it.

Joe said...

Interesting. What would be even more useful is if the card had only 2 physical ints, but 1 logical interface that the OS recognized...aka Harware interface bonding.

Charlton Glad said...
This comment has been removed by a blog administrator.
pci said...

This information is very helpful. It really helps me understand more about PCI. Keep posting. Will certainly try doing that myself. Your post/article really helped. Thanks a lot.