Monday, April 17, 2006

Profiling Sensors with Bpfstat

In the TaoSecurity lab I have three physical boxes that perform monitoring duties. I wanted to see how each of them performed full content data collection.

Note: I do not consider what I am about to blog as any sort of thorough or comprehensive test. In fact, I expect some of you to flail about in anger that I didn't take into account your favorite testing methodologies!

I would be happy to hear constructive feedback. I am aware that anything resembling a test brings out some of the worst flame wars known to man. With those caveats aside, let's move on!

These are rough specifications for each system.

  • bourque: Celeron 633 MHz midtower with 320 MB RAM, 9541 MB Quantum Fireball HDD, 4310 MB Quantum Fireball HDD, Adaptec ANA-62044 PCI quad NIC

  • hacom: Via Nehemia 1 GHz small form factor PC with 512 MB RAM, 238 MB HDD, and three Intel Pro/1000 Gigabit adapters

  • shuttle: Intel PIV 3.2 GHz small form factor PC with 2 GB RAM, 2x74 GB HDDs, integrated Broadcom BCM5751 Gigabit Ethernet and Intel Pro/1000 MT Dual Gigabit adapter

Each sensor runs FreeBSD 6.0 with binary security updates applied.

How does one verify that a sensor is logging full content data appropriately? I decided to have each sensor listen to the two outputs of a traditional Fast Ethernet tap, provided by Net Optics. The tap watched a link between a FreeBSD FTP client and a Debian Linux FTP server. I had the FTP client download a 87 MB .zip file (my first Sguil image, in fact) from the server while the sensor watched.

I bonded the two interfaces monitoring the tapped link so that a single ngeth0 interface would see both sides of the FTP data transfer. I ran Tcpdump on the sensor like so:

tcpdump -n -i ngeth0 -s 1515 -w testX.lpc

Before starting the FTP transfer, I started Bpfstat on the sensor to watch for packet drops.

Here's how the FTP transfer looked.

ftp> get
local: remote:
227 Entering Passive Mode (172,16,1,1,128,85)
150 Opening BINARY mode data connection for '' (91706415 bytes).
100% |*************************************| 89557 KB 9.72 MB/s 00:00 ETA
226 Transfer complete.
91706415 bytes received in 00:08 (9.72 MB/s)

As you can see I'm transferring at around 78 Mbps. This is a limitation of the hardware. I was able to run Iperf at over 90 Mbps, but no data was being saved.

After transferring the file via FTP from server to client, I used Tcpflow on the sensor to reassemble the FTP data stream carrying the 87 MB .zip file.

The original 87 MB file was 91706415 bytes. Every time I reassembled the FTP data session, I got files 91706415 in size. I ran a MD5 hash of each reconstructed file, however, and found none of them matched the original 87 MB .zip. This meant I was dropping packets somewhere.

To identify the bottleneck, I decided to use Bpfstat.

Here are the results when run on sensor bourque:

bourque:/root# bpfstat -i 5 -I ngeth0
pid netif flags recv drop match sblen hblen command
993 ngeth0 p--s- 11 0 11 0 0 tcpdump
993 ngeth0 p--s- 39 0 39 0 0 tcpdump
993 ngeth0 p--s- 21540 15475 21540 32740 31480 tcpdump
993 ngeth0 p--s- 75819 53392 75819 32740 32740 tcpdump
993 ngeth0 p--s- 95851 67142 95851 0 0 tcpdump

Wow, that's a lot of dropped packets. Here are the results for sensor hacom:

hacom:/root# bpfstat -i 5 -I ngeth0
pid netif flags recv drop match sblen hblen command
635 ngeth0 p--s- 41322 217 41322 32740 31480 tcpdump
635 ngeth0 p--s- 94843 420 94843 14124 0 tcpdump

That's a bit better. Let's look at shuttle, the most robust sensor available.

shuttle:/root# bpfstat -i 5 -I ngeth0
pid netif flags recv drop match sblen hblen command
689 ngeth0 p--s- 0 0 0 0 0 tcpdump
689 ngeth0 p--s- 7 0 7 0 0 tcpdump
689 ngeth0 p--s- 39 0 39 0 0 tcpdump
689 ngeth0 p--s- 23810 0 23810 17356 0 tcpdump
689 ngeth0 p--s- 77414 19 77414 15656 0 tcpdump
689 ngeth0 p--s- 95851 19 95851 0 0 tcpdump

That's excellent, but Bpfstat still reports dropping packets. That means I will not be able to reconstruct the FTP data session using this equipment.

None of this hardware is similar, but you can see how a progression from a slower CPU, less RAM, and less respected NICs to a faster CPU, more RAM, and Intel NICs results in better performance. All of these systems used 32 bit 33 MHz PCI buses for add-on cards, so I would expect PCI-X or PCI Express would improve performance.

If you're thinking that the ngeth0 bonded system might have degraded performance, that wasn't the case. Here is Bpfstat output for one of the transfers, when watching only the data from FTP server to client.

shuttle:/root# bpfstat -i 5 -I em1
pid netif flags recv drop match sblen hblen command
803 em1 p--s- 0 0 0 0 0 tcpdump
803 em1 p--s- 14106 0 14106 16852 0 tcpdump
803 em1 p--s- 49470 35 49470 27576 0 tcpdump
803 em1 p--s- 63341 35 63341 0 0 tcpdump

It appears to have dropped more traffic than the ngeth0 system. I had similar results on the other boxes.


Anonymous said...

Richard: You might want to reference 'tuning (7)' for sysctls that can be changed to improve network/system performance. Additionally, systat may help to identify hardware bottlnecks.

Richard Bejtlich said...

Anonymous, that is a really good comment. I did NO tuning on these systems. I bet I could make a few tweaks and drop fewer packets. I'll give that a try when I have some time.

Joe said...

I always see people getting flamed for asking "how can I tune my network stack" on mailing lists. The same response is always that you shouldn't touch it.

If there are some tunable options that would help an nsm/ids, why haven't we ever seen these published?

I'm not against this though.

Anonymous said...

Richard: From my previous post referencing 'tuning (7)', you also might want to look at device 'polling (4)' for your NIC (if you haven't already). Here are some links to reference regarding the combination of sysctl's and polling. The netperf port might also help you in benchmarking tests.

Anonymous said...

Sorry...the freebsd link seems to have been cut off

Anonymous said...

Are you aware of any tools similar to bpfstat that will run on linux?

Richard Bejtlich said...

No, that is a limitation of Linux. Someone please change my mind!

Anonymous said...

IPTraf is similar to bpfstat. However, it doesn't show the process associated with the interface. IPTraf will allow selections of hosts, interfaces, and port numbers.