Thoughts on High Speed Network Monitoring

I've been following an interesting thread on snort-users about collecting alert data on high speed networks. Users are debating how much traffic Snort can handle. One way to at least start answering this questions is to enable the performance monitor. Vjay Larosa's post was helpful, as it pointed me towards perfmon-graph. This Perl script works with Snort performance monitor output and RRDtool output to produce graphs of Snort performance statistics.

Using perfmon-graph requires two steps. First, enable Snort to output the statistics you need to a text file. Add the following line to your snort.conf file:

preprocessor perfmonitor: time 60 file /nsm/snort/perfmon.txt pktcnt 500

This tells Snort to output statistics every 60 seconds to a file called /nsm/snort/perfmon.txt The perfmon-graph README warns us not to set the number following 'pktcnt' too high. For example, if 500 packets are not collected in 60 seconds, then we will not get a statistics output.

After restarting Snort with this new preprocessor, your perfmon.txt file will begin collecting entries like these:

bourque# tail -f /nsm/snort/perfmon.txt 
1085192584,0.000,0.1,0.0,0.0,395,11.67,0.4,0.4,0.8,0.4,30,
30,0.0,0,1,0.0,0.0,0.0,0.0,0,0,0.0,0.0,99.9
1085192678,0.000,0.0,0.0,0.0,213,8.78,0.1,0.1,0.3,0.6,3,31,
0.0,0,3,0.0,0.0,0.0,0.0,0,0,0.0,0.0,100.0
1085192746,0.000,0.0,0.0,0.0,316,51.71,0.1,0.1,0.2,0.2,2,31,
0.0,0,1,0.0,0.0,0.0,0.0,0,0,0.0,0.0,100.0

Next decide where to run perfmon-graph and RRDtool. I chose to deploy both on a system other than the sensor running Snort, called janney. On FreeBSD use the /usr/ports/net/rrdtool port to install RRDtool. When running perfmon-graph I got an error about a missing tie/File.pm file, which I learned ships with /usr/ports/devel/p5-Tie-File.

To set up perfmon-graph itself, download and extract it to a convenient directory. I chose to follow the perfmon-graph author's suggestion to use SSH to periodically copy the perfmon.txt file to the system running perfmon-graph. I made the following entry in my crontab to do this:

0,5,10,15,20,25,30,35,40,45,50,55 * * * * ssh worker@bourque 
 "cat /nsm/snort/perfmon.txt" | /usr/local/bin/perl  
 /usr/local/src/perfmon-graph/perfmon-graph.pl  
 /usr/local/www/data/perfmon-graph/bourque/ -

This setup relies on user worker being able to SSH from janney to my sensor, bourque, using public key authentication. I'm using perfmon-graph.pl on FreeBSD 4.9 STABLE, which had an old version of Perl installed (5.005_03). I installed a newer version using the latest package (5.6.1), and executed 'use.perl port' to make it the default. (I can revert to the base with use.perl system.)

Once this is working, a set of files will appear in /usr/local/www/data/perfmon-graph/bourque. You may recognize this as a likely place where Apache could serve up Web pages, and that is the case here. A visit presents graphs like that above, which shows Snort's perception of bandwidth monitoring in Mbps.

The accuracy and value of these statistics are debatable, but they are at least a start.

In addition to setting up performance monitoring, I learned of a great source of information on high speed monitoring issues through this post. It led me to SCAMPI, "A Scaleable Monitoring Platform for the Internet." Their publications page lists many interesting papers with advice and research regarding monitoring high bandwidth networks. The SCAMPI project has the goal of monitoring traffic at 10 to 100 Gbps.

Finally, Brian Caswell posted details on a massive rule update designed to improve Snort performance. He directs interested readers to the updated section of the Snort manual.