Using Device Polling and More to Improve Packet Capture

I just read a fascinating paper by Luca Deri, author of Ntop, about "Improving Passive Packet Capture: Beyond Device Polling" (.pdf). Luca claims that out of the box, Windows 2000 performs better as a traffic collection platform under high loads (~80 Kpps), capturing 68% of traffic compared to 34% for FreeBSD and 0.2% for Linux kernel 2.4.x. Linux's performance improves to 1% if the mmap libpcap version is used, and up to 4% if a Netfilter-based loadable kernel module is used. These percentages sound off to me. Luca explains the results:

"An explanation for the poor performance figures is something called interrupt livelock. Device drivers instrument network cards to generate an interrupt whenever the card needs attention (e.g. for informing the operating system that there is an incoming packet to handle). In case of high traffic rate, the operating system spends most of its time handling interrupts leaving little time for other tasks. A solution to this problem is something called device polling."

FreeBSD has had device polling available in the kernel since FreeBSD 4.5 REL, but you need to recompile the kernel and use a polling-aware NIC. From /usr/src/sys/i386/conf/LINT on my 4.9 REL box:

# DEVICE_POLLING adds support for mixed interrupt-polling handling
# of network device drivers, which has significant benefits in terms
# of robustness to overloads and responsivity, as well as permitting
# accurate scheduling of the CPU time between kernel network processing
# and other activities. The drawback is a moderate (up to 1/HZ seconds)
# potential increase in response times.
# It is strongly recommended to use HZ=1000 or 2000 with DEVICE_POLLING
# to achieve smoother behaviour.
# Additionally, you can enable/disable polling at runtime with the
# sysctl variable kern.polling.enable (defaults off), and select
# the CPU fraction reserved to userland with the sysctl variable
# kern.polling.user_frac (default 50, range 0..100).
#
# Only the "dc" "fxp" and "sis" devices support this mode of operation at
# the time of this writing.

options         DEVICE_POLLING

However, the man page says other cards are supported, including the important em Gigabit Ethernet driver.

SUPPORTED DEVICES
     Polling requires explicit modifications to the device drivers.  As of
     this writing, the dc(4), em(4), fxp(4), rl(4), and sis(4) devices are
     supported, with other in the works.  The modifications are rather
     straightforward, consisting in the extraction of the inner part of the
     interrupt service routine and writing a callback function, *_poll(),
     which is invoked to probe the device for events and process them.  See
     the conditionally compiled sections of the devices mentioned above for
     more details.

According to Luca, the new Linux 2.6 kernel supports polling as well. He re-ran his tests against Linux kernel 2.6 and FreeBSD 4.8 (polling enabled). Linux with standard libpcap now captured 5.6% of traffic while FreeBSD captured 99.9%. (That's my boy!) When Linux implemented capture using a kernel module, Linux's performance matched FreeBSD at 99.5%.

Not happy with this outcome, Luca modified the Linux Gigabit driver and wrote patches to create a ring-buffer based version of libpcap. He found this works much better and plans to port it to FreeBSD as well.

Update: I asked the community to provide opinions on this paper. You can read the thread at unix.derkeiler.com. The netbsd-tech-kern list hotly debated this paper in this thread.

Update: On 20 Feb 04 Luca released a significantly modified version of the paper at the same URL. The big issue for FreeBSD seems to be poor performance with higher packet sizes.

Popular posts from this blog

Zeek in Action Videos

Image
This is a quick note to point blog readers to my Zeek in Action YouTube video series for the Zeek network security monitoring project .  Each video addresses a topic that I think might be of interest to people trying to understand their network using Zeek and adjacent tools and approaches, like Suricata, Wireshark, and so on.  I am especially pleased with Video 6 on monitoring wireless networks . It took me several weeks to research material for this video. I had to buy new hardware and experiment with a Linux distro that I had not used before -- Parrot .  Please like and subscribe, and let me know if there is a topic you think might make a good video.
Read more

MITRE ATT&CK Tactics Are Not Tactics

Image
Just what are "tactics"? Introduction MITRE ATT&CK  is a great resource, but something about it has bothered me since I first heard about it several years ago. It's a minor point, but I wanted to document it in case it confuses anyone else. The MITRE ATT&CK Design and Philosophy document from March 2020 says the following: At a high-level, ATT&CK is a behavioral model that consists of the following core components: • Tactics, denoting short-term, tactical adversary goals during an attack; • Techniques, describing the means by which adversaries achieve tactical goals; • Sub-techniques, describing more specific means by which adversaries achieve tactical goals at a lower level than techniques; and • Documented adversary usage of techniques, their procedures, and other metadata. My concern is with MITRE's definition of "tactics" as "short-term, tactical adversary goals during an attack," which is oddly recursive. The key word in the tacti...
Read more

New Book! The Best of TaoSecurity Blog, Volume 4

Image
  I've completed the TaoSecurity Blog book series . The new book is  The Best of TaoSecurity Blog, Volume 4: Beyond the Blog with Articles, Testimony, and Scholarship .  It's available now for Kindle , and I'm working on the print edition.  I'm running a 50% off promo on Volumes 1-3 on Kindle through midnight 20 April. Take advantage before the prices go back up. I described the new title thus: Go beyond TaoSecurity Blog with this new volume from author Richard Bejtlich. In the first three volumes of the series, Mr. Bejtlich selected and republished the very best entries from 18 years of writing and over 18 million blog views, along with commentaries and additional material.  In this title, Mr. Bejtlich collects material that has not been published elsewhere, including articles that are no longer available or are stored in assorted digital or physical archives. Volume 4 offers early white papers that Mr. Bejtlich wrote as a network defender, either for technica...
Read more