Notes on Network Security Monitoring

I've been performing a network security monitoring assessment for a client this week. I use interviews, observations, and documentation review to provide findings, discussion, and recommendations for improving your incident detection and response operations.

During this process I was asked if I knew ways to measure packet loss on open source sensors. (This client uses FreeBSD, which is helpful!) Today I remembered work by Christian SJ Peron on bpfstat, available only on FreeBSD 6.0. bpfstat provides statistics like the following. Here I am running Tcpdump and Trafshow, and bpfstat is reporting packet collection information on interface sf0 every 1 second.

 bpfstat -i 1 -I sf0
 pid   netif  flags  recv drop match sblen hblen command
1682     sf0  p--s-  6337    0  6337  3844     0 trafshow
 780     sf0  p--s- 38405    0 38405 11380     0 tcpdump
1682     sf0  p--s-  7142    0  7142 22046     0 trafshow
 780     sf0  p--s- 39210    0 39210 14588     0 tcpdump
1682     sf0  p--s-  7997    0  7997 12686     0 trafshow
 780     sf0  p--s- 40065    0 40065 24316     0 tcpdump
1682     sf0  p--s-  8851    0  8851  3222     0 trafshow
 780     sf0  p--s- 40919    0 40919  6412     0 tcpdump
1682     sf0  p--s-  9705    0  9705 26516     0 trafshow
 780     sf0  p--s- 41773    0 41773 21108     0 tcpdump
1682     sf0  p--s- 10467    0 10467  7484     0 trafshow
 780     sf0  p--s- 42535    0 42535  6516     0 tcpdump

recv, drop, and match are self-explanatory. Christian provided this explanation of flags:

p - promiscuous mode
i - BIOCIMMEDIATE has been set
f - BIOCSHDRCMPLT has NOT been set
s - BIOCSSEESENT has been set
a - packet reception generates a signal
l - descriptor has been locked (-CURRENT ONLY) BIOCLOCK

hblen refers to the BPF hold buffer and sblen refers to the store buffer. The first entry for Tcpdump, for example, indicate that the current size of Tcpdump's store buffer is 11380 bytes and the size of the hold buffer is zero.

I asked Christian "Do you know of any other ways to measure packet loss outside of the programs themselves? (In other words, upon exiting Snort reports packet
> loss.) This is an extremely interesting subject for people doing network
> security monitoring."

He replied:

"No, none that I know of. Which was the primary motivator behind what I did. The reason I added this functionality into FreeBSD was because we run a lot of various IDS and network monitoring processes. Some of them don't offer any stats at all while others require termination or signal delivery.

Rather than interrupting the processes, I figured since the kernel keeps real time counters in memory for these types of statistics, I would export them to userspace. The one exception was the "matched" counter, which I introduced.

You can expect to have packet loss, and now we have a non-intrusive way to monitor this packet loss.

It might also be worth noting that Rui Paulo from NetBSD thought this was a great idea and added support for this into the NetBSD and netstat. I am going to releasing bpfstat 2 which will compile on both NetBSD and FreeBSD.

CVS commit: src/sys/net
CVS commit: src/usr.bin/netstat

They also provided some good feedback for the netstat and PID processing."

---

My NSM Assessment client uses Bro to provide indicators for intrusion detection. I noticed the change log says Bro 0.9a10 was released on 6 Sep. The last version, 0.9a9, arrived 19 May. At some point, I intend to re-engage on Bro to learn what else it can provide in an operational environment. Unfortunately the security/bro port does not have a maintainer. That might be a good learning project.