TaoSecurity: 09/01/2006

Thursday, September 28, 2006

Preview: Hunting Security Bugs

Yesterday I received a copy of Hunting Security Bugs. One of this book's authors is Tom Gallagher, who posted thoughts on Microsoft's security initiatives.

This looks like a great book, especially as a companion to The Security Development Lifecycle, also by Microsoft authors.

A third book, The Practical Guide to Defect Prevention, arrives in the spring. This may be too developer-oriented for my needs, but I might take a look at it.

I am glad to see Microsoft sharing the knowledge it has gained through its ongoing security program.

You can look at my Amazon.com Wish List to track books I plan to read, but don't have copies. My reading page shows books I own that I plan to read. The reading page also links to my recommended books lists.

See Richard Bejtlich teach Enterprise Network Instrumentation for SANS CDI East 2006 on 14-15 Dec 06 in Washington, DC. Email training [at] taosecurity [dot] com for more information. Register by 25 Oct 06 for the best rates.

Security Scruples Poll

Dark Reading is conducting a Security Scruples Poll. Some of the preliminary results are disturbing. I'll withhold commentary until I see the poll is closed and results are disclosed. Please consider taking the poll. It has some interesting questions, and it takes about 5 minutes.

See Richard Bejtlich teach TCP/IP Weapons School for USENIX LISA 2006 on 3-4 Dec 06 and separately for TaoSecurity on 9-10 Dec 06 in Washington, DC. Email training [at] taosecurity [dot] com for more information. Register by 13 Oct 06 for the best rates.

Wednesday, September 27, 2006

Review of Apache Security Books Posted

Amazon.com just posted my two reviews on books about Apache. The first is Apache Security by Ivan Ristic. Here is a link to the five star review.

The second is Preventing Web Attacks with Apache by Ryan Barnett. Here is a link to the four star review.

Both reviews share the same introduction.

I recently received copies of Apache Security (AS) by Ivan Ristic and Preventing Web Attacks with Apache (PWAWA) by Ryan Barnett. I read AS first, then PWAWA. Both are excellent books, but I expect potential readers want to know which is best for them. The following is a radical simplification, and I could honestly recommend readers buy either (or both) books. If you are more concerned with a methodical, comprehensive approach to securing Apache, choose AS. If you want more information on offensive aspects of Web security, choose PWAWA.

These are my 39th and 40th reviews of 2006. I should break my previous high reading mark of 42 books, accomplished in 2001.

Congratulations to Ivan for the acquisition of Thinking Stone by Breach Security.

Monday, September 25, 2006

Symantec Internet Security Threat Report Volume X

Symantec has posted (for free, no registration!) the latest Internet Security Threat Report. I'm very pleased to see that such a high-profile report uses threat and vulnerability terms properly, and features details on the methodology used to produce the report. Here's some of the Executive Summary.

In contrast to previously observed widespread, network-based attacks, attackers today tend to be more focused, often targeting client-side applications... The current threat landscape is populated by lower profile, more targeted attacks, attacks that propagate at a slower rate in order to avoid detection and thereby increase the likelihood of successful compromise.

Instead of exploiting vulnerabilities in servers, as traditional attacks often did, these threats tend to exploit vulnerabilities in client-side applications that require a degree of user interaction, such as word processing and spreadsheet programs.

A number of these have been zero-day vulnerabilities. These types of threats also attempt to escape detection in order to remain on host systems for longer periods so that they can steal information or provide remote access.

Do you see how important it is to differentiate between threats and vulnerabilities when the terms are used in the same sentence? Bravo Symantec.

This volume of the Internet Security Threat Report will offer an analysis and discussion of threat activity that took place between January 1 and June 30, 2006. This brief summary will offer a synopsis of the data and trends discussed in the main report. Symantec will continue to monitor and assess threat activity in order to best prepare consumers and enterprises for the complex Internet security issues to come.

How does Symantec "monitor and assess threat activity"? By watching, of course.

The Symantec™ Global Intelligence Network comprehensively tracks attack activity across the entire Internet. The Global Intelligence Network, which includes the Symantec DeepSight™ Threat Management System and Symantec™ Managed Security Services, consists of over 40,000 sensors monitoring network activity in over 180 countries. As well, Symantec gathers malicious code data along with spyware and adware reports from over 120 million client, server, and gateway systems that have deployed Symantec’s antivirus products.

They're not using counts of vulnerabilities announced on mailing lists. They're watching exploitation of their customer base.

Their Vulnerability Trend Highlights are fascinating:

Symantec documented 2,249 new vulnerabilities, up 18% over the second half of
2005. This is the highest number ever recorded for a six-month period.

Web application vulnerabilities made up 69% of all vulnerabilities this period.

Mozilla browsers had the most vulnerabilities, 47, compared to 38 in Microsoft Internet Explorer.

In the first six months of 2006, 80% of vulnerabilities were considered easily exploitable, up from 79%.

Seventy-eight percent of easily exploitable vulnerabilities affected Web applications.

The window of exposure for enterprise vulnerabilities was 28 days.

Internet Explorer had an average window of exposure of nine days, the largest of any Web browser. Apple Safari averaged five days, followed by Opera with two days and Mozilla with one day.

In the first half of 2006, Sun operating systems had the highest average patch development time, with 89 days, followed by Hewlett Packard with 53 days, Apple with 37 days and Microsoft and Red Hat with 13 days.

I think it's interesting that Mozilla had more vulnerabilities, but a far smaller vulnerability window, than Internet Explorer.

I recommend reading the whole report, or at least the executive summary.

Review of The TCP/IP Guide Posted

Amazon.com just posted my 4 star review of The TCP/IP Guide. From the review:

Right away I must state that I did not read "The TCP/IP Guide" (TTG) cover-to-cover. I doubt anyone will, which raises interesting issues. This review is based on the sections I did read and my comparisons with other protocol books.

Protocol books should be divided into two eras. The first is the "Stevens era" meaning those written around the time Richard Stevens' "TCP/IP Illustrated, Vol 1: The Protocols" was published. For six years (1994-2000) Stevens' book was clearly the best protocol book, and it taught legions of networking pros TCP/IP. The second is the "modern era," beginning in 2000 and continuing to today. TTG fits in this group.

I question the approach taken by TTG. The book contains extremely basic information (what is networking, why use layers, what is a protocol, etc.) and extremely obscure information (PPP Link Control Protocol Frame Types and Fields, SNMPv2 PDU Error Status Field Values, Interpretation of Standard Telnet NVT ASCII Control Codes, etc.). If TTG were an introductory book, it wouldn't need the obscure material. If TTG were a reference, it wouldn't need the introductory material.

At 1616 pages and nearly 5 pounds, we should be dropping these books out of B-2s!

Saturday, September 23, 2006

Net Optics Think Tank Tuesday in Fairfax, VA

Don't forget to attend the free Net Optics Think Tank on Tuesday, 26 September 2006 in Fairfax, VA. It looks like I will be speaking during lunch from 1215 to 1315. Please register. I expect to see a lot of cool Net Optics gear on display, along with insights from those who make products for enterprise network instrumentation.

Throughput Testing Through a Bridge

In my earlier posts I've discussed throughput testing. Now I'm going to introduce an inline system as a bridge. You could imagine that this system might be a firewall, or run Snort in inline mode. For the purposes of this post, however, we're just going to see what effect the bridge has on throughput between a client and server.

This is the new system. It's called cel600, and it's running the same GENERIC.POLLING kernel mentioned earlier.


FreeBSD 6.1-RELEASE-p6 #0: Sun Sep 17 17:09:24 EDT 2006
    root@kbld.taosecurity.com:/usr/obj/usr/src/sys/GENERIC.POLLING
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel Celeron (598.19-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x686  Stepping = 6
  Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PA
T,PSE36,MMX,FXSR,SSE>
real memory  = 401260544 (382 MB)
avail memory = 383201280 (365 MB)

This system has two dual NICs in it. em0 and em1 are Gigabit fiber, and em2 and em3 are Gigabit copper.


cel600:/root# ifconfig em0
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        options=48<VLAN_MTU,POLLING>
        inet6 fe80::204:23ff:feb1:7f22%em0 prefixlen 64 scopeid 0x1 
        ether 00:04:23:b1:7f:22
        media: Ethernet autoselect (1000baseSX <full-duplex>)
        status: active
cel600:/root# ifconfig em1
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        options=48<VLAN_MTU,POLLING>
        inet6 fe80::204:23ff:feb1:7f23%em1 prefixlen 64 scopeid 0x2 
        ether 00:04:23:b1:7f:23
        media: Ethernet autoselect (1000baseSX <full-duplex>)
        status: active
cel600:/root# ifconfig em2
em2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        options=48<VLAN_MTU,POLLING>
        inet6 fe80::204:23ff:fec5:4e80%em2 prefixlen 64 scopeid 0x3 
        ether 00:04:23:c5:4e:80
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
cel600:/root# ifconfig em3
em3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        options=48<VLAN_MTU,POLLING>
        inet6 fe80::204:23ff:fec5:4e81%em3 prefixlen 64 scopeid 0x4 
        ether 00:04:23:c5:4e:81
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active

I configure them in /etc/rc.conf this way:


ifconfig_em0="polling up"
ifconfig_em1="polling up"
ifconfig_em2="polling up"
ifconfig_em3="polling up"
cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0="addm em0 addm em1 monitor up"
ifconfig_bridge1="addm em2 addm em3 monitor up"

The end result is two bridge interfaces.


bridge0: flags=48043<UP,BROADCAST,RUNNING,MULTICAST,MONITOR> mtu 1500
        ether ac:de:48:e5:e7:69
        priority 32768 hellotime 2 fwddelay 15 maxage 20
        member: em1 flags=3<LEARNING,DISCOVER>
        member: em0 flags=3<LEARNING,DISCOVER>
cel600:/root# ifconfig bridge1
bridge1: flags=48043>UP,BROADCAST,RUNNING,MULTICAST,MONITOR> mtu 1500
        ether ac:de:48:0c:26:66
        priority 32768 hellotime 2 fwddelay 15 maxage 20
        member: em3 flags=3<LEARNING,DISCOVER>
        member: em2 flags=3<LEARNING,DISCOVER>

Notice these two pseudo-interfaces are both in MONITOR mode. That was set automatically.

With the bridge in place, I can conduct throughput tests.

Here is the client's view.


asa633:/root# iperf -c 172.16.6.2 -t 60 -i 5
------------------------------------------------------------
Client connecting to 172.16.6.2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.1 port 57355 connected with 172.16.6.2 port 5001
[  3]  0.0- 5.0 sec  55.9 MBytes  93.9 Mbits/sec
[  3]  5.0-10.0 sec  51.6 MBytes  86.6 Mbits/sec
[  3] 10.0-15.0 sec  72.3 MBytes    121 Mbits/sec
[  3] 15.0-20.0 sec  54.6 MBytes  91.6 Mbits/sec
[  3] 20.0-25.0 sec  61.4 MBytes    103 Mbits/sec
[  3] 25.0-30.0 sec  75.4 MBytes    127 Mbits/sec
[  3] 30.0-35.0 sec  60.2 MBytes    101 Mbits/sec
[  3] 35.0-40.0 sec  47.8 MBytes  80.2 Mbits/sec
[  3] 40.0-45.0 sec  74.7 MBytes    125 Mbits/sec
[  3] 45.0-50.0 sec  59.0 MBytes  99.0 Mbits/sec
[  3] 50.0-55.0 sec  54.0 MBytes  90.6 Mbits/sec
[  3] 55.0-60.0 sec  76.8 MBytes    129 Mbits/sec
[  3]  0.0-60.0 sec    744 MBytes    104 Mbits/sec

Here is the server's view.


poweredge:/root# iperf -s -B 172.16.6.2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 172.16.6.2
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 172.16.6.2 port 5001 connected with 172.16.6.1 port 57355
[  4]  0.0-60.0 sec    744 MBytes    104 Mbits/sec

Compared to the straight-through tests, you can see the effect on throughput caused by the bridge.


[  4]  0.0-60.0 sec  1.19 GBytes    170 Mbits/sec

Of interest during the test is the interrupt count on the bridge.


last pid:   728;  load averages:  0.00,  0.09,  0.06    up 0+00:06:36  17:58:40
22 processes:  1 running, 21 sleeping
CPU states:  0.4% user,  0.0% nice,  0.4% system, 17.1% interrupt, 82.1% idle
Mem: 7572K Active, 4776K Inact, 16M Wired, 8912K Buf, 339M Free
Swap: 768M Total, 768M Free

Let's try the UDP test. First, the client view.


asa633:/root# iperf -c 172.16.6.2 -u -t 60 -i 5 -b 500M
------------------------------------------------------------
Client connecting to 172.16.6.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.1 port 51356 connected with 172.16.6.2 port 5001
[  3]  0.0- 5.0 sec    169 MBytes    284 Mbits/sec
[  3]  5.0-10.0 sec    169 MBytes    284 Mbits/sec
[  3] 10.0-15.0 sec    171 MBytes    287 Mbits/sec
[  3] 15.0-20.0 sec    171 MBytes    287 Mbits/sec
[  3] 20.0-25.0 sec    171 MBytes    287 Mbits/sec
[  3] 25.0-30.0 sec    171 MBytes    287 Mbits/sec
[  3] 30.0-35.0 sec    171 MBytes    287 Mbits/sec
[  3] 35.0-40.0 sec    172 MBytes    288 Mbits/sec
[  3] 40.0-45.0 sec    172 MBytes    288 Mbits/sec
[  3] 45.0-50.0 sec    172 MBytes    288 Mbits/sec
[  3] 50.0-55.0 sec    172 MBytes    288 Mbits/sec
[  3]  0.0-60.0 sec  2.00 GBytes    287 Mbits/sec
[  3] Sent 1463703 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec  1.93 GBytes    276 Mbits/sec  0.014 ms 53386/1463702 (3.6%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Now the server view.


poweredge:/root# iperf -s -u -B 172.16.6.2
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 172.16.6.2
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.2 port 5001 connected with 172.16.6.1 port 51356
[  3]  0.0-60.0 sec  1.93 GBytes    276 Mbits/sec  0.014 ms 53386/1463702 (3.6%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Here's the result from the straight-through test.


[  3]  0.0-60.0 sec  1.94 GBytes    277 Mbits/sec  0.056 ms 62312/1478219 (4.2%)

The results are almost identical.

Here is the bridge's interrupt count as shown in a top excerpt.


last pid:   751;  load averages:  0.00,  0.03,  0.04    up 0+00:10:20  18:02:24
22 processes:  1 running, 21 sleeping
CPU states:  0.0% user,  0.0% nice,  0.4% system, 19.8% interrupt, 79.8% idle
Mem: 7564K Active, 4788K Inact, 16M Wired, 8928K Buf, 339M Free
Swap: 768M Total, 768M Free

With the Gigabit fiber tests done, let's look at Gigabit copper.

First, a TCP test as seen by the client.


asa633:/root# iperf -c 172.16.7.2 -t 60 -i 5
------------------------------------------------------------
Client connecting to 172.16.7.2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.1 port 58824 connected with 172.16.7.2 port 5001
[  3]  0.0- 5.0 sec  76.3 MBytes    128 Mbits/sec
[  3]  5.0-10.0 sec  76.3 MBytes    128 Mbits/sec
[  3] 10.0-15.0 sec  76.8 MBytes    129 Mbits/sec
[  3] 15.0-20.0 sec  76.6 MBytes    129 Mbits/sec
[  3] 20.0-25.0 sec  76.8 MBytes    129 Mbits/sec
[  3] 25.0-30.0 sec  75.4 MBytes    127 Mbits/sec
[  3] 30.0-35.0 sec  76.3 MBytes    128 Mbits/sec
[  3] 35.0-40.0 sec  76.1 MBytes    128 Mbits/sec
[  3] 40.0-45.0 sec  76.5 MBytes    128 Mbits/sec
[  3] 45.0-50.0 sec  75.4 MBytes    126 Mbits/sec
[  3] 50.0-55.0 sec  76.7 MBytes    129 Mbits/sec
[  3] 55.0-60.0 sec  76.4 MBytes    128 Mbits/sec
[  3]  0.0-60.0 sec    916 MBytes    128 Mbits/sec

Here is the server's view.


poweredge:/root# iperf -s -B 172.16.7.2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 172.16.7.2
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 172.16.7.2 port 5001 connected with 172.16.7.1 port 58824
[  4]  0.0-60.0 sec    916 MBytes    128 Mbits/sec

That is better than the result for fiber from above.


[  4]  0.0-60.0 sec    744 MBytes    104 Mbits/sec

It's not as good as the result for straight-through copper.


[  4]  0.0-60.0 sec  1.16 GBytes    166 Mbits/sec

It seemed as though the bridge interrupt count was lower than the fiber TCP tests.


last pid:   754;  load averages:  0.00,  0.01,  0.02    up 0+00:13:48  18:05:52
22 processes:  1 running, 21 sleeping
CPU states:  0.0% user,  0.0% nice,  0.4% system, 16.7% interrupt, 82.9% idle
Mem: 7560K Active, 4792K Inact, 16M Wired, 8928K Buf, 339M Free
Swap: 768M Total, 768M Free

Finally, UDP copper tests. Here is the client view.


asa633:/root# iperf -c 172.16.7.2 -u -t 60 -i 5 -b 500M
------------------------------------------------------------
Client connecting to 172.16.7.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.1 port 62131 connected with 172.16.7.2 port 5001
[  3]  0.0- 5.0 sec    129 MBytes    217 Mbits/sec
[  3]  5.0-10.0 sec    129 MBytes    217 Mbits/sec
[  3] 10.0-15.0 sec    129 MBytes    217 Mbits/sec
[  3] 15.0-20.0 sec    129 MBytes    217 Mbits/sec
[  3] 20.0-25.0 sec    129 MBytes    217 Mbits/sec
[  3] 25.0-30.0 sec    129 MBytes    216 Mbits/sec
[  3] 30.0-35.0 sec    129 MBytes    216 Mbits/sec
[  3] 35.0-40.0 sec    129 MBytes    216 Mbits/sec
[  3] 40.0-45.0 sec    129 MBytes    216 Mbits/sec
[  3] 45.0-50.0 sec    129 MBytes    216 Mbits/sec
[  3] 50.0-55.0 sec    129 MBytes    216 Mbits/sec
[  3]  0.0-60.0 sec  1.51 GBytes    216 Mbits/sec
[  3] Sent 1103828 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec  1.46 GBytes    209 Mbits/sec  0.047 ms 35057/1103827 (3.2%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Here is the server view.


poweredge:/root# iperf -s -u -B 172.16.7.2
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 172.16.7.2
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.2 port 5001 connected with 172.16.7.1 port 62131
[  3]  0.0-60.0 sec  1.46 GBytes    209 Mbits/sec  0.047 ms 35057/1103827 (3.2%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Let's compare that to the fiber UDP test from above.


[  3]  0.0-60.0 sec  1.93 GBytes    276 Mbits/sec  0.014 ms 53386/1463702 (3.6%)

This time, the results are much worse than the UDP over fiber results.

When I tested UDP over crossover copper, this was the result.


[  3]  0.0-60.0 sec  1.86 GBytes    267 Mbits/sec  0.024 ms 40962/1401730 (2.9%)

The top excerpt is about the same as the fiber UDP test.


last pid:   754;  load averages:  0.01,  0.01,  0.01    up 0+00:16:21  18:08:25
22 processes:  1 running, 21 sleeping
CPU states:  0.0% user,  0.0% nice,  0.0% system, 17.1% interrupt, 82.9% idle
Mem: 7564K Active, 4788K Inact, 16M Wired, 8928K Buf, 339M Free
Swap: 768M Total, 768M Free

It's not really feasible to make any solid assumptions based on these tests. They're basically get to get a ballpark feel for the capabilities of a given architecture, but you need to repeat them multiple times to get some confidence in the results.

If you want built-in repeatability and confidence testing, try Netperf.

With these results, however, I have some idea of what I can expect from this particular hardware setup, namely a bridge between a client sending data to a server.

TCP over fiber: about 104 Mbps

UDP over fiber: about 276 Mbps

TCP over copper: about 128 Mbps

UDP over copper: about 209 Mbps

Rounding down, and acting conservatively, I would feel this setup could handle somewhere around 100 Mbps (aggregated) over fiber and around 125 Mbps over copper. Note this says nothing about any software running on the bridge and its ability to do whatever function it is designed to perform. This is just a throughput estimate.

In my next related posts I'll introduce bypass switches and see how they influence this process.

I'll also rework the configuration into straight-through, bridged, and switched modes to test latency using ping.

FreeBSD Device Polling Results for Gigabit Copper

In my post FreeBSD Device Polling I ran my tests over Gigabit fiber connections. I thought I would repeat the tests for Gigabit copper, connected by normal straight-through cables. (One benefit of Gigabit copper Ethernet NICs is there's no need for crossover cables.)

Although I booted my two test boxes, asa633 and poweredge, with kernels offering polling, neither interface had polling enabled by default. This is asa633's NIC:


em1: flags=8843 mtu 1500
        options=b
        inet6 fe80::20e:cff:feba:e726%em1 prefixlen 64 scopeid 0x4 
        inet 172.16.7.1 netmask 0xffffff00 broadcast 172.16.7.255
        ether 00:0e:0c:ba:e7:26
        media: Ethernet autoselect (1000baseTX )
        status: active

This is poweredge's NIC.


em1: flags=8843 mtu 1500
        options=b
        inet6 fe80::207:e9ff:fe11:a0a0%em1 prefixlen 64 scopeid 0x4 
        inet 172.16.7.2 netmask 0xffffff00 broadcast 172.16.7.255
        ether 00:07:e9:11:a0:a0
        media: Ethernet autoselect (1000baseTX )
        status: active

First I ran unidirectional TCP tests, from asa633 to poweredge, without polling.


asa633:/root# iperf -c 172.16.7.2 -t 60 -i 5
------------------------------------------------------------
Client connecting to 172.16.7.2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.1 port 58672 connected with 172.16.7.2 port 5001
[  3]  0.0- 5.0 sec  90.2 MBytes    151 Mbits/sec
[  3]  5.0-10.0 sec  91.1 MBytes    153 Mbits/sec
[  3] 10.0-15.0 sec  90.0 MBytes    151 Mbits/sec
[  3] 15.0-20.0 sec  91.2 MBytes    153 Mbits/sec
[  3] 20.0-25.0 sec  89.8 MBytes    151 Mbits/sec
[  3] 25.0-30.0 sec  90.9 MBytes    153 Mbits/sec
[  3] 30.0-35.0 sec  91.7 MBytes    154 Mbits/sec
[  3] 35.0-40.0 sec  92.0 MBytes    154 Mbits/sec
[  3] 40.0-45.0 sec  89.9 MBytes    151 Mbits/sec
[  3] 45.0-50.0 sec  90.1 MBytes    151 Mbits/sec
[  3] 50.0-55.0 sec  90.4 MBytes    152 Mbits/sec
[  3] 55.0-60.0 sec  91.0 MBytes    153 Mbits/sec
[  3]  0.0-60.0 sec  1.06 GBytes    152 Mbits/sec

Here is what the server saw.


poweredge:/root# iperf -s -B 172.16.7.2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 172.16.7.2
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 172.16.7.2 port 5001 connected with 172.16.7.1 port 58672
[  4]  0.0-60.0 sec  1.06 GBytes    152 Mbits/sec

Interrupt levels for both systems was similar to the Gigabit copper results.

Here is the change with polling enabled via 'ifconfig em1 polling'. First, the client.


asa633:/root# iperf -c 172.16.7.2 -t 60 -i 5
------------------------------------------------------------
Client connecting to 172.16.7.2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.1 port 52789 connected with 172.16.7.2 port 5001
[  3]  0.0- 5.0 sec  79.2 MBytes    133 Mbits/sec
[  3]  5.0-10.0 sec  76.3 MBytes    128 Mbits/sec
[  3] 10.0-15.0 sec  80.4 MBytes    135 Mbits/sec
[  3] 15.0-20.0 sec    123 MBytes    207 Mbits/sec
[  3] 20.0-25.0 sec    126 MBytes    212 Mbits/sec
[  3] 25.0-30.0 sec    110 MBytes    185 Mbits/sec
[  3] 30.0-35.0 sec  89.1 MBytes    149 Mbits/sec
[  3] 35.0-40.0 sec  77.0 MBytes    129 Mbits/sec
[  3] 40.0-45.0 sec  76.8 MBytes    129 Mbits/sec
[  3] 45.0-50.0 sec    103 MBytes    172 Mbits/sec
[  3] 50.0-55.0 sec    128 MBytes    215 Mbits/sec
[  3] 55.0-60.0 sec    120 MBytes    201 Mbits/sec
[  3]  0.0-60.0 sec  1.16 GBytes    166 Mbits/sec

Now the server.


poweredge:/root# iperf -s -B 172.16.7.2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 172.16.7.2
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 172.16.7.2 port 5001 connected with 172.16.7.1 port 52789
[  4]  0.0-60.0 sec  1.16 GBytes    166 Mbits/sec

Polling didn't improve the situation much for TCP.

Here are the results for unidirectional UDP tests, without polling.

This is the client.


asa633:/root# iperf -c 172.16.7.2 -u -t 60 -i 5 -b 500M
------------------------------------------------------------
Client connecting to 172.16.7.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.1 port 60193 connected with 172.16.7.2 port 5001
[  3]  0.0- 5.0 sec    129 MBytes    217 Mbits/sec
[  3]  5.0-10.0 sec    129 MBytes    217 Mbits/sec
[  3] 10.0-15.0 sec    129 MBytes    217 Mbits/sec
[  3] 15.0-20.0 sec    127 MBytes    212 Mbits/sec
[  3] 20.0-25.0 sec    129 MBytes    217 Mbits/sec
[  3] 25.0-30.0 sec    129 MBytes    217 Mbits/sec
[  3] 30.0-35.0 sec    129 MBytes    217 Mbits/sec
[  3] 35.0-40.0 sec    129 MBytes    217 Mbits/sec
[  3] 40.0-45.0 sec    129 MBytes    217 Mbits/sec
[  3] 45.0-50.0 sec    129 MBytes    216 Mbits/sec
[  3] 50.0-55.0 sec    129 MBytes    216 Mbits/sec
[  3]  0.0-60.0 sec  1.51 GBytes    216 Mbits/sec
[  3] Sent 1102470 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec    787 MBytes    110 Mbits/sec  0.042 ms 541153/1102469 (49%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Notice the huge drops. Here is the server.


poweredge:/root# iperf -s -u -B 172.16.7.2
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 172.16.7.2
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.2 port 5001 connected with 172.16.7.1 port 60193
[  3]  0.0-60.0 sec    787 MBytes    110 Mbits/sec  0.042 ms 541153/1102469 (49%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Here are results with polling enabled.

The client:


asa633:/root# iperf -c 172.16.7.2 -u -t 60 -i 5 -b 500M
------------------------------------------------------------
Client connecting to 172.16.7.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.1 port 53387 connected with 172.16.7.2 port 5001
[  3]  0.0- 5.0 sec    163 MBytes    274 Mbits/sec
[  3]  5.0-10.0 sec    164 MBytes    275 Mbits/sec
[  3] 10.0-15.0 sec    164 MBytes    275 Mbits/sec
[  3] 15.0-20.0 sec    164 MBytes    275 Mbits/sec
[  3] 20.0-25.0 sec    164 MBytes    275 Mbits/sec
[  3] 25.0-30.0 sec    164 MBytes    275 Mbits/sec
[  3] 30.0-35.0 sec    163 MBytes    274 Mbits/sec
[  3] 35.0-40.0 sec    164 MBytes    275 Mbits/sec
[  3] 40.0-45.0 sec    164 MBytes    275 Mbits/sec
[  3] 45.0-50.0 sec    164 MBytes    275 Mbits/sec
[  3] 50.0-55.0 sec    164 MBytes    275 Mbits/sec
[  3]  0.0-60.0 sec  1.92 GBytes    275 Mbits/sec
[  3] Sent 1401731 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec  1.86 GBytes    267 Mbits/sec  0.023 ms 40962/1401730 (2.9%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Now the server.


poweredge:/root# iperf -s -u -B 172.16.7.2
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 172.16.7.2
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
[  3] local 172.16.7.2 port 5001 connected with 172.16.7.1 port 53387
[  3]  0.0-60.0 sec  1.86 GBytes    267 Mbits/sec  0.024 ms 40962/1401730 (2.9%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Like the copper tests, polling really helps with UDP performance.

Given that polling can be enabled and disabled at will via ifconfig, I would like to see 'options DEVICE_POLLING' added to the GENERIC FreeBSD kernel.

Friday, September 22, 2006

The ZERT Evolution

In January during the WMF fiasco, I wrote The Power of Open Source. What we're now reading in Zero-Day Response Team Launches with Emergency IE Patch is the latest evolution of this idea. The Zeroday Emergency Response Team isn't a bunch of amateurs. These are some of the highest skilled security researchers and practitioners in the public arena. They are stepping up to meet a need not fulfilled by vendors, namely rapid response to security problems.

Why is this the case? Customers running closed operating systems and applications are stuck. They can't fix problems themselves, so they rely on their vendor. In fact, they are paying their vendor to perform the fixing service. To fund development of an alternative fix would be like paying for a fix twice.

ZERT is demonstrating that this model is broken. They are trying to respond as fast as possible to attacks. Because no one can be "ahead of the threat," reaction time is often key. ZERT can act faster than the vendor because ZERT operates in a freer environment:

Please keep in mind while the group performs extensive testing of any patches before releasing them, it is impossible for us to test our patches with each possible system configuration and in each usage scenario. We validate patches to the best of our ability, noting the environments in which the tests were performed and the test results.

So what shall it be? Wait and be owned, or turn to a third party? Perhaps we'll see a more rapid release of a use-at-your-own-risk patch from vendors, followed by a tested-for-stability patch. It's tough to believe that people without access to source code are developing fixes faster that the creators of software!

Generating Multicast Traffic

If you're a protocol junkie like me, you probably enjoy investigating a variety of network traffic types. I don't encounter multicast traffic too often, so the following caught my eye.

I'm using Iperf for some simple testing, and I notice it has a multicast option. Here's how I used it.

In the following scenario, I have two hosts (cel433 and cel600) on the same segment. This is important because the router(s) in this test network are not configured to support multicast.

I set up cel433 as a Iperf server listening on multicast address 224.0.55.55.


cel433:/root# iperf -s -u -B 224.0.55.55 -i 1
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 224.0.55.55
Joining multicast group  224.0.55.55
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)

Now I generate multicast traffic from cel600.


cel600:/root# iperf -c 224.0.55.55 -u -T 32 -t 3 -i 1
------------------------------------------------------------
Client connecting to 224.0.55.55, UDP port 5001
Sending 1470 byte datagrams
Setting multicast TTL to 32
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[  3] local 10.1.10.3 port 51296 connected with 224.0.55.55 port 5001
[  3]  0.0- 1.0 sec    129 KBytes  1.06 Mbits/sec
[  3]  1.0- 2.0 sec    128 KBytes  1.05 Mbits/sec
[  3]  2.0- 3.0 sec    128 KBytes  1.05 Mbits/sec
[  3]  0.0- 3.0 sec    386 KBytes  1.05 Mbits/sec
[  3] Sent 269 datagrams

Here is what cel433 sees:


------------------------------------------------------------
[  3] local 224.0.55.55 port 5001 connected with 10.1.10.3 port 51296
[  3]  0.0- 1.0 sec    128 KBytes  1.05 Mbits/sec  0.146 ms    0/   89 (0%)
[  3]  1.0- 2.0 sec    128 KBytes  1.05 Mbits/sec  0.100 ms    0/   89 (0%)
[  3]  2.0- 3.0 sec    128 KBytes  1.05 Mbits/sec  0.110 ms    0/   89 (0%)
[  3]  0.0- 3.0 sec    386 KBytes  1.05 Mbits/sec  0.098 ms    0/  268 (0%)
[  3]  0.0- 3.0 sec  1 datagrams received out-of-order

The traffic looks like this:


cel433:/root# tcpdump -n -i xl0 -s 1515 udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xl0, link-type EN10MB (Ethernet), capture size 1515 bytes
15:29:53.669508 IP 10.1.10.3.51296 > 224.0.55.55.5001: UDP, length 1470
15:29:53.680789 IP 10.1.10.3.51296 > 224.0.55.55.5001: UDP, length 1470
15:29:53.691934 IP 10.1.10.3.51296 > 224.0.55.55.5001: UDP, length 1470
...truncated...

This is a simple way to generate multicast traffic and ensure a member of the multicast group actually receives it.

Update: I forgot to show the IGMP messages one would see when starting a multicast listener.

This is the interface listening for multicast:


cel433:/root# ifconfig xl0
xl0: flags=8843 mtu 1500
        options=9
        inet6 fe80::2c0:4fff:fe1c:102b%xl0 prefixlen 64 scopeid 0x6 
        inet 10.1.10.2 netmask 0xffffff00 broadcast 10.1.10.255
        ether 00:c0:4f:1c:10:2b
        media: Ethernet autoselect (100baseTX )
        status: active

Here are IGMP report and leave messages.


cel433:/root# tcpdump -nevv -i xl0 -s 1515 igmp
tcpdump: listening on xl0, link-type EN10MB (Ethernet), capture size 1515 bytes
06:28:40.887868 00:c0:4f:1c:10:2b > 01:00:5e:00:37:37, ethertype IPv4 (0x0800),
 length 46: (tos 0x0, ttl   1, id 59915, offset 0, flags [none], proto: IGMP (2),
 length: 32, options
 ( RA (148) len 4 )) 10.1.10.2 > 224.0.55.55: igmp v2 report 224.0.55.55

06:28:42.196233 00:c0:4f:1c:10:2b > 01:00:5e:00:00:02, ethertype IPv4 (0x0800),
 length 46: (tos 0x0, ttl   1, id 59920, offset 0, flags [none], proto: IGMP (2),
 length: 32, options
 ( RA (148) len 4 )) 10.1.10.2 > 224.0.0.2: igmp leave 224.0.55.55

I used the -e option to show the MAC addresses. Notice the destination MAC for these multicast packets.


06:31:21.467919 00:b0:d0:14:b2:11 > 01:00:5e:00:37:37, ethertype IPv4 (0x0800),
 length 1512: (tos 0x0, ttl  32, id 1652, offset 0, flags [none], proto: UDP (17),
 length: 1498) 
 10.1.10.3.58479 > 224.0.55.55.5001: [udp sum ok] UDP, length 1470

The 01:00:5e:00:37:37 MAC address is a mapping derived from the 24-bit IANA multicast OUI 01:00:5e and the multicast IP address 224.0.55.55.

FreeBSD Device Polling

Not all of us work with the latest, greatest hardware. If we use open source software, we often find ourselves running it on old hardware. I have a mix of equipment in my lab and I frequently see what I can do with it.

In this post I'd like to talk about some simple network performance measurement testing. Some of this is based on the book Network Performance Toolkit: Using Open Source Testing Tools. I don't presume that any of this is definitive, novel, or particularly helpful for all readers. I welcome constructive ideas for improvements.

For the purposes of this post, I'd like to get a sense of the network throughput between two hosts, asa633 and poweredge.

This is asa633's dmesg output:


FreeBSD 6.1-RELEASE-p6 #0: Wed Sep 20 20:02:56 EDT 2006
    root@kbld.taosecurity.com:/usr/obj/usr/src/sys/GENERIC.SECURITY
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel Celeron (631.29-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x686  Stepping = 6
  Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PA
T,PSE36,MMX,FXSR,SSE>
real memory  = 334233600 (318 MB)
avail memory = 317620224 (302 MB)

This is poweredge's dmesg output:


FreeBSD 6.1-RELEASE-p6 #0: Wed Sep 20 20:02:56 EDT 2006
    root@kbld.taosecurity.com:/usr/obj/usr/src/sys/GENERIC.SECURITY
ACPI APIC Table: <DELL   PE2300  >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Pentium III/Pentium III Xeon/Celeron (498.75-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x673  Stepping = 3
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CM
OV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 536862720 (511 MB)
avail memory = 515993600 (492 MB)

Neither system has any tuning applied.

Each box has the following relevant interfaces.


asa633:/root# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        inet6 fe80::204:23ff:feb1:64e2%em0 prefixlen 64 scopeid 0x3 
        inet 172.16.6.1 netmask 0xffffff00 broadcast 172.16.6.255
        ether 00:04:23:b1:64:e2
        media: Ethernet autoselect (1000baseSX )
        status: active
asa633:/root# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        inet6 fe80::20e:cff:feba:e726%em1 prefixlen 64 scopeid 0x4 
        inet 172.16.7.1 netmask 0xffffff00 broadcast 172.16.7.255
        ether 00:0e:0c:ba:e7:26
        media: Ethernet autoselect (1000baseTX )
        status: active

poweredge:/root# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        inet6 fe80::204:23ff:feab:964%em0 prefixlen 64 scopeid 0x2 
        inet 172.16.6.2 netmask 0xffffff00 broadcast 172.16.6.255
        ether 00:04:23:ab:09:64
        media: Ethernet autoselect (1000baseSX )
        status: active
poweredge:/root# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        inet6 fe80::207:e9ff:fe11:a0a0%em1 prefixlen 64 scopeid 0x4 
        inet 172.16.7.2 netmask 0xffffff00 broadcast 172.16.7.255
        ether 00:07:e9:11:a0:a0
        media: Ethernet autoselect (1000baseTX )
        status: active

The 172.16.6.0/24 interfaces are connected directly via fiber. The 172.16.7.0/24 interfaces are connected directly via copper.

With this setup, let's use Iperf to transmit and receive traffic.

Poweredge runs the server, but let's show the client first.


asa633:/root# iperf -c 172.16.6.2 -t 60 -i 5
------------------------------------------------------------
Client connecting to 172.16.6.2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.1 port 52453 connected with 172.16.6.2 port 5001
[  3]  0.0- 5.0 sec  82.1 MBytes    138 Mbits/sec
[  3]  5.0-10.0 sec  83.4 MBytes    140 Mbits/sec
[  3] 10.0-15.0 sec  83.6 MBytes    140 Mbits/sec
[  3] 15.0-20.0 sec  83.6 MBytes    140 Mbits/sec
[  3] 20.0-25.0 sec  83.5 MBytes    140 Mbits/sec
[  3] 25.0-30.0 sec  84.2 MBytes    141 Mbits/sec
[  3] 30.0-35.0 sec  85.4 MBytes    143 Mbits/sec
[  3] 35.0-40.0 sec  85.7 MBytes    144 Mbits/sec
[  3] 40.0-45.0 sec  86.8 MBytes    146 Mbits/sec
[  3] 45.0-50.0 sec  88.8 MBytes    149 Mbits/sec
[  3] 50.0-55.0 sec  90.6 MBytes    152 Mbits/sec
[  3] 55.0-60.0 sec  91.6 MBytes    154 Mbits/sec
[  3]  0.0-60.0 sec  1.01 GBytes    144 Mbits/sec

Here is the server's view.


poweredge:/root# iperf -s -B 172.16.6.2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 172.16.6.2
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 172.16.6.2 port 5001 connected with 172.16.6.1 port 52453
[  4]  0.0-60.0 sec  1.01 GBytes    144 Mbits/sec

That's interesting. These boxes averaged 144 Mbps. While the tests were running I captured top output. First, the client asa633:


last pid:   840;  load averages:  0.24,  0.10,  0.03    up 0+01:03:10  15:48:51
27 processes:  2 running, 25 sleeping
CPU states:  2.7% user,  0.0% nice, 47.1% system, 49.4% interrupt,  0.8% idle
Mem: 8876K Active, 5784K Inact, 17M Wired, 9040K Buf, 273M Free
Swap: 640M Total, 640M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  840 root        2 102    0  2768K  1696K RUN      0:04  0.00% iperf

Now the server, poweredge.


last pid:   716;  load averages:  0.36,  0.12,  0.04    up 0+00:53:13  15:49:10
34 processes:  2 running, 32 sleeping
CPU states:  2.6% user,  0.0% nice, 39.0% system, 56.9% interrupt,  1.5% idle
Mem: 31M Active, 8768K Inact, 20M Wired, 12M Buf, 434M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  716 root        3 106    0  2956K  1792K RUN      0:07  0.00% iperf

Those seem like high interrupt counts. Before making changes to see if we can improve the situation, let's run Iperf in bidirectional mode. That sends traffic from the client to server and server to client simultaneously.

Here is the client's view.


asa633:/root# iperf -c 172.16.6.2 -d -t 60 -i 5
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.16.6.2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  5] local 172.16.6.1 port 64827 connected with 172.16.6.2 port 5001
[  4] local 172.16.6.1 port 5001 connected with 172.16.6.2 port 61729
[  5]  0.0- 5.0 sec  33.8 MBytes  56.8 Mbits/sec
[  4]  0.0- 5.0 sec  52.1 MBytes  87.5 Mbits/sec
[  5]  5.0-10.0 sec  41.2 MBytes  69.2 Mbits/sec
[  4]  5.0-10.0 sec  44.0 MBytes  73.8 Mbits/sec
[  5] 10.0-15.0 sec  44.2 MBytes  74.2 Mbits/sec
[  4] 10.0-15.0 sec  43.2 MBytes  72.5 Mbits/sec
[  5] 15.0-20.0 sec  41.7 MBytes  70.0 Mbits/sec
[  4] 15.0-20.0 sec  46.0 MBytes  77.1 Mbits/sec
[  4] 20.0-25.0 sec  44.5 MBytes  74.7 Mbits/sec
[  5] 20.0-25.0 sec  43.4 MBytes  72.8 Mbits/sec
[  5] 25.0-30.0 sec  40.7 MBytes  68.3 Mbits/sec
[  4] 25.0-30.0 sec  47.7 MBytes  80.0 Mbits/sec
[  5] 30.0-35.0 sec  44.4 MBytes  74.6 Mbits/sec
[  4] 30.0-35.0 sec  44.5 MBytes  74.7 Mbits/sec
[  5] 35.0-40.0 sec  40.7 MBytes  68.3 Mbits/sec
[  4] 35.0-40.0 sec  48.9 MBytes  82.1 Mbits/sec
[  5] 40.0-45.0 sec  44.3 MBytes  74.3 Mbits/sec
[  4] 40.0-45.0 sec  45.7 MBytes  76.6 Mbits/sec
[  4] 45.0-50.0 sec  46.8 MBytes  78.5 Mbits/sec
[  5] 45.0-50.0 sec  43.4 MBytes  72.8 Mbits/sec
[  5] 50.0-55.0 sec  42.6 MBytes  71.6 Mbits/sec
[  4] 50.0-55.0 sec  48.4 MBytes  81.2 Mbits/sec
[  5] 55.0-60.0 sec  45.3 MBytes  75.9 Mbits/sec
[  5]  0.0-60.0 sec    506 MBytes  70.7 Mbits/sec
[  4] 55.0-60.0 sec  46.0 MBytes  77.2 Mbits/sec
[  4]  0.0-60.0 sec    558 MBytes  78.0 Mbits/sec

Here is the server's view.


poweredge:/root# iperf -s -B 172.16.6.2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 172.16.6.2
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
bind failed: Address already in use
[  4] local 172.16.6.2 port 5001 connected with 172.16.6.1 port 64827
------------------------------------------------------------
Client connecting to 172.16.6.1, TCP port 5001
Binding to local address 172.16.6.2
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  6] local 172.16.6.2 port 61729 connected with 172.16.6.1 port 5001
[  6]  0.0-60.0 sec    558 MBytes  78.0 Mbits/sec
[  4]  0.0-60.0 sec    506 MBytes  70.7 Mbits/sec

Throughput is about half the previous, which makes sense because we are sending data in two directions.

Here is a snapshot of asa633's top output.


last pid:   868;  load averages:  0.34,  0.16,  0.08    up 0+01:09:33  15:55:14
27 processes:  2 running, 25 sleeping
CPU states:  1.2% user,  0.0% nice, 43.0% system, 54.7% interrupt,  1.2% idle
Mem: 8916K Active, 5848K Inact, 18M Wired, 10M Buf, 272M Free
Swap: 640M Total, 640M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  852 root        2 101    0  3112K  1852K RUN      0:10  0.00% iperf

Here is poweredge.


last pid:   739;  load averages:  0.49,  0.19,  0.10    up 0+00:59:47  15:55:44
34 processes:  2 running, 32 sleeping
CPU states:  1.9% user,  0.0% nice, 36.3% system, 61.8% interrupt,  0.0% idle
Mem: 31M Active, 8772K Inact, 20M Wired, 12M Buf, 434M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  722 root        3 103    0  3120K  1836K RUN      0:19  0.00% iperf
  507 mysql       5  20    0 57280K 26256K kserel   0:07  0.00% mysqld

Again, high interrupts. Let's try a undirectional UDP test.


asa633:/root# iperf -c 172.16.6.2 -u -t 60 -i 5 -b 500M
------------------------------------------------------------
Client connecting to 172.16.6.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.1 port 61919 connected with 172.16.6.2 port 5001
[  3]  0.0- 5.0 sec    131 MBytes    220 Mbits/sec
[  3]  5.0-10.0 sec    132 MBytes    221 Mbits/sec
[  3] 10.0-15.0 sec    131 MBytes    221 Mbits/sec
[  3] 15.0-20.0 sec    131 MBytes    220 Mbits/sec
[  3] 20.0-25.0 sec    131 MBytes    220 Mbits/sec
[  3] 25.0-30.0 sec    131 MBytes    220 Mbits/sec
[  3] 30.0-35.0 sec    131 MBytes    220 Mbits/sec
[  3] 35.0-40.0 sec    132 MBytes    221 Mbits/sec
[  3] 40.0-45.0 sec    132 MBytes    221 Mbits/sec
[  3] 45.0-50.0 sec    132 MBytes    221 Mbits/sec
[  3] 50.0-55.0 sec    132 MBytes    221 Mbits/sec
[  3]  0.0-60.0 sec  1.54 GBytes    221 Mbits/sec
[  3] Sent 1125481 datagrams
[  3] Server Report:
[  3]  0.0-60.3 sec    793 MBytes    110 Mbits/sec  15.711 ms 560027/1125479 (50%)
[  3]  0.0-60.3 sec  1 datagrams received out-of-order

Here is the server's view.


poweredge:/root# iperf -s -u -B 172.16.6.2
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 172.16.6.2
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.2 port 5001 connected with 172.16.6.1 port 61919
[  3]  0.0-60.3 sec    793 MBytes    110 Mbits/sec  15.712 ms 560027/1125479 (50%)
[  3]  0.0-60.3 sec  1 datagrams received out-of-order

Check out the interrupt levels. First, the client, which shows the iperf process working hard to generate packets.


last pid:   914;  load averages:  0.64,  0.34,  0.18    up 0+01:20:43  16:06:24
27 processes:  2 running, 25 sleeping
CPU states:  5.4% user,  0.0% nice, 75.5% system, 19.1% interrupt,  0.0% idle
Mem: 8956K Active, 6288K Inact, 18M Wired, 10M Buf, 271M Free
Swap: 640M Total, 640M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  914 root        2  20    0  2764K  1752K ksesig   0:16 80.33% iperf

On the server, however, the interrupt level. Packets are being lost, as we saw in the server report earlier.


last pid:   767;  load averages:  0.79,  0.42,  0.21    up 0+01:10:51  16:06:48
34 processes:  2 running, 32 sleeping
CPU states:  4.1% user,  0.0% nice, 35.2% system, 60.3% interrupt,  0.4% idle
Mem: 31M Active, 8776K Inact, 20M Wired, 12M Buf, 434M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  767 root        3 110    0  2944K  1760K RUN      0:17  0.00% iperf

Let's see if device polling improves any of these numbers.

Using the technique explained here, I create this kernel:


kbld:/root# cat /usr/src/sys/i386/conf/GENERIC.POLLING 
include GENERIC
options         DEVICE_POLLING

Now I boot asa633 and poweredge using that kernel.


FreeBSD 6.1-RELEASE-p6 #0: Sun Sep 17 17:09:24 EDT 2006
    root@kbld.taosecurity.com:/usr/obj/usr/src/sys/GENERIC.POLLING

Enabling polling gives access to a set of new sysctl knobs.


asa633:/root# sysctl -a | grep poll
kern.polling.burst: 5
kern.polling.burst_max: 150
kern.polling.each_burst: 5
kern.polling.idle_poll: 0
kern.polling.user_frac: 50
kern.polling.reg_frac: 20
kern.polling.short_ticks: 0
kern.polling.lost_polls: 0
kern.polling.pending_polls: 0
kern.polling.residual_burst: 0
kern.polling.handlers: 0
kern.polling.enable: 0
kern.polling.phase: 0
kern.polling.suspect: 0
kern.polling.stalled: 0
kern.polling.idlepoll_sleeping: 1
hw.nve_pollinterval: 0

You don't need to change the value of kern.polling.enable. In fact, doing so generates an error, e.g. kern.polling.enable is deprecated. Use ifconfig(8).

Instead, use ifconfig polling.


asa633:/root# ifconfig em0 polling
asa633:/root# ifconfig em0
em0: flags=8843 mtu 1500
        options=4b
        inet6 fe80::204:23ff:feb1:64e2%em0 prefixlen 64 scopeid 0x3 
        inet 172.16.6.1 netmask 0xffffff00 broadcast 172.16.6.255
        ether 00:04:23:b1:64:e2
        media: Ethernet autoselect (1000baseSX )
        status: active

I enable polling on both boxes em0 interfaces.

Here are the test results. First, the client.


asa633:/root# iperf -c 172.16.6.2 -t 60 -i 5
------------------------------------------------------------
Client connecting to 172.16.6.2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.1 port 62829 connected with 172.16.6.2 port 5001
[  3]  0.0- 5.0 sec  90.5 MBytes    152 Mbits/sec
[  3]  5.0-10.0 sec    128 MBytes    214 Mbits/sec
[  3] 10.0-15.0 sec    125 MBytes    209 Mbits/sec
[  3] 15.0-20.0 sec    105 MBytes    176 Mbits/sec
[  3] 20.0-25.0 sec  83.7 MBytes    140 Mbits/sec
[  3] 25.0-30.0 sec  76.7 MBytes    129 Mbits/sec
[  3] 30.0-35.0 sec  78.1 MBytes    131 Mbits/sec
[  3] 35.0-40.0 sec    121 MBytes    203 Mbits/sec
[  3] 40.0-45.0 sec    126 MBytes    212 Mbits/sec
[  3] 45.0-50.0 sec    115 MBytes    192 Mbits/sec
[  3] 50.0-55.0 sec  91.9 MBytes    154 Mbits/sec
[  3] 55.0-60.0 sec  77.9 MBytes    131 Mbits/sec
[  3]  0.0-60.0 sec  1.19 GBytes    170 Mbits/sec

The server:


poweredge:/root# iperf -s -B 172.16.6.2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 172.16.6.2
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 172.16.6.2 port 5001 connected with 172.16.6.1 port 62829
[  4]  0.0-60.0 sec  1.19 GBytes    170 Mbits/sec

Compare that to the previous test result without device polling.


[  4]  0.0-60.0 sec  1.01 GBytes    144 Mbits/sec

We get better throughput here, but not the amazing improvement we'll see with UDP (later).

The interrupt counts are much better. Here's the client.


last pid:   693;  load averages:  0.22,  0.07,  0.05    up 0+00:12:48  16:23:54
27 processes:  2 running, 25 sleeping
CPU states:  7.4% user,  0.0% nice, 59.1% system, 32.7% interrupt,  0.8% idle
Mem: 8928K Active, 5680K Inact, 17M Wired, 8928K Buf, 273M Free
Swap: 640M Total, 640M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  693 root        2 105    0  2768K  1684K RUN      0:08  0.00% iperf

Check out the server!


last pid:   633;  load averages:  0.40,  0.16,  0.10    up 0+00:12:33  16:24:20
34 processes:  2 running, 32 sleeping
CPU states:  1.1% user,  0.0% nice, 27.0% system,  0.4% interrupt, 71.5% idle
Mem: 31M Active, 9168K Inact, 21M Wired, 13M Buf, 433M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  632 root        3 103    0  2956K  1824K RUN      0:12  0.00% iperf

That's amazing.

Let's try a dual test. The client:


asa633:/root# iperf -c 172.16.6.2 -d -t 60 -i 5
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.16.6.2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  5] local 172.16.6.1 port 58192 connected with 172.16.6.2 port 5001
[  4] local 172.16.6.1 port 5001 connected with 172.16.6.2 port 57738
[  5]  0.0- 5.0 sec  54.8 MBytes  91.9 Mbits/sec
[  4]  0.0- 5.0 sec  75.3 MBytes    126 Mbits/sec
[  5]  5.0-10.0 sec  56.5 MBytes  94.8 Mbits/sec
[  4]  5.0-10.0 sec  75.3 MBytes    126 Mbits/sec
[  5] 10.0-15.0 sec  55.7 MBytes  93.4 Mbits/sec
[  4] 10.0-15.0 sec  76.3 MBytes    128 Mbits/sec
[  5] 15.0-20.0 sec  48.0 MBytes  80.5 Mbits/sec
[  4] 15.0-20.0 sec  85.4 MBytes    143 Mbits/sec
[  5] 20.0-25.0 sec  43.7 MBytes  73.3 Mbits/sec
[  4] 20.0-25.0 sec  89.1 MBytes    150 Mbits/sec
[  5] 25.0-30.0 sec  46.3 MBytes  77.7 Mbits/sec
[  4] 25.0-30.0 sec  83.6 MBytes    140 Mbits/sec
[  5] 30.0-35.0 sec  50.7 MBytes  85.1 Mbits/sec
[  4] 30.0-35.0 sec  80.8 MBytes    136 Mbits/sec
[  5] 35.0-40.0 sec  56.1 MBytes  94.2 Mbits/sec
[  4] 35.0-40.0 sec  75.1 MBytes    126 Mbits/sec
[  5] 40.0-45.0 sec  56.1 MBytes  94.2 Mbits/sec
[  4] 40.0-45.0 sec  76.4 MBytes    128 Mbits/sec
[  5] 45.0-50.0 sec  48.9 MBytes  82.0 Mbits/sec
[  4] 45.0-50.0 sec  84.4 MBytes    142 Mbits/sec
[  5] 50.0-55.0 sec  43.9 MBytes  73.6 Mbits/sec
[  4] 50.0-55.0 sec  91.0 MBytes    153 Mbits/sec
[  4]  0.0-60.0 sec    979 MBytes    137 Mbits/sec
[  5] 55.0-60.0 sec  44.6 MBytes  74.8 Mbits/sec
[  5]  0.0-60.0 sec    605 MBytes  84.6 Mbits/sec

The server:


poweredge:/root# iperf -s -B 172.16.6.2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 172.16.6.2
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
bind failed: Address already in use
[  4] local 172.16.6.2 port 5001 connected with 172.16.6.1 port 58192
------------------------------------------------------------
Client connecting to 172.16.6.1, TCP port 5001
Binding to local address 172.16.6.2
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  6] local 172.16.6.2 port 57738 connected with 172.16.6.1 port 5001
[  6]  0.0-60.0 sec    979 MBytes    137 Mbits/sec
[  4]  0.0-60.0 sec    605 MBytes  84.6 Mbits/sec

Compare those results with their non-device polling counterparts.


[  6]  0.0-60.0 sec    558 MBytes  78.0 Mbits/sec
[  4]  0.0-60.0 sec    506 MBytes  70.7 Mbits/sec

Here's the client top excerpt:


last pid:   697;  load averages:  0.21,  0.15,  0.09    up 0+00:16:02  16:27:08
27 processes:  2 running, 25 sleeping
CPU states:  5.1% user,  0.0% nice, 54.1% system, 40.9% interrupt,  0.0% idle
Mem: 9012K Active, 5688K Inact, 17M Wired, 8928K Buf, 272M Free
Swap: 640M Total, 640M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  697 root        2 108    0  3112K  1852K RUN      0:08  0.00% iperf

Server top excerpt:


last pid:   637;  load averages:  0.43,  0.21,  0.12    up 0+00:15:44  16:27:31
34 processes:  2 running, 32 sleeping
CPU states:  5.2% user,  0.0% nice, 56.3% system,  6.7% interrupt, 31.7% idle
Mem: 31M Active, 9168K Inact, 21M Wired, 13M Buf, 433M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  637 root        3 105    0  3120K  1868K RUN      0:15  0.00% iperf

Let's try the UDP tests again with device polling enabled. Here's the client side.


asa633:/root# iperf -c 172.16.6.2 -u -t 60 -i 5 -b 500M
------------------------------------------------------------
Client connecting to 172.16.6.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 9.00 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.1 port 54045 connected with 172.16.6.2 port 5001
[  3]  0.0- 5.0 sec    172 MBytes    289 Mbits/sec
[  3]  5.0-10.0 sec    173 MBytes    290 Mbits/sec
[  3] 10.0-15.0 sec    173 MBytes    290 Mbits/sec
[  3] 15.0-20.0 sec    173 MBytes    290 Mbits/sec
[  3] 20.0-25.0 sec    173 MBytes    290 Mbits/sec
[  3] 25.0-30.0 sec    174 MBytes    291 Mbits/sec
[  3] 30.0-35.0 sec    173 MBytes    291 Mbits/sec
[  3] 35.0-40.0 sec    173 MBytes    291 Mbits/sec
[  3] 40.0-45.0 sec    173 MBytes    291 Mbits/sec
[  3] 45.0-50.0 sec    173 MBytes    291 Mbits/sec
[  3] 50.0-55.0 sec    170 MBytes    284 Mbits/sec
[  3]  0.0-60.0 sec  2.02 GBytes    290 Mbits/sec
[  3] Sent 1478220 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec  1.94 GBytes    277 Mbits/sec  0.056 ms 62312/1478219 (4.2%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Here's the server side.


poweredge:/root# iperf -s -u -B 172.16.6.2
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 172.16.6.2
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
[  3] local 172.16.6.2 port 5001 connected with 172.16.6.1 port 54045
[  3]  0.0-60.0 sec  1.94 GBytes    277 Mbits/sec  0.056 ms 62312/1478219 (4.2%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Compare that to the results from the test without device polling.


[  3]  0.0-60.3 sec    793 MBytes    110 Mbits/sec  15.712 ms 560027/1125479 (50%)

Because so few packets were dropped, throughput was much higher for UDP.

Here's the client top excerpt:


last pid:   705;  load averages:  1.16,  0.59,  0.29    up 0+00:21:27  16:32:33
27 processes:  2 running, 25 sleeping
CPU states:  9.0% user,  0.0% nice, 80.5% system, 10.5% interrupt,  0.0% idle
Mem: 8952K Active, 5684K Inact, 17M Wired, 8928K Buf, 273M Free
Swap: 640M Total, 640M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  705 root        2  20    0  2764K  1752K ksesig   0:24 89.08% iperf

Server top excerpt:


last pid:   650;  load averages:  0.48,  0.25,  0.16    up 0+00:21:01  16:32:48
34 processes:  2 running, 32 sleeping
CPU states:  7.9% user,  0.0% nice, 88.8% system,  0.4% interrupt,  3.0% idle
Mem: 31M Active, 9168K Inact, 21M Wired, 13M Buf, 433M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  650 root        3 120    0  2944K  1792K RUN      0:25  0.00% iperf

The incredibly low interrupt count explains why so fewer packets were dropped.

The only downside to device polling may be that your NIC might not support it. Check man 4 polling. This is one of the reasons I like to use Intel NICs -- they are bound to be supported and they perform well.

Nisley on Failure Analysis

Since I'm not a professional software developer, the only reason I pay attention to Dr. Dobb's Journal is Ed Nisley. I cited him earlier in Ed Nisley on Professional Engineering and Insights from Dr. Dobb's. The latest issue features Failure Analysis, Ed's look at NASA's documentation on mission failures. Ed writes:

[R]eviewing your projects to discover what you do worst can pay off, if only by discouraging dumb stunts.

What works for you also works for organizations, although few such reviews make it to the outside world. NASA, however, has done a remarkable job of analyzing its failures in public documents that can help the rest of us improve our techniques.

Documenting digital disasters has been a theme of this blog, although my request for readers to share their stories went largely unheeded. This is why I would like to see (and maybe create/lead) a National Digital Security Board.

Here are a few excerpts from Ed's article. I'm not going to summarize it; it takes about 5 minutes to read. These are the concepts I want to remember.

NASA defines the "root" cause of mishap as [a]long a chain of events leading to a mishap, the first causal action or failure to act that could have been controlled systematically either by policy/practice/procedure or individual adherence to policy/practice/procedure.

The root causes of these mishaps (incorrect units, invalid inputs, inverted G-switches) seem obvious in retrospect. How could anyone have possibly made those mistakes?

In addition to the root cause, the MIB Reports also identify a "contributing" cause as [a] factor, event or circumstance which led directly or indirectly to the dominant root cause, or which contributed to the severity of the mishap.

The "chain of events" is symptomatic of disasters. A break in that chain prevents the disaster.

However, the MIB [Mishap Investigation Board] discovered that [t]he Software Interface Specification (SIS) was developed but not properly used in the small forces ground software development and testing. End-to-end testing ... did not appear to be accomplished. (emphasis added)

Lack of end-to-end testing appears to be a common theme with disasters.

Mars, the Death Planet for spacecraft, might not have been the right venue for NASA's then-new "Faster, Better, Cheaper" mission-planning process...

The Mars Program Independent Assessment Team (MPIAT) Report pointed out that overall project management decisions caused the cascading series of failed verifications and tests. One slide of their report showed the MCO and MPL project constraints: Schedule, cost, science requirements, and launch vehicle were established constraints and margins were inadequate. The only remaining variable was risk.

In this context, "Faster" means flying more missions, getting rid of "non-value-added" work, and reducing the cycle time by working smarter rather than harder. "Cheaper" has the obvious meaning: spending less to get the same result. The MCO [Mars Climate Orbiter] and MPL [Mars Polar Lander] missions together cost less than the previous (successful) Mars Pathfinder mission.

The term "Better" has an amorphous definition, which I believe is the fundamental problem. In general, management gets what it measures and, if something cannot be measured, management simply won't insist on getting it.

You can easily demonstrate that you're doing things faster, that you've eliminated "non-value-added" operations, and that you're spending less money than ever before. You cannot show that those decisions are better (or worse), because the only result that really matters is whether the mission actually returns science data. Regrettably, you can measure that aspect of "better" after the fact and, in space, there are no do-overs. (emphasis added)

The last part is crucial. For digital security, the only result that really matters is whether you preserve confidentiality, integrity, and availability, usually by preventing and/or mitigating compromise. All the other stuff -- "percentage of systems certified and accredited," "percentage of systems with anti-virus applied," "percentage of systems with current patch levels" -- is absolutely secondary. In the Mars mission context, who cares if you build the spacecraft quicker, launch on time, and spend less money, if the vehicle crashes and the mission fails?

Thankfully NASA is taking steps to learn from its mistakes by investigating and documenting these disasters. It's time the digital security world learned something from these rocket scientists.

Using tap0 with Tcpreplay

This thread on the Wireshark mailing list brought up the issue of not being able to use Tcpreplay with the loopback interface on FreeBSD, e.g.:


orr:/root# tcpreplay -i lo0 /data/lpc/1.lpc
sending out lo0 
processing file: /data/lpc/1.lpc
Unable to send packet: Address family not supported by protocol family

Here is an alternative: use tap0.


orr:/root# ifconfig tap0
ifconfig: interface tap0 does not exist
orr:/root# dd if=/dev/tap0 of=/dev/null bs=1500 &
[1] 9468
orr:/root# ifconfig tap0 up
orr:/root# ifconfig tap0
tap0: flags=8843 mtu 1500
        inet6 fe80::2bd:1dff:fe2d:4d00%tap0 prefixlen 64 scopeid 0x5 
        ether 00:bd:1d:2d:4d:00
        Opened by PID 9468
orr:/root# tcpreplay -i tap0 /data/lpc/1.lpc
sending out tap0 
processing file: /data/lpc/1.lpc
^C
Actual: 71 packets (6860 bytes) sent in 6.15 seconds
Rated: 1115.0 bps, 0.01 Mbps/sec, 11.54 pps

In a second window, sniff with Tcpdump or whatever program you want:


orr:/root# tcpdump -n -i tap0 -s 1515
tcpdump: WARNING: tap0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 1515 bytes
10:25:16.211443 00:0d:28:6c:f5:4f > 01:00:0c:cc:cc:cd sap aa ui/C
10:25:17.567563 IP 192.168.2.5.2882 > 10.20.2.19.22:
 P 1293772727:1293772779(52) ack 478395919 win 64444

I discussed this in my first book and in my network security monitoring class.

Wednesday, September 20, 2006

Does SecureWorks-LURHQ Count as Consolidation?

I think it does. Managed network security services is one arena where size is always a factor, and bigger is usually better. With more employees you have more analysts per shift. You have more customers, so you see more of the Internet. With enough customers your view of the Internet begins to resemble a statistically significant sample, from which you can make inferences about the health of the global network.

I thought this Dark Reading story on the merger (the new company will be called SecureWorks -- no more "how do I say LURHQ?") had an interesting quote:

But all of this doesn't mean IBM-ISS isn't on SecureWorks' radar: Prince says SecureWorks' main competitors on the enterprise side are Symantec, VeriSign, and "now IBM." On the commercial side, it will be local telcos and other service providers, he says.

Where is Counterpane? They must be desperate for a buyer. I expect to see more MSSPs combining to form Voltron as time progresses.

Multiple Kernels on FreeBSD

The following is a topic I would enjoy hearing more about. If you have helpful suggestions, please share them as a comment.

Two years ago I described my experiences with building a FreeBSD userland and kernel on one system and installing it on another. I found myself in the same situation recently, where I didn't want to sit around waiting for a couple slow boxes to build themselves custom kernels. I wanted to build the custom kernel on a fast box and use it on the slower boxes. I didn't want to replace the default kernel on any of the boxes. I wanted the new kernel(s) to be additional boot-time options.

This post gave me the answer I needed. Here's how I applied it.

I wanted to build a GENERIC-style kernel, but with security updates applied. First I installed cvsup-without-gui as a package. Next I created this /usr/local/etc/security-supfile file:


*default host=cvsup5.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_6_1
*default delete use-rel-suffix

*default compress

src-all

This would update my kernel sources and userland to the SECURITY branch effective the time I ran cvsup (next).


kbld# cvsup -g -L 2 /usr/local/etc/security-supfile
Parsing supfile "/usr/local/etc/security-supfile"
Connecting to cvsup5.FreeBSD.org
Connected to cvsup5.FreeBSD.org
Server software version: SNAP_16_1h
Negotiating file attribute support
Exchanging collection information
Establishing multiplexed-mode data connection
Running
Updating collection src-all/cvs
 Edit src/UPDATING
  Add delta 1.416.2.22.2.3 2006.05.31.22.31.41 cperciva
  Add delta 1.416.2.22.2.4 2006.06.14.15.59.27 cperciva
  Add delta 1.416.2.22.2.5 2006.07.07.07.25.21 cperciva
  Add delta 1.416.2.22.2.6 2006.08.23.22.02.25 cperciva
...edited...
Shutting down connection to server
Finished successfully

Next I created the file GENERIC.SECURITY in /usr/src/sys/i386/conf with the following:


include GENERIC

All that does is make GENERIC.SECURITY the same kernel as GENERIC, except with patches applied. At this point you might think I should just update the GENERIC kernel. I could do that, but I'm using this method because later steps show this system works best for my requirements.

Now I can build the kernel.


kbld# cd /usr/src
kbld# make buildkernel KERNCONF=GENERIC.SECURITY INSTKERNNAME=GENERIC.SECURITY
--------------------------------------------------------------
>>> Kernel build for GENERIC.SECURITY started on Wed Sep 20 19:54:46 EDT 2006
--------------------------------------------------------------
===> GENERIC.SECURITY
mkdir -p /usr/obj/usr/src/sys

--------------------------------------------------------------
>>> stage 1: configuring the kernel
--------------------------------------------------------------
...truncated...
--------------------------------------------------------------
>>> Kernel build for GENERIC.SECURITY completed on Wed Sep 20 20:12:42 EDT 2006
--------------------------------------------------------------

Next I installed the kernel.


kbld:/usr/src# make installkernel KERNCONF=GENERIC.SECURITY INSTKERNNAME=GENERIC.SECURITY
--------------------------------------------------------------
>>> Installing kernel
--------------------------------------------------------------
...edited...
kldxref /boot/GENERIC.SECURITY
kbld:/usr/src#

That's it. I make sure host kbld is exporting the appropriate directories via NFS by creating this /etc/exports file:


/usr    -alldirs

That's too loose but this is sufficient for my test network.

Now I move from the kernel builder to a slow system where I would like to make GENERIC.SECURITY available. 192.168.2.103 is kbld, where the new kernel is waiting.


asa633:/root# mount_nfs 192.168.2.103:/usr/src /usr/src
asa633:/root# mount_nfs 192.168.2.103:/usr/obj /usr/obj
asa633:/root# mount
/dev/ad0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad0s1f on /home (ufs, local, soft-updates)
/dev/ad1s1d on /nsm (ufs, local, soft-updates)
/dev/ad0s1g on /tmp (ufs, local, soft-updates)
/dev/ad0s1d on /usr (ufs, local, soft-updates)
/dev/ad0s1e on /var (ufs, local, soft-updates)
192.168.2.103:/usr/src on /usr/src (nfs)
192.168.2.103:/usr/obj on /usr/obj (nfs)
asa633:/usr/src# make installkernel KERNCONF=GENERIC.SECURITY INSTKERNNAME=GENERIC.SECURITY
--------------------------------------------------------------
>>> Installing kernel
--------------------------------------------------------------
...edited...
kldxref /boot/GENERIC.SECURITY

How do I get this GENERIC.SECURITY kernel to boot? If I were at the console at boot time, I could say 'boot GENERIC.SECURITY'. Since I am remote, I edit /boot/loader.conf to say


kernel=GENERIC.SECURITY

Now I reboot. After rebooting, I see the new kernel is installed:


asa633:/root# uname -a
FreeBSD asa633.taosecurity.com 6.1-RELEASE-p6 FreeBSD 6.1-RELEASE-p6 #0:
 Wed Sep 20 20:02:56 EDT 2006     
 root@kbld.taosecurity.com:/usr/obj/usr/src/sys/GENERIC.SECURITY  i386

Pretty easy. If I want to boot the default kernel, I remove the entry in /boot/loader.conf.

For example, asa633 is usually running the kernel provided by Colin Percival's FreeBSD-Update code:


asa633:/root# uname -a
FreeBSD asa633.taosecurity.com 6.1-SECURITY FreeBSD 6.1-SECURITY #0:
 Mon Aug 28 05:21:08 UTC 2006     
root@builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  i386

FreeBSD-Update, in fact, very nicely takes care of the latest security problems with Gzip:


asa633:/root# freebsd-update fetch
Fetching updates signature...
Fetching updates...
Fetching hash list signature...
Fetching hash list...
Examining local system...
Fetching updates...
/usr/bin/gunzip...
/usr/bin/gzcat...
/usr/bin/gzip...
/usr/bin/zcat...
Updates fetched

To install these updates, run: '/usr/local/sbin/freebsd-update install'
asa633:/root# freebsd-update install
Backing up /usr/bin/gunzip...
Installing new /usr/bin/gunzip...
Backing up /usr/bin/gzcat...
Recreating hard link from /usr/bin/gunzip to /usr/bin/gzcat...
Backing up /usr/bin/gzip...
Recreating hard link from /usr/bin/gunzip to /usr/bin/gzip...
Backing up /usr/bin/zcat...
Recreating hard link from /usr/bin/gunzip to /usr/bin/zcat...

Easy!

Changing Definitions of Network Security Monitoring

I first defined Network Security Monitoring in print through my contribution to the February 2003 book Hacking Exposed, 4th Edition. Prior to that I defined NSM in a December 2002 SearchSecurity Webcast. NSM probably became more recognized in my first book, where I repeated the same definition by writing "Network security monitoring is the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions."

I emphasized the role of indications and warning (I&W) because my Air Force intelligence background involved training specifically in that discipline. I recommend reading the last link above for additional insight into this approach.

Today, however, I reviewed some Department of Defense documentation that made me take a second look at my NSM definition. (You might say this proves I am not a slave to my prior writings. Then again, you won't ever hear me say a threat and a vulnerability are the same!)

I&W is defined as those intelligence activities intended to detect and report time-sensitive intelligence information on foreign developments that could involve a threat to the United States or allied and/or coalition military, political, or economic interests or to US citizens abroad. It includes forewarning of enemy actions or intentions; the imminence of hostilities; insurgency; nuclear/nonnuclear attack on the United States, its overseas forces, or allied and/or coalition nations; hostile reactions to US reconnaissance activities; terrorists' attacks; and other similar events. Also called I&W. See also information; intelligence.

Note the heavy emphasis on gaining intelligence on threats, namely their capabilities and intentions.

While reading a DoD document, I came across the term attack sensing and warning (AS&W), with which I was only vaguely familiar. AS&W is defined as the detection, correlation, identification and characterization of cyber attacks across a large spectrum coupled with the notification to command and decision makers so that an appropriate response can be developed. Attack sensing and warning also includes attack/intrusion related intelligence collection tasking and dissemination; limited immediate response recommendations; and limited potential impact assessments.

I have a feeling that AS&W might be derived from Army operations. A friend previously part of 1st Information Operations Command worked that unit's AS&W mission.

Looking at the AS&W definition, it seems more appropriate within the context of NSM than I&W. I haven't decided how I'll define NSM in my next book or major paper, but I will keep AS&W at the forefront of my thoughts.

Differentiating Among Assessment Services

Tate Hansen of Clear Net Security provides a great methodology for differentiating among vulnerability assessment and related network security services. Check out his flow chart and then see how your own provider compares.

Review of IPv6 Essentials Posted

Amazon.com just posted my five star review of IPv6 Essentials, 2nd Ed by Sylvia Hagen. From the review:

I read and reviewed IPv6 Network Administration (INA) in August 2005 and Running IPv6 (RI) in January 2006. I gave those books 5 stars, so I had high expectations for "IPv6 Essentials, 2nd Ed" (IE2E). INA and RI are very hands-on, implementation-specific books. IE2E is more concerned with explaining protocols and IPv6 features. In this respect, IE2E is the perfect complement to INA and RI.

My full review mentions IPv6 critiques by Daniel Bernstein and Todd Underwood. I intend to take a closer look at SEcure Neighbor Discovery (SEND) (RFC 3971) and Cryptographically Generated Addresses (CGA) (RFC 3972) after reading about attacks upon stateless autoconfiguration and duplicate address detection, which appear in IPv6 Neighbor Discovery (ND) Trust Models and Threats (RFC 3756). Authentication for DHCP Messages (RFC 3118) can also be a concern, thanks to DHCPv6 Reconfigure Messages.

I also plan to read (.pdf) and watch (Google Video) Van Hauser's Attacking the IPv6 Protocol Suite, nicely summarized here.

Tuesday, September 19, 2006

SANS Network IPS Testing Webcast

I'm listening to a SANS Webcast on Trustworthy IPS Testing and Certification. Jack Walsh from the Network Intrusion Prevention section of ICSA Labs spoke for about 45 minutes on his testing system. Jack spent a decent amount of time discussing the Network IPS Corporate Certification Testing Criteria (.pdf) and vulnerabilities set (.xls). The vulnerabilities set was just updated a week ago, after being criticized in July.

At present only three products are ICSA Labs certified, according to the ICSA Web site and this press release. ICSA Lab certification is a pass/fail endeavor; there are no grades.

ICSA does not release the name of the companies whose products fail. Looking at the members of the NIPS Product Developers Consortium, you can make some guesses about who participated.

Vendors pay for testing. They do so by paying for a year-long testing period, during which time they will receive at least one "full battery" of testing. Tests are rerun when the vulnerability set is updated or when then attacks used to exploit vulnerabilities change. Although ICSA Labs publishes the vulnerabilities they test, they do not say specifically how they exploit the vulnerabilities. Jack said they do use Metasploit, Core Impact, and home-grown programs. ICSA Labs relies on running real captured network traffic through a NIPS, during which they inject captured attack traffic.

I found the Webcast informative. I was surprised that Jack was so insistent that NIPS provide "mitigation" for denial of service attacks. I don't consider that an essential element of NIPS activity.

Looking at the vulnerability set, they appear to be dominated by "traditional" vulnerabilities, namely weaknesses in services running on servers. You will not see application-layer vulnerabilities like cross-site scripting, for example.

A competitor to ICSA Labs is NSS, who just announced their NSS Group IPS Testing Methodology V4.0 (060731) (.pdf) and a Certified IPS Products list.

How the FCC Handles Radio Denial of Service

I am a licensed Amateur Radio operator, but I'm about as active as packet radio. Today, though, I read how the Federal Communications Commission handles those who interfere with radio transmissions.

It was a day a lot of radio amateurs in Southern California had been waiting for a long time. On September 18, US District Court Judge R. Gary Klausner sentenced convicted radio jammer Jack Gerritsen, now 70, to seven years imprisonment and imposed $15,225 in fines on six counts -- one a felony -- that included transmitting without a license and willful and malicious interference with radio transmissions. Before sentencing, Gerritsen apologized to the federal government, the FCC and the local Amateur Radio community, which had endured the brunt of Gerritsen's on-air tirades and outright jamming.

Wow -- seven years in prison with a felony conviction. No wonder my Dad used to warn me about broadcasting without a license.

Suggestions for Testing Bypass Switches

I've acquired a number of bypass devices for testing in the TaoSecurity labs. I'd like to know if any of you have requests to know more about these devices. In other words, how would you like me to test them?

The devices in question include the following.

Shore Micro SM-2400 Programmable Bypass Switch: This device has TX copper connectors and may support Gigabit Ethernet.

Optical Bypass Switch with Heartbeat: This device has SX fiber connectors and supports Gigabit Ethernet.

10/100/1000 Bypass Switch with Heartbeat: This device has TX copper connectors and supports Gigabit Ethernet.

Interface Masters Niagara 2295RJ: This device has TX copper connectors and supports Gigabit Ethernet. I find it interesting that it does not require a power supply, but I wonder how it supports a heartbeat without power?

Niagara 2282: This is an internal NIC that acts as a bypass switch. It has SX fiber connectors and supports Gigabit Ethernet.

Niagara 2280: This is an internal NIC that acts as a bypass switch. It has SX fiber connectors and supports Gigabit Ethernet. I don't see functional differences between this NIC and the previous, but that is a preliminary assessment.

So those are the devices. This is how I intend to deploy them for testing.


traffic generator transmitter NIC

 |

bypass switch inbound NIC

bypass switch monitor NIC 1 --> sensor NIC 1

bypass switch monitor NIC 2 --> sensor NIC 2

bypass switch outbound NIC

 |

traffic generator receiver NIC

For the internal devices, I will have the internal NIC in the sensor feeding a second NIC in the same sensor.

At the moment my main goals are to fully understand how each device works, feature-wise. I plan to do some limited testing this week with the equipment on hand. Next week I plan to use commercial load generators to stress the devices.

Let me know as a comment on TaoSecurity Blog or email to richard [at] taosecurity.com if you have ideas regarding what I should do with these systems.

Teaching Possibilities in Australia

I've been invited to speak at the AusCERT Asia Pacific Information Technology Security Conference in Gold Coast, Australia. The conference takes place Sunday 20 May - Friday 25 May 2007.

I haven't decided if I will accept yet. I'd like to know if any TaoSecurity Blog readers in Australia, New Zealand, or nearby areas would be interested in attending a two (or maybe more) day class either directly before or after my presentation date (which is unknown right now).

I would need a location to host the training, in exchange for which I would provide two free seats for the hosting organization.

Is anyone interested in attending and/or hosting such a class? Please email training [at] taosecurity.com. I have to accept or decline the AusCERT invitation next week.

I am open to suggestions regarding the location of the class (if the Gold Coast is too remote) and the content of the class (Network Security Operations, TCP/IP Weapons School, etc.). Sydney is a possibility since I will fly through SYD on my way to and from BNE. Thank you.

Monday, September 18, 2006

Insider Threat Study

I received a copy of a study announced by ArcSight and conducted by the Ponemon Institute. I mention this for two reasons. One, it highlights issues regarding the meaning of security terms. Two, the content is worth a look.

First, the email I received bore the subject "Are Executives the Cause of Insider Threats?". I wondered if the study examined if executives were the parties with the intentions and capabilities to exploit weaknesses in assets. That's what a threat is, and a study that implied executives (and not corporate minions or IT staff) were the real problem would be noteworthy in its own right.

Near the beginning of the report I read the following:

The survey was sponsored by ArcSight, an enterprise security management company, and queried 461 respondents who are employed in corporate IT departments within U.S.-based organizations.

For purposes of this survey, we define the "insider threat" as the misuse or destruction of sensitive or confidential information, as well as IT equipment that houses this data, by employees, contractors and others.

They're actually talking about attacks caused by insiders, not "insider threats." Working with their language, an insider threat would be "those who misuse or destroy sensitive or confidential information, as well as IT equipment that houses this data."

The report continues:

"Insider threats occur because of human error such as mistakes, negligence, reckless behavior, and sometimes even corporate sabotage."

Not really. Insider threats take advantage of vulnerabilities caused by mistakes and negligence. Insider threats employ reckles behavior (if not truly intending to cause harm) or corporate sabotage (if intending to cause harm) as attack methods.

Our survey sought to answer the following three questions.

1. What are the root causes of insider threats and how do information security practitioners respond to this pervasive IT and business risk?

They actually mean "what are the root causes of vulnerabilities that are exploited by insider threats, and how to infosec practitioners mitigate risks?" To truly address root causes of insider threats, one would analyze the motivations of threats themselves, like greed, malice, etc.

2. What technologies, practices and procedures are employed by organizations to reduce or mitigate insider-related risks?

That's great. Risks is used appropriately.

3. What are the issues, challenges and possible impediments to effectively detecting and preventing insider threats?

I would say "detecting and preventing attacks by insider threats."

The following are the most salient findings in our study: Data breaches go unreported. While we seem to be inundated with reports of data breaches, we may not know the full extent of the problem. More than 78% of respondents said that there has been at least one and possibly more unreported insider-related security breaches within their company.

Wow, that's a lot. Let's look for evidence in the report.

Table 11 reports that over 78% of respondents know of an insider-related security incident that was not publicly disclosed.

Notice Table 11 asks "Do you know of an insider-related incident in your organization (or any other organization in your industry) which was not disclosed to the public or to law enforcement?" (emphasis added)

That 78% figure doesn't mean that "more than 78% of respondents said that there has been at least one and possibly more unreported insider-related security breaches within their company" at all! In fact, there could be zero unreported breaches in the surveyed companies, and all respondents answering "yes" could be pointing to the same incident at someone else's company.

This idea is backed up by the following finding:

Table 7 shows that over 59% of respondents believe that insider-related problems are more likely to occur outside of their departments or organizational units.

So almost 60% of respondents think problems are likely to happen someplace else. That reminds me of surveys that say parents think schools in general are poor, but the school their child attends is fine.

While I think there is some interesting data in the survey report, I would keep my analysis in mind while reading it.

ISSA NoVA Meeting Thursday

This Thursday is the next ISSA NoVA meeting. It will be held at the The MITRE Corporation in McLean, VA. The social hour starts at 1730 and the meeting starts at 1830. Dr. Gary McGraw is the speaker. I will probably bring his books so I can get them signed. RSVP as soon as possible.

Remember that the next NoVA Sec meeting is a week from Thursday.

Thoughts on Latest SANS Whitepaper

I read about the new SANS paper IT Security Industry Changes: Trouble on the Horizon (September 2006) (.pdf) in this NewsBites issue. Here are some excerpts and my reactions.

Over the past six months, SANS Technology Institute's Stephen Northcutt has been gathering data and stories from security managers in more than 100 US organizations searching for patterns in job changes of security managers and the consultants who support them. The research was triggered by multiple emails from security managers who were facing reorganizations. His conclusions, albeit preliminary, paint a worrisome picture of job prospects for ill-equipped security managers, but also offer promise of some opportunities for success and advancement.

That's an interesting project. Let's read more.

[S]enior executives began to feel more comfortable voicing their frustration that they were wasting money paying for hugely expensive people and compliance reports that probably were not needed and that often had no impact on their ability to stop attacks or avoid disclosure of private information. The senior executives pushed back on budget requests, asking exactly what they would get in decreased risk from each of the expenditures. When they got answers they didn't like, they looked for ways to reorganize. Numerous security managers were pushed out as their responsibilities were moved to IT operations or audit or risk management groups.

Stephen continues by implying that these fired security managers lacked real technical skills and could not do much more than write reports. However...

Government is the one area where soft security skills, like policy and report writing, are still in demand, both in security staff and in consultants. The US Congress and the White House passed and implemented legislation (the Federal Information Security Management Act) that rates federal agencies less on whether their systems are protected from attack and more on whether agencies have written security evaluation reports for every system. Consulting firms have gotten rich writing those reports. One CEO reported that his firm had grown from three people doing security evaluations to 175 people writing FISMA reports, in just five years.

Does that make any else sick? It makes me ill.

Recent public disclosure of huge security failings, however, have caused government officials to review FISMA, particularly how it is implemented. Change seems to be in the air. That same CEO reported that 75% of his 175 FISMA folk today have soft skills and only 25% have solid technical security skills. He sees the need for that mix of skills to be reversed, within the next year or so, or the business "will dry up."

That is awesome. It probably explains why TaoSecurity continues to receive calls from firms inside-the-Beltway (and beyond) wishing to "team" to provide technical services to .gov clients, instead of just certification and accreditation.

Latest Sguil Scripts

I last talked about installing Sguil in March 2006. Over the last few weeks I've worked on the scripts I use for FreeBSD platforms, mainly as a response to changes in the various libraries and components. For example, Snort 2.6.0.2 is now available, replacing the Snort 2.4.x line.

The idea behind these scripts is to replace an English-text description of what to install where with a computer syntax version. If properly configured, these commands can set up everything you need for Sguil -- sensor, database, server, and client.

One of the major problems I've encountered is making good choices about libraries and components. The various Tcl libraries are on the fringes of support, compared to more popular packages. This makes it difficult to provide scripts that work without any real user modification. I decided the best I can do for the "run-it-without-looking" crowd is to let the scripts install (by default) packages shipped with FreeBSD 6.1 RELEASE, assuming you're running 6.1. If you know how to install using newer packages, you're free to set the right environment variable in the script and deal with the consequences.

The scripts I finished today are available at www.bejtlich.net/sguil_scripts_18sep06b.tar.gz.

They include:

sguil_sensor_install.sh: Set up Sguil sensor components.

snort_pkg_install.sh or snort_src_install.sh: Run one or the other. These set up Snort and Barnyard.

sguil_sensor_install_patch.sh: This patches configuration files that you should modify as listed in the README, namely sensor_agent.conf.patch, snort.conf.patch, barnyard.conf.patch, sancp.conf.patch, and log_packets.sh.patch.

sguil_database_install_pt1.sh: Set up MySQL database, part 1.

sguil_database_install_pt2.sh: Set up MySQL database, part 2.

sguil_server_install.sh: Set up Sguil server.

sguil_client_install.sh: Set up Sguil client.

If you are careful you can choose which scripts to run in order to have an all-in-one distribution or a separate box for every component (sensor, database, server, and client).

These are the prerequisites for the sensor and server.

The client box should be installed with the X packages. Otherwise, you can add the following manually:


# pkg_add -r xorg-server
# pkg_add -r xorg-clients
# pkg_add -r bitstream-vera
# pkg_add -r perl
# pkg_add -r xorg-fonts-100dpi
# pkg_add -r xorg-fonts-75dpi
# pkg_add -r xorg-fonts-miscbitmaps

All systems should have a user sguil and a user analyst to support the components.

The smoothest use of the scripts involves the following.

FreeBSD 6.1 RELEASE or SECURITY.

Sensor name is taosecurity.

Sensor is a VMware image with lnc0 interface for management and monitoring.

Avoid the snort_pkg_install.sh, since the Snort package with FreeBSD 6.1 RELEASE is Snort 2.4.x. That will not work as I have written the scripts. Use snort_src_install.sh instead.

Those suggestions will require the least number of modifications.

There is still a problem with this setup, however. The mysqltcl package shipped with FreeBSD 6.1 RELEASE requires mysql40-client as a dependency. I install mysql50-client prior, which conflicts with mysql40-client. This means the addition of mysqltcl as a package fails while running the sguil_server_install.sh script. Without mysqltcl, you can't start sguild, which means you can't add a sguil client user and thereby can't access Sguil.

You can work around this problem by retrieving the package from my Web server www.bejtlich.net/mysqltcl-3.01.tbz and adding it manually as root:


# pkg_add -v mysqltcl-3.01.tbz

You'll then need to add a sguil client user manually.


# cd /usr/local/src/sguil-0.6.1/server
# ./sguild -c sguild.conf -u sguild.users -adduser sguil
# cp sguild.users /usr/local/etc/nsm/
# chown sguil:sguil /usr/local/etc/nsm/sguild.users

You could also use the sguild_adduser.sh script which contains basically the same, including a LD_LIBRARY_PATH in the even you have trouble creating the sguil client user manually.


#!/bin/sh -x
SGUIL=sguil-0.6.1
LD_LIBRARY_PATH=/usr/local/lib/mysql
export LD_LIBRARY_PATH
cd /usr/local/src/$SGUIL/server/
./sguild -c sguild.conf -u sguild.users -adduser sguil
cp sguild.users /usr/local/etc/nsm/
chown sguil:sguil /usr/local/etc/nsm/sguild.users

Note that as an example of grappling with problems with the FreeBSD ports tree, the current databases/mysqltcl port is broken.

I recommend viewing the included README to see what you should run in order to get Sguil installed with all components on a single box.

If you have questions I strongly recommend posting them to sguil-users [at] lists.sourceforge.net. Any question I receive I usually send to the list. You may also find faq.sguil.net helpful.

It is important to realize that these scripts do not check to see what is installed prior to acting. Parts may fail if something is missing. If something is already installed (say, Tcl) then the pkg_add for a second instance of Tcl will fail -- but that won't cause any problems.

Please consider all Sguil installation guidance prior to this to be obsolete. This post and the scripts are probably not as clear as I would like, but this is free work and the time I have allocated for it is done!

SwitchProxy and Tor

I just wrote about Web Browsing with Tor. You might wonder if there's an easy way to switch to using Tor while running Firefox. I looked at the Torbutton extension, but then I found SwitchProxy. I like SwitchProxy because can you configure multiple proxies and decide when to use them.

If you click on the thumb image above you'll see me accessing a Hidden Service using Tor while I have Privoxy and Tor working together. Notice the URL -- http://6sxoyfb3h2nvok2d.onion/

I can just as easily switch to my production proxy, or even import a list of anonymous proxies and have SwitchProxy cycle through them every X seconds.

Installing Privoxy

A task I'm going to blog shortly recommends that I install Privoxy. I encounted some troubles using FreeBSD so I thought I would document them.

First I installed the package.


orr:/root# pkg_add -vr privoxy
...edited...
Running pre-install for privoxy-3.0.3_4..
extract: Package name is privoxy-3.0.3_4
extract: CWD to /usr/local
extract: /usr/local/man/man1/privoxy.1.gz
extract: /usr/local/sbin/privoxy
extract: /usr/local/etc/privoxy/config
extract: /usr/local/etc/privoxy/default.action
extract: /usr/local/etc/privoxy/default.filter
extract: /usr/local/etc/privoxy/trust
...edited...

***********************************************************
** Before running privoxy you must modify the file       **
** /usr/local/etc/privoxy/config                         **
**                                                       **
** Start privoxy with:                                   **
** /usr/local/sbin/privoxy /usr/local/etc/privoxy/config **
**                                                       **
** For documentation see:                                **  
** /usr/local/share/doc/privoxy-manual or 'man privoxy'  **
***********************************************************

Next I enabled Privoxy in /etc/rc.conf.


orr:/root# echo "privoxy_enable=YES" >> /etc/rc.conf

Next I tried starting Privoxy. I ran into some problems that I fixed with the following:


orr:/usr/local/etc/rc.d# mkdir /var/run/privoxy
orr:/usr/local/etc/rc.d# chown privoxy:privoxy /var/run/privoxy
orr:/usr/local/etc/rc.d# mkdir /var/log/privoxy
orr:/usr/local/etc/rc.d# chown privoxy:privoxy /var/log/privoxy

Here's what Privoxy looks like while running.


orr:/usr/local/etc/rc.d# ./privoxy start
Starting privoxy.
orr:/usr/local/etc/rc.d# sockstat -4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
privoxy  privoxy    40053 3  tcp4   127.0.0.1:8118        *:*
richard  firefox-bi 37850 22 tcp4   192.168.2.5:62936     66.249.83.83:80
richard  ssh        691   3  tcp4   192.168.2.5:49499     172.16.3.2:22
root     sendmail   468   4  tcp4   127.0.0.1:25          *:*
root     sshd       462   4  tcp4   *:22                  *:*
root     syslogd    320   7  udp4   *:514                 *:*

So what is this good for? Well, now that I have Privoxy listening on port 8118 TCP I can point my Web browser toward it. I tell Firefox to use localhost port 8118 and now all my Web requests use Privoxy.

I can test the difference between normal Web browsing and Privoxy Web browsing by visiting http://config.privoxy.org/show-status. It shows information like the following.

Show-Request

Here you see the original headers that your client sent when requesting this page, along with the headers that Privoxy would have sent to the remote server if this request hadn't been intercepted.

Original Client Request:

GET http://config.privoxy.org/show-request HTTP/1.1
Host: config.privoxy.org
User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.0.7)
 Gecko/20060917 Firefox/1.5.0.7
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,
 text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Referer: http://config.privoxy.org/show-status
If-Modified-Since: Mon, 18 Sep 2006 15:25:41 GMT
Cache-Control: max-age=0

Processed Request:

GET /show-request HTTP/1.1
Host: config.privoxy.org
User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.0.7) 
 Gecko/20060917 Firefox/1.5.0.7
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,
 text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Referer: http://config.privoxy.org/
If-Modified-Since: Mon, 18 Sep 2006 15:25:41 GMT
Cache-Control: max-age=0
X-Actions-File-Version: 1.8
Connection: close

This doesn't appear to be a big deal, but I'm using Privoxy's default configuration. In my next post I'll combine Privoxy with Tor to facilitate (but not guarantee) anonymous Web browsing.

Web Browsing with Tor

In my Installing Privoxy post I said I needed to install Privoxy for a certain task. I decided to use Privoxy with Tor to facilitate anonymous Web browsing.

First I installed Tor via package.


orr:/root# pkg_add -vr tor
...edited...
Package 'tor-0.1.1.23' depends on 'tsocks-1.8.b5_3' with 'net/tsocks' origin.
...edited...
extract: Package name is tsocks-1.8.b5_3
extract: CWD to /usr/local
extract: /usr/local/man/man1/tsocks.1.gz
extract: /usr/local/man/man5/tsocks.conf.5.gz
extract: /usr/local/man/man8/tsocks.8.gz
extract: /usr/local/bin/tsocks
extract: /usr/local/etc/tsocks.conf.sample
extract: /usr/local/lib/libtsocks.so.1
extract: /usr/local/lib/libtsocks.so
extract: /usr/local/share/examples/tsocks/tsocks.conf.complex.example
extract: /usr/local/share/examples/tsocks/tsocks.conf.simple.example
extract: /usr/local/share/examples/tsocks/README
...edited...
Package 'tor-0.1.1.23' depends on 'libevent-1.2' with 'devel/libevent' origin.
 - already installed.
Running pre-install for tor-0.1.1.23..
Added group "_tor".
Added user "_tor".
extract: Package name is tor-0.1.1.23
extract: CWD to /usr/local
extract: /usr/local/man/man1/tor.1.gz
extract: /usr/local/man/man1/tor-resolve.1.gz
extract: /usr/local/man/man1/torify.1.gz
extract: /usr/local/bin/tor
extract: /usr/local/bin/tor-resolve
extract: /usr/local/bin/torify
extract: /usr/local/etc/tor/tor-tsocks.conf.sample
extract: /usr/local/etc/tor/torrc.sample
extract: CWD to /usr/local
extract: /usr/local/etc/rc.d/tor
...edited.
================================================================================
To enable the tor server, set tor_enable="YES" in your /etc/rc.conf
and edit /usr/local/etc/tor/torrc.  Also note that the rc.subr script overrides
many torrc options and is tunable.  See /usr/local/etc/rc.d/tor.sh for details
================================================================================
...truncated...

Next I made a copy of the config file and enabled Tor's startup script.


orr:/root# cp /usr/local/etc/tor/torrc.sample /usr/local/etc/tor/torrc
orr:/root# echo "tor_enable=YES" >> /etc/rc.conf

Finally I told Privoxy to accept connections and send them to Tor, which would listen on port 9050 TCP.


orr:/root# echo "forward-socks4a / localhost:9050 ." >> /usr/local/etc/privoxy/config

Using SOCKS4A means my local host will not make DNS requests. Instead, they will be made by the SOCKS server (ostensibly through Tor).

Thanks to this guide for help!

Now I start Privoxy.


orr:/root# /usr/local/etc/rc.d/privoxy start
Starting privoxy.

Finally I start Tor.


orr:/root# /usr/local/etc/rc.d/tor start
/usr/local/etc/rc.d/tor: WARNING: /var/db/tor is not a directory.

That's no good. I make the required directory. (Why isn't that a default?)


orr:/root# mkdir /var/db/tor
orr:/root# /usr/local/etc/rc.d/tor start
Starting tor.
Sep 18 10:50:59.336 [notice] Tor v0.1.1.23. This is experimental software. 
 Do not rely on it for strong anonymity.
Sep 18 10:50:59.346 [notice] Initialized libevent version 1.2 using method kqueue. Good.
Sep 18 10:50:59.348 [warn] /var/db/tor is not owned by this user (_tor, 256) but by root (0). 
 Perhaps you are running Tor as the wrong user?
Sep 18 10:50:59.349 [warn] Failed to parse/validate config: Couldn't access/create private data 
 directory "/var/db/tor"
Sep 18 10:50:59.350 [err] tor_init(): Reading config failed--see warnings above. For usage, try -h.

Shoot. I need to let the _tor user access the directory I just made.


orr:/root# chown _tor:_tor /var/db/tor

Now I start Tor.


orr:/root# /usr/local/etc/rc.d/tor start
Sep 18 11:12:06.587 [notice] Tor v0.1.1.23. This is experimental software.
 Do not rely on it for strong anonymity.
Sep 18 11:12:06.597 [notice] Initialized libevent version 1.2 using method kqueue. Good.
Sep 18 11:12:06.597 [notice] connection_create_listener(): Opening Socks listener on 
 127.0.0.1:9050
Sep 18 11:12:06.600 [warn] options_init_logs(): Can't log to stdout with RunAsDaemon set; 
 skipping stdout

Let's see what's listening.


orr:/root# sockstat -4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
_tor     tor        39325 4  tcp4   192.168.2.5:57518     62.35.214.207:9030
_tor     tor        39325 5  tcp4   127.0.0.1:9050        *:*
_tor     tor        39325 6  tcp4   192.168.2.5:56850     70.32.145.204:9001
_tor     tor        39325 8  tcp4   192.168.2.5:64675     218.189.210.17:4806
root     privoxy    39312 3  tcp4   127.0.0.1:8118        *:*
richard  ssh        691   3  tcp4   192.168.2.5:49499     172.16.3.2:22
root     sendmail   468   4  tcp4   127.0.0.1:25          *:*
root     sshd       462   4  tcp4   *:22                  *:*
root     syslogd    320   7  udp4   *:514                 *:*

Now I configure my Web browser to connect to port 8118 (where Privoxy is listening), and Privoxy will send my traffic to port 9050 TCP where Tor is listening.

Now if I browse to a site like whatismyip.com I get a result like 195.71.8.10, which is plug.rfc822.org.

You can see Tor node status at sites like serifos.eecs.harvard.edu/cgi-bin/exit.pl and node2.xenobite.eu/torstat.php.

Thursday, September 14, 2006

Simple Tiny Network Name Services

A great way to start a religious war is to discuss domain name services. I previously documented my experiences with BIND 9 on FreeBSD, and I really didn't want to repeat the process for my small lab network.

Looking in the ports tree I found Dnsmasq, "a lightweight, easy to configure DNS forwarder and DHCP server. It is designed to provide DNS and, optionally, DHCP, to a small network." Wow, that sounds perfect (but I don't need DHCP).

I decided to try this on a Debian host that had a fully populated /etc/hosts file.


macmini:~# apt-get install dnsmasq
Reading Package Lists... Done
Building Dependency Tree... Done
Suggested packages:
  resolvconf
The following NEW packages will be installed:
  dnsmasq
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 114kB of archives.
After unpacking 303kB of additional disk space will be used.
Get:1 http://mirrors.kernel.org stable/main dnsmasq 2.22-2 [114kB]
Fetched 114kB in 1s (78.8kB/s)  
Selecting previously deselected package dnsmasq.
(Reading database ... 13695 files and directories currently installed.)
Unpacking dnsmasq (from .../dnsmasq_2.22-2_powerpc.deb) ...
Setting up dnsmasq (2.22-2) ...
Starting DNS forwarder and DHCP server: dnsmasq.

macmini:/etc/init.d# netstat -natup | grep dnsmasq
tcp        0      0 0.0.0.0:53              0.0.0.0:*     LISTEN     3279/dnsmasq        
tcp6       0      0 :::53                   :::*          LISTEN     3279/dnsmasq        
udp        0      0 0.0.0.0:32770           0.0.0.0:*                3279/dnsmasq        
udp        0      0 0.0.0.0:53              0.0.0.0:*                3279/dnsmasq        
udp6       0      0 :::53                   :::*                     3279/dnsmasq

Note that by default, no DHCP server is started.

That's it. Now I point all my hosts to the IP address of this Debian box, and it resolves local and remote IPs. I made sure the Debian host had my ISP's DNS servers in its /etc/resolv.conf file. Easy.

Wednesday, September 13, 2006

Encrypted Snapshotted Remote Backup

I just read about Colin Percival's idea for an encrypted snapshotted remote backup service.

Please read his post for more information. Colin would like to know if you would find such a service useful. He appreciates any feedback you send, and since he reads this blog he will see comments posted here. Thank you.

Monday, September 11, 2006

TaoSecurity and USENIX Special Event

Are you attending days one and two of TCP/IP Weapons School at the USENIX LISA 2006 conference on 3 and 4 December, 2006? Do you want to do something with Ethereal/Wireshark besides inspecting normal traffic? Do you want to learn how networks can be abused and subverted, while analyzing the attacks methods and packets that make it happen? Are you ready for technical, packet-centric training that really matters?

If your answer to any of these questions is yes, join Richard Bejtlich for TCP/IP Weapons School Part 2 (TWS2), after USENIX LISA 2006, on 9 and 10 December 2006. TWS2 will be held at the Wardman Park Hotel, after LISA, in the same facility. It's the perfect way to finish LISA without leaving the hotel.

Days 1 and 2 of TCP/IP Weapons School, part of the official USENIX LISA Training Program, cover layers 1-3 of the TCP/IP stack. TWS2 is a separate but supported event, where day 3 addresses layer 4 and day 4 discusses layers 5-7. This training is unique and in-depth!

Registration fees are as follows. The first price column is for Non-LISA attendees. The second is for LISA attendees.

Register by 13 October 2006 $995/student $795/student (20% off)
Register by 10 November 2006 $1095/student $875/student (20% off)
Register after 10 November 2006 $1195/student $955/student (20% off)

To register for TWS2, please complete the registration form and submit it to TaoSecurity using one of the methods listed on the form.

I've also prepared a flyer to show your manager!

Don't forget to register for USENIX LISA 2006 before 10 November 2006 for the best deal. I will also teach Network Security Monitoring with Open Source Tools on 8 December 2006, right before I teach TWS2.

Friday, September 08, 2006

FreeBSD Minimum Memory Requirements

In my last post I mentioned installing FreeBSD 6.1 on a Pentium 200, specifically a Dell Dimenson XPS P200S. This box is so old-school it has that old bug that caused so much trouble for Intel. It has a "Windows 95" sticker on front and was built in 1996!


FreeBSD 6.1-SECURITY #0: Mon Aug 28 05:21:08 UTC 2006
    root@builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Pentium/P54C (199.43-MHz 586-class CPU)
  Origin = "GenuineIntel"  Id = 0x52c  Stepping = 12
  Features=0x1bf
real memory  = 33554432 (32 MB)
avail memory = 23379968 (22 MB)
Intel Pentium detected, installing workaround for F00F bug

Despite its age, CPU, and RAM, I am doing real work with it. Sure, I'm not serving Web pages or handling email, but I am testing IPv6. In fact, I just connected to IRC with it.


p200:/home/richard$ irssi -n helevius6 -c irc.ipv6.freenode.net
...edited...
22:14 -!- helevius6 [n=richard@2001:5c0:925d:0:204:5aff:fe79:43a7] has joined 
          #snort-gui

Using FreeBSD Update I installed all binary patches, and I am adding binary packages with pkg_add. The SSH session is very responsive. I could not run the latest version of Windows XP on this box, and Windows Vista is Right Out (TM).

If you want to see just how "low can you go," check out this great post by Nikolas Britton on FreeBSD's minimum memory requirements. I expect to see it in official documentation soon.

IPv6 Only FreeBSD Scenario

Earlier this year I described running Miredo on FreeBSD to gain access to the IPv6 Internet. Today I decided I would try to accomplish two goals. First, I would connect my FreeBSD gateway to the IPv6 Internet using Hexago/Freenet6 through the net/tspc2 port (Tunnel Setup Protocol Client). Second, I would deploy an IPv6-only host behind my FreeBSD gateway, and have it speak only IPv6 to the outside world.

I do not intend for this to be definitive by any means. Again, these are more or less personal notes. If someone else finds them useful, great.

First I registered with Hexago. This is not strictly necessary since anonymous access is apparently allowed. After registering I received an email with a username (I specified) and a password (provided) that I would add to the Tsp client. (I decided to try Tspc instead of manually deploying a tunnel because I heard Tspc was just too easy.)

After installing the net/tspc2 package, I literally added the information from the email to my /usr/local/etc/tspc.conf and started tspc2 manually.


mwmicro:/root# tspc -vvv
tspc - Tunnel Setup Protocol Client v2.1.1
Initializing (use -h for help)


Connecting to server with reliable udp
Using TSP protocol version 2.0.0
Establishing connection with tunnel broker...
Getting capabilities from server
Connection established
Authenticating taosecurity
Using authentification mecanism DIGEST-MD5
Authentication success
Asking for a tunnel
sent: Content-length: 204
<tunnel action="create" type="v6anyv4" proxy="no">
 <client>
  <address type="ipv4">69.143.202.28</address>
<keepalive interval="30"><address type="ipv6">::</address></keepalive> </client>
</tunnel>

recv:
200 Success
<tunnel action="info" type="v6v4" lifetime="604800">
  <server>
    <address type="ipv4">64.86.88.116</address>
    <address type="ipv6">2001:05c0:8fff:fffe:0000:0000:0000:5888</address>
  </server>
  <client>
    <address type="ipv4">69.143.202.28</address>
    <address type="ipv6">2001:05c0:8fff:fffe:0000:0000:0000:5889</address>
    <keepalive interval="30">
      <address type="ipv6">2001:05c0:8fff:fffe:0000:0000:0000:5888</address>
    </keepalive>
  </client>
</tunnel>


Processing response from server
sent: Content-length: 35
<tunnel action="accept"></tunnel>

Got tunnel parameters from server, setting up local tunnel
keepalive interval: 30

Going daemon, check /var/log/tspc.log for tunnel creation status

So far so good. Next I checked my routing table.


mwmicro:/root# netstat -nr -f inet6
Routing tables

Internet6:
Destination                       Gateway                       Flags      Netif Expire
default                           2001:5c0:8fff:fffe::5888      UGS        gif0
::1                               ::1                           UH          lo0
2001:5c0:8fff:fffe::5888          link#9                        UHL        gif0
2001:5c0:8fff:fffe::5889          link#9                        UHL         lo0
fe80::%sf1/64                     link#3                        UC          sf1
fe80::200:d1ff:feed:8c72%sf1      00:00:d1:ed:8c:72             UHL         lo0
fe80::%sf2/64                     link#4                        UC          sf2
fe80::200:d1ff:feed:8c73%sf2      00:00:d1:ed:8c:73             UHL         lo0
fe80::%fxp0/64                    link#6                        UC         fxp0
fe80::202:b3ff:fe0a:cd5e%fxp0     00:02:b3:0a:cd:5e             UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                   U           lo0
fe80::1%lo0                       fe80::1%lo0                   UHL         lo0
fe80::%gif0/64                    link#9                        UC         gif0
fe80::204:e2ff:fe29:4c3c%gif0     link#9                        UHL         lo0
ff01:3::/32                       link#3                        UC          sf1
ff01:4::/32                       link#4                        UC          sf2
ff01:6::/32                       link#6                        UC         fxp0
ff01:8::/32                       ::1                           UC          lo0
ff01:9::/32                       link#9                        UC         gif0
ff02::%sf1/32                     link#3                        UC          sf1
ff02::%sf2/32                     link#4                        UC          sf2
ff02::%fxp0/32                    link#6                        UC         fxp0
ff02::%lo0/32                     ::1                           UC          lo0
ff02::%gif0/32                    link#9                        UC         gif0

There's a lot of routes here, automatically created. This will be an issue for understanding IPv6 in the future. For now I was interested in the entries with gif0, for those represent the new tunnel.


mwmicro:/root# ifconfig gif0
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
        tunnel inet 69.143.202.28 --> 64.86.88.116
        inet6 2001:5c0:8fff:fffe::5889 --> 2001:5c0:8fff:fffe::5888 prefixlen 128 
        inet6 fe80::204:e2ff:fe29:4c3c%gif0 prefixlen 64 scopeid 0x9

I decided to see if I could ping6 the other end of the tunnel.


mwmicro:/root# ping6 2001:5c0:8fff:fffe::5888
PING6(56=40+8+8 bytes) 2001:5c0:8fff:fffe::5889 --> 2001:5c0:8fff:fffe::5888
16 bytes from 2001:5c0:8fff:fffe::5888, icmp_seq=0 hlim=64 time=26.540 ms

I could also ping6 an IPv6 host.


mwmicro:/root# ping6 www.6bone.net
PING6(56=40+8+8 bytes) 2001:5c0:8fff:fffe::5889 --> 2001:5c0:0:2::24
16 bytes from 2001:5c0:0:2::24, icmp_seq=0 hlim=61 time=32.894 ms

Note that I used IPv4 to resolve www.6bone.net:


21:05:41.961734 IP 69.143.202.28.61517 > 68.87.73.242.53:  59311+ AAAA? www.6bone.net. (31)
21:05:42.053465 IP 68.87.73.242.53 > 69.143.202.28.61517:  59311 2/0/0 CNAME 6bone.net., (73)

I wondered how I could resolve IPs using an IPv6-speaking DNS server. I hunted high and low for one that would respond to my queries. Finally someone in #ipv6 on Freenode mentioned that NetBSD's resolver pointed by default to 2001:240::1. Could I use that?


mwmicro:/root# host 2001:240::1
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0. 0.0.0.0.0.4.2.0.1.0.0.2.ip6.arpa
 domain name pointer ns9.iij.ad.jp.

2001:240::1 is ns9.iij.ad.jp. Count on the Japanese to have a working IPv6 system! Now will it resolve IPs?


mwmicro:/root# host www.6bone.net 2001:240::1
Using domain server:
Name: 2001:240::1
Address: 2001:240::1#53
Aliases:

www.6bone.net is an alias for 6bone.net.
6bone.net has address 206.162.147.152
Using domain server:
Name: 2001:240::1
Address: 2001:240::1#53
Aliases:

www.6bone.net is an alias for 6bone.net.
6bone.net has IPv6 address 2001:5c0:0:2::24
Using domain server:
Name: 2001:240::1
Address: 2001:240::1#53
Aliases:

www.6bone.net is an alias for 6bone.net.
6bone.net mail is handled by 10 quark.isi.edu.
6bone.net mail is handled by 20 darkstar.isi.edu.
6bone.net mail is handled by 0 venera.isi.edu.

Bingo. By the way, if you can suggest alternative IPv6 DNS servers, please leave a comment.

At this point I accomplished my first goal. On to the second. To get the gateway to work as an IPv6 gateway, I added the following to /usr/local/etc/tspc.conf:


#---------------------
# Router configuration
#
# In order to configure the machine as a router, a prefix must be requested
# and an interface must be specified.  The prefix will be advertised
# through that interface.
#
# host_type=host|router
#  default = host.
host_type=router

#
# prefixlen specifies the required prefix length for the TSP client 
#  network. Valid values are 64 or 48. 64 is for one link. 48 is for
#  a whole enterprise network (65K links).
prefixlen=48

#
# if_prefix is the name of the OS interface that will be configured
#  with the first /64 of the received prefix from the broker and the
#  router advertisement daemon is started to advertise that prefix
#  on the if_prefix interface.
if_prefix=sf3

Note sf3 is the internal interface, i.e., the one facing away from the Internet.


mwmicro:/root# ifconfig sf3
sf3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>mtu 1500
        inet6 fe80::200:d1ff:feed:8c74%sf3 prefixlen 64 scopeid 0x5 
        inet6 2001:5c0:925d::1 prefixlen 64 
        ether 00:00:d1:ed:8c:74
        media: Ethernet autoselect (10baseT/UTP)
        status: active

I also added these entries to /etc/rc.conf on my gateway:


ipv6_enable="YES"
ipv6_gateway_enable="YES"
rtadvd_enable="YES"
rtadvd_interfaces="sf3"
tspc2_enable="YES"

I next built a new FreeBSD host (on a P200 with 32 MB RAM, no less). The box did not have a working CD-ROM, so I had to use boot floppies. I found no easy way to do an IPv6-only network install, so I assigned a temporary IPv4 address for the network installation.

After installing FreeBSD, I rebooted the p200 system and removed the IPv4 address. Now my interface looked like this:


p200:/home/richard$ ifconfig dc0
dc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=8<VLAN_MTU>
        inet6 fe80::204:5aff:fe79:43a7%dc0 prefixlen 64 scopeid 0x1 
        inet6 2001:5c0:925d:0:204:5aff:fe79:43a7 prefixlen 64 autoconf 
        ether 00:04:5a:79:43:a7
        media: Ethernet autoselect (10baseT/UTP)
        status: active

My routing tables on p200 looked like this:


p200:/home/richard$ netstat -nr -f inet6
Routing tables

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0 =>
default                           fe80::200:d1ff:feed:8c74%dc0  UG          dc0
::1                               ::1                           UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2001:5c0:925d::/64                link#1                        UC          dc0
2001:5c0:925d::1                  00:00:d1:ed:8c:74             UHLW        dc0
2001:5c0:925d:0:204:5aff:fe79:43a7 00:04:5a:79:43:a7             UHL         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%dc0/64                     link#1                        UC          dc0
fe80::200:d1ff:feed:8c74%dc0      00:00:d1:ed:8c:74             UHLW        dc0
fe80::204:5aff:fe79:43a7%dc0      00:04:5a:79:43:a7             UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                   U           lo0
fe80::1%lo0                       fe80::1%lo0                   UHL         lo0
ff01:1::/32                       link#1                        UC          dc0
ff01:3::/32                       ::1                           UC          lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%dc0/32                     link#1                        UC          dc0
ff02::%lo0/32                     ::1                           UC          lo0

I achieved setting a default route manually with


route add -inet6 2000::/3 2001:5c0:925d::1

where 2001:5c0:925d::1 is the IPv6 address of my gateway (remember the output for interface sf3 earlier).

I configured p200's /etc/rc.conf like so:


hostname="p200.taosecurity.com"
ipv6_enable="YES"
ipv6_defaultrouter="2001:5c0:925d::1"
sshd_enable="YES"

From my gateway, I could now reach p200 using either of its IPv6 addresses (local or global):


mwmicro:/root# ping6 -c 1 fe80::204:5aff:fe79:43a7%sf3
PING6(56=40+8+8 bytes) fe80::200:d1ff:feed:8c74%sf3 --> fe80::204:5aff:fe79:43a7%sf3
16 bytes from fe80::204:5aff:fe79:43a7%sf3, icmp_seq=0 hlim=64 time=1.184 ms

--- fe80::204:5aff:fe79:43a7%sf3 ping6 statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.184/1.184/1.184/0.000 ms

Note using the above local method requires specifying an interface (%sf3) out of which the ICMP6 echo is sent.


mwmicro:/root# ping6 -c 1 2001:5c0:925d:0:204:5aff:fe79:43a7
PING6(56=40+8+8 bytes) 2001:5c0:925d::1 --> 2001:5c0:925d:0:204:5aff:fe79:43a7
16 bytes from 2001:5c0:925d:0:204:5aff:fe79:43a7, icmp_seq=0 hlim=64 time=1.205 ms

--- 2001:5c0:925d:0:204:5aff:fe79:43a7 ping6 statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.205/1.205/1.205/0.000 ms

The ping to 2001:5c0:925d:0:204:5aff:fe79:43a7 does not require the same interface specification.

After I connected via SSH from the gateway to p200, i.e.


mwmicro:/root# ssh richard@2001:5c0:925d:0:204:5aff:fe79:43a7

I was able to perform IPv6-only actions from p200. For example:


p200:/home/richard$ ping6 -c 1 www.6bone.net
PING6(56=40+8+8 bytes) 2001:5c0:925d:0:204:5aff:fe79:43a7 --> 2001:5c0:0:2::24
16 bytes from 2001:5c0:0:2::24, icmp_seq=0 hlim=60 time=29.471 ms

--- 6bone.net ping6 statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 29.471/29.471/29.471/0.000 ms

Here is what the traffic looked like between p200 and the gateway:


21:24:56.729796 IP6 2001:5c0:925d:0:204:5aff:fe79:43a7.49166 > 2001:240::1.53: 
 39700+ AAAA? www.6bone.net. (31)
21:24:57.188971 IP6 2001:240::1.53 > 2001:5c0:925d:0:204:5aff:fe79:43a7.49166: 
 39700 2/2/2[|domain]
21:24:57.193397 IP6 2001:5c0:925d:0:204:5aff:fe79:43a7 > 2001:5c0:0:2::24:
 ICMP6, echo request, seq 0, length 16
21:24:57.222194 IP6 2001:5c0:0:2::24 > 2001:5c0:925d:0:204:5aff:fe79:43a7:
 ICMP6, echo reply, seq 0, length 16

I also caught neighbor soliciation and advertisements between the two hosts.


21:24:59.309014 IP6 2001:5c0:925d:0:204:5aff:fe79:43a7 > 2001:5c0:925d::1:
 ICMP6, neighbor solicitation, who has 2001:5c0:925d::1, length 32
21:24:59.309325 IP6 2001:5c0:925d::1 > 2001:5c0:925d:0:204:5aff:fe79:43a7:
 ICMP6, neighbor advertisment, tgt is 2001:5c0:925d::1, length 24
21:25:01.728706 IP6 fe80::204:5aff:fe79:43a7 > fe80::200:d1ff:feed:8c74:
 ICMP6, neighbor solicitation, who has fe80::200:d1ff:feed:8c74, length 32
21:25:01.729104 IP6 fe80::200:d1ff:feed:8c74 > fe80::204:5aff:fe79:43a7:
 ICMP6, neighbor advertisment, tgt is fe80::200:d1ff:feed:8c74, length 24

How cool is this -- public IPv6 NTP servers:


p200:/root# ntpdate ntp6.space.net
 8 Sep 21:28:49 ntpdate[599]: adjust time server 2001:608::1000:1 offset -0.045629 sec

Here is the traffic.


21:28:47.456604 IP6 2001:5c0:925d:0:204:5aff:fe79:43a7.49168 > 2001:240::1.53: 
 36503+ A? ntp6.space.net. (32)
21:28:48.240130 IP6 2001:5c0:925d:0:204:5aff:fe79:43a7 > 2001:5c0:925d::1:
 ICMP6, neighbor solicitation, who has 2001:5c0:925d::1, length 32
21:28:48.240433 IP6 2001:5c0:925d::1 > 2001:5c0:925d:0:204:5aff:fe79:43a7:
 ICMP6, neighbor advertisment, tgt is 2001:5c0:925d::1, length 24
21:28:48.280169 IP6 2001:240::1.53 > 2001:5c0:925d:0:204:5aff:fe79:43a7.49168: 
 36503 2/3/0[|domain]
21:28:48.281557 IP6 2001:5c0:925d:0:204:5aff:fe79:43a7.49169 > 2001:240::1.53: 
 36504+ AAAA? ntp6.space.net. (32)
21:28:48.847565 IP6 2001:240::1.53 > 2001:5c0:925d:0:204:5aff:fe79:43a7.49169: 
 36504 2/3/0[|domain]
21:28:48.972514 IP6 2001:5c0:925d:0:204:5aff:fe79:43a7.123 > 2001:608::1000:1.123:
 NTPv4, Client, length 48
21:28:49.115271 IP6 2001:608::1000:1.123 > 2001:5c0:925d:0:204:5aff:fe79:43a7.123:
 NTPv4, Server, length 48

Even FTP works.


p200:/root# ftp ftp.freebsd.org
Trying 2001:4f8:0:2::e...
Connected to ftp.freebsd.org.
220 Welcome to freebsd.isc.org.
Name (ftp.freebsd.org:richard): ftp
331 Please specify the password.
Password:
230-
230-You have reached the freebsd.isc.org FTP server, serving the 
230-full FreeBSD FTP archive over IPv4 (204.152.184.73) and IPv6 
230-(2001:4f8:0:2::e) networks.  This server is also known as:

I think the key to understanding IPv6 is to start running deployments like this and watching traffic. I'm remembering that's how I started learning IPv4 in September 1998.

I welcome any constructive tips!

Ifconfig Monitor Option

I've been playing with IPv6 in my lab. I'm not going to say anything earth-shattering or definitive about IPv6 here or in future posts. I'm at the learning stage, but I figured I would record what I'm seeing.

One of my observations is that IPv6 ships with multiple discovery-type protocols that appear on active interfaces. This is especially interesting for people who want to build "stealthy" network security monitoring sensors.

In the following example, consider em1 to be an interface we wish to use as a passive sniffing NIC. It doesn't make a difference, but I have em1 plugged into the output of a Gigabit port aggregator tap.


hacom:/root# ifconfig em1
em1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICASTɚmtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        ether 00:40:48:b1:5c:dc
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

Note that this is a full duplex connection. The SIMPLEX in the output has nothing to do with half or full duplex. SIMPLEX refers to the NIC not being able to transmit and receive while operating on true CSMA/CD Ethernet, which is really not the case with switched networks.

Next we bring up em1.


hacom:/root# ifconfig em1 up
hacom:/root# ifconfig em1
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        inet6 fe80::240:48ff:feb1:5cdc%em1 prefixlen 64 scopeid 0x2 
        ether 00:40:48:b1:5c:dc
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active

Notice that em1 assigns itself a local IPv6 interface, fe80::240:48ff:feb1:5cdc. What is more interesting to me is that it begins sending IPv6 traffic.


hacom:/root# tcpdump -n -i em1 -s 1515
tcpdump: WARNING: em1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 1515 bytes
20:04:33.081021 IP6 :: > ff02::1:ffb1:5cdc: ICMP6, neighbor solicitation,
 who has fe80::240:48ff:feb1:5cdc, length 24
20:04:33.081029 IP6 fe80::240:48ff:feb1:5cdc > ff02::2:202:5c8d:
 HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2:202:5c8d, length 24
20:04:33.081083 IP6 fe80::240:48ff:feb1:5cdc > ff02::1:ffb1:5cdc:
 HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ffb1:5cdc, length 24

One of the tenets of proper passive sensor deployment is not letting the sniffing NIC inject packets onto the monitored network. People like to talk about cutting the transmit wires on Ethernet cables, but I always thought that was silly.

When I see traffic emitted from a supposedly passive interface, I do not like what I see. In the deployment scenario I have here (listening to a tap), these ICMP6 packets are really no problem. This is one of the reasons professionals use passive taps.

If you're placing a sniffing interface on a shared medium like half-duplex Ethernet, a common trick to prevent traffic from leaving the sniffing interface (without cutting wires) is to disable ARP, like so.


hacom:/root# ifconfig em1
em1: flags=8802 mtu 1500
        options=b
        ether 00:40:48:b1:5c:dc
        media: Ethernet autoselect (1000baseTX )
        status: active
hacom:/root# ifconfig em1 -arp up
hacom:/root# ifconfig em1
em1: flags=88c3 mtu 1500
        options=b
        inet6 fe80::240:48ff:feb1:5cdc%em1 prefixlen 64 scopeid 0x2 
        ether 00:40:48:b1:5c:dc
        media: Ethernet autoselect (1000baseTX )
        status: active

Without the interface being able to send and receive ARP traffic (NOARP), it can't send traffic. Or can it?


hacom:/root# tcpdump -n -i em1 -s 1515
tcpdump: WARNING: em1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 1515 bytes
20:13:27.711251 IP6 :: > ff02::1:ffb1:5cdc: ICMP6, neighbor solicitation,
 who has fe80::240:48ff:feb1:5cdc, length 24
20:13:29.062884 IP6 fe80::240:48ff:feb1:5cdc > ff02::2:202:5c8d:
 HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2:202:5c8d, length 24

Wow, there are those pesky ICMP6 messages.

The reason we see them is that IPv6 does not use ARP.

Let's try one more variation.


hacom:/root# ifconfig em1
em1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        ether 00:40:48:b1:5c:dc
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
hacom:/root# ifconfig em1 monitor up
hacom:/root# ifconfig em1 
em1: flags=48943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,MONITOR> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        inet6 fe80::240:48ff:feb1:5cdc%em1 prefixlen 64 scopeid 0x2 
        ether 00:40:48:b1:5c:dc
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active

When I look for traffic now, I see no IPv6 traffic at all. I recommend that we replace the use of "-arp" with "monitor". To disable monitor mode, use "-monitor".

Bejtlich Returns for SANS CDI East 2006

It's been three years since I spoke at at SANS conference; I last presented at SANS NIAL in 2003. After some friendly discussions with SANS staff at the recent SANS Log Management Summit, we've arranged for me to present a special event for SANS Cyber Defense Initiative East -- a two evening course called Enterprise Network Instrumentation (ENI).

I developed ENI for a private client, but no public class has ever seen the material. I will be presenting ENI for two evenings, 14 and 15 December, 2006, from 6 to 9 pm at the Hilton Washington & Towers in Washington, DC.

ENI is all about solving the difficult problems associated with gaining access to network traffic. It seems every book (with a few exceptions) assumes it's easy to deploy sensors to observe packets. In reality, achieving visibility in modern networks can be extremely difficult. ENI will share recommendations and concrete solutions for the most taxing enterprise network instrumentation issues seen today.

You can register for ENI immediately. The fees look reasonable -- especially if you're already attending a day track. Please let me know if you have any questions.

Thursday, September 07, 2006

SANS SCADA Security Webcast

I just listened to a SANS SCADA Security webcast. I found the first half or so more interesting than the last half. The audio should be available by tomorrow, and the slides are already posted.

I liked listening to Eric Byres. He described his work with the Industrial Security Incident Knowledgebase. This is real data supplied by 22 companies with SCADA implementations. The price of access to the database is providing at least one case study of a SCADA security event. I thought this was a novel way to encourage disclosure of security incidents.

I found the following slide surprising.

Yes, 17% of the incidents involved SCADA (or PLC or DCS) systems directly connected to the Internet. Eric said 80-90% of control systems are connected to business systems that are then connected to the Internet. He also said the so-called "air gap" is a "myth."

In 2001 Eric noted an increase in the number of attacks from outsiders. Does this sound familiar?

As I noted before, recidivist internal threats are the easiest to prevent because you can fire the perpetrator and preferably prosecute him/her (assuming you identify the perpetrator). Try doing the same with an unnamed recidivist attacker from a jump box in Romania!

This slide shows financial impact.

I recommend listening to at least the first half of the webcast before you jump all over my thoughts, Slashdot-style. If you heard the whole webcast already, please feel free to comment.

You might find all of the slides useful too.

NSM Wiki Created

David Bianco created a NSM Wiki to centralize public information about using Sguil and related NSM tools. I just posted my FreeBSD Netgraph Port Bonding script there. If you have something helpful to add about NSM, please do!

Mike Rothman Is Right

Mike Rothman is right:

I'm here at the Security Standard conference and I'm seeing the pendulum starting to swing back. What pendulum? The pendulum that swings like a metronome between security as a defense and security as an enabler...

I'll make it very very clear. Security is not a business enabler. It is a cost of doing business. You cannot do new things because of security. You do open up new revenue streams and add value to customers via new applications that reflect new (or updated) business processes. It may be ill advised to put these new business processes on the web without adequate security, but you CAN do it.

In extreme cases of incredible negligence or outright stupidity, a business may deploy an exceptionally insecure application or business process that must be shut down due to overwhelming fraud and theft. Barring those circumstances, however, I agree that businesses are willing to "put these new business processes on the web without adequate security" and suck up some level of "acceptable loss."

Richard Stiennon agrees:

My perspective is that treating IT security like a business process is like treating a tactical military strike force as a business. While maintaining the capability of military forces could be a process open for improvement by applying some business discipline, actually fighting battles and overcoming opposing forces does not have much of the "business process" about it. Security is much more akin to fighting a battle than it is to "aligning business objectives".

Hopefully someone at this conference will address security as a cost, like insurance or legal teams.

Further Discussion of Chinese Cyber Threat

As a follow-up to this post, I found this forum transcript to be a mildly informative overview of the Chinese cyber threat. This question is really troubling, if true:

Joe in Groton, CT: I am an administrator of a DoD network. Why haven't I heard anything from up above about what types of attacks they are using, and whether or not Sysadmins need to take any extra steps to secure our networks? As a matter of fact, I haven't even heard anything from the DoD that there was a compromise at all. There was not even a post at the infosec web site about any compromise. If it wasn't for the SANS newsletter, I wouldn't have even found the GCN website. I feel that we need to share information within our community so we can all be more proactive in protecting our networks and our data. I get the impression that without this cohesion, we are sitting ducks.

That is sad. DoD is being owned and the people in one of the best positions to resist, and potentially detect and respond, are not aware of what's happening!

Port Independent Protocol Identification

One of the Holy Grails of network security monitoring is Port Independent Protocol Identification (PIPI -- lousy acronymn, but technically useful). PIPI allows inspection of protocols regardless of the port in use. PIPI has many security implications for discovery and (preferably) denial of covert channels, back doors, and other policy-violating channels. PIPI can also help network engineers better understand the legitimate use of protocols on their networks.

Some implementations exist. Last year after visiting Fidelis Security I mentioned their appliance uses port-neutral methods to identify protocols. Sourcefire's RNA also does PIPI. The Linux-only Application Layer Packet Classifier for Linux (L7-filter) and IPP2P projects use signatures to discover protocols on arbitrary ports. I'd like to hear of other approaches.

Today, thanks to geek00l, I read the paper Dynamic Application-Layer Protocol Analysis
for Network Intrusion Detection by an all-star team from Technische Universität München and Berkeley's ICSI Center for Internet Research. From the abstract:

In this paper, we discuss the design and implementation of a NIDS extension to perform dynamic application-layer protocol analysis. For each connection, the system first identifies potential protocols in use and then activates appropriate analyzers to verify the decision and extract higher-level semantics. We demonstrate the power of our enhancement with three examples: reliable detection of applications not using their standard ports, payload inspection of FTP data transfers, and detection of IRC-based botnet clients and servers.

Even better, their implementation is scheduled for integration in the next release of Bro, perhaps next month.

On a related PIPI note, in the future I expect we will not create firewall policies using port numbers as a major component. A security policy enforcement system might instead allow an administrator to implement a policy like "deny all outbound HTTP except [real] HTTP on port 80 and HTTPS on port 443." In other words, network (i.e., traffic-centric) security policy will be decoupled from ports and instead focus on applications and data.

Wednesday, September 06, 2006

Net Optics Think Tank on 26 September

In about three weeks I will be speaking at the next Net Optics Think Tank on 26 September 2006 in Fairfax, VA. It looks like I will be speaking during lunch from 1215 to 1315. Please register. Speaking of Net Optics, I see they've announced a GigaBit Port Aggregator with SFP Monitor Ports and a New iBypass™ Switch with Heartbeat, Unique Utilization Display, and Remote Management. I expect those will be on display in Fairfax.

Comment on Draft NIST Publications

Thanks to SANS I read this FCW story about new NIST draft publications, specifically Draft Special Publication 800-94, Guide to Intrusion Detection and Prevention (IDP) Systems (.pdf). I am worried about this document because it seems to imply that detection and prevention are equivalent functions. Recent Dark Reading stories like IDS/IPS: Too Many Holes? and IPS Technology: Ready for Overhaul have been critical of both technologies, but especially IPS.

Despite some vendor claims to the contrary, customers realize that the same inspection logic used to "detect intrusions" is supposed to be applied to the "prevent intrusions" problem. Since so-called "intrusion prevention" products were sold as devices that overcame "false positive" problems, many customers are disappointed are end up running their "IPS" in detect only mode. From this perspective, it makes sense for NIST to lump IDS and IPS in the same category.

From an operational and consequences-based perspective, IDS and IPS are completely different. Management generally permits an IDS team with passive sensors to do just about anything it wants, shy of using RST tricks to deny traffic. Management does not take the same approach with IPS, since an IPS is just a smarter firewall. One bad IPS rule and business traffic is interrupted.

While on this subject, I consider it unfortunate that the terms IDS and IPS even exist. Neither describes the actual function of either device. IDS as used by the vast majority of people seldom "detects intrusions." Rather, the technology is an Attack Indication System.

IPS is even more poorly named. An IPS is just a layer 7 firewall, so I would just as soon call an IPS a smarter firewall.

Finally, I read this press release:

[McAfee] announced that it has been selected to be deployed as the standard network intrusion prevention solution for the U.S. Air Force. The Air Force will use McAfee® IntruShield Network Intrusion Prevention System (IPS) and McAfee IntruShield Security Manager appliances to provide comprehensive and proactive protection for its worldwide non-classified and classified networks. The task order was awarded by the Air Force's Combat Information Transport System (CITS) program office to prime contractor Booz Allen Hamilton under the U.S. Air Force NETCENTS contract.

I'm guessing this is the end of ASIM?

Tuesday, September 05, 2006

Personal LinkedIn Policy

Every so often I someone finds my LinkedIn profile and asks me to join their network, and vice-versa. I am always pleased to hear from TaoSecurity Blog readers who would like to contact me. However, I stick to a fairly consistent policy about accepting invitations to LinkedIn.

If you look at my LinkedIn connections, you might notice a few groupings. First, you'll see an overwhelming number of ex-Foundstone consultants, ex- or current-Air Force personnel and contractors, and others with whom I've directly worked. Second, you'll see some members of the digital security community with whom I have regular or semi-regular contact. Finally, you will see a few students of my multi-day training classes, people I've met through NoVA Sec and related northern Virginia security business, and maybe a few people I chat with regularly in IRC.

In other words, I hardly ever accept invitations from people I've never met in person. Meeting once or twice at security conferences doesn't count, unless it's part of an ongoing conversation. I realize some people use LinkedIn like an extended address book. I don't use LinkedIn like that. I use the site to keep ties with people for whom I can vouch or testify to their technical skills or interests. I never want to look at my LinkedIn contacts list and ask myself "Who's that again?"

I expect some people might not care for this policy, but that's the way I use the site. Thank you.

Troubleshooting New Sguil Installation

Today I worked on installing a new Sguil sensor. I am trying to do so using scripts that I've mentioned before, but I am adapting the scripts to work with the newest FreeBSD and packages available. None of them are ready for public viewing. In fact, they are exceedingly pathetic. All they do is save me from typing commands. There is no error checking, intelligence, or anything of redeemable value. If I ever get some time I am going to help InstantNSM support FreeBSD, instead of getting my lousy scripts to production grade.

In any event, I was having trouble running sguild and sensor_agent.tcl. I forgot to capture the error I received, but one was complained that Tclx was not installed and the other said mysqltcl was not installed. I knew both were installed via package.


sensor-v:/home/sguil$ pkg_info | grep -i tcl
mysqltcl-3.01       TCL module for accessing MySQL databases based on msqltcl
tcl-8.4.13_1,1      Tool Command Language
tclX-8.4            Extended TCL
tcllib-1.7_1        A collection of utility modules for Tcl
tcltls-1.5.0        SSL extensions for TCL; dynamicly loadable

You can see the packages on sensor-c.


sensor-c:/home/analyst$ pkg_info | grep -i tcl
mysqltcl-3.01       TCL module for accessing MySQL databases based on msqltcl
tcl-8.4.13,1        Tool Command Language
tclX-8.4            Extended TCL
tcllib-1.7_1        A collection of utility modules for Tcl
tcltls-1.5.0        SSL extensions for TCL; dynamicly loadable

I decided to check using the Tcl interpreter.


sensor-v:/home/sguil$ tclsh
% package require Tclx
couldn't load file "/usr/local/lib/tclx8.4/libtclx8.4.so":
 /usr/local/lib/tclx8.4/libtclx8.4.so: Undefined symbol "__h_errno"
% package require mysqltcl
couldn't load file "/usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3": 
 /usr/local/lib/mysql/libmysqlclient.so.15: Undefined symbol "gethostbyname_r"

That's not good. Here's what should happen, on a working sensor.


sensor-c:/home/analyst$ tclsh
% package require mysqltcl
3.01
% package require Tclx
8.4
% package require mysqltcl
3.01

Next I checked hashes of the critical files, first on the good sensor.


sensor-c:/home/analyst$ md5 /usr/local/lib/tclx8.4/libtclx8.4.so
MD5 (/usr/local/lib/tclx8.4/libtclx8.4.so) = 51c311403e358a8c369fc9d195b94de8
sensor-c:/home/analyst$ md5 /usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3
MD5 (/usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3) = 071d8d664cde06611adf333dc065b1d6

Now the nonworking sensor.


sensor-v:/home/analyst$ md5 /usr/local/lib/tclx8.4/libtclx8.4.so
MD5 (/usr/local/lib/tclx8.4/libtclx8.4.so) = 2db9e4320ffeee548f2768ca9aa7c353
sensor-v:/home/analyst$ md5 /usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3
MD5 (/usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3) = 96957f535baf6b77df0cc5cb56ed45be

They hashes are different. Let's see what these files are.


sensor-c:/home/analyst$ file /usr/local/lib/tclx8.4/libtclx8.4.so
/usr/local/lib/tclx8.4/libtclx8.4.so: ELF 32-bit LSB shared object, 
 Intel 80386, version 1 (FreeBSD), stripped
sensor-c:/home/analyst$ file /usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3
/usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3: ELF 32-bit LSB shared object,
 Intel 80386, version 1 (FreeBSD), not stripped

Now sensor-v.


sensor-v:/home/analyst$ file /usr/local/lib/tclx8.4/libtclx8.4.so
/usr/local/lib/tclx8.4/libtclx8.4.so: ELF 32-bit LSB shared object, 
 Intel 80386, version 1 (FreeBSD), stripped
sensor-v:/home/sguil$ file /usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3
/usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3: ELF 32-bit LSB shared object,
 Intel 80386, version 1 (FreeBSD), not stripped

I decided to move the bad files on sensor-v out of the way and replace them with good copies from sensor-c.


sensor-v:/root# mv /usr/local/lib/tclx8.4/libtclx8.4.so 
 /usr/local/lib/tclx8.4/libtclx8.4.so.noworkie
sensor-v:/root# cp /tmp/libtclx8.4.so /usr/local/lib/tclx8.4/
sensor-v:/root# mv /usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3 
 /usr/local/lib/tcl8.4/mysqltcl/libmysqltcl.so.3.noworkie
sensor-v:/root# cp /tmp/libmysqltcl.so.3 /usr/local/lib/tcl8.4/mysqltcl/

Now I ran the Tcl interpreter again on sensor-v.


sensor-v:/home/sguil$ tclsh
% package require Tclx
8.4
% package require mysqltcl
3.01

Everything is in order. I am suspicious though. Why would those two files not be correct on a fresh installation?

Snort 2.6.0 FreeBSD Port Problem

You may have read that Snort 2.6.0 is in the FreeBSD ports tree now. I installed the package this morning and learned there is a problem with the specification for the dynamic components. Specifically, from snort.conf:


dynamicpreprocessor directory /usr/local/lib/snort_dynamicpreprocessor/
...
dynamicengine /usr/local/lib/snort_dynamicengine/libsf_engine.so

FreeBSD does not use this structure. Change those entries to:


dynamicpreprocessor directory /usr/local/lib/snort/dynamicpreprocessor/
...
dynamicengine /usr/local/lib/snort/dynamicengine/libsf_engine.so

You can also pass the necessary locations via the command line.

I've submitted a FreeBSD PR.

Snort 2.6.0 High Memory Usage on FreeBSD

I've been working with Snort 2.6.0 on FreeBSD.

When you look at the snort.conf you'll see a bunch of rules commented out.


# include $RULE_PATH/web-attacks.rules
# include $RULE_PATH/backdoor.rules
# include $RULE_PATH/shellcode.rules
# include $RULE_PATH/policy.rules
# include $RULE_PATH/porn.rules
# include $RULE_PATH/info.rules
# include $RULE_PATH/icmp-info.rules
# include $RULE_PATH/virus.rules
# include $RULE_PATH/chat.rules
# include $RULE_PATH/multimedia.rules
# include $RULE_PATH/p2p.rules
# include $RULE_PATH/spyware-put.rules

When you start Snort you'll see it uses much more memory compared to earlier versions.


  654 root        1 -58    0   248M   247M bpf    0   0:01  3.30% snort

If this is too much, and you are willing to sacrifice Snort performance, you can enable the following in snort.conf:


config detection: search-method lowmem

This results in less memory usage.


  656 root        1 -58    0 39800K 39128K bpf    0   0:01  0.00% snort

With this option enabled you can even uncomment all rules.


  661 root        1 -58    0 59224K 58580K bpf    0   0:01  1.95% snort

What if you want the best performance and all rules? Well, on FreeBSD you are going to encounter a 512 MB default RAM limitation that will prevent Snort from running.

 
# limit
cputime      unlimited
filesize     unlimited
datasize     524288 kbytes
stacksize    65536 kbytes
coredumpsize unlimited
memoryuse    unlimited
vmemoryuse   unlimited
descriptors  11095 
memorylocked unlimited
maxproc      5547 
sbsize       unlimited

You can change this by making the following entries in /boot/loader.conf:


kern.dfldsiz="1G"                # Set the initial data size limit
kern.maxdsiz="1G"                # Set the max data size

Reboot when done. Here is the result.


# limit
cputime      unlimited
filesize     unlimited
datasize     1048576 kbytes
stacksize    65536 kbytes
coredumpsize unlimited
memoryuse    unlimited
vmemoryuse   unlimited
descriptors  11095 
memorylocked unlimited
maxproc      5547 
sbsize       unlimited

Now you can run Snort with all rules and best performance, and see it occupies over 900 MB.


  645 root        1 -58    0   925M   926M bpf    0   0:04  0.00% snort

I have not tested Snort to see the effect of various options, although I prefer to run as many rule sets as makes sense for my environment. Note I did not add Bleeding or Community rule sets for this example.

Using Root Certificates with OpenSSL on FreeBSD

I'm reading a great book on Apache security.

One of the examples involves using the openssl client program to analyze a chain of certificates.

In the following example I use openssl to connect to www.thawte.com, but I do not provide a location to find root certificates.


orr:/home/richard$ openssl s_client -host www.thawte.com -port 443
CONNECTED(00000003)
 depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting (Pty) 
      Ltd/OU=Security/CN=www.thawte.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDRDCCAq2gAwIBAgIQaS8vV/TrQ/S1Mg+/74dipDANBgkqhkiG9w0BAQQFADBM
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wNjA3MDQxMTM5MTdaFw0w
NzA3MTYwOTMzMDlaMIGKMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBD
YXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xJDAiBgNVBAoTG1RoYXd0ZSBDb25zdWx0
aW5nIChQdHkpIEx0ZDERMA8GA1UECxMIU2VjdXJpdHkxFzAVBgNVBAMTDnd3dy50
aGF3dGUuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC/h9nzmXfQkZsJ
CjznxdLV8/Ebms8qzEjO8KnHyT4+94cI0WjjlyqJH1ZgLM2/dccKS4vj6X7nl/vY
6o1ZT3pn+o6C3vl3EdMiLzKcQLpq+da03Y2xrtgIe070zCbQb4r7um4A/QFDrHAR
fgC4PllHGnmPFDXsiA4OKGIQ65vcGwIDAQABo4HnMIHkMCgGA1UdJQQhMB8GCCsG
AQUFBwMBBggrBgEFBQcDAgYJYIZIAYb4QgQBMDYGA1UdHwQvMC0wK6ApoCeGJWh0
dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVTR0NDQS5jcmwwcgYIKwYBBQUHAQEE
ZjBkMCIGCCsGAQUFBzABhhZodHRwOi8vb2NzcC50aGF3dGUuY29tMD4GCCsGAQUF
BzAChjJodHRwOi8vd3d3LnRoYXd0ZS5jb20vcmVwb3NpdG9yeS9UaGF3dGVfU0dD
X0NBLmNydDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUAA4GBAINguft3Ek1x
X6aiP9/QCrwHJErYrwEtRkURG+ghVVTCEhtZrqD5LdUHionIkmfgu5FSvrSN3VH/
JdOv3qngubpcTfhOHLxNuML5WB6Rwul5gfONsUFEoEPOMGxF4x22b7iG6QA5j42U
eop6NkYq4HtuOqKNcy9zuyDpzVA/EZus
-----END CERTIFICATE-----
subject=/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting (Pty) 
 Ltd/OU=Security/CN=www.thawte.com
issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
---
No client certificate CA names sent
---
SSL handshake has read 2214 bytes and written 340 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: BF365536023EF272C68C3A9420364D68BF6363FB59461B1BF9BB9C36A6BA8FA9
    Session-ID-ctx: 
    Master-Key: 9B6224C8083E007C2D4797B9D92317B213AD53FC6C5EF693FC17D
                8D3B3B9B918D7E03317C67BCE5699CF102ED707B5C6
    Key-Arg   : None
    Start Time: 1157455445
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Date: Tue, 05 Sep 2006 11:25:22 GMT
Server: Apache
Accept-Ranges: bytes
Content-Length: 25062
Connection: close
Content-Type: text/html

closed

The return code


    Verify return code: 20 (unable to get local issuer certificate)

is the important item here.

FreeBSD's base installation does not include the ca-root.crt file expected with other Unix-like systems. That file is available as the security/ca-roots port, however.


orr:/root# setenv PACKAGESITE 
 ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-6-stable/Latest/
orr:/root# pkg_add -vr ca-roots
looking up ftp2.freebsd.org
connecting to ftp2.freebsd.org:21
setting passive mode
opening data connection
initiating transfer
Fetching ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-6-stable/Latest/
 ca-roots.tbz...x +CONTENTS
x +COMMENT
x +DESC
x +MTREE_DIRS
x share/certs/ca-root.crt
tar command returns 0 status
 Done.
extract: Package name is ca-roots-1.2
extract: CWD to /usr/local
extract: execute 'mkdir -p /usr/local/share/certs'
extract: /usr/local/share/certs/ca-root.crt
extract: execute 'ln -s /usr/local/share/certs/ca-root.crt /etc/ssl/cert.pem'
extract: CWD to .
Running mtree for ca-roots-1.2..
mtree -U -f +MTREE_DIRS -d -e -p /usr/local >/dev/null
Attempting to record package into /var/db/pkg/ca-roots-1.2..
Package ca-roots-1.2 registered in /var/db/pkg/ca-roots-1.2

With these SSL Certificate Authority root certificates installed, I use openssl in this manner.


orr:/home/richard$ openssl s_client -host www.thawte.com -port 443
 -CAfile /etc/ssl/cert.pem
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
verify return:1
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify return:1
depth=0 /C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting (Pty) Ltd/OU=Security/CN=www.thawte.com
verify return:1
---
Certificate chain
 0 s:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting (Pty) 
   Ltd/OU=Security/CN=www.thawte.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDRDCCAq2gAwIBAgIQaS8vV/TrQ/S1Mg+/74dipDANBgkqhkiG9w0BAQQFADBM
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wNjA3MDQxMTM5MTdaFw0w
NzA3MTYwOTMzMDlaMIGKMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBD
YXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xJDAiBgNVBAoTG1RoYXd0ZSBDb25zdWx0
aW5nIChQdHkpIEx0ZDERMA8GA1UECxMIU2VjdXJpdHkxFzAVBgNVBAMTDnd3dy50
aGF3dGUuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC/h9nzmXfQkZsJ
CjznxdLV8/Ebms8qzEjO8KnHyT4+94cI0WjjlyqJH1ZgLM2/dccKS4vj6X7nl/vY
6o1ZT3pn+o6C3vl3EdMiLzKcQLpq+da03Y2xrtgIe070zCbQb4r7um4A/QFDrHAR
fgC4PllHGnmPFDXsiA4OKGIQ65vcGwIDAQABo4HnMIHkMCgGA1UdJQQhMB8GCCsG
AQUFBwMBBggrBgEFBQcDAgYJYIZIAYb4QgQBMDYGA1UdHwQvMC0wK6ApoCeGJWh0
dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVTR0NDQS5jcmwwcgYIKwYBBQUHAQEE
ZjBkMCIGCCsGAQUFBzABhhZodHRwOi8vb2NzcC50aGF3dGUuY29tMD4GCCsGAQUF
BzAChjJodHRwOi8vd3d3LnRoYXd0ZS5jb20vcmVwb3NpdG9yeS9UaGF3dGVfU0dD
X0NBLmNydDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUAA4GBAINguft3Ek1x
X6aiP9/QCrwHJErYrwEtRkURG+ghVVTCEhtZrqD5LdUHionIkmfgu5FSvrSN3VH/
JdOv3qngubpcTfhOHLxNuML5WB6Rwul5gfONsUFEoEPOMGxF4x22b7iG6QA5j42U
eop6NkYq4HtuOqKNcy9zuyDpzVA/EZus
-----END CERTIFICATE-----
subject=/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting (Pty) 
 Ltd/OU=Security/CN=www.thawte.com
issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
---
No client certificate CA names sent
---
SSL handshake has read 2214 bytes and written 340 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 3121A8A4A5DFE70DABA68C5354C2DE89996F622CE7A323F97EB3E87390594F6B
    Session-ID-ctx: 
    Master-Key: E4C5A3B6B5DFF98BFAF2C9CCF8E86083B2D03EE06984707EF238
                F431531F00D9CEA3579E6386E9DB13C57EB84B3E2BAF
    Key-Arg   : None
    Start Time: 1157455485
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Date: Tue, 05 Sep 2006 11:26:02 GMT
Server: Apache
Accept-Ranges: bytes
Content-Length: 25062
Connection: close
Content-Type: text/html

closed

Now we see


    Verify return code: 0 (ok)

which means the certificate presented by the Web server is legitimate.

Eliminating Serverauth Files

I've noticed an accumulation of files like .serverauth.NUM (e.g., .serverauth.626) in my home directory. After some searching I found this helpful blog post indicating that a change in X11R6.9 requires a change in the startx script I use to launch X.


orr:/usr/X11R6/bin$ diff startx startx.new 
138c138,139
< xserverauthfile=$HOME/.serverauth.$$
---
> #xserverauthfile=$HOME/.serverauth.$$
> xserverauthfile=$XAUTHORITY

By specifying xserverauthfile=$XAUTHORITY we remove the need to create .serverauth files.

Monday, September 04, 2006

Review of Nagios Books Posted

Amazon.com just posted my two reviews on books about Nagios. The first is Pro Nagios 2.0 by James Turnbull. Here is a link to the five star review.

The second is Nagios: System and Network Administration by Wolfgang Barth. Here is a link to the four star review.

Both reviews share the same introduction.

I recently received review copies of Pro Nagios 2.0 (PN2) by James Turnbull and Nagios: System and Network Monitoring (NSANM) by Wolfgang Barth. I read PN2 first, then NSANM. Both are excellent books, but I expect potential readers want to know which is best for them. The following is a radical simplification, and I could honestly recommend readers buy either (or both) books. If you are completely new to Nagios and want a very well-organized introduction, I recommend PN2. If you are somewhat familiar with Nagios and want detailed descriptions of a wide variety of Nagios plug-ins, I recommend NSANM.

MIB Browser

While reading a book on Nagios, I learned of net-mgmt/mbrowse, pictured above. It's not fancy -- just a graphical SNMP v1 MIB browser.

Saturday, September 02, 2006

Working SNMP v3 Trap Using Net-SNMP Tools 5.1.2

I managed to get a SNMP v3 trap to work when sending the trap with Debian.

This is important because it confirms a bug was introduced into snmptrap somewhere in the 5.2.x line of Net-SNMP tools.

The version of snmptrap installed by Debian stable is 5.1.2. Here is what I set up.

The Debian host is macmini. I created /etc/snmp/snmpd.conf with the following.


createUser              doit MD5 doitpassword DES doitpassword

When I ran snmpd, I saw the user created along with the engine ID for this host.


macmini:~# snmpd -f -Lo -Dusm
usmUser: created a new user doit at 80 00 07 E5 80 54 D7 15 E8 44 FA 12 65 
Warning: no access control information configured.
  It's unlikely this agent can serve any useful purpose in this state.
  Run "snmpconf -g basic_setup" to help you configure the snmpd.conf file for this agent.
NET-SNMP version 5.1.2

This step also created /var/lib/snmp/snmpd.conf with the following:


usmUser 1 3 0x800007e58054d715e844fa1265 0x646f697400 0x646f697400 NULL .1.3.6.1.6.3.10.1.1.2 
 0x7118d87274c4aa4e22c27c003bf92add .1.3.6.1.6.3.10.1.2.2 0x7118d87274c4aa4e22c27c003bf92add ""
engineBoots 1
oldEngineID 0x800007e58054d715e844fa1265

0x800007e58054d715e844fa1265 is my engine ID. I need this when I set up snmptrapd.conf on hacom, which simulates a NMS using snmptrapd.

On hacom I create /usr/local/etc/snmp/snmptrapd.conf with the following:


createUser -e 0x800007e58054d715e844fa1265 doit MD5 doitpassword DES doitpassword

Next I start snmptrapd on hacom.


hacom:/root# snmptrapd -f -Lo -Dusm
usmUser: created a new user doit at 80 00 07 E5 80 54 D7 15 E8 44 FA 12 65 
2006-09-02 19:25:10 NET-SNMP version 5.2.2 Started.

Finally I can send a trap from macmini to hacom.


richard@macmini:~$ snmptrap -Ddumph_send,dumpv_send,usm -v 3
 -e 0x800007e58054d715e844fa1265 
 -u doit -a MD5 -A doitpassword -l authNoPriv 192.168.2.18 ''
 SNMPv2-SMI::enterprises.3.1
dumph_send: SNMPv3 Message
dumph_send:   PDU-TRAP2
dumph_send:     VarBind
dumph_send:       Value    ObjID: SNMPv2-SMI::enterprises.3.1
dumph_send:       Name    ObjID: SNMPv2-MIB::snmpTrapOID.0
dumph_send:     VarBind
dumph_send:       Value    UInteger:    637099911 (0x25F95F87)
dumph_send:       Name    ObjID: SNMPv2-MIB::sysUpTime.0
dumph_send:   error index    Integer:   0 (0x00)
dumph_send:   error status    Integer:  0 (0x00)
dumph_send:   request_id    Integer:    209733159 (0xC804627)
dumph_send:   ScopedPdu
dumph_send:     contextName    String: [NULL]
dumph_send:     contextEngineID    String:      ...å.J4..Dú.Z
dumph_send: msgSecurityModel    Integer:        3 (0x03)
dumph_send: msgFlags    String: .
dumph_send: msgMaxSize    Integer:      65507 (0xFFE3)
dumph_send: msgID    Integer:   29075524 (0x1BBA844)
dumph_send: SNMP Version Number    Integer:     3 (0x03)
dumph_send: SM msgSecurityParameters
usm: USM processing has begun (offset 76)
usm: getting user doit
dumph_send:   msgPrivacyParameters    String: [NULL]
dumph_send:   msgAuthenticationParameters    String:    ............
dumph_send:   msgUserName    String:    doit
dumph_send:   msgAuthoritativeEngineTime    Integer:    637099911 (0x25F95F87)
dumph_send:   msgAuthoritativeEngineBoots    Integer:   1 (0x01)
dumph_send:   msgAuthoritativeEngineID    String:       ...å.T×.èDú.e
usm: USM processing completed.

Here is what snmptrapd saw.


usm: USM processing begun...
usm: Verification succeeded.
usm: USM processing completed.
2006-09-02 19:26:50 macmini.taosecurity.com [UDP: [192.168.2.12]:34061]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (637099911) 73 days, 17:43:19.11   
    SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.3.1

Here is the packet that was sent.


Simple Network Management Protocol
    msgVersion: snmpv3 (3)
    msgGlobalData
        msgID: 29075524
        msgMaxSize: 65507
        msgFlags: 01
        .... .0.. = Reportable: Not set
        .... ..0. = Encrypted: Not set
        .... ...1 = Authenticated: Set
        msgSecurityModel: USM (3)
    msgAuthoritativeEngineID: 800007E58054D715E844FA1265
    1... .... = Engine ID Conformance: RFC3411 (SNMPv3)
    Engine Enterprise ID: U.C. Davis, ECE Dept. Tom (2021)
    Engine ID Format: Reserved/Enterprise-specific (128): UCD-SNMP Random
    Engine ID Data: 54D715E8
    Engine ID Data: Creation Time: Sep 26, 2023 11:35:32
    msgAuthoritativeEngineBoots: 1
    msgAuthoritativeEngineTime: 637099911
    msgUserName: doit
    msgAuthenticationParameters: 90E951108773145325537BF0
    msgData: plaintext (0)
        plaintext
            contextEngineID: 800007E5804A34181044FA135A
            data: sNMPv2-Trap (7)
                sNMPv2-Trap
                    request-id: 209733159
                    error-status: noError (0)
                    error-index: 0
                    variable-bindings: 2 items
                        Item
                            name: 1.3.6.1.2.1.1.3.0 (SNMPv2-MIB::sysUpTime.0)
                            valueType: value (0)
                                value: simple (4294967295)
                                value: simple (4294967295)
                                    application-wide: timeticks-value (3)
                                        timeticks-value: 637099911
                        Item
                            name: 1.3.6.1.6.3.1.1.4.1.0 (SNMPv2-MIB::snmpTrapOID.0)
                            valueType: value (0)
                                value: simple (4294967295)
                                    simple: objectID-value (2)
                                        Value: OID: SNMPv2-SMI::enterprises.3.1

0000  00 40 48 b1 5c db 00 14 51 17 6a b2 08 00 45 00   .@H.\...Q.j...E.
0010  00 b3 00 00 40 00 40 11 b4 cb c0 a8 02 0c c0 a8   ....@.@.........
0020  02 12 85 0d 00 a2 00 9f de 3d 30 81 94 02 01 03   .........=0.....
0030  30 11 02 04 01 bb a8 44 02 03 00 ff e3 04 01 01   0......D........
0040  02 01 03 04 30 30 2e 04 0d 80 00 07 e5 80 54 d7   ....00........T.
0050  15 e8 44 fa 12 65 02 01 01 02 04 25 f9 5f 87 04   ..D..e.....%._..
0060  04 64 6f 69 74 04 0c 90 e9 51 10 87 73 14 53 25   .doit....Q..s.S%
0070  53 7b f0 04 00 30 4a 04 0d 80 00 07 e5 80 4a 34   S{...0J.......J4
0080  18 10 44 fa 13 5a 04 00 a7 37 02 04 0c 80 46 27   ..D..Z...7....F'
0090  02 01 00 02 01 00 30 29 30 10 06 08 2b 06 01 02   ......0)0...+...
00a0  01 01 03 00 43 04 25 f9 5f 87 30 15 06 0a 2b 06   ....C.%._.0...+.
00b0  01 06 03 01 01 04 01 00 06 07 2b 06 01 04 01 03   ..........+.....
00c0  01                                                .

If I want to send the trap encrypted, I do the following.


richard@macmini:~$ snmptrap -Ddumph_send,dumpv_send,usm -v 3 
 -e 0x800007e58054d715e844fa1265
 -u doit -a MD5 -A doitpassword -x DES -X doitpassword -l authPriv 192.168.2.18 '' 
 SNMPv2-SMI::enterprises.3.1
dumph_send: SNMPv3 Message
dumph_send:   PDU-TRAP2
dumph_send:     VarBind
dumph_send:       Value    ObjID: SNMPv2-SMI::enterprises.3.1
dumph_send:       Name    ObjID: SNMPv2-MIB::snmpTrapOID.0
dumph_send:     VarBind
dumph_send:       Value    UInteger:    637119304 (0x25F9AB48)
dumph_send:       Name    ObjID: SNMPv2-MIB::sysUpTime.0
dumph_send:   error index    Integer:   0 (0x00)
dumph_send:   error status    Integer:  0 (0x00)
dumph_send:   request_id    Integer:    472573359 (0x1C2AE5AF)
dumph_send:   ScopedPdu
dumph_send:     contextName    String: [NULL]
dumph_send:     contextEngineID    String:      ...å.ox¿9Dú..
dumph_send: msgSecurityModel    Integer:        3 (0x03)
dumph_send: msgFlags    String: .
dumph_send: msgMaxSize    Integer:      65507 (0xFFE3)
dumph_send: msgID    Integer:   56841470 (0x36354FE)
dumph_send: SNMP Version Number    Integer:     3 (0x03)
dumph_send: SM msgSecurityParameters
usm: USM processing has begun (offset 76)
usm: getting user doit
  String:       æ/Øá:ë......â¡¯Qlpª.z.u.Á?ó8t5b_$V.Rq.ð³¥3..¦ºIÏnÇ.
                ?.¥ó·}Û?».c.YPü÷Ã_I®èö.Î...§m
usm: Encryption successful.
dumph_send:   msgPrivacyParameters    String:   ....ÜF..
dumph_send:   msgAuthenticationParameters    String:    ............
dumph_send:   msgUserName    String:    doit
dumph_send:   msgAuthoritativeEngineTime    Integer:    637119304 (0x25F9AB48)
dumph_send:   msgAuthoritativeEngineBoots    Integer:   1 (0x01)
dumph_send:   msgAuthoritativeEngineID    String:       ...å.T×.èDú.e
usm: USM processing completed.

Here is what snmptrapd sees.


2006-09-02 19:30:05 macmini.taosecurity.com [UDP: [192.168.2.12]:34061]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (637119304) 73 days, 17:46:33.04      
    SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.3.1

Here is what the trace looks like.


Simple Network Management Protocol
    msgVersion: snmpv3 (3)
    msgGlobalData
        msgID: 56841470
        msgMaxSize: 65507
        msgFlags: 03
        .... .0.. = Reportable: Not set
        .... ..1. = Encrypted: Set
        .... ...1 = Authenticated: Set
        msgSecurityModel: USM (3)
    msgAuthoritativeEngineID: 800007E58054D715E844FA1265
    1... .... = Engine ID Conformance: RFC3411 (SNMPv3)
    Engine Enterprise ID: U.C. Davis, ECE Dept. Tom (2021)
    Engine ID Format: Reserved/Enterprise-specific (128): UCD-SNMP Random
    Engine ID Data: 54D715E8
    Engine ID Data: Creation Time: Sep 26, 2023 11:35:32
    msgAuthoritativeEngineBoots: 1
    msgAuthoritativeEngineTime: 637119304
    msgUserName: doit
    msgAuthenticationParameters: 2BBB80DD5668B46281AFDF39
    msgPrivacyParameters: 00000001DC469993
    msgData: encryptedPDU (1)
        encryptedPDU: E62FD8E13AEB7F088B0B111EE2A1AF516C70AA177A9E759C...

0000  00 40 48 b1 5c db 00 14 51 17 6a b2 08 00 45 00   .@H.\...Q.j...E.
0010  00 c1 00 00 40 00 40 11 b4 bd c0 a8 02 0c c0 a8   ....@.@.........
0020  02 12 85 0d 00 a2 00 ad 40 32 30 81 a2 02 01 03   ........@20.....
0030  30 11 02 04 03 63 54 fe 02 03 00 ff e3 04 01 03   0....cT.........
0040  02 01 03 04 38 30 36 04 0d 80 00 07 e5 80 54 d7   ....806.......T.
0050  15 e8 44 fa 12 65 02 01 01 02 04 25 f9 ab 48 04   ..D..e.....%..H.
0060  04 64 6f 69 74 04 0c 2b bb 80 dd 56 68 b4 62 81   .doit..+...Vh.b.
0070  af df 39 04 08 00 00 00 01 dc 46 99 93 04 50 e6   ..9.......F...P.
0080  2f d8 e1 3a eb 7f 08 8b 0b 11 1e e2 a1 af 51 6c   /..:..........Ql
0090  70 aa 17 7a 9e 75 9c c1 3f f3 38 74 35 62 5f 24   p..z.u..?.8t5b_$
00a0  56 8a 52 71 01 f0 b3 a5 33 91 14 a6 ba 49 cf 6e   V.Rq....3....I.n
00b0  c7 1e 3f 7f a5 f3 b7 7d db 3f bb 18 63 7f 59 50   ..?....}.?..c.YP
00c0  fc f7 c3 5f 49 ae e8 f6 1a ce 14 13 1e a7 6d      ..._I.........m

I am so glad I can get this to work. Everyone recommends using SNMP v3 but it's frustrating to figure it out when facing a bug in snmptrapd. Net-SNMP tools are really powerful, though.

The next challenge is figuring out the access control model in Net-SNMP 5.3.x. Apparently it's different from 5.2.x.

When 5.2.4 is released I plan to test out snmptrap on FreeBSD as well.

Friday, September 01, 2006

FreeBSD Update in FreeBSD 6.2

I've been using Colin Percival's FreeBSD Update for about three years. Today Colin blogged that he expects FreeBSD Update to be included in FreeBSD 6.2, arriving in October or November. At that point you won't need to install the port. Great work Colin!

Thursday, September 28, 2006

Wednesday, September 27, 2006

Monday, September 25, 2006

Saturday, September 23, 2006

Friday, September 22, 2006

Wednesday, September 20, 2006

Tuesday, September 19, 2006

Monday, September 18, 2006

Show-Request

Original Client Request:

Processed Request:

Thursday, September 14, 2006

Wednesday, September 13, 2006

Monday, September 11, 2006

Friday, September 08, 2006

Thursday, September 07, 2006

Wednesday, September 06, 2006

Tuesday, September 05, 2006

Monday, September 04, 2006

Saturday, September 02, 2006

Friday, September 01, 2006

About Me

twitter.com/taosecurity

TCP/IP Weapons School 2.x at Black Hat DC 2010 31 Jan - 01 Feb

SANS Incident Detection Summit

Books by Richard Bejtlich

Association of Former Information Warriors

Labels

Blog Archive

Hakin9 Magazine

BSD Magazine

Google Analytics