Monday, January 31, 2005

Review of Forensic Discovery Posted just posted my five star review of Forensic Discovery. I read a pre-publication draft of the book which resulted in my quote on the Addison-Wesley site. From the review:

"Farmer and Venema do for digital archaeology what Indiana Jones did for historical archaeology. 'Forensic Discovery' unearths hidden treasures in enlightening and entertaining ways, showing how a time-centric approach to computer forensics reveals even the cleverest intruder. I highly recommend reading this book."

In the chapter 7 (available online as a .pdf) Farmer and Venema mention the Veeco Nanotheater. Veeco makes products which can scan the surface of disks at nanotechnology scales. They show the image at right, and describe it as "residuals of overwritten information on the sides of magnetic disk tracks." This demonstrates the difficulty of truly "destroying" digital evidence. Forensic Discovery explains the problem this way:

"Although memory chips and magnetic disks are designed to store digital information, the underlying technology is analog. With analog storage of digital information, the value of a bit is a complex combination of past stored values. Memory chips have undocumented diagnostic modes that allow access to values smaller than a bit. With modified electronic circuitry, signals from disk read heads can reveal older data as modulations on the analog signal."

At 198 pages this book is a quick read, which explains how I was able to read and review it while writing a new book!

Sunday, January 30, 2005

Creating favicon.ico Files

Visitors to may notice that the icon appearing in the Web browser address bar has changed from the FreeBSD daemon to the yin-yang "S" pictured at left. I created this icon using the following process, detailed by DHCPDump author Edwin Groothuis.

First I used xv to crop the TaoSecurity logo, set the image size at 16x16, and save the image (taosecurity.png) in png format.

Next I installed Netpbm using the graphics/netpbm FreeBSD port. Once installed I executed the following command to convert my png file to the desired favicon.ico:

orr:/home/richard$ pngtopnm taosecurity.png | ppmtowinicon -output favicon.ico

When done I had the icon I needed:

orr:/home/richard$ file favicon.ico
favicon.ico: ms-windows icon resource - 1 icon

I copied the favicon.ico file to my Web root, and then added the following to the HEAD section of my index.html file:


I had to clear my Web browser cache to see the new icon. Incidentally, I don't see the icon in Internet Explorer. Using Firefox, I do see it.

In-Depth Book Review at

Benjamin D. Thomas posted a very thorough review of The Tao of Network Security Monitoring at Benjamin writes:

"To be honest, this was one of the best books that I've read on network security. Others books often dive too deeply into technical discussions and fail to provide any relevance to network engineers/administrators working in a corporate environment. Budgets, deadlines, and flexibility are issues that we must all address. The Tao of Network Security Monitoring is presented in such a way that all of these are still relevant. One of the greatest virtues of this book is that is offers real-life technical examples, while backing them up with relevant case studies. Network security engineers, system administrations, and security management will find value in this book. It is a must-read for anyone interested in getting into the field, but would still be useful as a reference for the experienced expert."

Friday, January 28, 2005

Flowgrep: Flow-oriented Content Matching

Last month I found Meling Mudin's IDS blog, and learned of Jose Nazario's tool Flowgrep. Flowgrep is written in Python. It is similar to Ngrep, which I addressed in my first book. Ngrep is packet-oriented, meaning the strings for which Ngrep searches must all appear in a single packet. If you search for 'bejtlich', and 'bejt' is in one packet and 'lich' another, then Ngrep won't find anything.

Flowgrep, in contrast, is conversation-oriented. Flowgrep assembles TCP sessions, as well as pseudo-sessions for UDP and ICMP. Flowgrep will rebuild a conversation where 'bejt' is in one packet and 'lich' another, and report seeing 'bejtlich'.

Flowgrep relies on Mike Schiffman's Libnet and Mike Pomraning's Pynids, a Python wrapper for Rafal Wojtczuk's Libnids. Mike was kind enough to work with me over the last week to get Pynids operational on FreeBSD 5.3.

Here's how I ended up with a working Flowgrep implementation. First I ensured Python was on my system:

janney:/usr/local/src/pynids-0.5rc1$ which python2.4

I installed the net/libnet-devel port. Note this installs Libnet, rather than the deprecated Libnet 1.0.2a found in net/libnet.

Next I downloaded and extracted pynids-0.5rc1.tar.gz.

janney:/usr/local/src$ tar -xzvf pynids-0.5rc1.tar.gz
x pynids-0.5rc1/
x pynids-0.5rc1/CHANGES
x pynids-0.5rc1/COPYING
x pynids-0.5rc1/Example
x pynids-0.5rc1/
x pynids-0.5rc1/README
x pynids-0.5rc1/dispatch.diff
x pynids-0.5rc1/libnids-1.19.tar
x pynids-0.5rc1/nidsmodule.c
x pynids-0.5rc1/setup.cfg
x pynids-0.5rc1/
x pynids-0.5rc1/PKG-INFO

Note I did not install Libnids myself. Mike packages a patched version of Libnids with his Pynids distribution.

I used these commands to install Pynids:

janney:/usr/local/src$ cd pynids-0.5rc1
janney:/usr/local/src/pynids-0.5rc1$ python2.4 build
running build
tar -xf libnids-1.19.tar
patch -c -p1 -i ../dispatch.diff
janney:/usr/local/src/pynids-0.5rc1$ sudo python2.4 install
running install
running build
running build_ext
running install_lib
copying build/lib.freebsd-5.3-RELEASE-p2-i386-2.4/
-> /usr/local/lib/python2.4/site-packages

Now that Pynids was installed, I downloaded Flowgrep and installed it.

janney:/usr/local/src$ tar -xzvf flowgrep-0.8.tar.gz
x flowgrep-0.8
x flowgrep-0.8/
x flowgrep-0.8/flowgrep.8
x flowgrep-0.8/
x flowgrep-0.8/Makefile
x flowgrep-0.8/README
janney:/usr/local/src$ cd flowgrep-0.8
janney:/usr/local/src/flowgrep-0.8$ make
python build
running build
janney:/usr/local/src/flowgrep-0.8$ sudo make install
cp flowgrep
python install
running install
running build
running install_data
copying flowgrep -> /usr/local/sbin/
copying flowgrep.8 -> /usr/local/man/man8/

Now that Flowgrep is installed, let's see how it works. For demonstration purposes, I have telnet running on host janney on an odd port -- 47557 TCP. I tell Tcpdump to collect all traffic on janney involving that port. This gives us traffic for later analysis:

janney:/home/richard$ sudo tcpdump -n -i xl0 -s 1515
-w flowgrep2.lpc port 47557
tcpdump: listening on xl0,
link-type EN10MB (Ethernet), capture size 1515 bytes
^C88 packets captured
810 packets received by filter
0 packets dropped by kernel

On janney (or really any host with visibility of janney) to watch for flows containing the string 'hackerpassword':

janney:/home/richard$ sudo flowgrep -d xl0 -l /tmp
-i -a hackerpassword

From host neely, I telnet to janney on port 47557 TCP and try to login as user 'bejtlich' (echoed to my screen) and password 'hackerpassword' (not echoed to my screen):

richard@neely:~$ telnet janney 47557
Connected to janney.
Escape character is '^]'.

FreeBSD/i386 ( (ttyp6)

login: bejtlich
Login incorrect
telnet> quit
Connection closed.

Let's see what Flowgrep wrote to the /tmp directory:


We can use hd to view the contents of these files:

janney:/home/richard$ hd /tmp/1106916564-
00000000 ff fc 25 ff fe 26 ff fb 18 ff fb 20 ff fc 23 ff |..%..&..... ..#.|
00000010 fb 27 ff fc 24 ff fa 20 00 33 38 34 30 30 2c 33 |.'..$.. .38400,3|
00000020 38 34 30 30 ff f0 ff fa 27 00 ff f0 ff fa 18 00 |8400....'.......|
00000030 52 58 56 54 ff f0 ff fd 03 ff fc 01 ff fb 22 ff |RXVT..........".|
00000040 fa 22 03 01 00 00 03 62 03 04 02 0f 05 00 00 07 |.".....b........|
00000050 62 1c 08 02 04 09 42 1a 0a 02 08 0b 02 15 0c 02 |b.....B.........|
00000060 17 0d 02 12 0e 02 16 0f 02 11 10 02 13 11 02 ff |................|
00000070 ff 12 02 ff ff ff f0 ff fb 1f ff fa 1f 00 50 00 |..............P.|
00000080 18 ff f0 ff fd 05 ff fb 21 ff fa 22 01 14 ff f0 |........!.."....|
00000090 ff fd 01 ff fc 22 62 65 6a 74 6c 69 63 68 0d 00 |....."bejtlich..|
000000a0 68 61 63 6b 65 72 70 61 73 73 77 6f 72 64 0d 00 |hackerpassword..|
000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

That is what the client sent to the server. Here is what the server sent to the client:

janney:/home/richard$ hd /tmp/1106916564-
00000000 ff fd 25 ff fb 26 ff fd 18 ff fd 20 ff fd 23 ff |..%..&..... ..#.|
00000010 fd 27 ff fd 24 ff fa 20 01 ff f0 ff fa 27 01 ff |.'..$.. .....'..|
00000020 f0 ff fa 18 01 ff f0 ff fb 03 ff fd 01 ff fd 22 |..............."|
00000030 ff fd 1f ff fb 05 ff fd 21 ff fa 22 01 10 ff f0 |........!.."....|
00000040 ff fa 21 00 ff f0 ff fa 21 03 ff f0 ff fb 01 ff |..!.....!.......|
00000050 fe 22 ff fa 22 03 03 e2 03 04 82 0f 07 e2 1c 08 |.".."...........|
00000060 82 04 09 c2 1a 0a 82 08 0b 82 15 0c 82 17 0d 82 |................|
00000070 12 0e 82 16 0f 82 11 10 82 13 11 82 ff ff 12 82 |................|
00000080 ff ff ff f0 0d 00 0d 0a 46 72 65 65 42 53 44 2f |........FreeBSD/|
00000090 69 33 38 36 20 28 6a 61 6e 6e 65 79 2e 74 61 6f |i386 (janney.tao|
000000a0 73 65 63 75 72 69 74 79 2e 63 6f 6d 29 20 28 74 | (t|
000000b0 74 79 70 36 29 0d 00 0d 0a 0d 00 0d 0a 6c 6f 67 |typ6)........log|
000000c0 69 6e 3a 20 62 65 6a 74 6c 69 63 68 0d 0a 50 61 |in: bejtlich..Pa|
000000d0 73 73 77 6f 72 64 3a 0d 0a 4c 6f 67 69 6e 20 69 |ssword:..Login i|
000000e0 6e 63 6f 72 72 65 63 74 0d 0a 6c 6f 67 69 6e 3a |ncorrect..login:|
000000f0 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ...............|
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

"So what," you might say. I can reconstruct streams with Tcpflow. True enough, but how did we end up with these two streams? These are the result of Flowgrep searching streams for the content 'hackerpassword'. How did the packets which built that stream look? Let's jump to the packets where the server presented "Password:" to the client:

07:49:16.711854 IP >
P 207:216(9) ack 161 win 33304
<nop,nop,timestamp 101794307 107723922>
0x0000: 4510 003d 9342 4000 4006 d09b c0a8 0207 E..=.B@.@.......
0x0010: 0a0a 0a14 b9c5 8011 af45 1755 e813 c462 .........E.U...b
0x0020: 8018 8218 7e23 0000 0101 080a 0611 4203 ....~#........B.
0x0030: 066b bc92 5061 7373 776f 7264 3a .k..Password:

My client acknowledges receipt of the server's 'Password:' packet but sends no data of its own:

07:49:16.713050 IP >
. ack 216 win 5840 <nop,nop,timestamp 107723923 101794307>
0x0000: 4510 0034 790c 4000 3f06 ebda 0a0a 0a14 E..4y.@.?.......
0x0010: c0a8 0207 8011 b9c5 e813 c462 af45 175e ...........b.E.^
0x0020: 8010 16d0 d11b 0000 0101 080a 066b bc93 .............k..
0x0030: 0611 4203 ..B.

Now I enter 'hackerpassword'. This packet has the 'h' -- it's the last ASCII character:

07:49:17.703919 IP >
P 161:162(1) ack 216 win 5840
<nop,nop,timestamp 107724022 101794307>
0x0000: 4510 0035 790d 4000 3f06 ebd8 0a0a 0a14 E..5y.@.?.......
0x0010: c0a8 0207 8011 b9c5 e813 c462 af45 175e ...........b.E.^
0x0020: 8018 16d0 68af 0000 0101 080a 066b bcf6 ....h........k..
0x0030: 0611 4203 68 ..B.h

The server ACKs the packet with the 'h':

07:49:17.796820 IP >
. ack 162 win 33304 <nop,nop,timestamp 101794416 107724022>
0x0000: 4510 0034 9347 4000 4006 d09f c0a8 0207 E..4.G@.@.......
0x0010: 0a0a 0a14 b9c5 8011 af45 175e e813 c463 .........E.^...c
0x0020: 8010 8218 6502 0000 0101 080a 0611 4270 ....e.........Bp
0x0030: 066b bcf6 .k..

I send the 'a'. Again, it's the very last byte of application layer data:

07:49:17.811595 IP >
P 162:163(1) ack 216 win 5840
<nop,nop,timestamp 107724033 101794416>
0x0000: 4510 0035 790e 4000 3f06 ebd7 0a0a 0a14 E..5y.@.?.......
0x0010: c0a8 0207 8011 b9c5 e813 c463 af45 175e ...........c.E.^
0x0020: 8018 16d0 6f36 0000 0101 080a 066b bd01 ....o6.......k..
0x0030: 0611 4270 61 ..Bpa

Next, another server ACK and a client character, 'c':

07:49:17.906823 IP >
. ack 163 win 33304 <nop,nop,timestamp 101794427 107724033>
0x0000: 4510 0034 934a 4000 4006 d09c c0a8 0207 E..4.J@.@.......
0x0010: 0a0a 0a14 b9c5 8011 af45 175e e813 c464 .........E.^...d
0x0020: 8010 8218 64eb 0000 0101 080a 0611 427b ....d.........B{
0x0030: 066b bd01 .k..
07:49:18.087160 IP >
P 163:164(1) ack 216 win 5840
<nop,nop,timestamp 107724061 101794427>
0x0000: 4510 0035 790f 4000 3f06 ebd6 0a0a 0a14 E..5y.@.?.......
0x0010: c0a8 0207 8011 b9c5 e813 c464 af45 175e ...........d.E.^
0x0020: 8018 16d0 6d0e 0000 0101 080a 066b bd1d ....m........k..
0x0030: 0611 427b 63 ..B{c

By now everyone should appreciate just how powerful and useful Flowgrep can be. Even though every character of my 'hackerpassword' string appeared in separate packets, Flowgrep assembled the stream and logged the traffic that matched the filter I specified.

Flowgrep is not the only tool with this capability, since more robust intrusion detection systems offer similar features. However, this is the only stand-alone tool I know that offers rapid string matching on stream contents on arbitrary ports.

I chose to demonstrate telnet because I knew it would place virtually every character I typed into separate packets. The principle applies anywhere you are concerned that content of interest could be split between multiple packets. Furthermore, Flowgrep is designed to watch UDP and ICMP conversations as well.

If you'd like to know more about Flowgrep, check out Jose's presentation to UMeet 2004 and the transcript of the talk. Jose also maintains the Worm Blog, which I intend to peruse, and is part of the Internet Motion Sensor project.

I think Python is a very promising language for rapidly developing these sorts of tools. I have the following books on Python on my to-read list, courtesy of APress:

Expect reviews when I get a chance to read, which is difficult as I work on my own new book.

Wednesday, January 26, 2005

SANS Confuses Threats with Vulnerabilities

In late 2003 I published Dynamic Duo Discuss Digital Risk. This was my light-hearted attempt to reinforce the distinction between a threat and a vulnerability. Specifically, a threat is a party with the capabilities and intentions to exploit a vulnerability in an asset. A vulnerability is a weakness in an asset that could lead to exploitation. An intruder (the threat) exploits a hole (the vulnerability) in Microsoft IIS to gain remote control of a Web server. In other words, threats exploit vulnerabilities.

This is a simple concept, yet it is frequently confused by security prophets like Bruce Schneier in Beyond Fear. Now SANS is making the same mistake in the latest Incident Handler's Diary. In a posting to announce work on the upcoming SANS Top 20 List, the Diary calls the new report the "SANS CRITICAL INTERNET THREATS 2005" and says:

"SANS Critical Internet Threats research is undertaken annually and provides the basis for the SANS 'Top 20' report. The 'Top 20' report describes the most serious internet security threats in detail, and provides the steps to identify and mitigate these threats."

So, are we going to read a ranking of identified Romanian intruders, followed by Russian organized crime, Filipino virus writers, and then Zimbabwean foreign intelligence services? Will mitigation include prosecution, incarceration, and the like? Probably not, as the announcement continues:

"The current 'Top 20' is broken into two complimentary yet distinct sections:
- The 10 most critical vulnerabilities for Windows systems.
- The 10 most critical vulnerabilities for UNIX and Linux systems."

So now we're talking about vulnerabilities. That's what last year's "Twenty Most Critical Internet Security Vulnerabilities" addressed. The announcement concludes:

"The 2005 Top 20 will once again create the experts' consensus on threats - the result of a process that brings together security experts, leaders, researchers and visionaries... In addition to the Windows and UNIX vulnerabilities, this year's research will also focus on the 10 most severe vulnerabilities in the Cisco platforms."

I sincerely hope at least one expert will clue in the announcement-writer concerning the difference between a threat and a vulnerability. Words matter!

Update: While doing some research I found a 1999 report by the Navy's Center on Terrorism and Irregular Warfare called Cyberterror: Prospects and Implications. It says in footnote 11:

"Vulnerability is not synonymous with threat. A vulnerability is a weakness in a system that may be exploited. A threat requires an actor with the motivation, resources, and intent to exploit a vulnerability."

University of California, San Diego Intrusion

I read in the latest SANS NewsBites that UC San Diego suffered another intrusion in November 2004, jeopardizing the personal information of about 3,500 people who had taken courses at UCSD Extension. This incident follows a well-publicized intrusion in April 2004 putting at risk personal data on 380,000 people. In both cases UC appears to have caught unstructured threats, as each intruder used the systems as warez depositories for pirated movies and music.

I was shocked by this claim concerning the latest intrusion:

"Officials said it took two months to notify those who were affected because officials first needed to determine the extent of the breach."

This is exactly why I promote network security monitoring as a means to rapidly scope the extent of intrusions. First, generating indicators and warnings in the form of alert data (usually from IDSs) and statistical data gives security professionals a good chance of identifying an intrusion as it happens or shortly thereafter. I would bet the University saw an increase in traffic when its systems began hosting warez. Second, collecting session and full content data would give the University a chance to inspect data not tied to IDS alerts. Third, all of this information could potentially describe the intruder's activities, and validate if he or she stole sensitive personal information.

Tuesday, January 25, 2005

Snort 2.3.0 Released

Snort 2.3.0 has been released. There appear to be only bug fixes and documentation updates since RC2 arrived last month. At the moment the online manual still shows 2.2.0, but the .pdf packaged with the tarball is the 2.3.0 version. I have not seen any problems with RC2, so I believe the upgrade process should be smooth.

I will get to work on an updated Sguil installation guide shortly, as I've accumulated enough minor fixes to warrant an update.

In related news, Sourcefire was named Frost & Sullivan’s 2005 Network Security Infrastructure Protection Company of the Year. From the announcement:

"Frost & Sullivan presents this Award to the company that demonstrated excellence in all operations. Sourcefire is recognized for its numerous achievements including unique product strategy, important technological developments, and significant gain in market share."

Congratulations Sourcefire!

FreeBSD 4.11 RELEASE Now Available

Those of you running the FreeBSD 4.x tree will be happy to know that FreeBSD 4.11 is now available. The release announcement calls 4.11 "the latest release of the FreeBSD Legacy development branch" and states the following:

"FreeBSD 4.11 will become the first 'Errata Branch.' In addition to Security fixes other well-tested fixes to basic functionality will be committed to the RELENG_4_11 branch after the release... This is expected to be the last release from the RELENG_4 branch."

I am no longer running any 4.x systems and have migrated everything to 5.3.

US Supreme Court Rules on Real False Positives

Last year when US Senator Ted Kennedy was detained for being on a no-fly list, I discussed his plight in relation to intrusion detection system "false positives." If an IDS is operating correctly, every alert it sees is the result of an action it was programmed to take. In other words, when a functioning IDS sees "cmd.exe", it reports seeing "cmd.exe".

It doesn't matter if the appearance of "cmd.exe" on the wire is not part of an actual intrusion; a rule to alert on "cmd.exe" does not cause "false positives" if the IDS reports seeing "cmd.exe". A real false positive involves the IDS reporting "cmd.exe" when no such content passed on the wire. Therefore, there are no such things as false positives. Blame the signature writer or IDS developer, not the IDS.

Let's move from the realm of IDS false positives to the land of canine false positives. Yesterday the US Supreme Court issued its opinion in the case of ILLINOIS, PETITIONER v. ROY I. CABALLES. This is a case where false positives involve a dog's ability to sniff for illegal drugs. Justice Ginsberg's dissenting opinion summarizes the facts of the case:

"Illinois State Police Trooper Daniel Gillette stopped Roy Caballes for driving 71 miles per hour in a zone with a posted speed limit of 65 miles per hour. Trooper Craig Graham of the Drug Interdiction Team heard on the radio that Trooper Gillette was making a traffic stop. Although Gillette requested no aid, Graham decided to come to the scene to conduct a dog sniff.

Gillette informed Caballes that he was speeding and asked for the usual documents–driver’s license, car registration, and proof of insurance. Caballes promptly provided the requested documents but refused to consent to a search of his vehicle. After calling his dispatcher to check on the validity of Caballes’ license and for outstanding warrants, Gillette returned to his vehicle to write Caballes a warning ticket. Interrupted by a radio call on an unrelated matter, Gillette was still writing the ticket when Trooper Graham arrived with his drug-detection dog.

Graham walked the dog around the car, the dog alerted at Caballes’ trunk, and, after opening the trunk, the troopers found marijuana."

Justice Stevens' majority opinion held that "the dog sniff was performed on the exterior of respondent's car while he was lawfully seized for a traffic violation. Any intrusion on respondent's privacy expectations does not rise to the level of a constitutionally cognizable infringement... A dog sniff conducted during a concededly lawful traffic stop that reveals no information other than the location of a substance that no individual has any right to possess does not violate the Fourth Amendment." In other words, it's ok for police to use dogs to inspect cars for drugs during traffic violation stops (or at other times), even if there is no suspicion of drugs involved.

I do not agree with this opinion, for several reasons. The first reason involves false positives, and was correctly diagnosed in Justice David Souter's dissenting opinion:

"I would hold that using the dog for the purposes of determining the presence of marijuana in the car’s trunk was a search unauthorized as an incident of the speeding stop and unjustified on any other ground...

The infallible dog, however, is a creature of legal fiction... [T]heir supposed infallibility is belied by judicial opinions describing well-trained animals sniffing and alerting with less than perfect accuracy, whether owing to errors by their handlers, the limitations of the dogs themselves, or even the pervasive contamination of currency by cocaine...

In practical terms, the evidence is clear that the dog that alerts hundreds of times will be wrong dozens of times.

Once the dog’s fallibility is recognized, however... the sniff alert does not necessarily signal hidden contraband, and opening the container or enclosed space whose emanations the dog has sensed will not necessarily reveal contraband or any other evidence of crime."

Justice Ginsberg expresses the second reason for my disagreement. Returning to her dissent, we see that beyond a Fourth Amendment violation, there are other problems with allowing canine searches prone to false positives:

"A drug-detection dog is an intimidating animal... Injecting such an animal into a routine traffic stop changes the character of the encounter between the police and the motorist. The stop becomes broader, more adversarial, and (in at least some cases) longer. Caballes –- who, as far as Troopers Gillette and Graham knew, was guilty solely of driving six miles per hour over the speed limit -– was exposed to the embarrassment and intimidation of being investigated, on a public thoroughfare, for drugs...

Under today’s decision, every traffic stop could become an occasion to call in the dogs, to the distress and embarrassment of the law-abiding population...

Today’s decision... clears the way for suspicionless, dog-accompanied drug sweeps of parked cars along sidewalks and in parking lots... Nor would motorists have constitutional grounds for complaint should police with dogs, stationed at long traffic lights, circle cars waiting for the red signal to turn green."

My third and final reason for disagreeing with the Court's opinion is based on Justice Stevens' majority opinion. He writes for the Court:

"We have held that any interest in possessing contraband cannot be deemed 'legitimate,' and thus, governmental conduct that only reveals the possession of contraband 'compromises no legitimate privacy interest.'"

Now, what if the definition of contraband is extended beyond illegal drugs? How about music or movies in digital form, or pirated software? Is the Court opening the door to knock down privacy rights, since means to discover contraband do not infringe Fourth Amendment rights? The Court continues:

"The legitimate expectation that information about perfectly lawful activity will remain private is categorically distinguishable from respondent’s hopes or expectations concerning the nondetection of contraband in the trunk of his car."

The Court also brushes aside the false positive concerns:

"Although respondent argues that the error rates, particularly the existence of false positives, call into question the premise that drug-detection dogs alert only to contraband, the record contains no evidence or findings that support his argument."

I find this ruling very disturbing. I expect to see canine units used in increasing numbers in the coming months, where false positives will continue to plague innocent people. For example, yesterday National Public Radio reported that a man carrying cash to close on his house purchase was arrested when a dog alerted to supposed traces of illegal drugs on the money. Apparently traces of drugs on US currency is not an urban legend!

Saturday, January 22, 2005

FreeBSD Release Engineer Scott Long on SMP

I highly recommend that FreeBSD enthusiasts read Scott Long's interview at He succinctly explains FreeBSD's new Symmetric MultiProcessing Next Generation (SMPng) code and answers good questions posed by interviewer Federico Biancuzzi.

Since I maintain multiple Dell PowerEdge 750 servers with Hyper-Threading Technology (HTT), I found Scott's comments on gains from HTT to be interesting. It seems that HTT will only be useful once the new ULE scheduler is equipped to make use of HTT and ULE replaces the 4BSD scheduler. Scott says:

"The other design goal of ULE was to have it map out and understand the CPU topology and make good scheduling choices for features like Hyper-Threading. Unfortunately, to my knowledge this work is not yet complete.

[Frederic asks:] Does SMPng improve performance on Intel Hyper-Threading capable CPUs?

Scott Long: As of right now, very little. The scheduler really needs to be aware of Hyper-Threading and schedule threads and processes appropriately so that the caches and TLBs [Transaction Look-aside Buffers] can be shared and not get thrashed. The ULE scheduler will fill this role in the future, but it's not there yet."

Scott also discusses how the network stack will benefit from the removal of the GIANT lock, and how the Pf firewall imported from OpenBSD already runs without GIANT, unlike IPFW. This means Pf is believed to be faster on FreeBSD than IPFW.

Friday, January 21, 2005

Application Vulnerabilities Are Not New

This morning I read the new @RISK: The Consensus Security Alert from SANS and friends. It begins with this comment:

"Prediction: This is the year you will see application level attacks mature and proliferate. As hackers focus more on applications, Oracle may start competing with Microsoft as the vendor delivering software with the most critical vulnerabilities."

I hear this focus on "applications" constantly, but this is old news. First look at the problem by separating the operating system (OS) kernel from the OS applications. If we look at vulnerabilities in this respect, "applications" have been under attack for decades. Perusing the CERT Advisories list (transitioned to the US-CERT's Technical Cyber Security Alerts in 2004), we see warnings about application vulnerabilities since 1988. For example, in December 1998 we have CA-1988-01: ftpd Vulnerability.

You might say that my separation of OS kernel and OS applications doesn't capture the spirit of SANS' "prediction." You might think that their new warning means we should focus on applications that don't ship with the "OS." In other words, look at widely deployed applications that aren't bundled with an OS installation CD. Using that criteria, "application attacks" are still old news. Check out this July 2001 advisory, CA-2001-16: Oracle 8i contains buffer overflow in TNS listener. That was followed a month later by CA-2001-24: Vulnerability in OpenView and NetView and three months later by CA-2001-29: Oracle9iAS Web Cache vulnerable to buffer overflow.

Maybe my background as a history major is at work here, but I think "hackers" have been attacking applications for years.

Wednesday, January 19, 2005

Microsoft Third-Party Patch Testing

I just read an article titled Microsoft Turns to External Patch Testers. The goal is "is to provide a small number of dedicated external evaluation teams with access to the [beta] patches to test for application compatibility, stability and reliability in simulated production environments." This article cites a Microsoft rep saying "'This is a very controlled program... We have only invited participants with whom we have a close relationship, where we are sure that confidentiality will be maintained.'"

This comment makes me question if Microsoft understands what it is doing: Stephen Toulouse, program manager at the Microsoft Security Response Center, "made it clear that the outside testers had no access to information on the vulnerability addressed by the patch. 'They're evaluating the updates in a private, closed-lab environment. They are required to sign an NDA [nondisclosure agreement] and they don't ever know what the patch is correcting. They're simply simulating a real-world deployment in a lab environment and looking for potential problems,' Toulouse said."

At the very least, patch recipients will be able to see what files were changed on the target system if they use file integrity verification software. The testers may not know exactly what problem is being corrected, but any competent tester will know that XYZ.dll and ABC.dll have been replaced by Microsoft's beta versions.

Any program involving greater testing of patches is probably a good idea. However, Microsoft should have realistic expectations concerning the sharing of information on replacement of .dlls and other Windows components.

Tuesday, January 18, 2005

Security Subject Matter Experts Sought as Authors

Last night I started working on my next book: Extrusion Detection: Security Monitoring for Internal Intrusions. The goal of this book is to help security architects and engineers control and instrument their networks, and help analysts investigate security events.

Extrusion Detection is a sequel to my first book, The Tao of Network Security Monitoring: Beyond Intrusion Detection.

Extrusion Detection explains how to engineer an organization's internal network to control and detect intruders launching client-side attacks. Client-side attacks are more insidious than server-side attacks, because the intruder targets a vulnerable application anywhere inside a potentially hardened internal network. A powerful means to detect the compromise of internal systems is to watch for outbound connections from the victim to systems on the Internet operated by the intruder. Here we see the significance of the word "extrusion" in the book's title. In addition to watching connections inbound from the Internet, we watch for suspicious activity exiting the protected

Readers will learn theory, techniques, and tools to implement network security monitoring (NSM) for internal intrusions. I have already received several case studies from LURHQ and I have contacted an expert on p2p networks who plans to write a chapter. I am interested in hearing from any blog readers who might want to contribute a case study, section, chapter, on appendix on one or more of the following subjects:

- Interpreting Microsoft Server Message Block (SMB) (port 139, 445 TCP) protocols
- Microsoft's Network Access Protection (NAP)
- Cisco's Network Admission Control (NAC) technologies.
- 802.1x
- VLANs and VLAN access control lists
- Cisco Network Access Module and similar means to collect traffic on network hardware
- Using FPGAs, network processors, or other non-libpcap methods to capture network traffic in high bandwidth environments
- Using proxies to inspect and carry traffic from internal systems to the Internet -- the more exotic, the better
- Any case studies involving compromise of internal systems, such as via VPN to partner networks, attaching rogue laptops, opening malicious email or visiting evil Web sites
- Anything else you think would be cool to discuss in a book on controlling, detecting, and responding to internal threats -- as long as it doesn't appear in other books!

If you have an idea you'd like to discuss, please email taosecurity at gmail dot com. You will receive full credit for anything you submit that makes it in some form into the final book, even if I have to rewrite some or all of it to meet publishing guidelines. Thank you!

Register for ShmooCon 2005 Today

I'd like to thank higB of for reminding me to register for ShmooCon 2005. This is "an all-new, annual East coast hacker convention hell-bent on offering an interesting and new atmosphere for demonstrating technology exploitation, inventive software & hardware solutions, as well as open discussion of critical information security issues." The program looks great, and you can't beat the $199 price tag (pay before 1 Feb) for a 3 day con (Fri 4 Feb - Sun 6 Feb). If you plan to join me for ShmooCon in DC, reply to this post.

Marty Roesch Explains Snort Stream Strategy

Today in #snort-gui on, Marty Roesch of Snort fame explained how Snort handles stream reassembly:

roesch: when stream4 is doing it's thing it queues the tcp segments as they come in

roesch: in stream4 we actually queue the entire packet and keep a pointer to the payload to management reassembly

roesch: "flushing" is what happens when we accumulate a certain number of bytes on a stream that's in excess of the "flush point" for that stream

roesch: when we flush, we reassemble the segments into a pseudopacket and run it back thru the preprocessor stack and detection engine

roesch: if there's a detect, we ask stream4 to log all the queued *packets* on the stream

roesch: the first packet gets identified as the attack packet and the rest of them are tagged off of that event

roesch: so if you're detecting on "foobar" and it's been spread across three packets as "fo" "ob" "ar" then you're going to get one even packet and two tagged packets

roesch: this was in 2.1.x or maybe 2.2

roesch: the idea is that we don't want to log the pseudopacket since it's pretty much "inadmissable" from a evidence standpoint

qru: roesch: Yeah, I always hated that thing. What do you do w/the pseudo packet then?

roesch: we chuck it

roesch: as an analyst you'll need to have something that can reassemble the segments and present them to you

roesch: which in theory is pretty easy but in implementation is a pain if you've got an evasive attacker

This explanation is important for several reasons. First, it's important to understand how your IDS works. If you don't understand how it works, you're less likely to trust the alert data it generates. If you don't trust IDS alerts, why are you collecting them?

Second, this stream implementation represents a trade-off between capability and performance. Sensors are not built with unlimited ability to capture and reassemble traffic. Anything you can do to make the traffic stream cleaner for your sensor, like packet scrubbing, helps.

Third, Marty demonstrates that the pseudopacket that Snort presents to an analyst may not be an actual packet that crossed the wire. If an analyst wants to see exactly what passed by the sensor, she must turn to full content data collected independently of the alert data generation with Snort.

Confirmation That I Need a Mac Mini

When I last wrote about buying a Mac mini, I wondered if it would boot headless. This helpful blog entry explains that the Mac mini will indeed boot headless. I don't plan to buy a Mac mini until Mac OS X Tiger arrives later this year. If I need help installing X on a Mac, I can reference Configuring and Running X11 Applications on Mac OS X. If I need an office suite, I'll use NeoOffice.

Monday, January 17, 2005

FreeBSD July-December 2004 Status Report Highlights

The FreeBSD July-December 2004 Status Report is now available. I found the following entries to be important.

From Release engineering:

"FreeBSD 5.4 release engineering will start in March, and FreeBSD 5.5 release engineering will likely start in June. These releases are expected to be more conservative than previous 5.x releases and will follow the same philosophy as previous -STABLE branches of fixing bugs and adding incremental improvements while maintaining API stability.

For the 6-CURRENT development branch as well as all future development and stable branches, we are planning to move to a schedule with fixed timelines that move away from the uncertainty and wild schedule fluctuations of the previous 5.x releases. This means that major branches will happen at 18 month intervals, and releases from those branches will happen at 4 month intervals. There will also be a dedicated period of testing and bug fixing at the beginning of each branch before the first release is cut from that branch. With the shorter and more defined release schedules, we hope to lessen the problem of needed features not reaching users in a reasonable time, as happened too often with 5.x. This is a significant change in our strategy, and we look forward to realizing the benefits of it. This will kick off with the RELENG_6 branch happing in June of 2005, followed by the 6.0 release in August of 2005.

Also on the roadmap is a plan to combine the live-iso disk2 and the install distributions of disk1 into a single disk which can be used for both installation and for recovery. 3rd party packages that currently reside on disc1 will be moved to a disk2 that will be dedicated to these packages. This move will allow us to deal with the ever growing size of packages and also provide more flexibility to vendors that wish to add their own packages to the releases. It also opens the door to more advanced installers being put in place of sysinstall."

From OpenBSD packet filter:

"FreeBSD 5.3 is the first release to include PF. It went out okay, but some bugs were discovered too late to make it on the CD. It is recommend to update `src/sys/contrib/pf' to RELENG_5. The specific issues addressed are:

* Possible NULL-deref with user/group rules.
* Crash with binat on dynamic interfaces.
* Silent dropping of IPv6 packets with option headers.
* Endless loops with `static-port' rules."

From port status:

" 2.0 is planned to be released in March 2005. Currently developer snapshot versions are available. Now one of the developer version has been ported, and committed to ports tree (/usr/ports/editors/openoffice-2.0-devel)."

1.1.4 has been ported and committed to ports tree. Packages are available.

Invoking from command line has been changed. Now `.org' is mandatory. e.g. openoffice-1.1.4 -> Since the name of the software is, not OpenOffice. We are also considering the name of the ports (/usr/ports/editors/openoffice-2.0-devel -> openoffice.org2-devel etc)."

Saturday, January 15, 2005

More Tools for Network Security Monitoring

The February issue of Sys Admin magazine features a new article I wrote titled More Tools for Network Security Monitoring. It discusses Edwin Groothuis' Dhcpdump, Matt Shelton's Passive Asset Detection System (PADS), and John Curry's Security Analyst Network Connection Profiler (SANCP).

On a related note, the January 2005 issue of Information Security magazine mentions this blog, along with two others (Bruce Schneier and Mikko Hypponen), as reader favorites. Thank you to the anonymous contributor who said "Richard Bejtlich provides some of the best insights into network security... his blog has given me an ongoing extension to his book." Some of my favorite blogs include Aaron Higbee and friends, Chris Reining, David Bianco, Anton Chuvakin, Dru Lavigne, and Beau Monday.

Earler this month Slashdot announced the publication of Matt Blaze's new paper Safecracking for the Computer Scientist (.pdf, 2.5 MB). Bruce Schneier's blog pointed me to an alt.locksmithing thread where locksmiths debate full disclosure in light of Matt's article.

The thread starts with the usual defense of "security through obscurity" one might expect:

"As many of you know Matt Blaze a professor at Pennsylvania University has published an article that reveals proprietary techniques of safe penetration. It was featured on well known hacker website recently, and it came to our attention on Saturday. It includes information normally reserved to the trade, for good reasons that need not be discussed here. The article is available to the general public without any restrictions whatsoever. We as professionals in the security field are outraged and concerned with the damage that the spread of this sensitive information will cause to security and to our profession."

Here is an educated response to this foolish opinion:

"I think you meant to say: We have to nip it in the bud or soon there will be no __APPEARANCE_OF__ security left. This is so silly on so many levels. You sell a product that has known deficiencies so that you can break in when you need to. Then you act like it's a big deal when someone talks about it! On top of that you act like it's a matter of national security when, in fact, it changes nothing.

It does not take a brain surgeon to figure out that anyone can buy a safe, disassemble it and figure out it's weaknesses. The fact that every single copy of model X is built the same way is planned insecurity. Now THAT's a crime. That they are sold as secure when they are not is a crime.

If you want to get Blaze to protect your job, that's understandable. To villify him for openly discussing what is known within the industry to be common shortcomings is shear hypocrisy.

I'm still waiting for SCHLAGE to notify folks that it's recalling their defective entry locks. Wait, they can't so that without disclosing that they are insecure, so only the locksmiths and burglers know."

One response shows that lock vendors are acting exactly like software vendors not held accountable for producing flawed software:

"The fact of the matter is the lock manufactuers, Ingersol Rand and Black and Decker being the two largest ones here in the states, dont want to spend a dollar or two more on their locks to improve them. They would rather put out pot metal junk that offers only a since of security. If the public in general only knew what I know, that being the fact that Kwikset and Titan locks are junk, the famous Schlage 'Maximam Security Deadbolt' is pot metal, Yale is no longer up to par, Sentry safes are worthless."

For a 1991 document on picking locks, check out the Guide to Lock Picking, hosted at a real "hacker site" -- MIT.

Friday, January 14, 2005

Can You Trust Your Cell Phone, PDA, Blackberry...?

SecurityFocus writer Kevin Poulsen continues to provide some of the best original security reporting in his new story Hacker penetrates T-Mobile systems. This is an absolutely fascinating must-read. From the story:

"By August 5th the agents already had a good idea what was going on, when Ethics made a fateful mistake. The hacker asked the Secret Service informant for a proxy server -- a host that would pass through Web connections, making them harder to trace. The informant was happy to oblige. The proxy he provided, of course, was a Secret Service machine specially configured for monitoring, and agents watched as the hacker surfed to "My T-Mobile," and entered a username and password belonging to Peter Cavicchia, a Secret Service cyber crime agent in New York.

Cavicchia was the agent who last year spearheaded the investigation of Jason Smathers, a former AOL employee accused of stealing 92 million customer e-mail addresses from the company to sell to a spammer. The agent was also an adopter of mobile technology, and he did a lot of work through his T-Mobile Sidekick -- an all-in-one cellphone, camera, digital organizer and e-mail terminal. The Sidekick uses T-Mobile servers for e-mail and file storage, and the stolen documents had all been lifted from Cavicchia's T-Mobile account, according to the affidavit. (Cavicchia didn't respond to an e-mail query from SecurityFocus Tuesday.)

By that time the Secret Service already had a line on Ethic's true identity. Agents had the hacker's ICQ number, which he'd used to chat with the informant. A Web search on the number turned up a 2001 resume for the then-teenaged Jacobsen, who'd been looking for a job in computer security. The e-mail address was listed as

The trick with the proxy honeypot provided more proof of the hacker's identity: the server's logs showed that Ethics had connected from an IP address belonging to the Residence Inn Hotel in Buffalo, New York. When the Secret Service checked the Shadowcrew logs through a backdoor set up for their use -- presumably by the informant -- they found that Ethics had logged in from the same address. A phone call to the hotel confirmed that Nicolas Jacobsen was a guest."

I strongly recommend reading the whole article for context, but the four italicized sections yield some interesting lessons:

- Intruders: Don't trust proxies you don't own. Don't post your resumes online. Trust no one!
- Agents: Don't store sensitive information on technology or services you don't control or thoroughly understand.

Review of Gray Hat Hacking Posted just published my four star review of Gray Hat Hacking. From the review:

"'Gray Hat Hacking' (GHH) is positioned as a next-generation book for so-called ethical hackers, moving beyond the tool-centric discussions of books like 'Hacking Exposed.' The authors leave their definition of 'gray hat' unresolved until ch 3, where they claim that a 'white hat' is a person who 'uncovers a vulnerability and exploits it with authorization;' a 'black hat' is one who 'uncovers a vulnerability and illegally exploits it and/or tells others how to;' and a 'gray hat' is one who 'uncovers a vulnerability, does not illegally exploit it or tell others how to do it, but works with the vendor.' I disagree and prefer's definitions, where white hats find vulnerabilities and tell vendors without providing public exploit code; black hats find vulnerabilities, code exploits, and maliciously attack victims; and gray hats find vulnerabilities, publish exploits, but do not illegally use them. According to these more common definitions, the book should have been called 'White Hat Hacking.' I doubt it would sell as well with that title!"

My review echoes most of Patrick Mueller's review in Information Security magazine, except for his comment that "The authors did, however, deliver on their ethical obligations to provide accurate countermeasures to the attack methods they describe--a true value to readers." This makes no sense to me. Defense gets a short 10 page chapter, which should have been dropped and replaced by a reference to any of the extensive tomes written about network defense.

Metasploit 2.3 Released

I wrote about the Metasploit Framework in April 2004. The Metasploit Framework is an advanced open-source platform for developing, testing, and using exploit code. This week they released version 2.3, which offers 3 user interfaces, 46 exploits, and 68 payloads. One of the more interesting additions is the Meterpreter (.pdf). This is a replacement for calling cmd.exe on Windows after an exploit succeeds. Windows support is currently offered and UNIX (to replace calling /bin/sh) is planned. The Meterpreter is extensible, so you can add features once you gain control of the target. You can browse the exploits and payloads using their Web-based interface.

Tuesday, January 11, 2005

Got to Get Me a Mini

I think Sun and Apple are doing real innovation in the commercial software and hardware spaces, unlike many of their competitors. I already own an old Sun Ultra 30, and I plan to buy several Sun Ray thin clients at work. I've been looking for an excuse to get an Apple system of some kind ever since Mac OS X was released. Since I run FreeBSD on my Thinkpad a20p laptop, I don't need another desktop or laptop system. I've also vowed to never buy another tower form factor PC again. It's either small form factor, laptops, or rackmounts from here on. That left buying an Apple Xserve, which was more horsepower than I could justify buying.
Today, Apple released the Mac mini, pictured at right and in Steve Jobs' hands above. This looks like a great little box. Take a look at the back side below. Although it only has one built-in Ethernet port, those two USB 2.0 ports say "additional NICs" to me, assuming I can get a USB-based NIC to work with OS X.
I don't plan to use this Mac as a desktop. Rather, I'll deploy it on my wire shelving in my lab and access it remotely. I'm wondering if I can boot this baby without a monitor attached. This MacFixIt tip suggests connecting an Apple display adapter to fool the Mac into booting without a display attached. It looks like Fink provides VNC packages, so I can access the whole Mac OS X desktop remotely.

If I buy a Mac mini, I'll report how I use it. Maybe once I finish this new book?

Using a Router to Exchange VLAN Traffic

In late 2003 I described how part of my home network looked:

cable modem - cisco router - freebsd fw/gw - cisco switch - clients

This past weekend I decided to remove the firewall/gateway from the picture. When the router is deployed like this, it's called a "router on a stick."

cable modem - cisco router - cisco switch - clients

In that late 2003 story I explained how I set up 802.1q on the FreeBSD system to pass traffic between VLANs on the Cisco switch. Without that FreeBSD in place, I needed to configure my Cisco 2651XM router to exchange inter-VLAN traffic.

Luckily this Cisco document came to the rescue. The process was fairly simple. I administered the router via console cable, so none of my changes resulted in being locked out of one of the interfaces. I don't recommend letting anyone be able to connect to a Cisco router interface, in any case. (For a great presentation on router security, check out this .pdf of a presentation by Sean Convery and Matthew Franz.)

First I removed the IP address previously assigned to the interface facing the switch:

int fa0/1
no ip address

Next I created an IP address to handle VLAN 10, which is a network. Note the use of '0/1.1' instead of '0/1':

int fa0/1.1
encapsulation dot1Q 10
ip address

Then I created an IP address to handle VLAN 20, which is a network. Note the use of '0/1.2':

int fa0/1.2
encapsulation dot1Q 20
ip address

That's it. Now if a system on VLAN 10 needs to talk to a system on VLAN 20, the router will pass the traffic.

Monday, January 10, 2005

Linux Kernel Development Problems

Today's Slashdot features Security Holes Draw Linux Developers' Ire. Essentially the GRSecurity Linux security patch developers are upset about the lack of response to their discovery of Linux kernel vulnerabilities. This article by Brad Spengler features the 31337 technique used to find the holes:

"Using 'advanced static analysis':

cd drivers; grep copy_from_user -r ./* |grep -v sizeof

I discovered 4 exploitable vulnerabilities in a matter of 15 minutes. More vulnerabilities were found in 2.6 than in 2.4. It's a pretty sad state of affairs for Linux security when someone can find 4 exploitable vulnerabilities in a matter of minutes."

I am disappointed that this is the case. I am not a kernel developer so I won't comment on the difficulties associated with removing these sorts of vulnerabilities. However, some of those that are kernel developers do not seem to be heeding the warnings in books like Building Secure Software, which I reviewed last week. This is an unfortunate indictment of part of our software engineering community, especially when Linux is being deployed in ever more important places.

More disturbing for me was this email from kernel developer Ted Ts'o in the linux-kernel mailing list:

"Not all 2.6.x kernels will be good; but if we do releases every 1 or 2 weeks, some of them *will* be good."

I could be accused of taking this out of context, but to me this sort of thinking is not what I want to hear associated with a kernel called stable. This is exactly the point of the Slashdot commentator who brought this email to my attention. I saw the same mentality in The Hacker Ethic, where ESR criticizes the BSD development model:

BSD is "carefully coordinated... by a relatively small, tightly knit group of people" [in comparison with Linux, where] quality was maintained not by rigid standards or autocracy but by the naively simple strategy of releasing every week and getting feedback."

I prefer the BSD model, where users and administrators know that CURRENT is bleeding edge and STABLE is more or less that -- "stable." Those that need even more "stability" can track a security release, where the primary changes are security fixes and critical bux fixes.

I think if we continue to see this sort of development process, Linux vendors will have no choice but to heavily patch the "vanilla" Linux kernel and provide that patched version in their distros. They of course can do that, but I believe such patching contributes to the fragmentation of the Linux community. That increases the level of difficulty of writing projects like l7-filter, which itself requires patches for the Linux kernel to operate.

Sunday, January 09, 2005

Setting Up BIND 9.3 on FreeBSD 5.3

Today I moved my local name resolution duties from a FreeBSD 4.x system to a FreeBSD 5.3 system. I found the FreeBSD Handbook sparse reading, but this article gave a few more pointers. Here's what I ended up doing.

The first step is to recognize that /etc/namedb is a symlink:

janney:/home/richard$ ls -al /etc/namedb
lrwxrwxrwx 1 root wheel 23 Nov 8 09:14 /etc/namedb -> ../var/named/etc/namedb

This supports the new default of running BIND in a jail.

Next I created the localhost-v6.rev and localhost.rev files in /var/named/etc/namedb/master.

janney:/home/richard$ cd /var/named/etc/namedb/
janney:/var/named/etc/namedb$ sudo sh make-localhost

I altered the serial numbers by adding '01' to the end to allow 99 edits per day. (Using the default '20050109' yields one edit per day, if you want your serial number to be related to the day you change it. This is totally optional but I find it helpful.)

janney:/var/named/etc/namedb/master$ cat localhost.rev
; From: @(#)localhost.rev 5.1 (Berkeley) 6/30/90
; $FreeBSD: src/etc/namedb/PROTO.localhost.rev,v 1.6 2000/01/10 15:31:40 peter Exp $
; This file is automatically edited by the `make-localhost' script in
; the /etc/namedb directory.

$TTL 3600

@ IN SOA (
2005010901 ; Serial
3600 ; Refresh
900 ; Retry
3600000 ; Expire
3600 ) ; Minimum

Next I ran rndc-confgen to create a key which authorizes me to administer the BIND server. Notice that rndc replaces ndc for controlling BIND.

janney:/home/richard$ sudo rndc-confgen -a
wrote key file "/etc/namedb/rndc.key"
janney:/home/richard$ sudo cat /etc/namedb/rndc.key
key "rndc-key" {
algorithm hmac-md5;
secret "OBSCURED";

I then added that information plus a control statement to /var/named/etc/namedb/named.conf:

controls {
inet allow { localhost; } keys { rndc-key; };

key "rndc-key" {
algorithm hmac-md5;
secret "OBSCURED";

I found I had a file /etc/rndc.conf that had a matching key:

options {
default-server localhost;
default-key "rndc-key";

server localhost {
key "rndc-key";

key "rndc-key" {
algorithm hmac-md5;
secret "OBSCURED";

With this infrastructure in place, I essentially copied my old zone configuration files into /var/named/etc/namedb. I made sure to update all of the serial numbers on files with changes. Once done I used the new rc scripts to restart named:

janney:/etc/namedb$ sudo /etc/rc.d/named restart
Stopping named.
Starting named.

I checked to see if I could query BIND's status as a user, but then had to use sudo because they rndc.key file was not readable for users:

janney:/etc/namedb$ rndc status
rndc: error: none:0: open: /etc/namedb/rndc.key: permission denied
rndc: could not load rndc configuration
janney:/etc/namedb$ sudo rndc status
number of zones: 7
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is OFF
recursive clients: 0/1000
tcp clients: 0/100
server is up and running

Everything seems to be working. My /etc/rc.conf file has these DNS-related entries:


Saturday, January 08, 2005

Investigative Leads for Network Security Monitoring

When I worked incident response for Foundstone, my boss Kevin Mandia taught me about "investigative leads." This is a Bureau/law enforcement term for items which are recognized as important in a report but require additional scrutiny. I have several network security monitoring investigative leads which I have not yet had time to follow. I list them here in the event one or more of my readers have checked them out:

  • In November Dave Aitel of Immunity, Inc. posted an announcement of his company's CANVAS Reference Implementation (CRI). CANVAS is a penetration testing toolkit consisting of private exploits written by Immunity, Inc. The CRI is a subset of CANVAS, available for free under NDA, aimed at those wishing to test IDS and layer 7 firewalls (aka "IPS"). I plan to try this out soon, but don't expect public results due to the NDA.

  • There's an extended focus-ids thread discussing the need for packet capture and the problems of doing so in high bandwidth environments. Anyone who has seen my Wish List will notice I am researching hardware-based approaches to the problem, like network processors, FPGAs, and microcontrollers.

  • A friend pointed me to l7-filter, an "Application Layer Packet Classifier for Linux." This looks really cool. Along with the upcoming release of Snort 2.3 with integrated inline capabilities, I'm being forced to deploy one or more Linux boxes to try these features. If l7-filter is able to profile traffic running on arbitrary ports, it will give open-source-bound NSM analysts a powerful new capability.

  • If you have trouble justifying your monitoring duties, you'll face less resistance if you share Wanted: Chief Espionage Officer with the doubting parties. I have yet to read all of this article, but it's a detailed look at (illegal) corporate intelligence gathering.

Regarding the third point -- would anyone care to suggest a Linux distro for my snort-inline and l7-filter projects? I'm going to be running on minimal hardware without X. I'm leaning toward Debian or Slackware and away from Fedora Core, Mandrake, and Gentoo. I'd like a Linux distro that uses the kernel as-is, or as much as possible. Is there such a thing? Coming from BSD-land, I'm not current on the Linux scene. Thank you.

Happy 2nd Birthday TaoSecurity Blog

Today is the 2nd birthday of the TaoSecurity blog. Thank you to all of my readers. The primary purpose of this blog is to be a "hard drive for my brain." In other words, I mainly record how I accomplish certain tasks, or I put context around security events and related developments. I hope you find the content useful and relevant.

Friday, January 07, 2005

Thoughts on "Benchmark Comparison of NetBSD 2.0 and FreeBSD 5.3

Several people have asked me to comment on George McGarry's Benchmark Comparison of NetBSD 2.0 and FreeBSD 5.3. My initial reaction to this article is disappointment. I am not upset because the author says his "results indicate that NetBSD has surpassed FreeBSD in performance on nearly every benchmark and is poised to grab the title of the best operating system for the server environment." I am disappointed that the author has decided to use his "results" in a divisive manner. Rather than seek to learn from each BSD project and potentially compete better with Linux or Windows, George decides to drive a wedge between the NetBSD and FreeBSD communities.

Regarding the nature of the delivery of this report, George demonstrates he apparently doesn't know how to use email. Instead of posting his findings himself, he relies on Hubert Feyrer to post the abstract for the report in multiple newsgroups, like freebsd-advocacy, freebsd-performance, netbsd-advocacy, and NetBSD's tech-perform.

While I find Hubert's blog to be a good independent source of NetBSD information, I question why he is the delivery man. Why can't George post for himself, and answer replies publicly? If you're going to make strong claims, the author should be willing to stand up for his beliefs in person. None of these postings have been followed by responses from George or Hubert.

Regarding the "results" themselves, I found three responses interesting. First, Stheg Olloydson points out that George's test results do not support his conclusions. Stheg uses excerpts from George's paper to make his case.

Second, Robert Watson explains the technical decisions behind FreeBSD's architecture. Robert says:

"Regarding SMP or not -- the path the FreeBSD Project has taken (and this choice was before I was really all that involved, to be honest) was a re-architecture of the kernel to improve performance, scalability, and structure via a movement to a parallelizable, preemptible, threaded kernel. I think this is the right architecture to move to, as it not only improves performance and scalability, but it also closes a lot of existing race conditions in the kernel that only became more exposed as threading and SMP became more predominant. This has had a lot of performance benefits, but comes with initial costs that aren't all immediately offset by initial benefits. Now that this model is largely adopted, we'll see a nice increase in benefits over time -- i.e., it was an investment."

The third response comes from Kris Kennaway, who mentions the controversial nature of the benchmarks. Kris also says:

"There's a leap from 'NetBSD performs better in microbenchmarks' to 'NetBSD is a better-performing server' (macrobenchmarks often do not reflect the same performance characteristics as microbenchmarks, although of course they are influenced by them)."

Slashdot is covering this story now, and there are a few helpful comments in the responses.

I personally use whatever operating system best suits the project at hand. For example, I prefer to use OpenBSD for firewalls and FreeBSD for general purpose servers. I have plans for a NetBSD system to become a terminal server, since it supports an obscure piece of hardware on that box and the machine offers only 32 MB RAM.

In conclusion, I would be much happier to see performance comparisons, especially between BSD versions, used to improve the collective performance of each variant. Instead, we see trolls here and here using these "results" to justify personal vendettas. I hope to see personal replies from George in the near future, or I will continue to be suspicious of his motives.

Tuesday, January 04, 2005

Review of Building Secure Software Posted just posted my five star review of Building Secure Software. From the review:

"'Building Secure Software' (BSS) is an excellent book. I can't believe it was published in the fall of 2001, and I've only gotten to it now. Negative reviewers should remember that a single book can't address every security topic under the sun. BSS is the first of several titles by authors Viega and McGraw; those looking for additional details can peruse their later books."

Monday, January 03, 2005

IPSec Tunnels with FreeBSD

Although the FreeBSD Handbook offers a VPN over IPSec section, it doesn't describe the scenario I face when deploying network security monitoring sensors. That document also references commands that no longer exist in FreeBSD 5.3, like 'gifconfig.' My architecture looks like this (all IP addresses are obfuscated):

remote sensor
interface em0
public management IP
VPN concentrator
and NAT gateway
interface em0
external public IP
interface em1
internal private IP
monitoring backend
interface em0
internal private IP

I need to encrypt communications from the sensor to the monitoring backend. This can involve multiple individual sockets. I don't like to use OpenSSH port forwarding or Stunnel because I must set up a separate port forwarding or tunnel session for each channel. I would much rather use IPSec, since that can carry any communications between the sensor and the backend.

Complicating matters, I need to communicate between a sensor with a public management IP and a backend with an internal private IP address. That backend internal private IP address is transformed using NAT on the VPN concentrator and NAT gateway. All boxes in this scenario run FreeBSD 5.3 RELEASE.

One answer to this problem, and the approach I use, is to create a virtual tunnel from the sensor to the gateway, through which traffic to and from the backend can pass. I will use the gif facility in FreeBSD. This will create an IP-in-IP tunnel, which I will then wrap inside IPSec ESP.

Here is the architecture, with gif tunnels added:

remote sensor
interface gif0
private tunnel IP
interface em0
public management IP
VPN concentrator
and NAT gateway
interface em0
external public IP
interface gif0
private tunnel IP
interface em1
internal private IP
monitoring backend
internal private IP

The monitoring backend will communicate with when it needs to talk to the sensor. The sensor will communicate with when it needs to talk to the backend. The gateway will take care of connecting the two endpoints.

The first step is to recompile the kernels of the sensor and gateway to suit their roles. Here is what I add to the sensor's kernel config file before recompiling the kernel:

options FAST_IPSEC
device crypto

Here is what I add to the gateway's kernel config file before recompiling the kernel. The last two lines are completely optional, but the IPFIREWALL_DEFAULT_TO_ACCEPT means I don't need to add rules to permit later traffic:

options FAST_IPSEC
device crypto
options IPDIVERT

Next I modify the /etc/rc.conf on each system to add support for IPSec and the gif tunnel. Here is the sensor's additions to /etc/rc.conf:

ifconfig_gif0=" netmask"
route_gif0_0=" -interface gif0"

The gifconfig statement defines the public IPs used as the tunnel endpoints. The ifconfig_gif0 statement sets up the tunnel, with as the local endpoint and as the remote endpoint. The static_routes and route_gif0_0 statements tell the sensor how to reach the backend network.

Here is the gateway's additions to /etc/rc.conf:

#natd_flags="-redirect_port tcp 8080"
ifconfig_gif0=" netmask"

First I tell the gateway to act as a gateway, and I enable the firewall. I also enable NAT, with em0 being the Internet-facing interface with the external public IP address. I have commented out a natd_flags line showing how to do port forwarding. For example, a connection to port 8080 TCP on the gateway's external IP would be sent to the internal system

Next I set up the gif interfaces for this end of the tunnel. They are mirror images of the entries for the sensor.

Note that in both cases I have commented out ipsec_enable="YES" for the moment. It is important to get a working IPSec configuration before one enables ipsec_enable="YES" in /etc/rc.conf. If you reboot a system with ipsec_enable="YES" uncommented, and your /etc/ipsec.conf configuration file is faulty, the system will not completely boot up. You will end up needing physical access to the system or remote serial access to fix the problem.

We have done enough at this point to try sending traffic without using IPSec, but with the gif tunnel. To create the gif interface manually on the sensor, use syntax like this:

ifconfig gif0 create
ifconfig gif0 tunnel
ifconfig gif0 netmask

To manually create the static route for the network on the sensor, use this syntax.

route add -interface gif0

Checking ifconfig, you will see a result like this:

fedorov:/root# ifconfig gif0
gif0: flags=8051 mtu 1280
tunnel inet -->
inet6 fe80::2c0:9fff:fe40:1818%gif0 prefixlen 64 scopeid 0x6
inet --> netmask 0xffffffff

To create the gif interface manually on the gateway, use syntax like this:

ifconfig gif0 create
ifconfig gif0 tunnel
ifconfig gif0 netmask

You will see a result like this:

forsberg:/root# ifconfig gif0
gif0: flags=8051 mtu 1280
tunnel inet -->
inet6 fe80::2c0:9fff:fe3f:4fc4%gif0 prefixlen 64 scopeid 0x6
inet --> netmask 0xffffffff

With these interfaces live, you can use them. Here we ping the gateway from the sensor:
fedorov:/root# ping
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=8.395 ms

Here is what the traffic looks like:

16:01:20.112213 IP >
IP > icmp 64: echo request seq 0 (ipip-proto-4)
16:01:20.120540 IP >
IP > icmp 64: echo reply seq 0 (ipip-proto-4)

As it passes on the Internet, this traffic is not TCP (IP protocol 6). It is IP-in-IP, or IP protocol 4. Tcpdump reports it correctly as 'ipip-proto-4). Your /etc/protocols file shows it to be this:

ipencap 4 IP-ENCAP # IP encapsulated in IP (officially ``IP'')

If you add the routes necessary for the sensor to know how to reach the network, you can ping the backend:

fedorov:/root# ping
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=63 time=8.800 ms

Again, the traffic:

16:04:59.618465 IP >
IP > icmp 64: echo request seq 0 (ipip-proto-4)
16:04:59.627196 IP >
IP > icmp 64: echo reply seq 0 (ipip-proto-4)

Now that the traffic is being passed appropriately, we need to apply IPSec ESP to it. We create the following /etc/ipsec.conf file on the sensor. All of the spdadd statements should occupy a single unbroken line:


spdadd any -P out ipsec

spdadd any -P in ipsec

spdadd any -P out ipsec

spdadd any -P in ipsec

The first two lines flush IPSec Security Association Database (SAD) entries and Security Policy Database (SPD) entries. The first spdadd statement says traffic sent out from to should go via the IPSec tunnel from to

The second spdadd statement says traffic sent in from to should go via the IPSec tunnel from to These two entries are enough to protect traffic sent between the sensor and gateway.

The third spdadd statement says traffic sent out from to the network should go via the IPSec tunnel from to

The fourth spdadd statement says traffic sent in from the network to should go via the IPSec tunnel from to These two entries protect traffic sent between the sensor and the backend.

The /etc/ipsec.conf file on the gateway is a mirror image of the sensor's /etc/ipsec.conf:


spdadd any -P in ipsec

spdadd any -P out ipsec

spdadd any -P in ipsec

spdadd any -P out ipsec

Now that the /etc/ipsec.conf files are ready, the last step is to install a program to manage key negotiations. We'll use Racoon, which can be installed via the security/racoon port. First, I made this change to the /usr/local/etc/racoon/racoon.conf file on the sensor to tell Racoon where to listen for key exchange packets:

isakmp [500];

On the gateway, the modification looks like this:

isakmp [500];

Now, both public IP endpoints are listening on port 500 UDP for key exchange traffic.

Next I enabled a secret key. On the sensor, /usr/local/etc/racoon/psk.txt looks like this, which says use the specified key with the gateway thisisabadsecret

On the gatewat, /usr/local/etc/racoon/psk.txt looks like this, which says use the specified key with the sensor thisisabadsecret

Make sure the permissions for the /usr/local/etc/racoon/psk.txt file are 600, or the Racoon daemon will complain.

Now we are ready to start up Racoon, and enable IPSec. I recommend starting Racoon on the sensor and gateway in separate windows, using the 'racoon -F' syntax to show Racoon running in the foreground. Next enable IPSec via 'setkey -f /etc/ipsec.conf' on each system.

You can test your IPSec tunnel by pinging the gateway's gif IP address from the sensor:

fedorov:/root# ping
PING ( 56 data bytes
64 bytes from icmp_seq=1 ttl=64 time=7.428 ms
64 bytes from icmp_seq=2 ttl=64 time=7.343 ms
--- ping statistics ---
3 packets transmitted, 2 packets received, 33% packet loss
round-trip min/avg/max/stddev = 7.343/7.386/7.428/0.042 ms

We lost the first packet during negotiation of the tunnel. Here is how the traffic looks. Notice first the tunnel set-up, and then the ESP IPSec packets:

16:25:22.918098 IP >
isakmp: phase 1 I agg
16:25:22.972442 IP >
isakmp: phase 1 R agg
16:25:22.991453 IP >
isakmp: phase 1 I agg
16:25:22.993060 IP >
isakmp: phase 2/others I inf[E]
16:25:23.029160 IP >
isakmp: phase 2/others R inf[E]
16:25:23.039564 IP >
isakmp: phase 2/others I oakley-quick[E]
16:25:23.074885 IP >
isakmp: phase 2/others R oakley-quick[E]
16:25:23.081259 IP >
isakmp: phase 2/others I oakley-quick[E]
16:25:23.909566 IP >
16:25:23.916908 IP >
16:25:24.919570 IP >
16:25:24.926837 IP >

That's it. You will also be able to ping the backend ( from the sensor, or ping the sensor ( from the backend. It will all be encrypted via IPSec.

If you've been trying to deploy IPSec on FreeBSD, and have followed certain threads, you'll see I encountered no issues with enabling FAST_IPSEC and INET6 in the kernel. I also did not have to exempt port 500 UDP key exchange traffic in my /etc/ipsec.conf file. Those two problems seem to have been ironed out with FreeBSD 5.3 RELEASE.