Tuesday, January 31, 2006

Bejtlich/Bianco ShmooCon DVD

Thanks to David Bianco, I received a copy of a DVD of our ShmooCon 2006 presentation, Network Security Monitoring with Sguil (.ppt). The cover is posted at left, and clicking on it will show a larger version. I am not sure if the Feds will appreciate the Che Guevara theme the next time my security clearance is reviewed. If you want to order you own copy, you can visit MediaArchives.com. As far as I know I do not get a penny from DVD sales, unless there was some hidden clause in the form Heidi Potter asked me to sign!

By the way, this blog has been on fire this month. Where are you all coming from? If you started reading this blog this month, would you mind posting a comment saying where you heard of it, and if you plan to return next month? Thank you!

Miss the Internet of the 1970s? It's still here.

Imagine the following conversation took place some time before 15 January 2001.

Alice: "Why don't we create a Web page that anyone can edit?"

Bob: "Cool. How do we prevent 'bad people' from posting 'bad things'?" [Note that "bad people" and "bad things" are entirely subjective.]

Alice: "Don't worry, people will be nice."

Bob: "What if they are not nice?"

Alice: "We'll keep track of the IP addresses people use to post content. We'll block bad IP addresses."

Bob: "What if bad people post bad content using anonymous proxy servers? What about NAT, such that hundreds of people can be using the same public IP address?"

Alice: "Don't frighten me with your sorcerer's ways."

Bob: "So what do we call this system?"

Alice: "Wikipedia!"

Now, people are shocked -- shocked I say -- when anyone can edit pages they would wish said something else.

The Wikipedia model works when the user community is small and the participants trust each other. When was the last time that was true? Oh yes -- the Internet of the 1970s. Is that true now, or at least in 2001 when Wikipedia was founded? Well, the community was certainly smaller back then. But a small user base does not hold up well as a defense model. As Wikipedia's activity has grown, it has attracted the attention of people who are more likely to act maliciously. Sounds the Internet as a whole, from the 1970s into the 1980s, followed by the explosion of users in the 1990s.

I would personally never use Wikipedia as a resource for any serious research. I might use it as a starting point, but why should I trust what it says? Am I going to go back through the editing history and note that made a change that looks suspicious, but seems more reliable? That is ridiculous.

I think Wikipedia is fundamentally broken. Here's what would reduce it to scrap -- a MalWikiBot. The MalWikiBot would edit Wikipedia pages at random. Maybe it would replace whole sentences with material found on other Web pages. Perhaps it would change dates, measurements, and other numeric quantities. MalWikiBot couldn't be blocked using existing Wikipedia techniques because it would use bot nets to appear to come from legitimate IP addresses. On some days it would delete whole pages, but that would be far too obvious. Better to silently corrupt small sections of data in a manner not immediately obvious.

Wikipedia is going to need to at least restrict changes to authenticated users. Sign up with a username and an email address. At the very least a MalWikiBot writer would need to overcome that small hurdle before changing pages on the fly. If the current Wikipedia "security model" continues, I predict ongoing decline as users lose faith in the integrity of its data.

Update (28 July 2019): I was wrong about Wikipedia! I use it regularly. I'm glad it's withstood the attacks upon its content.

Update (26 August 2020): I was right about so-called Scottish Wikipedia! Since 2013, most of it is fake!

DoD 8570.01-M Posted

Thanks to David Bianco for sending me to this article about the manual for DoD 8570.1 being posted here. The .pdf looks like a scan of a hard copy document. I couldn't search it using xpdf.

Monday, January 30, 2006

Review of Running IPv6 Posted

Amazon.com just posted my five-star review of Running IPv6 by Iljitsch van Beijnum. It is so much easier to write reviews for great books! From the review:

"When I read and reviewed O'Reilly's IPv6 Network Administration by Niall Richard Murphy and David Malone, I called their book "a must-have book for all network administrators." Upon seeing Apress' Running IPv6 by Iljitsch van Beijnum, I wondered if I would waste my time reading and reviewing another book on IPv6. Now I'm glad I digested Running IPv6 -- it's my first must-read book of 2006. The books are complementary, so I recommend them both."

What a great book.

IPv6 Behind NAT Using FreeBSD and Miredo

Thanks to the generosity of a TaoSecurity Blog reader, I have been experimenting with a dual-stack IPv4 and IPv6 system at a university. I connect to the IPv4 address using OpenSSH. Once on the box, I can use IPv6.

I've been looking for ways to connect my home network directly to IPv6. At the moment I'm using a common gateway/router to perform NAT for my cable network connection. I needed a way to provide IPv6 for systems behind the NAT. Enter Teredo and the Miredo project.

Now, before you decide that I'm giving this protocol my "thumbs up," I'm going to explicitly tell you I just wanted to get the software working and use ping6. That's it for now.

Teredo, which is now a draft RFC, is a Microsoft protocol. Basically you take IPv6 traffic, tunnel it in UDP, and send it to a relay server. The relay pulls off the UDP and sends the traffic using IPv6 to the destination. The process is reversed for return traffic. Obviously sending your traffic elsewhere, especially to one of the Microsoft relays, is enough to scare most people.

Installing Miredo is simple. Thanks to author RĂ©mi Denis-Courmont responding to my troubleshooting emails, the latest version compiles flawlessly on FreeBSD 6.0. The standard ./configure, make, make install is all that is needed.

Once installed, I run Miredo in the foreground.

orr:/home/richard$ sudo miredo --foreground

* *
* At the time of release of this version of the program, the IETF had *
* not yet published the Teredo protocol specification (RFC). As such, *
* this version of the program still uses experimental provisional *
* settings, which will most likely be altered when the specification *
* is published. A new version of the program will then be released to *
* take these changes into consideration. Until then, this program *
* might not work properly and should be considered experimental. *

miredo[684]: Starting...
miredo[685]: Qualified (NAT type: restricted)
miredo[685]: Teredo pseudo-tunnel started
miredo[685]: (address: 3ffe:831f:8ac3:9ddd:0:3650:ba0c:d759, MTU: 1280)

Miredo creates a tun0 interface for IPv6.

orr:/home/richard$ ifconfig -a
fxp0: flags=8943 mtu 1500
inet6 fe80::203:47ff:fe0f:1f3c%fxp0 prefixlen 64 scopeid 0x1
inet netmask 0xffffff00 broadcast
ether 00:03:47:0f:1f:3c
media: Ethernet autoselect (100baseTX )
status: active
plip0: flags=108810 mtu 1500
lo0: flags=8049 mtu 16384
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet netmask 0xff000000
tun0: flags=80d1 mtu 1280
inet6 fe80::203:47ff:fe0f:1f3c%tun0 prefixlen 64 scopeid 0x4
inet6 fe80::5445:5245:444f%tun0 prefixlen 64 scopeid 0x4
inet6 3ffe:831f:8ac3:9ddd:0:3650:ba0c:d759 prefixlen 32
Opened by PID 685

Teredo encapsulates IPv6 inside UDP packets sent to port 3544.

Here is what it looks like to Tcpdump when Teredo starts. All we can see at this point is Miredo doing a DNS lookup for its default relay server, followed by UDP traffic to that server.

11:08:15.389255 IP > 58453+ A? teredo.via.ecp.fr. (35)
11:08:15.394528 IP > 58453 1/0/0 A (51)
11:08:15.395024 IP > UDP, length 77
11:08:19.396616 IP > UDP, length 77
11:08:23.396031 IP > UDP, length 77
11:08:27.396404 IP > UDP, length 77
11:08:27.517795 IP > UDP, length 117
11:08:27.518031 IP > UDP, length 77
11:08:27.639923 IP > UDP, length 117
11:09:01.396212 IP > UDP, length 77
11:09:01.514967 IP > UDP, length 117

Tethereal strips off the UDP traffic by default and shows the underlying IPv6 traffic. Keep this in mind if you're using Tethereal and think you're seeing native IPv6. This is the same trace as examined above with Tcpdump.

Now with the help of Tethereal, we see Miredo making ICMPv6 router solicitations. Later we see ICMPv6 router advertisements from fe80::8000:dd8:753c:6222.

1 2006-01-30 11:08:15.389255 -> DNS Standard query A
2 2006-01-30 11:08:15.394528 -> DNS Standard query response A
3 2006-01-30 11:08:15.395024 fe80::8000:5445:5245:444f -> ff02::2 ICMPv6 Router solicitation
4 2006-01-30 11:08:19.396616 fe80::8000:5445:5245:444f -> ff02::2 ICMPv6 Router solicitation
5 2006-01-30 11:08:23.396031 fe80::8000:5445:5245:444f -> ff02::2 ICMPv6 Router solicitation
6 2006-01-30 11:08:27.396404 fe80::5445:5245:444f -> ff02::2 ICMPv6 Router solicitation
7 2006-01-30 11:08:27.517795 fe80::8000:dd8:753c:6222 -> fe80::5445:5245:444f ICMPv6 Router advertisement
8 2006-01-30 11:08:27.518031 fe80::5445:5245:444f -> ff02::2 ICMPv6 Router solicitation
9 2006-01-30 11:08:27.639923 fe80::8000:dd8:753c:6222 -> fe80::5445:5245:444f
ICMPv6 Router advertisement
12 2006-01-30 11:09:01.396212 fe80::5445:5245:444f -> ff02::2 ICMPv6 Router solicitation
13 2006-01-30 11:09:01.514967 fe80::8000:dd8:753c:6222 -> fe80::5445:5245:444f ICMPv6 Router advertisement

Here is what the routing table for IPv6 looks like once Miredo is running.

orr:/home/richard$ netstat -nr -f inet6
Routing tables

Destination Gateway Flags Netif Expire
default link#4 ULS tun0
::1 ::1 UH lo0
3ffe:831f::/32 link#4 UC tun0
3ffe:831f:8ac3:9ddd:0:3650:ba0c:d759 link#4 UHL lo0
fe80::%fxp0/64 link#1 UC fxp0
fe80::203:47ff:fe0f:1f3c%fxp0 00:03:47:0f:1f:3c UHL lo0
fe80::%lo0/64 fe80::1%lo0 U lo0
fe80::1%lo0 link#3 UHL lo0
fe80::%tun0/64 link#4 UC tun0
fe80::5445:5245:444f%tun0 link#4 UHL lo0
fe80::203:47ff:fe0f:1f3c%tun0 link#4 UHL lo0
ff01::/32 ::1 U lo0
ff02::%fxp0/32 link#1 UC fxp0
ff02::%lo0/32 ::1 UC lo0
ff02::%tun0/32 link#4 UC tun0

Interface tun0 is the default for IPv6. That is good news. Let's try to ping6 an IPv6 enabled host.

orr:/home/richard$ ping6 -c 2 www6.olympus-zone.net
PING6(56=40+8+8 bytes) 3ffe:831f:8ac3:9ddd:0:3650:ba0c:d759 --> 2001:1638:305:4::1
16 bytes from 2001:1638:305:4::1, icmp_seq=0 hlim=56 time=766.023 ms
16 bytes from 2001:1638:305:4::1, icmp_seq=1 hlim=56 time=394.536 ms

--- www6.olympus-zone.net ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 394.536/580.279/766.023/185.744 ms

Awesome. Here is how Tcpdump sees the traffic.

11:09:16.646506 IP > 58174+ AAAA? www6.olympus-zone.net. (39)
11:09:16.648419 IP > 58174 1/0/0 AAAA 2001:1638:305:4::1 (67)
11:09:16.649738 IP > UDP, length 64
11:09:16.977910 IP > UDP, length 48
11:09:16.978090 IP > UDP, length 40
11:09:17.130790 IP > UDP, length 64
11:09:17.130940 IP > UDP, length 56
11:09:17.415124 IP > UDP, length 56
11:09:17.649746 IP > UDP, length 56
11:09:18.043784 IP > UDP, length 56

Packet 3 would appear to be the ICMPv6 request, with packet 4 the response. But what about the last 6 packets?

Here is the same traffic in Tethereal.

14 2006-01-30 11:09:16.646506 -> DNS Standard query AAAA www6.olympus-zone.net
15 2006-01-30 11:09:16.648419 -> DNS Standard query response AAAA 2001:1638:305:4::1
16 2006-01-30 11:09:16.649738 3ffe:831f:8ac3:9ddd:0:3650:ba0c:d759 -> 2001:1638:305:4::1 ICMPv6 Echo request
17 2006-01-30 11:09:16.977910 fe80::8000:5445:5245:444f -> 3ffe:831f:8ac3:9ddd:0:3650:ba0c:d759 IPv6 IPv6 no next header
18 2006-01-30 11:09:16.978090 -> UDP Source port: 51631 Destination port: 51246
19 2006-01-30 11:09:17.130790 -> UDP Source port: 51246 Destination port: 51631
20 2006-01-30 11:09:17.130940 -> UDP Source port: 51631 Destination port: 51246
21 2006-01-30 11:09:17.415124 -> UDP Source port: 51246 Destination port: 51631
22 2006-01-30 11:09:17.649746 -> UDP Source port: 51631 Destination port: 51246
23 2006-01-30 11:09:18.043784 -> UDP Source port: 51246 Destination port: 51631

Tethereal sees the ICMPv6 request and reply, but it can't decode the last 6 packets.

I plan to investigate this further.

FreeBSD Networking over FireWire

You might be familiar with Apple's implementation of IP over FireWire. This allows connecting two computers directly over FireWire ports. FreeBSD offers two drivers that provide networking over FireWire. fwe is a non-standard protocol, but it is implemented by default in the GENERIC kernel. fwip implements RFC 2734 (IPv4 over IEEE 1394) and RFC 3146 (Transmission of IPv6 Packets over IEEE 1394 Networks); it is available via kernel module.

I decided to have my laptop orr talk to my server janney using FireWire. To implement FireWire, orr uses an Adaptec DuoConnect PC Card Adapter and janney uses an Adaptec DuoConnect PCI Adapter. Both provide FireWire and USB 2.0.

Each system is running FreeBSD 6.0.

The laptop dmesg sees the following when the FireWire adapter is inserted.

cardbus0: CIS pointer is 0!
cardbus0: Resource not specified in CIS: id=10, size=800
cardbus0: Resource not specified in CIS: id=14, size=4000
fwohci0: mem 0x88004000-0x880047ff,0x88000000-0x88
003fff irq 11 at device 0.0 on cardbus0
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 08:00:28:56:02:00:49:8a
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: on fwohci0
fwe0: on firewire0
if_fwe0: Fake Ethernet address: 0a:00:28:00:49:8a
fwe0: Ethernet address: 0a:00:28:00:49:8a
fwe0: if_start running deferred for Giant
sbp0: on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc000ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
cardbus0: CIS pointer is 0!
cardbus0: Resource not specified in CIS: id=10, size=1000
ohci0: mem 0x88000000-0x88000fff irq 11 at device
0.4 on cardbus0
usb1: OHCI version 0.0
usb1: unsupported OHCI revision
ohci0: USB init failed

The server dmesg sees the following at boot.

ohci0: mem 0xf7fff000-0xf7ffffff irq 22 at device
8.0 on pci4
usb1: OHCI version 1.0
usb1: on ohci0
usb1: USB revision 1.0
uhub1: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
ohci1: mem 0xf7ffe000-0xf7ffefff irq 21 at device
8.1 on pci4
usb2: OHCI version 1.0
usb2: on ohci1
usb2: USB revision 1.0
uhub2: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ehci0: mem 0xf7ffdc00-0xf7ffdcff irq 20 at d
evice 8.2 on pci4
usb3: EHCI version 0.95
usb3: companion controllers, 3 ports each: usb1 usb2
usb3: on ehci0
usb3: USB revision 2.0
uhub3: NEC EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 5 ports with 5 removable, self powered
fwohci0: mem 0xf7ffd000-0xf7ffd7ff,0xf7ff8000-0xf7
ffbfff irq 22 at device 12.0 on pci4
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:50:42:b5:c0:0d:0d:af
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: on fwohci0
fwe0: on firewire0
if_fwe0: Fake Ethernet address: 02:50:42:0d:0d:af
fwe0: Ethernet address: 02:50:42:0d:0d:af
fwe0: if_start running deferred for Giant
sbp0: on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc1, gen=1, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1 (me)
firewire0: bus manager 1 (me)

Here is the fwe interface that is created when the FireWire adapter is active on orr.

fwe0: flags=108802 mtu 1500
ether 0a:00:28:00:49:8a
ch 1 dma -1

Here is the fwe interfaced that is created when the FireWire adapter is active on janney.

fwe0: flags=108943 mtu 1500
inet6 fe80::50:42ff:fe0d:daf%fwe0 prefixlen 64 scopeid 0x2
inet netmask 0xffffff00 broadcast
ether 02:50:42:0d:0d:af
ch 1 dma 0

When I plug a FireWire cable into each host, dmesg on orr sees the following.

fwohci0: BUS reset
fwohci0: node_id=0xc800ffc1, gen=2, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1 (me)
firewire0: bus manager 1 (me)
firewire0: New S400 device ID:005042b5c00d0daf

Server janney sees similar messages.

fwohci0: BUS reset
fwohci0: node_id=0x8800ffc0, gen=4, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1
firewire0: bus manager 1
firewire0: New S400 device ID:080028560200498a

First I assign an IP to fwe0 on orr.

orr:/home/richard$ sudo ifconfig fwe0 inet netmask up
orr:/home/richard$ ifconfig fwe0
fwe0: flags=108943 mtu 1500
inet6 fe80::800:28ff:fe00:498a%fwe0 prefixlen 64 scopeid 0x4
inet netmask 0xffffff00 broadcast
ether 0a:00:28:00:49:8a
ch 1 dma 0

Next I assign an IP to fwe0 on janney.

janney:/home/richard$ sudo ifconfig fwe0 inet netmask up
janney:/home/richard$ ifconfig fwe0
fwe0: flags=108943 mtu 1500
inet6 fe80::50:42ff:fe0d:daf%fwe0 prefixlen 64 scopeid 0x2
inet netmask 0xffffff00 broadcast
ether 02:50:42:0d:0d:af
ch 1 dma 0

Now the two systems can communicate over FireWire.

janney:/home/richard$ ping
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.694 ms
64 bytes from icmp_seq=1 ttl=64 time=0.333 ms
64 bytes from icmp_seq=2 ttl=64 time=0.312 ms
--- ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.312/0.446/0.694/0.175 ms

One can sniff the fwe0 interface with Tcpdump.

07:36:42.479420 IP6 fe80::50:42ff:fe0d:daf > ff02::1:ff0d:daf: HBH ICMP6, multicast listener
reportmax resp delay: 0 addr: ff02::1:ff0d:daf, length 24
07:36:42.479667 IP6 fe80::50:42ff:fe0d:daf > ff02::2:f2f0:bec6: HBH ICMP6, multicast listener
reportmax resp delay: 0 addr: ff02::2:f2f0:bec6, length 24
07:36:42.479818 arp who-has tell
07:36:42.498496 IP6 :: > ff02::1:ff0d:daf: ICMP6, neighbor solicitation, who has
fe80::50:42ff:fe0d:daf, length 24
07:36:43.756817 IP6 fe80::50:42ff:fe0d:daf > ff02::1:ff0d:daf: HBH ICMP6, multicast listener
reportmax resp delay: 0 addr: ff02::1:ff0d:daf, length 24
07:36:50.962703 IP6 fe80::50:42ff:fe0d:daf > ff02::2:f2f0:bec6: HBH ICMP6, multicast listener
reportmax resp delay: 0 addr: ff02::2:f2f0:bec6, length 24
07:36:54.422718 arp who-has tell
07:36:54.422793 arp reply is-at 0a:00:28:00:49:8a
07:36:54.423012 IP > ICMP echo request, id 34306, seq 0, length 64
07:36:54.423050 IP > ICMP echo reply, id 34306, seq 0, length 64

You can see that the fwe driver supports IPv6. The last four packets are IPv4.

Next I decided to try the fwip driver. I loaded the fwip kernel module on each system. First, orr.

orr:/home/richard$ sudo kldload fwip
orr:/home/richard$ kldstat
Id Refs Address Size Name
1 12 0xc0400000 63072c kernel
2 2 0xc0a31000 74b0 snd_csa.ko
3 3 0xc0a39000 1d408 sound.ko
4 1 0xc0a57000 c3a4 r128.ko
5 2 0xc0a64000 eeec drm.ko
6 16 0xc0a73000 568dc acpi.ko
7 1 0xc2236000 5000 if_fwip.ko

This created the fwip0 interface, which I gave an IP address.

orr:/home/richard$ ifconfig fwip0
fwip0: flags=108802 mtu 1500
orr:/home/richard$ sudo ifconfig fwip0 inet netmask up
orr:/home/richard$ ifconfig fwip0
fwip0: flags=108843 mtu 1500
inet6 fe80::a00:2856:200:498a%fwip0 prefixlen 64 scopeid 0x5
inet netmask 0xffffff00 broadcast

Notice dmesg output for orr.

fwip0: on firewire0
fwip0: Firewire address: 08:00:28:56:02:00:49:8a @ 0xfffe00000000, S400, maxrec 2048

Now, janney.

janney:/home/richard$ sudo kldload fwip
janney:/home/richard$ kldstat
Id Refs Address Size Name
1 5 0xc0400000 641298 kernel
2 16 0xc0a42000 568dc acpi.ko
3 1 0xc214a000 5000 if_fwip.ko
janney:/home/richard$ ifconfig fwip0
fwip0: flags=108802 mtu 1500
lladdr 0.50.42.b5.c0.d.d.af.a.2.ff.fe.
janney:/home/richard$ sudo ifconfig fwip0 inet netmask up
janney:/home/richard$ ifconfig fwip0
fwip0: flags=108843 mtu 1500
inet6 fe80::250:42b5:c00d:daf%fwip0 prefixlen 64 scopeid 0x6
inet netmask 0xffffff00 broadcast
lladdr 0.50.42.b5.c0.d.d.af.a.2.ff.fe.

Notice janney's dmesg.

fwip0: on firewire0
fwip0: Firewire address: 00:50:42:b5:c0:0d:0d:af @ 0xfffe00000000, S400, maxrec 2048

Now the two systems can communicate using the fwip driver.

janney:/home/richard$ ping -c 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.733 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.733/0.733/0.733/0.000 ms

You can also sniff fwip0. Take a look at the differences.

orr:/home/richard$ tcpdump -n -r fwe0.short.lpc -c 2 icmp
reading from file fwe0.short.lpc, link-type EN10MB (Ethernet)
07:36:54.423012 IP > ICMP echo request, id 34306, seq 0, length 64
07:36:54.423050 IP > ICMP echo reply, id 34306, seq 0, length 64

orr:/home/richard$ tcpdump -n -r fwip0.short.lpc -c 2 icmp
reading from file fwip0.short.lpc, link-type APPLE_IP_OVER_IEEE1394 (Apple IP-over-IEEE 1394)
07:41:52.593940 IP > ICMP echo request, id 38914, seq 0, length 64
07:41:52.593970 IP > ICMP echo reply, id 38914, seq 0, length 64

The first reports "Ethernet" because fwe is an Ethernet emulation layer. The second reports Apple IP-over-IEEE 1394 because that is an entirely new protocol. Check it out in Tethereal.

Frame 8 (102 bytes on wire, 102 bytes captured)
Arrival Time: Jan 30, 2006 07:41:52.593940000
Time delta from previous packet: 0.000231000 seconds
Time since reference or first frame: 13.327087000 seconds
Frame Number: 8
Packet Length: 102 bytes
Capture Length: 102 bytes
Protocols in frame: ap1394:ip:icmp:data
Apple IP-over-IEEE 1394, Src: 005042B5C00D0DAF, Dst: 80B2FFC100000000
Destination: 80B2FFC100000000
Source: 005042B5C00D0DAF
Type: IP (0x0800)
Internet Protocol, Src: (, Dst: (
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 84
Identification: 0x0696 (1686)
Flags: 0x00
0... = Reserved bit: Not set
.0.. = Don't fragment: Not set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: ICMP (0x01)
Header checksum: 0x19f0 [correct]
Good: True
Bad : False
Source: (
Destination: (
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0
Checksum: 0xe88d [correct]
Identifier: 0x9802
Sequence number: 0x0000
Data (56 bytes)

0000 43 de 09 a3 00 09 3e e2 08 09 0a 0b 0c 0d 0e 0f C.....>.........
0010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f ................
0020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f !"#$%&'()*+,-./
0030 30 31 32 33 34 35 36 37 01234567

I noticed each host reported error messages like the following. I believe these were caused by the fwip driver.

fwohci0: txd err= 3 miss Ack err

I believe fwip has a future going forward, since it implements a standard.

For a rough idea of how fast these interfaces were, I transferred the same 100 MB file using the fwe, fwip, and fxp/xl wired Ethernet interfaces. Here are the results.

  1. fwe: 104857600 bytes received in 00:09 (10.16 MB/s)

  2. fwip: 104857600 bytes received in 00:09 (11.01 MB/s)

  3. fxp/xl through a switch: 104857600 bytes received in 00:08 (11.21 MB/s)

  4. fxp/xl via crossover cable: 104857600 bytes received in 00:09 (10.75 MB/s)

All the results are roughly the same, so the bottleneck is probably the hard drive of the laptop.

This opens some interesting possibilities for networking. Maybe I will buy a FireWire hub and connect multiple machines simultaneously.

I hope to try some of the debugging and memory access features of FireWire in the future.

Saturday, January 28, 2006

QEMU on FreeBSD, with Networking

Maybe you've heard of QEMU, an "open source processor emulator." It's not quite VMware, since there doesn't seem to be a concept of persistent state and there are definitely not snapshots. However, when I saw the variety of ready-to-run system images at OSZoo.org, I decided to try it on FreeBSD 6.0.

Luckily there are several QEMU ports. I installed emulators/qemu from the latest FreeBSD 6.0 package. I next installed emulators/kqemu-kmod using the port.

janney:/root# cd /usr/ports/emulators/kqemu-kmod
janney:/usr/ports/emulators/kqemu-kmod# make
=> kqemu-0.7.2.tar.gz doesn't seem to exist in /usr/ports/distfiles/kqemu.
=> Attempting to fetch from http://fabrice.bellard.free.fr/qemu/.
kqemu-0.7.2.tar.gz 100% of 77 kB 102 kBps
===> Extracting for kqemu-kmod-0.7.2_1
=> MD5 Checksum OK for kqemu/kqemu-0.7.2.tar.gz.
=> SHA256 Checksum OK for kqemu/kqemu-0.7.2.tar.gz.
===> Patching for kqemu-kmod-0.7.2_1
===> Applying FreeBSD patches for kqemu-kmod-0.7.2_1
===> Configuring for kqemu-kmod-0.7.2_1
===> Building for kqemu-kmod-0.7.2_1
Warning: Object directory not changed from original /usr/ports/emulators/kqemu-kmod/work/kqemu
@ -> /usr/src/sys
machine -> /usr/src/sys/i386/include
cc -O2 -fno-strict-aliasing -pipe -D_KERNEL -DKLD_MODULE -nostdinc -I- -I. -I@
-I@/contrib/altq -I@/../include -I/usr/include -finline-limit=8000 -fno-common
-mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2
-ffreestanding -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes
-Wpointer-arith -Winline -Wcast-qual -fformat-extensions -std=c99 -c kqemu-freebsd.c
ld -d -warn-common -r -d -o kqemu.kld kqemu-mod-i386.o kqemu-freebsd.o
touch export_syms
awk -f /sys/conf/kmod_syms.awk kqemu.kld export_syms | xargs -J% objcopy % kqemu.kld
ld -Bshareable -d -warn-common -o kqemu.ko kqemu.kld
objcopy --strip-debug kqemu.ko
janney:/usr/ports/emulators/kqemu-kmod# make install
===> Installing for kqemu-kmod-0.7.2_1
===> Generating temporary packing list
===> Checking if emulators/kqemu-kmod already installed
install -o root -g wheel -m 555 kqemu.ko /boot/kernel
kldxref /boot/kernel
if mount |/usr/bin/grep ^devfs >/dev/null ; then : ; else if [ ! -e /dev/kqemu ]; then mknod
/dev/kqemu c 250 0 ; fi ; /bin/chmod 666 /dev/kqemu ; fi
===> Registering installation for kqemu-kmod-0.7.2_1

Next I had to enable network connectivity. The host OS had xl0 as a live interface. QEMU would use the tap0 interface.

# cat /dev/null > /dev/tap0
# ifconfig tap0 create

# kldload bridge.ko
# sysctl net.link.ether.bridge_cfg=xl0,tap0
net.link.ether.bridge_cfg: -> xl0,tap0
# sysctl net.link.ether.bridge.enable=1
net.link.ether.bridge.enable: 0 -> 1

I needed to create this small script to enable networking as well.

$ cat /etc/qemu-ifup
ifconfig ${1}

Now I needed an image to run. I decided to use the NetBSD 2.0.2 x86 image, since it wasn't too large and I figured I would be familiar enough with NetBSD once it was running.

After downloading and extracting the image, I was ready to try running it.

# qemu -net nic -net tap netbsd_2.0.2.img

Here is the initial boot screen.

Now the system is booted.

The root login for this system image is "piripicchio" (it's Italian). Once logged in, I configured the ne2 interface with an IP address on the same segment as the host OS.

Next I added a default route, a nameserver, and a normal user account with a password.

# route add default
# echo "nameserver" > /etc/resolv.conf
# useradd -m richard
# passwd richard

Now I wanted to enable sshd. After trying to just start the daemon, I realized I needed to generate at least a DSA key as shown below.

With sshd listening, I could log in remotely.

$ ssh
The authenticity of host ' (' can't be established.
DSA key fingerprint is 5d:7f:a2:08:b0:3f:f7:e2:45:35:79:60:45:50:09:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (DSA) to the list of known hosts.
richard@'s password:
NetBSD 2.0.2 (GENERIC) #0: Wed Mar 23 08:53:42 UTC 2005

Welcome to NetBSD!

Erase is backspace.
If one studies too zealously, one easily loses his pants.
-- A. Einstein.
: {1} uname -a
NetBSD 2.0.2 NetBSD 2.0.2 (GENERIC) #0: Wed Mar 23 08:53:42 UTC 2005
-obj/home/builds/ab/netbsd-2-0-2-RELEASE/src/sys/arch/i386/compile/GENERIC i386
: {2} df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/wd0a 846M 291M 513M 36% /
kernfs 1.0K 1.0K 0B 100% /kern

I think that is pretty cool for a free virtual machine. As I learn more about QEMU I will share it here.

I found these three posts to be helpful when getting QEMU working.

Friday, January 27, 2006

Black Hat Federal 2006 Wrap-Up, Part 5

Please see part 1 for an introduction if you are reading this article separately.

Next I heard Stefano Zanero discuss problems with testing intrusion detection systems. He said that researchers prefer objective means with absolute results, while users prefer subjective means with relative results. This drives the "false positive" debate. Researchers see false positives as failures of the IDS engine to work properly, while users see any problem as the fault of the whole system.

Stefano mentioned work done by Giovannii Vigna and others on the Python-based Sploit, which creates exploit templates and mutant operators to test IDS'. He also cited a ICSA Labs project that doesn't appear to have made much progress developing IDS testing methodologies. Stefano said that good IDS tests must include background traffic; running an exploit on a quiet network is a waste of time. Stefano is developing a test bed for network traffic generation in the context of testing IDS'. He lamented there was no "zoo list" of attacks currently seen in the wild, as is the case with viruses and related malware.

Stefano claimed that vendors who say they perform "zero day detection" are really "detecting new attacks against old vulnerabilities." I don't necessarily agree with this, and I asked him about "vulnerability filter"-type rules. He said those are a hybrid of misuse and anomaly detection. Stefano declared that vendors who claim to perform "anomaly detection" are usually doing protocol anomaly detection, meaning they identify odd characteristics in protocols and not odd traffic in aggregate. He reminded the audience of Bob Colwell's paper If You Didn't Test It, It Doesn't Work. He also said that vendor claims must be backed up by repeatable testing methodologies. Stefano made the point that a scientist could never publish a paper that offered unsubstantiated claims.

I spoke with Stefano briefly and was happy to hear he uses my first book as a text for his undergraduate security students.

After Stefano's talk I listened to Halvar Flake explain attacks on uninitialized local variables. I admit I lost him about half way through his talk. Here is what I can relate. Non-initialized stack variables are regions on the stack that have not been initialized before being used. The trick is _not_ to insert code into these variables to be later executed, but to _control_ these values (as they might contain array indices or pointers). These can then be abused to gain control. The problem revolves around the fact that the stack is not cleaned when items are popped off, for performance reasons. Ok, that's about it. [Note: thanks to Halvar below for correcting this text!]

Here are a few general thoughts on the talk. Twice Halvar noted that finding 0-days is not as easy as it was before. Bugs are getting harder to exploit. In order to test new exploitation methods, Halvar can't simply find 20 0-days in an application and run his exploits against those vulnerabilities. He also can't write simple yet flawed code snippets as test against those, since they do not adequately reflect the complexity of modern applications. What he ends up doing is "patching in" flawed code into existing applications. That way he ends up with a complex program with known problems, against which he can try novel exploitation methods.

Halvar made heavy use of graphical depictions of code paths, as shown by his company's product BinNavi. This reminded me of the ShmooCon reverse engineering BoF, where many of the younger guns expressed their interest in graphical tools. As the problem of understanding complex applications only grows, I see these graphical tools as being indispensible for seeing the bigger picture.

For points of future research, Halvar wonder if there were uninitialized heap variables that could be exploited. He said that complexity cuts two ways in exploit development. Complex applications sometimes give intruders more freedom of manuever. They also may exploitation more difficult because the order in which an application allocates memory often matters. Halvar mentioned that felinemenace.org/~mercy addressed the same subject as his talk.

Robert Graham, chief scientist of ISS, gave the last talk I saw. He discussed the security of Supervisory Control And Data Acquisition (SCADA) systems. SCADA systems control power, water, communication, and other utilities. Robert does domestic and foreign penetration testing of SCADA systems, and he included a dozen case studies in his talk.

For example, he mentioned that the Blaster worm shut down domestic oil production for several days after a worker connected an infected laptop into a diagnostic network! Who needs a hurricane? Robert also told how a desktop negotiation for a pen test resulted in a leap from the corporate conference room, via wireless to a lab network, via Ethernet to the office network, through a dual-home Solaris box to the SCADA network. At that point the prospective client said "Please stop." (ISS received approval from the client to make each step, I should note.) In another case, Robert's team found a lone Windows PC sitting in a remote unlocked shed that had complete connectivity via CDPD modem to a SCADA network.

The broad outline of his conclusions include:

  • Patches are basically forbidden on SCADA equipment.

  • SCADA systems require little or no authentication.

  • Little or not SCADA traffic is encrypted.

  • Despite SCADA industry claims, SCADA systems are publicly accessible from the Internet, wireless networks, and dial-up. The "air gap" is a myth.

  • There is little to no logging of activity on the SCADA networks.

In sum, Robert said that SCADA "executives are living in a dreamworld." They provide network diagrams with "convenient ommissions" of links between office and SCADA/production segments. They are replacing legacy, dumb, RS-232 or 900 MHz-based devices with WinCE or Linux, Ethernet or 802.11-based devices. Attackers can jump from the Internet, to the DMZ, to the office network, to the SCADA network. They do not have to be "geniuses" to figure out how to operate SCADA equipment. The manuals they need are either online or on the company's own open file servers. SCADA protocols tend to be firewall-unfriendly, as they often need random ports to communicate. SCADA protocols like the Inter Control Center Protocol (ICCP) or OLE Process Control (OPC, a DCOM-based Microsoft protocol) are brittle. OPC, for example, relies on ASN.1.

At the end of the talk Robert said there was no need to panic. I asked "Why?" and so did Brian Krebs. Robert noted that SCADA industry threat models point to "accidents, not determined adversaries." That is a recipe for disaster. The SCADA network reminds me of the Air Force in the early 1990s. I will probably have more to say on that in a future article.

I hope you enjoyed this conference wrap-up. I look forward to your comments.

Black Hat Federal 2006 Wrap-Up, Part 4

Please see part 1 for an introduction if you are reading this article separately.

I finished Wednesday listening to Irby Thompson and Mathew Monroe discuss FragFS, a way to use the Windows Master File Table (MFT) on NTFS to store data covertly. The MFT can be read as a file if you open C:\$MFT as the administrator. That file can even be written to by administrators, hence the proof of concept tools "hammer.exe" and "looker.exe" provided by the presenters. Their research indicates the average MFT can store around 36 MB of hidden data, and that commercial tools neither review nor understand data hidden in the MFT. Beyond their userland implementation, the pair also wrote a Windows device driver that provides greater functionality. They will not release that code for fear of its misuse.

Incidentally, prior to this talk I met Sam Stover, who gave me two FragFS stickers for my laptop. Thanks Sam.

On Thursday I started with Dr. Arun Lakhotia, who explained problems with analyzing adversarial code. His main point was that tools currently used to investigate malware were built for programmers solving development problems, not secuirty people analyzing suspicious binaries. He outlined three types of analysis problems.

  1. Some problems can be solved in polynomial time, meaning finding a solution is not difficult.

  2. Some problems can only be solved in nonpolynomial time, meaning that with infinite resources, they theoretically can be solved.

  3. Some problems, however, are undecidable. There is no exact solution, regardless of resources. Approximation is the best answer.

Dr. Lakhotia said disassembly is an undecidable problem. He described a few examples to make his point.

He also noted that although there are huge numbers of distinct pieces of malware (12,000 in the last Symantec Internet Threat Report), there are really a very small number of malware families. In other words, code reuse and plagarism is rampant in the malware world. Using this fact, Dr. Lakhotia demonstrated novel ways to decipher call obfuscation and reverse malware to a single "common form." Keep an eye on his Web site for a place where malware will be categorized according to families in a system called "VILO". I also learned of VX Heavens, a malware collection site. I guess Offensive Computing is similar.

Next I heard my friend Kevin Mandia talk about recent incident response and computer forensics cases he and his team have worked. He stated that while investigating 215 suspected compromised systems in the last three years, he could only conclusively say 103 were 0wned. Of those, only 32 revealed enough evidence to demonstrate the intruder's point of entry. Why? The team seldom had the time, audit records, or network logs to figure out what had happened.

Kevin said that modern incident response in corporate America is characterized by "vague reporting channels" and "processes that are shelf-ware." Companies approach IR as a "directionless infantry march" instead of a "precision blitzkrieg." Kevin then outlined common indicators of compromise.

  1. The number one detection method is internal end users. When their systems crash, their anti-virus refuses to start, they cannot "save as" documents, install new applications, run common applications, or start Task Manager, they are likely compromised.

  2. Surge in bandwidth usage

  3. Anti-virus hits: these do not mean the problem has been dealt with. Rather, it's the tip of the iceberg.

  4. IDS detection is rare, but can be effective during the IR.

  5. Customers are often a helpful source of detection indicators.

Kevin recommends enabling process tracking, to which I would add exporting the resulting logs via syslog to a central reporting server farm. He shared a few tips for understanding system processes, like using "tasklist /svc" on Windows XP and "psservice -a" to see what processes are started by the Windows svchost.exe, a sort of "inetd" for Windows.

Kevin's team demonstrated their new First Response client-server architecture for Windows. You will see me describe this as soon as I can try the beta. Suffice it to say that this free program will rock the IR world. It makes retrieving and analyzing live response data a snap.

Black Hat Federal 2006 Wrap-Up, Part 3

Please see part 1 for an introduction if you are reading this article separately.

Staying on the rootkit theme, I next heard Joanna Rutkowska discuss "Rootkit Hunting vs. Compromise Detection." She has done some impressive work on network-based covert channels, but she is also a rootkit guru. Joanna talked about "Explicit Compromise Detection," and the need to scan kernel memory for integrity checking. She challenged many of the ideas of traditional rootkits, such as the need to survive a reboot, the desire to hide processes, open sockets, and so on. It seems like her new DeepDoor rootkit is an all-in-one package that hooks the Windows Network Driver Interface Specification (NDIS) code by modifying four words in the NDIS data section of memory.

She demonstrated her ddcli client talking to a DeepDoor'd victim. The client communicated with the server over port 445 TCP. Fair enough, but port 445 TCP was also able to handle normal SMB traffic, even with the rootkit active! That is insane. She showed how her rootkit could still function even with Zone Alarm denying access.

Joanna emphasized that there is no safe way to read kernel memory on Windows. She said that even reading physical memory can be tough. She requested that Microsoft implement a means to let third party vendors reliably read kernel memory. She said that such a new feature would not aid attackers, since they do not care if their unreliable methods end up crashing a target. A security vendor, however, must take extra care. Joanna noted that next generations operating systems should ship with more than two CPU privilege modes, and that Trusted Computing will not prevent the attacks she described. She mentioned the introduction of a hypervisor that runs at ring -1 (todays systems descend to ring 0). Joanna also postulated that there may be a finite number of places for malware to hook an OS, so perhaps it would be helpful to enumerate them in a public place. A related project is her Open Methodology for Compromise Detection.

Joanna was not able to release her DeepDoor rootkit for reasons of "NDAs." She was also not able to discuss ongoing work on network covert channels for the same reason. On a personal note, I spoke with Dave Aitel (note he has cut his hair WAY back from what's shown in the photo!) who had a tough time pronouncing my name. I guessed that as a fellow Eastern European (I'm American but my ancestors are from that area), Joanna (who is Polish) would be able to pronounce "bate-lik." Joanna was sitting nearby, and sure enough, she could!

After hearing about rootkits for three straight talks, I took a break by hearing Simson Garfinkel discuss new directions for disk forensics. (He reminded the audience of his company Sandstorm Enterprises, and I learned by speaking with him that he sells a laptop version of NetIntercept for consultants like me. )

Simson spoke for a long time discussing his ongoing used hard drive analysis project. He introduced his cross-drive forensic analysis methodology, which involves finding interesting data on groups of hard drives. One of the most powerful techniques was building histograms of email addresses. On a single hard drive, the most frequently seen email address is usually the address of the hard drive owner. He also searched hard drives for patterns associated with credit cards. The interesting aspect of this sort of analysis is that he is reviewing raw data in all cases, such that he can even review something like an Oracle data drive that has no conventional partitions.

I was most excited to hear about Simson's Advanced Forensic Format project. He noted that images produced by dd are big and contain no metadata. Proprietary formats like the Encase E01 are "bad an undocumented." Simson promotes AFF as an open standard that will be intergrated into a future release of Brian Carrier's Sleuth Kit. AFF contains tools that do more than efficiently image and describe drives. They acquisition tools can even help bring old drives to life by pulsing and otherwise manipulating them.

The most thought-provoking aspect of Simson's presentation was his discussion of the market for used hard drives on eBay. He says people pay unreasonable amounts for small old hard drives, and defintely odd amounts for hard drives reported as broken. The implication is that those hard drives might be bought by criminals hunting for sensitive information. (Simson gave examples of such data during his presentation.) He is working to educate people that "format" does not mean "erase," and he hopes Microsoft will replace the current format command with a tool that truly zeroes out a drive. Simson also said he is unaware of any technique to retrieve data from a zeroed-out hard drive, saying that Peter Gutmann's 1996 techniques would no longer work on drives built since then due to the density of modern drives.

Black Hat Federal 2006 Wrap-Up, Part 2

Please see part 1 for an introduction if you are reading this article separately.

The first technical talk I attended was presented by Mariusz Burdach, titled "Finding Digital Evidence In Physical Memory." Mariusz really needed two hours or more to give his topic justice. He started his talk buy holding up DoD and DoJ manuals which recommend pulling the plug as an incident response step (argh), and he said commercial tools all focus on inspecting hard drives. Unfortunately, modern rootkits may stay in non-swappable memory pages, and will not touch the hard drive. Therefore, traditional victim hard drive forensic practices may be useless against modern techniques.

Mariusz named three anti-forensic methods.

  1. Data contraception: do not create data on the hard drive; keep everything in memory

  2. Data hiding: keep processes from appearing in task or process lists

  3. Data destruction: remove suspicious information on the file system

He mentioned a few cool examples.

  • The Core Security syscall proxy as a means to not write any files to disk when loading malicious programs into memory on a target system.

  • The Metasploit SAM Juicer dumps Windows
    password hashes from a Meterpreter shell without writing any files to disk.

  • Hacker Defender has commercial antidetection modules.

  • Jamie Butler's FU and Shadow Walker (collaboration with Sherri Sparks) are impressive.

Marius briefly discussed software- and hardware-based means to acquire victim memory. On the hardware side he noted Tribble, a PCI card that can read system memory. On a related noted, I ate lunch on Wednesday with Jamie Butler. His new company Komoku is working on the Copilot host monitor, another PCI card mentioned by Mariusz. If you don't have a PCI card already in the victim system, you might be able to acquire or change memory via Firewire. I missed this when it was originally announced, but now I realize it's a huge issue.

The most relevant aspect of Mariusz's talk was his announcement of two tools for reviewing physical memory dumps. The first is the Windows Memory Forensic Toolkit (WMFT) and the second is Idetect, for Linux. These look very interesting, and I believe Mariusz will release new versions once he returns to Poland. Mariusz' talk and several that followed emphasized that memory absolutely must be analyzed when performing incident response.

Next I saw John Heasman from NGS Software present "Implementing and Detecting an ACPI BIOS Rootkit." John was the best speaker at BH Federal, in terms of content and delivery style. His presentation (.pdf) has already seen some coverage. The problem centers on the fact that Advanced Configuration and Power Interface can be used to read and write sensitive areas of targets, like system memory. For example, ACPI could be used to disable all access control on a Windows system by extracting ACPI Machine Language (AML) from a target BIOS, finding inititialization control methods, appending ACPI Source Language (ASL) to implement the SeAccessCheck exploit, recompiling into machine language, flashing the BIOS, and rebooting the system. Linux has a similar problem where the sys_ni_syscall exception handler could be patched.

John brought up very interesting points about rootkits. He asked whether they always need to be active, or if they could simply activate at random times to frustrate detection. He said the bootable CDs that use ACPI would be as vulnerable as the OS installed on a hard drive, making life tougher for incident responders. Sure, ACPI can be disabled, but that may disable some device drivers too. John said that ACPI debugging and Windows event logs may yield clues to ACPI exploitation, so stay alert. He also mentioned that ACPI could be modified such that fans never activate. Combine that with a process that starts the CPUs spinning and you have a software-based way to destroy a machine!

Keep in mind that a BIOS rootkit would not be a traditional rootkit. It would be used to infect a target, and then code on the target would open back doors and so on. The BIOS only offers "tens of KB" of space, according to John.

This reinforces my point that rootkits make NSM more relevant than ever. Now all we need is a Cisco router or switch rootkit.

Black Hat Federal 2006 Wrap-Up, Part 1

I attended two days of Black Hat Federal Briefings 2006. I paid my own way, and I must say the conference was worth every penny. If you didn't attend, I highly recommend registering for next year's conference. I spoke briefly with Jeff Moss, who said Black Hat will return to DC in February 2007 for another Federal conference. This is welcome news. I taught Foundstone's Ultimate Hacking: Expert class at Black Hat Federal 2003, which was the last Black Hat conference in DC.

My summaries cannot do most of the speakers justice. I will attempt to offer highlights for most talks, along with links to relevant techniques or tools.

Jeff Moss began the conference by noting its main theme: paranoia. After attending many of the sessions, I understand why. Jeff didn't want Federal to be "Las Vegas-lite," and I think he succeeded in assembling a conference that truly delivered.

Dr. Linton Wells II from DoD offered the keynote. He briefly discussed the Quadrennial Defense Review, which will be delivered to Congress on 6 Feb. He lamented the fact that the DoD budgets in 6 year increments, beyond which the department has to look 10 more years. He asked the audience to consider what the world was like in 1990 compared to today. How could planners in a pre-Gulf War, Soviet-facing, Internet-minimal world anticipate the current landscape? He mentioned that the DoD Directive 3000.05, "Military Support for Stability, Security, Transition, and Reconstruction (SSTR) Operations," dated November 28, 2005, emphasizes the traditional non-combat activities like network defense are on par with combat operations.

With regards to threats facing DoD, Dr. Wells said the threat is the "patient, skilled, well-resourced adversary with intent to do harm." (Dr. Wells did not say a hole in OpenSSH is a threat!) He noted that US Strategic Command has command over DoD networks now, and that DISA is trying to "minimize the number of connections from the Internet to the NIPRNet." DoD has recognized and is beginning to treat NIPRNet as the "command network" that it is, especially for logistics and health care users. Dr. Well said classic security labels (unclassified, secret, etc.) "just don't work anymore," and current 30, 45, or 60 day patch cycles "have to change." DoD has even spoken with Google about how that company decides how to internally select and fund projects on 3-6 development cycles.

I asked Dr. Wells about the security stand-down that happened in November. (More details here He said "we have a problem, and people need to pay attention to it." He said the stand-down included a password change and patching of applications, and that DoD has about 100,000 people with sys admin duties. I followed up with a question to two of Dr. Wells' team about DoD usage of Snort, given that Sourcefire was purchased by Checkpoint -- an Israeli company. They said there was "concern at high levels," and that a deputy secretary of defense had just been briefed on the issue on Tuesday. They emphasized that, in the future, DoD might require vendors to provide source code of their products to "assure the pedigree of their software." DoD is worried about foreign elements introducing back doors into code. Finally, of the $450 billion spent by DoD each year, $29-30 billion is IT-related. Of that amount, about $2 billion is IA-related.

Soekris Dies, What Replacement?

Yesterday the UPS powering my Soekris Net4801 died. Now the Soekris no longer finds its internal 2.5 hard drive running FreeBSD 6.0. I was able to update the BIOS using this guide and the comms/lrzsz, but it had no effect. The process was simple

> download

Shift ~
Shift C

lsz -X b4801_128.bin

If I want to stick with the Soekris, I may try one of the OS installation options listed here. However, I'm wondering if I should just abandon the Soekris for something more powerful. I saw the 256 MB Net4801 will arrive soon, but I've been looking at these OpenBrick and newer systems.

Does anyone have any recommendations for new small form factor systems? Here are my ideal requirements:

  • Very small and flat -- ideally something that would fit in a consultant's brief case for carrying on a plane, along with a laptop.

  • 3 NICs, preferably one or more with Gigabit capability

  • Can use flash or a laptop HDD

  • Runs FreeBSD 6.x

  • Video and keyboard outputs are not required, but I'm starting to like that option

  • At least 128 MB RAM, preferably 256 MB or more

This is starting to sound like a laptop, but I would prefer not to use a laptop. I do not like leaving laptops at client sites. The temptation to open the screen and touch the keyboard is too great for some clients. I like using a small appliance like a Soekris.

Snort.org Posts BlackWorm Packet Captures

The folks at Sourcefire have done the analyst community a great service by posting traffic captures of CME-24, aka "BlackWorm". Kudos also to the Common Malware Enumeration project for providing an easy way to reference malware! Once OpenPacket.org gets going, I hope to host these sorts of captures there.

Update: Check out this Sourcefire VRT analysis.

Thursday, January 26, 2006

Additional Thoughts on Amazon.com Reviews

I received some good comments on my previous post about my Amazon.com reviews. A few people at Black Hat Federal yesterday asked similar questions, namely: "Why don't you post bad reviews? We think they are more helpful than good reviews."

First, let's consider the definition of "bad review." I've never given a book 1 star. I've only given a few books two stars. For example, this book was awful. It's also got the highest number of fake positive reviews I've ever seen. (Many are written by people who have only reviewed the author's books, which is an indicator of being planted by the author.) The author somehow got Amazon.com to reject my original review. In the second review (which is now posted), I restricted my comments to quoting outrageously bad technical details that neither the author nor Amazon.com could deny.

My reading and reviewing habits are usually contrary to posting bad reviews. I am not the typical "reviewer" who gets a free book from the publisher, skims the contents, looks at the back cover, and then posts a so-called "review." In almost every case I read the whole book, or at least enough of the book to ensure I cover the author's main points. (Sometimes I do skip material. For example, I am not going to read about compiling software using "./configure, make, make install" in a sys admin book.)

Therefore, reading and reviewing is a fairly serious time commitment. As a result my reading list is about 50 books deep now. Publishers send me dozens of books per year. 95% of the time I have already identified the ones I want to read by adding them to my Wish List. If you're a publisher and you want a book review, please first check that list. If your book is not on it, I will probably not ever review it.

Books sent to me by publishers that go unread are not sold on eBay. I give them away at conferences or classes. I gave about 30 books away at ShmooCon, with the hope that attendees read and review them.

Given this scenario, sometimes I do write three star reviews. That sort of assessment means I thought the book had potential, so I decided to read it. While reading, I became disappointed by the content or technical accuracy. In 2005, I wrote 26 reviews. Of those, four were three-star reviews. This is a good example. I hoped that would be a good book, but the material covered plus the technical inaccuracies really sunk it for me.

The books most likely to get a low review are those I personally purchase. I may buy a book because it seems very good on the surface. If while reading it I find errors or other problems, I will definitely provide a bad review. I will probably not read all of a bad book in this case. The fact that I paid for the book will make me feel better about reviewing a book I have not thoroughly read.

I do not enjoy writing bad reviews. As an author, I do not like seeing my books receive low reviews. Thankfully that has only happen infrequently, and in the most recent case I can cite personal problems with one reviewer. As an author I also know that writing a book represents committing a lot of time and effort. If a lousy book somehow manages to slip through the editing process, or if the publisher refuses to correct deficiencies despite being notified, then I am more likely to post a bad review.

In some cases I even dislike writing 4 star reviews, since I can tell the author spent a lot of time on the book. A book that is technically sound can still receive a four star review. I usually deduct stars for covering material that has appeared elsewhere. I am a proponent for publishing new material, and I am disappointed when I see one good book followed by a handful of copycats who provide little original material.

I welcome your comments on this issue.

Wednesday, January 25, 2006

Issue 5 of (IN)SECURE Magazine Released

The new (IN)SECURE magazine is out. Issue 5 features another set of interesting articles. I plan to pay particular attention to Ivan Ristic's article on Web application firewalls. Ivan wrote modsecurity, O'Reilly's Apache Security, and the Web Security Blog. The new (IN)SECURE also gives brief but positive reviews of my two newest books, Real Digital Forensics and Extrusion Detection.

3000 Helpful Review Votes at Amazon.com

This morning my Amazon.com reviews "helpful votes" count hit 3,000. This means my reviews were considered "helpful" 3,000 times. (Conversely, 299 people thought they were not helpful. Sorry!) Thank you to everyone who answered yes to the question "Was this review helpful to you?"

I reported hitting the 1,500 mark in December 2003. Since then I reviewed 62 more books, but my reviewer rank has dropped from 336 to 390. On the positive side, my average number of helpful votes per review has risen from 12 (or 1,500 / 125 ) to 16 (3,000 / 187).

Competition is tough when many high ranking "reviewers" post several times per day, showing they only glanced at a book's contents and read the back cover. The person I have in mind when writing this, however, has received 13,236 votes for 2260 reviews. His vote-to-review ratio is less than 6, indicating his reviews are, on average, not that helpful. Justice, perhaps?

I've only just started reading and reviewing regularly again. Right now I'm enjoying Running IPv6, which is excellent. I'd like to thank all of the publishers who send books for review. I never sell them on eBay, nor do I receive money from publishers or Amazon.com to write reviews. I'm in it for learning and sharing word of good technical books.

Tuesday, January 24, 2006

Nepenthes Discoveries

Earlier today I posted how I installed Nepenthes. Within a few minutes I started getting hits. Monitoring with Sguil makes analysis much easier.

Consider this first attack:

Sensor Name: soekris
Timestamp: 2006-01-24 18:14:34
Connection ID: .soekris_4888215984542487947
Src IP: (69-11-44-99.regn.hsdb.sasknet.sk.ca)
Dst IP: (pcp0010708738pcs.manass01.va.comcast.net)
Src Port: 1734
Dst Port: 80
OS Fingerprint: - Windows 2000 SP2+, XP SP1 (seldom 98 4.10.2222) [priority1]
OS Fingerprint: -> (distance 24, link: ethernet/modem)

SRC: Host:
DST: Microsoft Windows 2000 [Version 5.00.2195]
DST: (C) Copyright 1985-2000 Microsoft Corp.
DST: C:\WINDOWS\System32>
SRC: .

In the middle of that mess you can see Nepenthes pretending to return a Windows command shell.

Here is the text represenation of the corresponding Snort alert as reported through Sguil.

Count:1 Event#1.2039 2006-01-24 18:14:34
IPVer=4 hlen=5 tos=32 dlen=1500 ID=60064 flags=2 offset=0 ttl=104 chksum=16980
Protocol: 6 sport=1734 -> dport=80

Seq=2259253829 Ack=960260189 Off=5 Res=0 Flags=***A**** Win=17520 urp=61185 chks
46 42 51 55 46 42 51 55 46 42 51 51 4D 41 49 34 FBQUFBQUFBQQMAI4
49 4D 56 77 4F 43 42 41 6F 41 6B 45 4B 51 51 70 IMVwOCBAoAkEKQQp
31 2F 55 31 5A 58 5A 6F 48 73 67 41 43 4A 35 75 1/U1ZXZoHsgACJ5u
6A 74 41 41 jtAA

This is interesting, but what happens next is cooler.

Here are the three sessions involving the attacking source IP, as recorded by SANCP and reported by Sguil.

Sensor:soekris Session ID:4888215984542349888
Start Time:2006-01-24 18:14:34 End Time:2006-01-24 18:14:34 ->
Source Packets:4 Bytes:0
Dest Packets:3 Bytes:0
Sensor:soekris Session ID:4888215984542487947
Start Time:2006-01-24 18:14:34 End Time:2006-01-24 18:14:35 ->
Source Packets:9 Bytes:5699
Dest Packets:5 Bytes:104
Sensor:soekris Session ID:4888215988836780153
Start Time:2006-01-24 18:14:35 End Time:2006-01-24 18:15:27 ->
Source Packets:11 Bytes:71
Dest Packets:10 Bytes:191

The first session is recon. The second is the attack as seen by Snort.

Here is a transcript of the last session.

Sensor Name: soekris
Timestamp: 2006-01-24 18:14:35
Connection ID: .soekris_4888215988836780153
Src IP: (pcp0010708738pcs.manass01.va.comcast.net)
Dst IP: (69-11-44-99.regn.hsdb.sasknet.sk.ca)
Src Port: 50748
Dst Port: 11624
OS Fingerprint: - UNKNOWN [65535:63:1:64:M1460,N,W1,N,N,T,S,E:P:?:?] (up: 1264 hrs)
OS Fingerprint: -> (link: ethernet/modem)

DST: 220 StnyFtpd 0wns j0
DST: 331 Password required
DST: 230 User logged in.
SRC: PORT 192,168,2,7,211,48
DST: 200 PORT command successful.
SRC: RETR commdlg32.exe
DST: 150 Opening BINARY mode data connection
DST: 425 Can't open data connection.
DST: 221 Goodbye happy r00ting.

"lol" indeed. Nepenthes was unable to retrieve the commdlg32.exe binary because of this PORT command:

SRC: PORT 192,168,2,7,211,48

The attacker FTP server does not know how to connect to!

I am sniffing this activity outside of my router, which should have translated into the router's public IP. Why didn't that happen? The reason is the destination port for this FTP command channel -- it's 11624 TCP. The standard FTP command channel uses dest port 21 TCP.

In this case, the malware was unable to retrieve additional executables because it chose to use active FTP. If it had chosen passive FTP, then the client would have connected to the server. No client PORT command would have been required.

Installing Tor

In my last post I mentioned that by default Nepenthes is configured to use Tor to carry IRC traffic. This post documents what I did to get Tor running on FreeBSD 6.0 STABLE.

I installed Tor using the security/tor-devel page. Remember to set the environment variable to use the newest package.

janney:/root# pkg_add -vr tor-devel

Next I added the following to /etc/rc.conf so I could use the /usr/local/etc/rc.d/tor.sh script.


Next I edited /usr/local/etc/rc.d/tor.sh, because I had an issue with the %%PREFIX%% specification.

janney:/usr/local/etc/rc.d# diff tor.sh.orig tor.sh
< TORCTL=%%PREFIX%%/bin/torctl
> TORCTL=/usr/local/bin/torctl

I used the default config file.

janney:/root# cp /usr/local/etc/tor/torrc.sample /usr/local/etc/tor/torrc

I needed to create this tor data directory.

janney:/root# mkdir -p /var/db/tor/data
janney:/root# chown -R _tor:_tor /var/db/tor/data

I also needed to create this log file owned by user _tor.

janney:/root# touch /var/log/tor
janney:/root# chown _tor:_tor /var/log/tor

I debated running Tor directory lookups through my Squid proxy. If I want to do that, I add the following TORARGS section:

TORARGS="$TORARGS --HttpProxy --HttpsProxy"

Tor doesn't run as a server by default. If I wanted to be a Tor server, I would consider adding the following TORARGS:

TORARGS="$TORARGS --BandwidthRate 10KB --BandwidthBurst 20KB --MaxAdvertisedBandwidth 10KB"

After I made the changes listed above, I started Tor using the tor.sh script.

janney:/root# /usr/local/etc/rc.d/tor.sh start
Starting tor:
Jan 24 15:43:21.144 [notice] Tor v0.1.1.12-alpha. This is experimental software.
Do not rely on it for strong anonymity.
Jan 24 15:43:21.156 [notice] Initialized libevent version 1.1a using method kqueue. Good.
Jan 24 15:43:21.156 [notice] connection_create_listener(): Opening Socks listener on
/usr/local/bin/torctl start: tor started

janney:/root# sockstat -4 | grep tor
_tor tor 7810 4 tcp4
_tor tor 7810 5 tcp4 *:*
_tor tor 7810 9 tcp4

Now when I start Nepenthes and tell it to use Tor, it appears like the following in my specified IRC channel:

15:44 -!- mynep [i=debian-t@dsl093-038-182.pdx1.dsl.example.com] has
joined #myfakeircchannel

That Debian system is not my own -- it's the other side of the Tor connection. There are many ways to use Tor. I just wanted to document how I got it working to support Nepenthes.

Monday, January 23, 2006

Nepenthes Installation

I've been interested in trying Nepenthes since I saw it added to the FreeBSD ports collection as net/nepenthes. According to the Nepenthes Web site, "Nepenthes is a versatile tool to collect malware. It acts passively by emulating known vulnerabilities and downloading malware trying to exploit these vulnerabilities."

I tried to install Nepenthes using the precompiled package for FreeBSD, like this:

janney:/root# setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-6-stable/Latest/

janney:/root# pkg_add -vr nepenthes

I ran into two problems. First, I had to install the ftp/curl port manually since the package seemed unavailable.

cd /usr/ports/ftp/curl
make install

Second, and more problematic, I found that the package which offered Nepenthes 0.1.5 did not work properly. Using the package, I could not get my Nepenthes client to connect to a specific IRC channel protected by a key.

I decided to install Nepenthes using the FreeBSD port. I made these changes to make the ports tree install version 0.1.6 instead of the older 0.1.5:

janney:/usr/ports/net/nepenthes# diff Makefile.orig Makefile
janney:/usr/ports/net/nepenthes# diff distinfo.orig distinfo
< MD5 (nepenthes-0.1.5.tar.gz) = d7eae244a5adef66ca504a233f1c51e1
< SHA256 (nepenthes-0.1.5.tar.gz) = 7c74614cb3027f0c9a409f68ed81baed4793673509e09138bd6296d72b04b08a
< SIZE (nepenthes-0.1.5.tar.gz) = 780788
> MD5 (nepenthes-0.1.6.tar.gz) = 317afd3dc86d57a22570632bdf839ef2
> SHA256 (nepenthes-0.1.6.tar.gz) = f9bae290d49df9658b7f27a2f4c304fd671cc1f2f344a3b960a181c12416d94b
> SIZE (nepenthes-0.1.6.tar.gz) = 794938

Once Nepenthes was installed, I began editing configuration files in /usr/local/etc/nepenthes.

First I edited log-irc.conf to log to my IRC channel of choice.

janney:/usr/local/etc/nepenthes# diff log-irc.conf.orig log-irc.conf
< nick "nep-noname";
< ident "nepenthes";
< userinfo "http://nepenthes.sf.net";
> nick "mynep";
> ident "mynep";
> userinfo "mynep";
< name "#nepenthesirc";
< pass "foo";
> name "#myfakeircchannel";
> pass "myfakepw";

Note that the default log-irc.conf wants to use Tor. I show how I set that up in a future post. To disable using Tor, change

use-tor "1";


use-tor "0";

Next I made changes to nepenthes.conf to reflect using /var/log instead of var/log.

janney:/usr/local/etc/nepenthes# diff nepenthes.conf.dist nepenthes.conf
< ring_logging_file "var/log/nepenthes/nepenthes.%d.log";
< file_logging_file "var/log/nepenthes/nepenthes.log";
> ring_logging_file "/var/log/nepenthes/nepenthes.%d.log";
> file_logging_file "/var/log/nepenthes/nepenthes.log";
< filesdir "var/nepenthes/binaries/";
> filesdir "/var/nepenthes/binaries/";
< hexdump_path "var/nepenthes/hexdumps/";
> hexdump_path "/var/nepenthes/hexdumps/";
< cache_path "var/cache/nepenthes/geolocation/";
> cache_path "/var/cache/nepenthes/geolocation/";

I made similar changes to log-download.conf.

janney:/usr/local/etc/nepenthes# diff log-download.conf.orig log-download.conf
< downloadfile "var/log/logged_downloads"; // log download attempts
< submitfile "var/log/logged_submissions"; // log successfull downloads
> downloadfile "/var/log/logged_downloads"; // log download attempts
> submitfile "/var/log/logged_submissions"; // log successfull downloads

And submit-file.conf:

janney:/usr/local/etc/nepenthes# diff submit-file.conf.orig submit-file.conf
< path "var/binaries/";
> path "/var/nepenthes/binaries/";

And submit-norman.conf:

janney:/usr/local/etc/nepenthes# diff submit-norman.conf.orig submit-norman.conf
< email "malware@mac.com";
> email "myemail@gmail.com";

download-nepenthes.conf needed a more radical change because I don't have /share/hda3/opt/ on FreeBSD.

janney:/usr/local/etc/nepenthes# diff download-nepenthes.conf.orig download-nepenthes.conf
< filespath "/share/hda3/opt/nepenthes";
> filespath "/var/nepenthes/";

Next I create directories needed by Nepenthes.

janney:/root# mkdir -p /var/nepenthes/binaries
janney:/root# mkdir /var/log/nepenthes
janney:/root# touch /var/log/nepenthes/nepenthes.log

When done I was able to start Nepenthes with the simple command 'nepenthes'. I recommend running it in the foreground within script(1). By default Nepenthes generates a ton of debug info.

As configured, Nepenthes is listening on a slew of ports:

janney:/usr/local/etc/rc.d# sockstat -4 | grep nep
root nepenthes 7669 3 udp4 *:50838 *:*
root nepenthes 7669 6 tcp4 *:21 *:*
root nepenthes 7669 7 tcp4 *:25 *:*
root nepenthes 7669 8 tcp4 *:110 *:*
root nepenthes 7669 9 tcp4 *:143 *:*
root nepenthes 7669 10 tcp4 *:220 *:*
root nepenthes 7669 11 tcp4 *:465 *:*
root nepenthes 7669 12 tcp4 *:993 *:*
root nepenthes 7669 13 tcp4 *:995 *:*
root nepenthes 7669 14 tcp4 *:2745 *:*
root nepenthes 7669 15 tcp4 *:6129 *:*
root nepenthes 7669 16 tcp4 *:135 *:*
root nepenthes 7669 17 tcp4 *:445 *:*
root nepenthes 7669 18 tcp4 *:1025 *:*
root nepenthes 7669 19 tcp4 *:443 *:*
root nepenthes 7669 20 tcp4 *:17300 *:*
root nepenthes 7669 21 tcp4 *:2103 *:*
root nepenthes 7669 22 tcp4 *:2105 *:*
root nepenthes 7669 23 tcp4 *:2107 *:*
root nepenthes 7669 24 tcp4 *:3372 *:*
root nepenthes 7669 25 udp4 *:1434 *:*
root nepenthes 7669 26 tcp4 *:3127 *:*
root nepenthes 7669 27 tcp4 *:139 *:*
root nepenthes 7669 28 tcp4 *:3140 *:*
root nepenthes 7669 29 tcp4 *:5554 *:*
root nepenthes 7669 30 tcp4 *:1023 *:*
root nepenthes 7669 31 tcp4 *:27347 *:*
root nepenthes 7669 32 tcp4 *:5000 *:*
root nepenthes 7669 33 tcp4 *:10000 *:*
root nepenthes 7669 34 tcp4 *:42 *:*
root nepenthes 7669 35 tcp4 *:80 *:*

The system running Nepenthes has a private IP and it sits behind my Comcast cable modem. I decided to tell my router connected to the cable modem to forward ports 80 and 443 to the Nepenthes system.

Now, if I connect to port 80 on my router, Nepenthes handles the connection.

janney:/root# nc -v bej.dyndns.org 80
Connection to bej.dyndns.org 80 port [tcp/http] succeeded!

Here is what Nepenthes reports:

[ debug net mgr ] Socket TCP (bind) ->
DialogueFactory ASN1 Dialogue Factory creates dialogues for the SMB and IIS flaw
killbill showed us could Accept a Connection
[ spam net handler ]
[ spam net handler ] Socket TCP (accept) ->
[ spam net handler ] Adding Dialogue ASN1 Dialogue Factory
[ spam mgr event ]
[ debug net mgr ] Accepted Connection Socket TCP (accept) ->
32 Sockets in list
[ spam net handler ]
[ spam mgr event ]
[ spam net handler ] doRecv() 16
[ debug net handler ] Dialogue IISDialogue inactive, returned CL_DROP
[ debug net handler ] Socket TCP (accept) ->
has no active Dialogues left, closing
[ debug net mgr ] Deleting Socket TCP (accept) ->
due to closed connection
[ spam net handler ]
[ spam net handler ] Socket TCP (accept) ->
clearing DialogueList (1 entries)
[ spam net handler ] Removing Dialog "IISDialogue"
[ warn dia ] Unknown IIS 16 bytes State 0
[ dia ] =------------------[ hexdump(0x0808c300 , 0x00000010) ]-------------------=
[ dia ] 0x0000 48 45 41 44 20 2f 20 48 54 54 50 2f 31 2e 30 0a HEAD / H TTP/1.0.
[ dia ] =-------------------------------------------------------------------------=

If I find anything interesting, I will pass it on. Here I just wanted to document what I had to do to get Nepenthes running.

Web Site Discovery with SensePost SP-DNS-mine.pl

Today I needed to discover Web sites for a client. I'll demonstrate part of my methodology here, using sun.com as a sample domain. I relied on a technique outlined in Johnny Long's Google Hacking for Penetration Testers. He mentions a SensePost tool called SP-DNS-mine.pl. The script uses Google to extract sub domains and DNS names for a given domain. You have to register with SensePost to retrieve SP-DNS-mine.pl; they email a username and password once you register.

The first requirement is having a license key for the Google API. You put your key into SP-DNS-mine.pl, thus:

#$key = "----YOUR GOOGLE API KEY HERE----";

Since I am running the script on FreeBSD, I realized I needed the net/p5-SOAP-Lite package. I added the latest version from the STABLE package collection.

Finally I needed the file http://api.google.com/GoogleSearch.wsdl.

orr:/home/richard$ fetch http://api.google.com/GoogleSearch.wsdl
fetch: http://api.google.com/GoogleSearch.wsdl: size of remote file is not known
GoogleSearch.wsdl 7496 B 145 kBps

Now I'm ready to find sun.com Web sites.

orr:/home/richard$ perl ./SP-DNS-mine.pl sun.com

Adding word [site]
0 1 0 1
Adding word [web]
0 1 0 1
Adding word [document]
0 1 0 1
Adding word [sun.com]
0 1 0 1
DNS names:

Sub domains:

That's it. You'll notice I found domains that end in sun.com but are not part of sun.com, like www.gainesvillesun.com. Still, this is a powerful way to use Google to identify Web servers.