Wednesday, December 31, 2008

Best Book Bejtlich Read in 2008

If I read and reviewed a book you wrote in 2008, this was one of the better years to win my Best Book Bejtlich Read award. I only read and reviewed 20 books this year, compared to 17 in 2000, 42 in 2001, 24 in 2002, 33 in 2003, 33 in 2004, 26 in 2005, 52 in 2006, and 25 in 2007.

My 2007 and 2006 winners are posted too. Although I've been reviewing books seriously since 2000 and blogging since 2003, I only started listing my favorite books in 2006.

I did not spend enough time "hanging in the sky" (to quote John Denver) reading a book, and too much of my day job spilled into my evening reading hours. I prefer to avoid long-haul air travel, so I don't expect to read more on planes in 2009. Regarding work-life balance, I have more help at work for detection and response duties. We'll see how 2009 fares with respect to reading overall.

My ratings for 2008 can be summarized as follows:

  • 5 stars: 7 books

  • 4 stars: 8 books

  • 3 stars: 4 books

  • 2 stars: 1 book

  • 1 star: 0 books

Here's my overall ranking of the five star reviews; this means all of the following are excellent books.

  • 7. Beginning Perl, 2nd Ed by James Lee. Lee's book is excellent from start to finish. I found his explanations very clear and his writing style lively. He covered just about everything I hoped to read in a book of roughly 400 pages.

  • 6. OSSEC HIDS by Rory Bray, Daniel Cid and Andrew Hay. I have to congratulate the author team for OHG. Writing a book for Syngress with many contributors is usually a recipe for disaster. OHG features three lead authors, four contributors, and one foreword author -- and they don't step on each others' toes.

  • 5. Virtual Honeypots: From Botnet Tracking to Intrusion Detection by Niels Provos and Thorsten Holz. If you are at all interested in potentially deceiving intruders, buy and read Virtual Honeypots. You'll learn about more than VMware (QEMU, UML, etc.) as well as numerous open source tools you can download and try for free.

  • 4. Googling Security: How Much Does Google Know About You? by Greg Conti. There's no question that Greg Conti writes excellent books. Last year's Security Data Visualization book earned 5 stars, and I put Googling Security in the same league. Conti takes a thorough and methodical look at the privacy consequences of Google's services, incorporating technical realities and thoughtful analysis.

  • 3. Nmap Network Scanning by Gordon "Fyodor" Lyon. If you are looking for *the* book on Nmap, the search is over: NNS is a winner.

  • 2. Applied Security Visualization by Raffy Marty. I think ASV is a great book on security visualization, but it will also help general security practitioners.

And, the winner of the Best Book Bejtlich Read in 2008 award is...

1. Malware Forensics: Investigating and Analyzing Malicious Code by Cameron H. Malin, Eoghan Casey, and James M. Aquilina. Malware Forensics is an awesome book. Last year Syngress published Harlan Carvey's 5-star Windows Forensic Analysis, and now we get to enjoy this new title. I should disclose that I co-wrote a forensics book with Curtis Rose, and I just delivered a guest lecture in a class taught by Eoghan Casey. However, I still call books as I see them, regardless of the author.

I can confidently say that anyone interested in learning how to analyze malware, or perform incident response, will benefit from reading Malware Forensics. The authors even maintain a Web site -- -- to support the book.

Looking at the publisher count, top honors in 2008 go to Addison-Wesley for 3 titles, followed by Syngress with 2, and finally Apress and a self-published title, each with one. Thank you to all publishers who sent me books in 2008. I have plenty more to read in 2009.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Bejtlich Speaking at DC BSDCon 2009

Jason Dixon just announced that I will be speaking at DC BSDCon 2009, either 5 or 6 February 2009. I'm looking forward to this conference since it's local and thus easier to attend. I'll be discussing something about Network Security Monitoring that applies to FreeBSD.

Registration ends 31 Jan 09 and is limited to the first 150 attendees.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Monday, December 29, 2008

Installing Sguil Using NSMNow

In my post NSM-Friendly VMware Lab Setup I mentioned wanting to use NSMNow to install Sguil on Ubuntu 8.04 for student use in my next class. I had tried the Securix-NSM live CD but I had not tried installing Sguil using the same project's NSMNow scripts. I just did it:

root@twsu804:/usr/local/src# wget
=> `NSMnow-1.1.1.tar.gz'
Connecting to||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 164,613 (161K) [application/x-gzip]

100%[====================================>] 164,613 53.85K/s

22:14:42 (53.80 KB/s) - `NSMnow-1.1.1.tar.gz' saved [164613/164613]

root@twsu804:/usr/local/src# tar -xzvf NSMnow-1.1.1.tar.gz

root@twsu804:/usr/local/src# cd NSMnow-1.1.1/

root@twsu804:/usr/local/src/NSMnow-1.1.1# ls
INSTALL MANUAL NSMnow-core README.apparmor templates
libs NSMnow NSMnow.log RELEASE.NOTES
LICENCE NSMnow.conf README run-init

root@twsu804:/usr/local/src/NSMnow-1.1.1# ./NSMnow -i

Allow pre-checks to install requisite packages [Y]:
[2008/12/29 22:18:05] #1 - Performing NSMnow pre-checks.
[2008/12/29 22:21:06] #1 - Pre-checks completed successully
[2008/12/29 22:21:06] #1 - Detected platform: UBUNTU
[2008/12/29 22:21:06] #1 - Action: Installing package(s).

Download Directory
Path where all downloaded files will be saved to

Source Directory
Path where all source tarballs will be extracted to

Sensor Name
A unique name given to deliniate sensors from one another
[sensor1]: twsu804a

Sensor Interface
Enter the interface that this sesnor will be monitoring
[eth0]: eth1

Configuration Path
Path to where all sensor related configuration files will be stored

Sensor Data Path
Path to where all sensor captured information will be stored

Server Host
Hostname or IP of the server component that this sensor will connect to

Server Name
A unique name given to deliniate servers from one another

Server Data Path
Path to where all server collected information will be stored

Server Database Name
Name of the sguil database which will store all sguil correlated information.

Server Database User
Name of the user who will have access rights to the sguil database.

Server Database Password
Password of the user who will have access rights to the sguil database.
[password]: sguil

Client User
Name of the sguil client user who will have access the sguil server.

Client Password
Password of the sguil client user who will have access to the sguil server.
[password]: sguil

Server Host
Hostname or IP of the server component that this client will connect to
[2008/12/29 22:23:16] #1 - Installing package: mysql
[2008/12/29 22:23:16] #1 - Installing with: apt-get -y install mysql-server
[2008/12/29 22:28:22] #1 - Installing package: tcl
[2008/12/29 22:28:22] #1 - Installing with: apt-get -y install tcl8.3
itcl3 mysqltcl tcltls tcllib tcl8.3-dev iwidgets4 tclx8.4 itk3 tcl8.4 tk8.4
[2008/12/29 22:29:23] #1 - Installing package: buildessential
[2008/12/29 22:29:23] #1 - Installing with: apt-get -y install libpcre3-dev
libpcap0.8-dev build-essential
[2008/12/29 22:29:43] #1 - Installing package: snort
Download snort tarball? [Y]: y
[2008/12/29 22:39:37] #1 - Configuring with: ./configure --enable-perfprofiling
[2008/12/29 22:40:29] #1 - Compiling with: make
[2008/12/29 22:43:50] #1 - Installing with: make install
[2008/12/29 22:44:02] #1 - Installing package: barnyard2
Download barnyard2 tarball? [Y]: y
[2008/12/29 22:44:29] #1 - Configuring with: ./configure --with-tcl=/usr/lib/tcl8.3
[2008/12/29 22:44:54] #1 - Compiling with: make
[2008/12/29 22:45:10] #1 - Installing with: make install
[2008/12/29 22:45:10] #1 - Installing package: sancp
Download sancp tarball? [Y]: y
[2008/12/29 22:45:18] #1 - Compiling with: make linux
[2008/12/29 22:45:37] #1 - Installing with: cp sancp /usr/local/bin
[2008/12/29 22:45:37] #1 - Installing package: sguilsensor
Download sguil-sensor (sguil) package(s)? [Y]: y
[2008/12/29 22:46:09] #1 - Installing sguil-sensor binaries
[2008/12/29 22:46:10] #1 - Installing package: sguilclient
[2008/12/29 22:46:10] #1 - Installing sguil-client library files
[2008/12/29 22:46:10] #1 - Installing sguil-client binary
[2008/12/29 22:46:10] #1 - Installing package: sguilserver
[2008/12/29 22:46:10] #1 - Installing sguil-server library files
[2008/12/29 22:46:10] #1 - Installing sguil-server binary
[2008/12/29 22:46:10] #1 - Installing package: sguiltools
[2008/12/29 22:46:10] #1 - Installing with: apt-get -y install wireshark p0f tcpflow tcpdump
[2008/12/29 22:47:24] #1 - Configuring package: mysql
* Stopping MySQL database server mysqld [ OK ]
* Stopping MySQL database server mysqld [ OK ]
Reloading AppArmor profiles : done.
* Starting MySQL database server mysqld [ OK ]
* Checking for corrupt, not cleanly closed and upgrade needing tables.
[2008/12/29 22:48:07] #1 - Configuring package: tcl
[2008/12/29 22:48:10] #1 - Configuring package: buildessential
[2008/12/29 22:48:10] #1 - Configuring package: snort
[2008/12/29 22:48:10] #1 - Generating snort config file: /etc/nsm/twsu804a/snort.conf
[2008/12/29 22:48:11] #1 - Configuring package: barnyard2
[2008/12/29 22:48:11] #1 - Generating barnyard2 config file: /etc/nsm/twsu804a/barnyard2.conf
[2008/12/29 22:48:12] #1 - Configuring package: sancp
[2008/12/29 22:48:12] #1 - Generating sancp config file: /etc/nsm/twsu804a/sancp.conf
[2008/12/29 22:48:12] #1 - Configuring package: sguilsensor
[2008/12/29 22:48:12] #1 - Generating sensor agent config file(s)
[2008/12/29 22:48:12] #1 - Configuring package: sguilclient
[2008/12/29 22:48:12] #1 - Generating sguil-client config file: /etc/sguil/sguil.conf
[2008/12/29 22:48:12] #1 - Configuring package: sguilserver
[2008/12/29 22:48:12] #1 - Configuring AppArmor profile
[2008/12/29 22:48:12] #1 - Ensure you restart AppArmor to apply changes
[2008/12/29 22:48:12] #1 - Generating sguil-server config file: /etc/sguild/sguild.conf
[2008/12/29 22:48:13] #1 - Updating sguild init file: /etc/init.d/sguild
Copy default rules file(s)? [Y]: y
What Sensor name is to be associated with these rules [sensor1]: twsu804a
[2008/12/29 22:49:20] #1 - Creating the CA certificate
[2008/12/29 22:49:22] #1 - Creating certificate request for: server1
[2008/12/29 22:49:22] #1 - Signing server certificate for: server1
[2008/12/29 22:49:22] #1 - Adding client user "sguil" to sguil server ACL.
[2008/12/29 22:49:22] #1 - Creating database and initial user.

You will need the mysql root password.
Enter password:
[2008/12/29 22:49:29] #1 - Configuring package: sguiltools
[2008/12/29 22:49:29] #1 - Completed installing package(s) successfully.

NOTE: Snort can log in either UTC or the localtime, so firstly make sure that all machines
are synced together.Secondly, either set the timezone on all machines to UTC or set the
timezone on all machines to the same andremove the $UTC variable from the OPTIONS variable
in both /etc/init.d/snortu and /etc/init.d/snortl

I decided to comment out the $UTC variable in the two scripts. Then I started the programs.

root@twsu804:/usr/local/src/NSMnow-1.1.1# ./run-init start
Starting - sguil server (sguild) [ OK ]
Starting - sguil: sensor snort_agent (snort_agent) [ OK ]
Starting - sguil: sensor pcap_agent (pcap_agent) [ OK ]
Starting - sguil: sensor sancp_agent (sancp_agent) [ OK ]
Starting - snort: IDS mode, unified output (snort_unified) [ OK ]
* output in /nsm/sensor_data/twsu804a, /ssn_logs, /portscans
Starting - barnyard2 (barnyard2) [ OK ]
* created directory: /var/log/barnyard2
* created directory: /var/log/barnyard2/twsu804a
Starting - sancp: session logging (sancpd) [ OK ]
* output in /nsm/sensor_data/twsu804a/sancp
Starting - snort: logging mode (snort_packetlogging) [ OK ]
* output in /nsm/sensor_data/twsu804a/dailylogs/2008-12-30
* created directory: /nsm/sensor_data/twsu804a/dailylogs
* created directory: /nsm/sensor_data/twsu804a/dailylogs/2008-12-30
* disk space currently at 43%

At this point I could start a user terminal, type, and start using the Sguil console. The only real change I made was to alter the default fonts. I would probably consolidate the three panels into 1 as well.

Very impressive! Great work guys.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

NSM-Friendly VMWare Lab Setup

I'm working on labs for my all-new TCP/IP Weapons School 2.0 class (early registration ends Wednesday). Almost the whole class is labs; I'll have between 10 and 12 scenarios for students to investigate.

As you might imagine, network traffic will play a key role. I wanted to set up a VM running Ubuntu that could watch traffic involving other VMs. (Why not FreeBSD? Ubuntu is easier for students to use, and NSMnow makes it easy to get Sguil running. FreeBSD has also never seemed to run well in VMs due to some weird timing issues that have never been resolved.)

The problem, as I noted in Using VMware for Network Security Monitoring last year, is that modern versions of VMware Server (I run 1.0.8 now) act as switches and not hubs. That means each VM is connected to a virtual switch, effectively sheltered from other traffic. This is good for performance but bad for my monitoring needs.

Monitoring on the VMware server itself is not an option. Although it can see the traffic, I want to distribute a VM to the students that was running and capturing the traffic using Sguil and other tools as necessary.

Incidentally, here are two options for sniffing on the VMware server itself, for reference. VMware mentions its vmnet-sniffer, which is a console-output application with basically no features used only for troubleshooting:

richard@neely:~$ sudo vmnet-sniffer -e /dev/vmnet1

len 98 src 00:0c:29:7f:d6:a1 dst 00:0c:29:0a:0f:c1 IP src
dst ICMP ping request - len=64 type=8
00:0c:29:7f:d6:a1 08 00 88 e6 c0 17 00 01 ae 85 59 49 b5 2e 07 00 08
09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36

len 98 src 00:0c:29:0a:0f:c1 dst 00:0c:29:7f:d6:a1 IP src
dst ICMP ping reply

You could just as easily run Tcpdump or any other sniffer of your choice:

richard@neely:~$ sudo tcpdump -n -i vmnet1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmnet1, link-type EN10MB (Ethernet), capture size 96 bytes
20:41:51.272555 IP > ICMP echo request, id 49175, seq 1, length 64
20:41:51.273469 IP > ICMP echo reply, id 49175, seq 1, length 64

One note: vmnet-sniff can watch /dev/vmnet0 even though vmnet0 is not listed by ifconfig. vmnet0 is the bridged interface so you just watch it directly (e.g., eth0, etc.) with Tcpdump.

What to do? I decided that I could deploy the NSM sensor VM as a gateway, and put any hosts which I want to monitor as legs on that gateway. Consider this three-host scenario:

  1. NSM sensor VM / gateway with 1) eth0 as, default gateway is, the VMware NAT /dev/vmnet8 gateway; 2) eth1 as, on a random subnet; and 3) eth2 as, on another random subnet

  2. Windows victim with interface, default gateway

  3. Linux attacker with interface, default gateway

I configured the NSM sensor to be a gateway, and told it to NAT connections outbound to the VMware server NAT interface:

root@tws-u804:~# echo "1" > /proc/sys/net/ipv4/ip_forward
root@tws-u804:~# iptables -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE
root@tws-u804:~# iptables -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE

Why this "second" level of NAT via MASQUERADE? It turns out that if you send traffic from, say, through a gateway that doesn't NAT, when that gateway sends the traffic with source IP to the NAT interface on the VMware server, the VMWare server doesn't know how to handle the replies. I saw the traffic exit properly (i.e., it was NATed out), but when the reply arrived the VMware server didn't know how to return it to With this "second" NAT on the NSM sensor / gateway, the VMware server thinks the gateway is originating all traffic, so the hosts can reach the Internet.

With this setup I can now monitor traffic from to, because the traffic is routed through the NSM sensor / gateway.

This seems kludgy, and I wish there were a way to just configure VMware Server to act like a hub and have all hosts see all traffic. If anyone knows how to do that, please let me know.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Thursday, December 25, 2008

Snort Report 22 Posted

My 22nd Snort Report titled Snort vs. Microsoft Security Bulletin MS08-068 has been posted. From the article:

Welcome to the 22nd edition of the Snort Report! On Nov. 11, 2008, Microsoft published Microsoft Security Bulletin MS08-068 -- Important Vulnerability in SMB Could Allow Remote Code Execution (957097). Server Message Block (SMB) is an old and integral aspect of Microsoft Windows file sharing and related functions...

I continue by describing how Snort's rule set dealt with this super-old vulnerability.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

OSSEC and Pf on FreeBSD to Limit SSH Brute Forcing

Disclaimer: This post is neither original nor particularly illuminating. It does, however, document how I configured software on systems I administer. Therefore, I post it here mainly for my own future reference, but know it might be useful to someone else.

If you run OpenSSH on any Internet-facing server, you're likely to see entries like these every day:

r200a:/root# bzcat /var/log/auth.log.0.bz2 | head -n 5 | grep -v turned
Dec 23 20:00:02 r200a sshd[33320]: Invalid user httpd from
Dec 23 20:00:03 r200a sshd[33322]: Invalid user dima from
Dec 23 20:00:04 r200a sshd[33324]: Invalid user bane from
Dec 23 20:00:05 r200a sshd[33326]: Invalid user juan from

I like to run OSSEC on servers as a means to monitor and analyze log files. OSSEC would report that activity as follows.

2008 Dec 23 20:00:44 Rule Id: 5712 level: 10
Location: (r200a)>/var/log/auth.log
Src IP:
SSHD brute force trying to get access to the system.
Dec 23 20:00:02 r200a sshd[33320]: Invalid user httpd from

In the setup for this post I am running an OSSEC agent on an Internet-facing gateway with an internal IP of (r200a). My OSSEC server is (macmini). My simulated attack box is (debian40r0).

The first thing I need is a program to brute force SSH on I wanted something simple so I installed by d3hydr8.

tws@debian40r0:~$ wget
tws@debian40r0:~$ su -
debian40r0:~# apt-get install python-pexpect
debian40r0:~# logout
tws@debian40r0:~$ python sshBrute v1.0

Usage : ./
Eg: ./ root words.txt

I decided to use /etc/dictionaries-common/words for my wordlist because this is only a test and I don't really care if I can brute force my own user accounts in this scenario. I just want to be sure I can configure OSSEC to tell Pf to block the offending source IP for a period of time.

Now I need to configure Pf on the FreeBSD gateway to ensure OSSEC can work with it. I make certain OSSEC changes to /etc/pf.conf.

ext_if = "bge0"
int_if = "bge1"
localnet = $int_if:network
# needed by OSSEC
table <ossec_fwtable> persist come some rules...
# needed by OSSEC
block in quick from <ossec_fwtable> to any
block out quick from any to <ossec_fwtable>

To verify the OSSEC firewall table is currently blank I run this:

r200a:/root# pfctl -t ossec_fwtable -T show
No ALTQ support in kernel
ALTQ related functions disabled

The next step is to configure my OSSEC server to take an action when an offending IP address takes a sufficiently hostile step against a host running an OSSEC agent. This took a second thought, because I am configuring the OSSEC server to tell the OSSEC agent to take a blocking action using the script on the host running the OSSEC agent reporting the SSH brute forcing. That isn't the only way to configure this option, but it works for me. The block will persist for 600 seconds or 10 minutes.


<!-- Active Response Config -->

<!-- Firewall Drop response. Block the IP for
- 600 seconds on the firewall (iptables,
- ipfilter, etc).

I restart OSSEC on both client and server using /var/ossec/bin/ossec-control restart.

Now it's show time; I run the brute force from the attack box.

tws@debian40r0:~$ python root /etc/dictionaries-common/words sshBrute v1.0
[+] Loaded: 98569 words
[+] Server:
[+] User: root
[+] BruteForcing...
Trying: A
Trying: A's
Trying: AOL
Trying: AOL's
Trying: Aachen
Trying: Aachen's
Trying: Aaliyah
Trying: Aaliyah's

Checking /var/ossec/logs/alerts/alerts.log on the OSSEC server shows messages like this:

** Alert 1230260722.392334: - syslog,sshd,authentication_failed,
2008 Dec 25 22:05:22 (r200a)>/var/log/messages
Rule: 5716 (level 5) -> 'SSHD authentication failed.'
Src IP:
User: root
Dec 25 22:05:19 r200a sshd[49425]: error: PAM: authentication error
for root from

Then I see OSSEC's report of a level 10 event.

** Alert 1230260730.394490: mail - syslog,sshd,authentication_failures,
2008 Dec 25 22:05:30 (r200a)>/var/log/messages
Rule: 5720 (level 10) -> 'Multiple SSHD authentication failures.'
Src IP:
User: root
Dec 25 22:05:25 r200a sshd[49450]: error: PAM: authentication error
for root from
Dec 25 22:05:24 r200a sshd[49445]: error: PAM: authentication error
for root from

A look at the active-responses.log on the gateway shows a new rule adding that blocks the offending IP.

r200a:/root# tail -n 1 /var/ossec/logs/active-responses.log
Thu Dec 25 22:05:28 EST 2008 /var/ossec/active-response/bin/
add - 1230260730.395394 5720

Checking Pf we see the new rule.

r200a:/root# pfctl -t ossec_fwtable -T show
No ALTQ support in kernel
ALTQ related functions disabled

If I want to manually remove the block I can do this:

r200a:/root# /var/ossec/active-response/bin/
delete -
r200a:/root# pfctl -t ossec_fwtable -T show
No ALTQ support in kernel
ALTQ related functions disabled

That worked just as I hoped. Now I have a way to limit scanners who hammer at SSH on port 22. Yes, I could take a lot of other actions, but this is what I wanted to document.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Tuesday, December 23, 2008

Traffic for Revoked TLSv1 Certificate

I read the Slashdot post Perfect MITM Attacks With No-Check SSL Certs with some interest, mainly from a traffic perspective. Basically Eddy Nigg managed to obtain a certificate for a domain he should not have had access to via a reseller for a company called Comodo. You can check your Firefox certificate authorities list to see their presence in the screenshot below.

Eddy managed to get a certificate allowing him to masquerade as, because the party issuing the certificate did not properly validate his authorization to act on behalf of

I wondered what this might look like when I read this comment suggesting visiting a site using a Comodo-provided certificate. When I visited I saw this:

The question is, how did Firefox know to avoid this problem? I decided to sniff traffic while revisiting the site. Here's what happened.

The first session of interest is from client to Comodo-certificate-possessing Web server The second session of interest is from client to Online Certificate Status Protocol (OSCP) server

So, in frames 36-51 you see the setup of a TLSv1 session with the Web server. Frame 49 shows the certificate presented by the Web server.

In frames 52 and 53 we see a DNS query and response for In frames 54-58 and 62-67 we see our Web browser checking the status of the certificate we received from the Web server. Frame 64 shows the OCSP replying that the certificate has been revoked.

This leads to frame 68 where we see the TLSv1 connection fail due to the revoked certificate.

After that the connections close.

Taking a closer look at Firefox settings, we find this is what controls OCSP, and these are the defaults.

Taking a closer look at the cert we see it does specify an OCSP server, which shows how our browser knew to try to find it.

I am not sure how other certificates set this part of their certs, but it would be easy to find out.

This scenario just demonstrates the importance of trust, but also rapid detection and response when that trust is violated. Building visibility in through monitoring and validation mechanisms like OCSP (but perhaps to a third party rather than the company duped into issuing a cert!) are good lessons here.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Physical Security Lessons for Digital Security

The newest CSO magazine featured a great article by Bill Brenner on jewelry store security. It's online via PCWorld at How Tech Caught the Jewelry Thief. I'd like to cite several excerpts and relate them to digital security.

It used to be that after a robbery, the police would review a surveillance tape for clues into who broke in, at what time and what the bad guys looked like. Since the thieves would be long gone by the time the tape was reviewed, there would often be little the authorities could do about it.

That sounds like a traditional digital forensics scenario, with the problem that it can be difficult to apprehend criminals well after the crime occurs.

But thanks to 21st-Century technology, the crooks are being watched in real time and, as a result, getting caught a lot more often.

Notice the word "watched" -- this frames the problem as one of faster detection and response.

In this Q&A, Dennis Thomas, regional loss prevention manager and certified field trainer at Zale Corp., explains how the retailer's IT operation is playing an increasingly important role in the physical security effort...

CSO: Your organization seems to be fighting back in more of a real-time fashion, as opposed to surveillance camera recordings where you would see the burglary take place long after the fact.

Thomas: Keep in mind, in the old days a crime could occur in a store with the employees there and they wouldn't always notice what was happening. With remote technology our trained operators at the command center can observe a theft in progress and notify the police in real time with important time-sensitive details like description, method of operation and where the merchandise is on the person. The police in turn are a lot more successful in making an arrest than they were five years ago.

Two points: first, Zale Corp. uses a centralize and specialize method where experts provide a service to the entire company, remotely. Second, the result is removing a threat via police arrest.

The real benefit is the increase in time notification. Let's say the operator doesn't immediately see the theft as it's happening. They can still e-mail camera images to the police, which is still faster than trying to pull video off an old VCR tape.

This sounds like Network Security Monitoring, where prevention eventually fails and sometimes intruders are smarter than you. When you know you were victimized, however, you can review your forensic evidence quickly and efficiently.

CSO: Who are you using as a vendor to operate the command center?

Thomas: We own and operate our own command center.

CSO: So you built the whole thing in house.

Zale Corp. is big enough to staff their own centralized "security operations center (SOC)". Smaller players might want to outsource, but I see more large companies building their own.

Thomas: Exactly. We worked with a local vendor to develop the technology and devised everything right down to the terminology that the operators use to communicate with the stores.

CSO: Did your command center develop gradually and organically, or was it based off of one big plan from the outset?

Thomas: It was a gradual process that took years. There were three phases: developing the technology, implementing the technology and further enhancing the system once it was operational, working out the kinks. We had our challenges as we basically ventured into uncharted territory but the technology was proven and successfully implemented the vision into the business.

No one does this correctly from day one. Developing an effective security operation is a multi-year process.

CSO: How much has this cut down on the time it takes on average to either catch the thief or at least solve a crime?

Thomas: I'll give you two statistics: First: The corporation has achieved record shrink lows for the last seven consecutive years. Second: a significant reduction in shrink [lost merchandise/revenue] as a result of burglaries. You can directly attribute that to the technology we've put in place.

This is a crucial point: Zale Corp's security department has performed a cost-benefit analysis that demonstrates how their security operation is saving money. First they had to quanitfy loss, and now they are showing how their team has reduced that loss. Note that the security team isn't "making money;" they are preventing loss.

There has been a significant increase in the number of criminals apprehended because we can get three to five cruisers out there immediately, because the police know if Zales calls, we are seeing a burglary unfolding before our eyes. We are able to verify to them immediately that it's not a false alarm.

Zale Corp. is avoiding the problem facing many MSSPs. Many MSSPs just call the customer when one of a million Snort alerts appear on an analyst's console. The customer is left to do an investigation to validate the alert. Good MSSPs (including internal ones) use an alert as an indicator to start their own investigation, backed by the necessary actionable evidence to make a decision. Then they call the customer to inform them that a problem is happening, not to ask the customer "is anything wrong?" The customer learns to trust the MSSP, because when the MSSP does call it means something.

CSO: If you are a retailer just coming to the realization that you need to adopt a system like Zale's, what are the first items you should be thinking about?

Thomas: The first thing you need to do is determine where your risk is. Is it the employee? Does the general public have access to your merchandise? Where is your shrink occurring and where will those precious dollars get the most benefit? The second thing you should do is go out and look at what your competitors are doing technologically to ensure security. Then you are able to build your system to meet the specific needs of your organization.

Again, Zale Corp. demonstrates where to begin. You can determine risk by performing preliminary monitoring to observe actual problems before implementing countermeasures. Bruce Schneier calls this monitor first.

Great article Bill Brenner!

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Download Free Hakin9 Issue

I noticed that Hakin9 magazine is running a one-day special free download of issue 1/2008. If you'd like to check out this magazine, visit the link to download the magazine in .pdf format.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Monday, December 22, 2008

Justifying National Security Spending

Recently I posted Jeremiah Grossman on Justifying Security Spending. Yesterday I read Noah Schachtman's article Jets vs. Grunts in Pentagon Spending Showdown. I realized DoD (and really any other global military) has the same problem facing digital security practitioners: how do you justify security spending? DoD spending doesn't make the country richer. As I've said elsewhere, spending on security only makes security vendors richer. (See Security ROI Revisited for my reference to the broken window fallacy. By the way, if you are a politically-minded first-time blog visitor, you can forget about posting comments. This blog is for digital security; I'm not taking political sides here.)

One major difference between digital security justification and military justification is the latter's emphasis on threats, especially their capabilities and intentions. We are not worried if the United Kingdom builds a 5th generation fighter aircraft. We are worried if China, Russia, or Iran does. You don't see discussions of vulnerabilities, e.g., "we have to do something about the exposures and vulnerabilities in our domestic fuel storage facilities that allow 5th generation fighters to bomb them!" Instead the conversation focuses on designing, building, and deploying fighters that can deter or destroy enemy fighters. This is the case because a national military is in a position to take these actions, unlike the owners of the fuel storage facilities.

Also notice that owners of domestic fuel storage facilities are not buying their own fighter aircraft to defend their assets. Obviously, you might think. Well, not if you are a digital security practitioner. We're expected to protect all of our assets, against any range of threats, with little to no help from the governments we elect to "provide for the common defense." I mentioned this last year in US Needs Cyber NORAD.

Until this situation changes you can expect me to point out the absurdity of our situation. Maybe in 25 years we'll look back at this time as the "Wild Cyber West" that it is.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Sunday, December 21, 2008

Command Information Securing, Hacking and Defending IPv6

Last week I had the good fortune to attend Securing, Hacking and Defending IPv6, a class offered by Command Information in Herndon, VA. I've experimented with IPv6, as noted most recently in my May 2007 post Freenet6 on FreeBSD. I thought I knew a decent amount about IPv6, although I recognized a class like this would be helpful.

One word: wow. IPv6 is more complicated than I expected. I only began to realize this as the two Command Information instructors, Joe Klein and TJ Evans, explained what they know about the protocol and how it is used and abused. When IPv6 becomes even moderately deployed, intruders are going to have a field day. The network teams who have been hiding in the shadows of the Web app folks are going to have to step into the light and learn quickly. You can forget any hype about IPv6 bringing "security" when deployed, at least in the short-to-mid-term. The operational realty of designing, building, and running IPv6 networks properly is going to rock everyone's world.

The instructors were the best aspect of the class. They could answer any IPv6 question anyone asked. The overall content needs to be adjusted, but the instructors were very open to feedback. The 3-day class I attended was only the second session taught thus far, so I expect continuous improvement during the next few sessions.

I haven't seen training on this subject anywhere else, by anyone I consider authoritative on the subject. Joe and TJ are helping shape the nature and use of IPv6 in Federal and other locations, and you will learn a lot in this class.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Saturday, December 20, 2008

Traffic Talk 4 Posted

My fourth edition of Traffic Talk, titled Daemonlogger for Packet Capture and Redirection, has been posted. From the article:

Welcome to the 4th edition of Traffic Talk, a regular series for network solution providers and consultants who troubleshoot business networks.

In this article I'll demonstrate two novel features of Marty Roesch's Daemonlogger tool.

I compare Daemonlogger's ring buffer to Tcpdump's ring buffer, and then show how to use the Daemonlogger soft tap function.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Thursday, December 18, 2008

Colin Percival and Craig Balding on Amazon Cloud Security

If you're a security professional, it would be worth your time to read Craig Balding's post What’s New in the Amazon Cloud?: Security Vulnerability in Amazon EC2 and SimpleDB Fixed (7.5 Months After Notification), a summary and analysis of Colin Percival's post AWS signature version 1 is insecure. These posts demonstrate the changing nature of our jobs. We will become increasingly reliant on others hosting, processing, and ostensibly "protecting" our data, but our ability to measure the effectiveness of these services is likely to erode over time. In this case it sounds like worked slowly but very effectively with Colin, and their example should be followed.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

You Get What You Inspect

There are some great security catch phrases, like "Trust but verify." I found my new favorite in Fresher Cookers, an Economist article about designing stoves for the developing world.

“You don’t get what you expect—you get what you inspect,” says Dr [Kirk] Smith, [an expert on the impact of stove air-pollution on health.]

I think that maxim holds very true for anyone who inspects their enterprise to see how it is really used and abused. That saying holds true at every level -- network, platform, operating system, or application. All of these components are so complicated and ever-changing that you are likely to be surprised every time you stop to look at what's happening.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Wednesday, December 17, 2008

Traffic Talk 3 Posted

My third edition of Traffic Talk, titled Network Security Monitoring: Knowing Your Network has been posted. From the article:

Recently I read an interview with network security pioneer Marcus Ranum, who was asked the following question about network security monitoring: "In your opinion, what is the current weakest link in the network security chain that will need to be dealt with next year and beyond?"

Read my article to see what Marcus wrote and how I responded.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Saturday, December 13, 2008

Indian Navy Demonstrates that Offense Stops Pirates

Clearly the Indian Navy doesn't understand vulnerability-centric security. If they did, they wouldn't have captured 23 pirates "who tried to take over a merchant vessel in the Gulf of Aden, between the Horn of Africa and the Arabian Peninsula." They also wouldn't have "exchanged fire with a pirate "mother vessel" off the hijacking-plagued Horn of Africa, leaving the ship ablaze." Someone needs to teach these Indian sailors that the best way to stop pirates is to "build security in" when merchants construct ships!

I guess the Indians read my Offense Kills Pirates post. Maybe they decided to Take the Fight to the Enemy. Whatever the reason, good for them. Instead of commercial shippers being the only party suffering higher costs in this piracy environment (due to losses, higher insurance, increased salaries, etc.), now it's more expensive for pirates too.

Yo ho ho, pirates. We're coming for you soon. When will we take the same attitude to cyber pirates?*

*Note: I don't mean those the RIAA/MPAA calls "pirates."

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Thursday, December 11, 2008

Supporting FreeBSD

I've been using FreeBSD in production environments since early 2000, and I continue to rely on it at home and at work. Even though I could download the operating system for free, I still subscribe through to support the project.

However, I seem to ask for public support for FreeBSD every two years, with my call for Helping the FreeBSD Foundation Retain Non-profit Status in 2004 and Supporting FreeBSD Security Coding in 2006. Today I read FreeBSD Foundation End-of-Year Fund Raising Drive Update.

The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and building the FreeBSD Project and community worldwide. You can see all the good work they are doing on their Web site.

The Foundation set a $300,000 goal for 2008 fundraising, and it's 2/3 of the way there. I just donated $100. Will anyone else donate? Please let me know here. I thank you and the project thanks you.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Jeremiah Grossman on Justifying Security Spending

I liked the way Jeremiah Grossman listed five ways to justify security spending:

1) Risk Mitigation
"If we spend $X on Y, we’ll reduce of risk of loss of $A by B%."

2) Due Diligence
"We must spend $X on Y because it’s an industry best-practice."

3) Incident Response
"We must spend $X on Y so that Z never happens again."

4) Regulatory Compliance
"We must spend $X on Y because PCI-DSS says so."

5) Competitive Advantage
"We must spend $X on Y to make the customer happy."

Jeremiah expands on each in his blog, which makes for good reading.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Monday, December 08, 2008

3rd Issue of BSD Magazine

I recently received a copy of the 3rd issue of BSD Magazine. This issue turns to NetBSD, the BSD project with the most stylish BSD Web site I've seen. The next issue will be devoted to PC-BSD, which I have never used (but should probably try).

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Review of Nmap Network Scanning Posted

Earlier this year I posted Review of Nmap Network Scanning. Now Fyodor's book is available through Therefore, I expanded my earlier story into a five star review:

Earlier this year Fyodor sent me a pre-publication review copy of his new self-published book, Nmap Network Scanning (NNS). I had heard of Fyodor's book when I wrote my 3 star review of Nmap in the Enterprise in June, but I wasn't consciously considering what could be in Fyodor's version compared to the Syngress title. Although the copy I read was labelled "Pre-Release Beta Version," I was very impressed by this book. Now that I have the final copy (available from Amazon) in my hands, I am really pleased with the product. In short, if you are looking for *the* book on Nmap, the search is over: NNS is a winner.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Review of Googling Security Posted just posted my five star review of Greg Conti's Googling Security. From the review:

There's no question that Greg Conti writes excellent books. Last year's Security Data Visualization book earned 5 stars, and I put Googling Security in the same league. Conti takes a thorough and methodical look at the privacy consequences of Google's services, incorporating technical realities and thoughtful analysis. My only question is whether this book will matter to the intended audience.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Sunday, December 07, 2008

Review of Software Security Engineering Posted just posted my three star review of Software Security Engineering: A Guide for Project Managers. From the review:

The Addison-Wesley Software Security Series is generally a great collection, with titles like Software Security: Building Security In (my rating: 5 stars), Rootkits: Subverting the Windows Kernel (my rating: 4 stars), and Exploiting Software: How to Break Code (my rating: 4 stars). I particularly liked the first of those three (SS:BSI), which I reviewed last year. I felt Gary McGraw wrote "a powerful book with deep truths for secure development." Software Security Engineering (SSE), by a collection of authors, pales in comparison to SS:BSI. You can skip SSE and stick with SS:BSI.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Thursday, December 04, 2008

Bejtlich Cited in Economist

I've been a subscriber of the Economist magazine since 1997. Although I have not been working to achieve this goal, I am happy to report that a personal ambition of mine has been reached today: I was cited in the 6 Dec 08 edition, in an article titled Cyberwarfare: Marching off to cyberwar.

One way for governments to do this [to become resilient to cyber attack], says Richard Bejtlich, a former digital-security officer with the United States Air Force who now works at GE, an American conglomerate, might be to make greater use of open-source software, the underlying source code of which is available to anyone to inspect and improve. To those outside the field of computer security, and particularly to government types, the idea that such software can be more secure than code that is kept under lock and key can be difficult to accept. But from web-browsers to operating systems to encryption algorithms, the more people can scrutinise a piece of code, the more likely it is that its weak spots will be found and fixed. It may be that open-source defence is the best preparation for open-source attack.

Thank you to Evgeny Morozov for including my comment and to the Economist editors for not cutting it.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

BPF for IP or VLAN Traffic

Four years ago I did a second post on Understanding Tcpdump's -d Option, showing how you can using the -d option to understand how Berkeley Packet Filter syntax works.

Recently my colleagues and I encountered a problem where we were monitoring traffic on a tap, but the traffic contained traffic with and without 802.1q VLAN tags. We wanted to create a BPF that would catch traffic whether or not it had VLAN tags. It turns out there is a difference between these two BPFs:

ip or vlan

is not the same as

vlan or ip

The first accomplishes our goal, but the second does not.

To understand why, I used Tcpdump's -d option.

$ tcpdump -d -n -r sample.pcap ip or vlan
reading from file sample.pcap, link-type EN10MB (Ethernet)
(000) ldh [12]
(001) jeq #0x800 jt 3 jf 2
(002) jeq #0x8100 jt 3 jf 4
(003) ret #65535
(004) ret #0

That looks right. Load the half word at offset 12. If it's the IP Ethertype, you get the whole packet. If it's not IP, go to the next instruction. If it's a 802.1Q VLAN tag, again you get the whole packet. Otherwise, return nothing.

This is the other option.

$ tcpdump -d -n -r sample.pcap vlan or ip
reading from file sample.pcap, link-type EN10MB (Ethernet)
(000) ldh [12]
(001) jeq #0x8100 jt 4 jf 2
(002) ldh [16]
(003) jeq #0x800 jt 4 jf 5
(004) ret #65535
(005) ret #0

That doesn't work. Load the half word at offset 12. If it's a 802.1Q VLAN tag, you get the whole packet. If it's not a 802.1Q VLAN tag, load the half word at offset 16. If that half word is an IP Ethertype (which it won't be), you get the whole packet. Otherwise, return nothing.

For an example of how you would combine a host and port filter with this syntax, see the following:

tcpdump -n -r ip.pcap \(ip and host and port 80\) or \(vlan and host and port 80\)

You might see this new option appear in Sguil CVS soon.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Wednesday, December 03, 2008

Letters You Will Need to Know: 201 CMR 17.00

Props to Ed at SecurityCurve for informing me of 201 CMR 17.00: Standards for The Protection of Personal Information of Residents of the Commonwealth, a new Massachusetts law. Section 17.03 sets the basic tone;

Every person that owns, licenses, stores or maintains personal information about a resident of the Commonwealth shall develop, implement, maintain and monitor a comprehensive, written information security program applicable to any records containing such personal information.

Unless you're prepared to figure out how to separate PII on Massachusetts residents from non-MA residents, this law now applies to all PII in your organization.

Jack Daniel has written several great posts on what this new law means. References for Mass 201 CMR 17.00 is really helpful. You can also access a video of a presentation he just made to the Boston chapter of the National Information Security Group. The slides don't render in Firefox but I was able to download the .wmv video and I'm viewing it now.

If you don't want to download the video (large) you can access an audio recording.

Bill Brenner wrote a good article titled Why Mass. 201 CMR 17 Deadline Was Extended, explaining why the compliance deadline moved from 1 Jan 09 to 1 May 09.

Cynthia Larose and Elissa Flynn-Poppey wrote Privacy Compliance 101: Why Massachusetts Data Security Standards DO Affect You for CIO magazine. They mention potential financial penalties:

What Happens If You DON'T Comply: Penalties

It is crucial for businesses to understand and comply with the newly enacted data breach legislation to avoid potentially severe monetary penalties. Massachusetts, unlike the majority of states, provides for civil penalties in cases of non-compliance with its data breach notification statute, Massachusetts General Law 93H [the law which created the guidelines of 201 CMR 17.00]. In particular, a civil penalty of $5,000 may be awarded for each violation of 93H. In addition, under the portion of 93H concerning data disposal, businesses can be subject to a fine of up to $50,000 for each instance of improper disposal.
(emphasis added)

I decided to see how the law might affect detection and response. Looking for references to monitoring or response in the law found the following:

[E]very comprehensive information security program shall include, but shall not be limited to...

(iii) means for detecting and preventing security system failures...

(j) Regular monitoring to ensure that the comprehensive information security program is operating in a manner reasonably calculated to prevent unauthorized access to or unauthorized use of personal information; and upgrading information safeguards as necessary to limit risks...

(l) Documenting responsive actions taken in connection with any incident involving a breach of security, and mandatory post-incident review of events and actions taken, if any, to make changes in business practices relating to protection of personal information...

Every person that owns, licenses, stores or maintains personal information about a resident of the Commonwealth and electronically stores or transmits such information shall include in its written, comprehensive information security program the establishment and maintenance of a security system covering its computers, including any wireless system, that, at a minimum, shall have the following elements...

(4) Reasonable monitoring of systems, for unauthorized use of or access to personal information
(emphasis added)

I think this law is going to have a real impact. I'm not sure when; companies aren't going to be ready by 1 May 09.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Sunday, November 30, 2008

Craig Balding Podcast on Cloud Security

I noticed Craig Balding's post Podcast: Cloud Computing, Software Development, Testing and Security, so I just listened to all three segments. Readers of this blog may choose to concentrate on the third segment, Cloud computing's effect on application security. Craig is a thought leader on cloud security so I enjoy hearing his ideas.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Tuesday, November 25, 2008

Splunk on FreeBSD 7.0

Although there is not a version of Splunk compiled natively for FreeBSD 7.0, I was told to try using Splunk 3.4.1 on FreeBSD 7.0 via FreeBSD's compat6x libraries.

I did the following:

freebsd70:/usr/local/src# pkg_add -v splunk-3.4.1-45588-freebsd-6.1-intel.tgz
Requested space: 106458852 bytes, free space: 1565927424 bytes in
Running pre-install for splunk-3.4.1-45588-freebsd-6.1-intel..
extract: Package name is splunk-3.4.1-45588-freebsd-6.1-intel
extract: CWD to /opt
extract: /opt/splunk/README.txt
extract: /opt/splunk/bin/btool
extract: /opt/splunk/bin/bunzip2
extract: /opt/splunk/splunk-3.4.1-45588-FreeBSD-i386-manifest
extract: CWD to .
Running post-install for splunk-3.4.1-45588-freebsd-6.1-intel..
Splunk has been installed in:

To start Splunk, run the command:
/opt/splunk/bin/splunk start

To use the Splunk Web interface, point your browser at:

Complete documentation is at
Attempting to record package into /var/db/pkg/splunk-3.4.1-45588-freebsd-6.1-intel..
Package splunk-3.4.1-45588-freebsd-6.1-intel registered in

If you try to start Splunk at this point you'll get an error like the following:

freebsd70:/usr/local/src# /opt/splunk/bin/splunk start
/libexec/ Shared object "" not found, required by "splunk"

To fix the problem I installed compat6:

freebsd70:/usr/local/src# pkg_add -vr
scheme: [ftp]
user: []
password: []
host: []
port: [0]
document: [/pub/FreeBSD/ports/i386/packages-7.0-release/misc/
looking up
connecting to
<<< 220 NcFTPd Server (licensed copy) ready.
>>> USER anonymous
<<< 331 Guest login ok, send your complete e-mail address as password.
>>> PASS analyst@freebsd70.localdomain
<<< 230-You are user #147 of 800 simultaneous users allowed.
<<< 230-
<<< 230 Logged in anonymously.
>>> PWD
<<< 257 "/" is cwd.
>>> CWD pub/FreeBSD/ports/i386/packages-7.0-release/misc
<<< 250 "/pub/FreeBSD/ports/i386/packages-7.0-release/misc" is new cwd.
>>> MODE S
<<< 200 Mode okay.
>>> TYPE I
<<< 200 Type okay.
setting passive mode
>>> PASV
<<< 227 Entering Passive Mode (62,243,72,50,214,227)
opening data connection
initiating transfer
>>> RETR compat6x-i386-6.3.602114.200711.tbz
<<< 150 Data connection accepted from; transfer starting for compat6x-
i386-6.3.602114.200711.tbz (3164256 bytes).
i386-6.3.602114.200711.tbz...x +CONTENTS
extract: CWD to /usr/local
extract: /usr/local/libdata/ldconfig/compat6x
extract: CWD to .
Running mtree for compat6x-i386-6.3.602114.200711..
mtree -U -f +MTREE_DIRS -d -e -p /usr/local >/dev/null
Attempting to record package into /var/db/pkg/compat6x-i386-6.3.602114.200711..
Package compat6x-i386-6.3.602114.200711 registered in

* *
* Do not forget to add COMPAT_FREEBSD6 into *
* your kernel configuration (enabled by default). *
* *
* To configure and recompile your kernel see: *
* *
* *

Then I could start Splunk:

freebsd70:/usr/local/src# /opt/splunk/bin/splunk start
Splunk Free Software License Agreement
Do you agree with this license? [y/n]: y
Copying '/opt/splunk/etc/myinstall/splunkd.xml.cfg-default'
to '/opt/splunk/etc/myinstall/splunkd.xml'.
Copying '/opt/splunk/etc/openldap/ldap.conf.default'
to '/opt/splunk/etc/openldap/ldap.conf'.
Copying '/opt/splunk/etc/modules/distributedSearch/config.xml.default'
to '/opt/splunk/etc/modules/distributedSearch/config.xml'.
/opt/splunk/etc/auth/audit/private.pem generated.
/opt/splunk/etc/auth/audit/public.pem generated.

/opt/splunk/etc/auth/audit/private.pem generated.
/opt/splunk/etc/auth/audit/public.pem generated.

This appears to be your first time running this version of Splunk.
Validating databases...
Creating /opt/splunk/var/lib/splunk/audit/thaweddb
Creating /opt/splunk/var/lib/splunk/blockSignature/thaweddb
Creating /opt/splunk/var/lib/splunk/_internaldb/thaweddb
Creating /opt/splunk/var/lib/splunk/fishbucket/thaweddb
Creating /opt/splunk/var/lib/splunk/historydb/thaweddb
Creating /opt/splunk/var/lib/splunk/defaultdb/thaweddb
Creating /opt/splunk/var/lib/splunk/sampledata/thaweddb
Creating /opt/splunk/var/lib/splunk/splunkloggerdb/thaweddb
Creating /opt/splunk/var/lib/splunk/summarydb/thaweddb
Validated databases: _audit, _blocksignature, _internal, _thefishbucket,
history, main, sampledata, splunklogger, summary

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Verifying configuration. This may take a while...
Finished verifying configuration.
Checking index directory...
Verifying databases...
Verified databases: _audit, _blocksignature, _internal, _thefishbucket,
history, main, sampledata, splunklogger, summary

Checking index files
All index checks passed.
All preliminary checks passed.
Starting splunkd...
Starting splunkweb.../opt/splunk/share/splunk/certs does not exist. Will create
Generating certs for splunkweb server
Generating a 1024 bit RSA private key
writing new private key to 'privkeySecure.pem'
Signature ok
Getting CA Private Key
writing RSA key

Splunk Server started.

The Splunk web interface is at http://freebsd70.localdomain:8000

I was then able to connect to the Splunk Web interface, add a directory (/var/log) to monitor, and access results.

Documentation for FreeBSD installation is also available. Thanks Splunk!

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Monday, November 24, 2008

Defining the Win

In March I posted Ten Themes From Recent Conferences, which included the following:

Permanent compromise is the norm, so accept it. I used to think digital defense was a cycle involving resist -> detect -> respond -> recover. Between recover and the next attack there would be a period where the enterprise could be considered "clean." I've learned now that all enterprises remain "dirty" to some degree, unless massive and cost-prohibitive resources are directed at the problem.

We can not stop intruders, only raise their costs. Enterprises stay dirty because we can not stop intruders, but we can make their lives more difficult. I've heard of some organizations trying to raise the $ per MB that the adversary must spend in order to exfiltrate/degrade/deny information.‏
(emphasis added)

Since then I've grappled with this idea of how to define the win. If you used to define the win as detecting and ejecting all intruders from your enterprise, you are going to be perpetually disappointed (unless your enterprise is sufficiently small). Are there are alternative ways to define the win if you have to accept permanent compromise as the norm? The following are a few ideas, credited where applicable.

The first two come from my post Intellectual Property: Develop or Steal, but I repost them here for easy reference.

  1. Information assurance (IA) is winning, in a broad sense, when the cost of stealing intellectual property via any means is more expensive than developing that intellectual property independently. Nice idea, but probably too difficult to measure.

  2. IA is winning, in a narrow sense, when the cost of stealing intellectual property via digital means is more expensive than stealing that data via nontechnical means (such as human agents placed inside the organization). Still difficult to measure, but might be estimated using red teaming/adversary simulation/penetration testing.

  3. IA is winning when detection operations can see the adversary's actions. This relates to Bruce Schneier's classic advice to Monitor First. The more mature answer is next.

  4. IA is winning when incident responders can anticipate the adversary's next target. I credit Kevin Mandia with this idea. I like it because it shows that complex enterprises will always have vulnerabilities and will always be targeted, but a sufficiently mature detection and response operation will at least be able to guess the intruder's next move. You can even test this by keeping a track record.

  5. IA is winning when the time to detect and remediate has been reduced to B. Insert your own value there. You can track your progress from time A to time B.

  6. IA is winning when your enterprise security integrity assessments show less than D percent of your assets are compromised. You can track progress from C percent to D percent over time. This leads to the more mature version which follows.

  7. IA is winning when your enterprise intrusion debt is reduced to F. You can measure intrusion debt as you like and take steps to reduce it from E to F.

Does anyone else have ideas on how to define the win?

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Live Incident Map

I think this is fascinating: a map depicting naval piracy.

One of the most interesting aspects of this map is that it concerns commercial entities (i.e. ships carrying cargo) and anyone can quickly learn the fate of each vessel. It's a giant incident map for 2008. Previous years (2007, 2006) are also available.

The closest equivalent for digital security is probably the narrative of the Breach Blog and similar sites.

Only when we can openly talk about this problem and share lessons learned can we improve. We still need a National Digital Security Board.

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.