Thursday, November 14, 2013

Linux Covert Channel Explains Why NSM Matters

I just read a post by Symantec titled Linux Back Door Uses Covert Communication Protocol. It describes a new covert channel on Linux systems. A relevant excerpt follows:

[T]he attackers devised their own stealthy Linux back door to camouflage itself within the Secure Shell (SSH) and other server processes. This back door allowed an attacker to perform the usual functionality—such as executing remote commands—however, the back door did not open a network socket or attempt to connect to a command-and-control server (C&C).

Rather, the back door code was injected into the SSH process to monitor network traffic and look for the following sequence of characters: colon, exclamation mark, semi-colon, period (“:!;.”).

After seeing this pattern, the back door would parse the rest of the traffic and then extract commands which had been encrypted with Blowfish and Base64 encoded.


Figure. Example of injected command

The attacker could then make normal connection requests through SSH or other protocols and simply embed this secret sequence within some otherwise legitimate traffic to avoid detection. The commands would be executed and the result sent back to the attacker. Symantec then recommends:

To identify the presence of this back door on your network, look for traffic that contains the “:!;.” string (excluding quotes).

How are you supposed to do that? If you implement NSM operations, perhaps using Security Onion, you could do something like the following:

$ for i in `ls`; do echo $i && ngrep -q -I $i \:\!\;\.; done
If you run that command in the directory where you store your full content data traffic (i.e., raw pcap files), Ngrep will search each packet for the string of interest.

You might also want to use the -O flag to dump matching packets to a pcap-formatted file.

Results look like the following:

input: snort.log.1384440248
match: :!;.

T -> [AP]
  ?...T.<...x&~...F.:.d..`.Vg/h.........rN.....a.......yq...q.f..N_m.D.F@C>v.]&...l{....m ..4.......`.f..Y.u...WQq.(...9........qq.
 --- SNIP ---
This appears to be part of a SSL-encrypted session. Running through some of my lab traffic, the :!;. string appears several times in encrypted traffic involving Windows computers.

This method doesn't provide prima facie evidence of compromise. For example, you might have to reference personal knowledge or an asset inventory to determine that the system in question is a Windows computer, not a Linux system. You are more likely to see a stream of commands involving this string and follow that trail to identify compromise. You could also cross-reference using the memory-based indicators of compromise Symantec includes in the post.

Regardless, the fact that you are already collecting as much full content data as your technical, legal, and political processes allow means you have a chance to perform retrospective security analysis, thanks to the collection of pcap within an NSM operation.

Without taking an NSM approach, you would have to set up new signatures or data collection from this point forward and hope the adversary doesn't change his covert channel, now that Symantec has publicized it.

The point of this approach is to help you determine the scope of the compromise as efficiently and rapidly as possible, so that you can conduct effective remediation once the scope is understood.

If you want to know how to set up a NSM operation, check out my Network Security Monitoring 101 class at Black Hat Seattle next month. The class outline is here. You may also like my latest book on NSM. Use code "NSM101" to save 30% off, and get the digital edition free with a print copy.

Thank you to Symantec for sharing their IOCs for this covert channel.

Sunday, October 27, 2013

Mozilla Lightbeam Add-On Shows Risk of Third Party Sites

The slide above shows an experiment I just conducted using the Lightbeam addon with NoScript.

  • The image at left shows the results of visiting,,, and while NoScript is denying JavaScript and similar content.
  • The image at left shows the results of visiting,,, and while NoScript is disabled to allow JavaScript and similar content.

The Lightbeam add-on renders the primary and third party Web sites visited in each case.

  • When NoScript is denying Javascript and similar content, only 9 third party sites are called in order to render the 4 primary Web sites.
  • When NoScript is disabled to allow JavaScript and similar content, 66 third party Web sites are called.

Only a few minutes after taking the original images, the count for the second case increased from 66 to 90.

Why is this a problem? From a security perspective:

  • The more third party Web sites required to render a primary site, the more opportunities intruders have to introduce malicious content.
  • The more third party Web sites required to render a primary site, the more complex the primary site becomes, and the less likely it will perform as intended. We're seeing this at work (or not at work, perhaps) with

From a privacy perspective:

  • The Lightbeam rendering shows relationships among the 4 primary Web sites and the third party sites. In the first image, and share third party sites and therefore could potentially access data about users from each other. and are separate from each other.
  • In the second image, all of the primary Web sites are interconnected. This means it is possible for them to share data about user activities. This is how Web sites track you on the Internet.
I recommend installing Lightbeam with NoScript to try for yourself.

Update: Up to 95 sites now...

Update 2: Up to 105 sites now...

Thursday, September 12, 2013

Bejtlich Teaching at Black Hat West Coast Trainings

I'm pleased to announce that I will be teaching at Black Hat West Coast Trainings 9-10 December 2013 in Seattle, Washington. This is a brand new class, only offered thus far in Las Vegas in July 2013. I posted Feedback from Network Security Monitoring 101 Classes last month as a sample of the student feedback I received.

Several students asked for a more complete class outline. So, in addition to the outline posted currently by Black Hat, I present the following that shows what sort of material I cover in my new class.

Please note that discounted registration ends 11:59 pm EDT October 24th. You can register here. I have only one session available in Seattle and fewer seats than in Las Vegas, so please plan accordingly. Thank you.


Is your network safe from intruders? Do you know how to find out? Do you know what to do when you learn the truth? If you are a beginner, and need answers to these questions, Network Security Monitoring 101 (NSM101) is the newest Black Hat course for you. This vendor-neutral, open source software-friendly, reality-driven two-day event will teach students the investigative mindset not found in classes that focus solely on tools. NSM101 is hands-on, lab-centric, and grounded in the latest strategies and tactics that work against adversaries like organized criminals, opportunistic intruders, and advanced persistent threats. Best of all, this class is designed *for beginners*: all you need is a desire to learn and a laptop ready to run a virtual machine. Instructor Richard Bejtlich has taught over 1,000 Black Hat students since 2002, and this brand new, 101-level course will guide you into the world of Network Security Monitoring.


Day One

·         Introduction
·         Enterprise Security Cycle
·         State of South Carolina case study
·         Difference between NSM and Continuous Monitoring
·         Blocking, filtering, and denying mechanisms
·         Why does NSM work?
·         When NSM won’t work
·         Is NSM legal?
·         How does one protect privacy during NSM operations?
·         NSM data types
·         Where can I buy NSM?

·         Break

·         SPAN ports and taps
·         Making visibility decisions
·         Traffic flow
·         Lab 1: Visibility in ten sample networks
·         Security Onion introduction
·         Stand-alone vs server plus sensors
·         Core Security Onion tools
·         Lab 2: Security Onion installation

·         Lunch

·         Guided review of Capinfos, Tcpdump, Tshark, and Argus
·         Lab 3: Using Capinfos, Tcpdump, Tshark, and Argus

·         Break

·         Guided review of Wireshark, Bro, and Snort
·         Lab 4: Using Wireshark, Bro, and Snort
·         Using Tcpreplay with NSM consoles
·         Guided review of process management, key directories, and disk usage
·         Lab 5: Process management, key directories, and disk usage

Day Two

·         Computer incident detection and response process
·         Intrusion Kill Chain
·         Incident categories
·         CIRT roles
·         Communication
·         Containment techniques
·         Waves and campaigns
·         Remediation
·         Server-side attack pattern
·         Client-side attack pattern

·         Break

·         Guided review of Sguil
·         Lab 6: Using Sguil
·         Guided review of ELSA
·         Lab 7: Using ELSA

·         Lunch

·         Lab 8. Intrusion Part 1 Forensic Analysis
·         Lab 9. Intrusion Part 1 Console Analysis

·         Break

·         Lab 10. Intrusion Part 2 Forensic Analysis
·         Lab 11. Intrusion Part 2 Console Analysis


Students must be comfortable using command line tools in a non-Windows environment such as Linux or FreeBSD. Basic familiarity with TCP/IP networking and packet analysis is a plus.


NSM101 is a LAB-DRIVEN course. Students MUST bring a laptop with at least 8 GB RAM and at least 20 GB free on the hard drive. The laptop MUST be able to run a virtualization product that can CREATE VMs from an .iso, such as VMware Workstation (minimum version 8, 9 is preferred); VMware Player (minimum version 5 -- older versions do not support VM creation); VMware Fusion (minimum version 5, for Mac); or Oracle VM VirtualBox (minimum version 4.2). A laptop with access to an internal or external DVD drive is preferred, but not mandatory.

Students SHOULD test the open source Security Onion ( NSM distro prior to class. The students should try booting the latest version of the 12.04 64 bit Security Onion distribution into live mode. Students MUST ensure their laptops can run a 64 bit virtual machine. For help with this requirement, see the VMware knowledgebase article “Ensuring Virtualization Technology is enabled on your VMware host (1003944)” ( Students MUST have the BIOS password for their laptop in the event that they need to enable virtualization support in class. Students MUST also have administrator-level access to their laptop to install software, in the event they need to reconfigure their laptop in class.


Students will receive a paper class handbook with printed slides, a lab workbook, and the teacher’s guide for the lab questions. Students will also receive a DVD with a recent version of the Security Onion NSM distribution.


Richard Bejtlich is Chief Security Officer at MANDIANT. He was previously Director of Incident Response for General Electric, where he built and led the 40-member GE Computer Incident Response Team (GE-CIRT). Prior to GE, he operated TaoSecurity LLC as an independent consultant, protected national security interests for ManTech Corporation's Computer Forensics and Intrusion Analysis division, investigated intrusions as part of Foundstone's incident response team, and monitored client networks for Ball Corporation.  Richard began his digital security career as a military intelligence officer in 1997 at the Air  Force Computer Emergency Response Team (AFCERT), Air Force Information Warfare Center (AFIWC), and Air Intelligence Agency (AIA).  Richard is a graduate of Harvard University and the United States Air Force Academy.  He wrote "The Tao of Network Security Monitoring" and "Extrusion Detection," and co-authored "Real Digital Forensics."  His latest book is "The Practice of Network Security Monitoring" ( He also writes for his blog ( and Twitter (@taosecurity), and teaches for Black Hat.

Tuesday, August 13, 2013

Feedback from Network Security Monitoring 101 Classes

At Black Hat in Las Vegas I taught two Network Security Monitoring 101 (NSM101) classes. This is a new class that I developed this year, after retiring the third edition of my TCP/IP Weapons School. Once again I was glad to have Steve Andres from Special Ops Security there to help students with questions and lab issues.

I wanted to share some feedback from the classes, in case any of you are considering attending an upcoming edition. Currently I'm scheduled to teach at Black Hat Seattle on 9-10 December. I plan to continue offering my class through Black Hat as they expand their training location offerings.

Student feedback from NSM101 included:
  • Great tools, fun labs, very prepared -- a lot of experience from interesting real world scenarios.
  • This course was everything I hoped for and more. Very impressive considering the course is new.
  • One of the best training classes I have ever taken.
  • Richard hosted an exemplary class.
  • I thought the class was excellent, and the content was relevant and informative.
  • The instructor was there when help was needed. I can easily take what I learned here and apply it to my work.
  • Excellent instructor and class. It is nice to learn from true pros who are humble and willing to help.
  • Richard is an excellent speaker. His use of real world examples added value to each lab. The material was easy to understand and very well thought out.
  • The stories behind the scenes and the practical notes (i.e., how to create a team) really helped.
  • Great balance of hands-on and theory.
  • Easy to follow and inspiring, even for an NSM beginner like me.
  • Great companion to the new NSM book.
  • This class was fantastic. I wish I could send my whole department.
  • I look forward to using your book and teaching some of your techniques to my students.

In the "constructive criticism" category, several students recommended that I modify the class description to better suit the class structure. For example, some students didn't realize they would be using Security Onion in the class. A few students told me they would have sent more people from their team if they had a better sense of what the class was going to include. I will fix that for the Seattle edition and future events.

Overall I very much enjoyed teaching the new class. I will make a few tweaks to fix typos but otherwise I am ready to teach again in December. Once the registration form is active I will post it via Twitter.

If you have any questions, please post them as comments here or via Twitter to @taosecurity. Thank you.

Tuesday, June 18, 2013

President Obama Is Right On US-China Hacking

I strongly recommend watching the excerpt on the Charlie Rose show titled Obama: Blunt Conversation With China on Hacking. I reproduced the relevant part of the transcript below and added emphasis to key points.

CHARLIE ROSE: Speaking of pushing back, what happened when you pushed back on the question of hacking and serious allegations that come from this country that believe that the Chinese are making serious strides and hacking not only private sector but public sector?

BARACK OBAMA: We had a very blunt conversation about cyber security.

CHARLIE ROSE: Do they acknowledge it?

BARACK OBAMA: You know, when you’re having a conversation like this I don’t think you ever expect a Chinese leader to say "You know what? You’re right. You caught us red-handed."

CHARLIE ROSE: You got me. Yes.

BARACK OBAMA: We’re just stealing all your stuff and every day we try to figure out how we can get into Apple --

CHARLIE ROSE: But do they now say "Look? See you’re doing the same thing. We’ve been reading about what NSA is doing and you’re doing the same thing that we’re doing and there are some allegations of that. And the man who is now unleashing these secrets who’s telling everybody is in Hong Kong.



CHARLIE ROSE: And may be talking to the Chinese.

BARACK OBAMA: Well, let’s separate out the NSA issue which I’m sure you’re going to want to talk to and the whole full balance of privacy and security with -- with the specific issue of cyber security and our concerns --

CHARLIE ROSE: And cyber warfare and cyber espionage.

BARACK OBAMA: Right. Every country in the world, large and small, engages in intelligence gathering and that is an occasional source of tension but is generally practiced within bounds. There is a big difference between China wanting to figure out how can they find out what my talking points are when I’m meeting with the Japanese which is standard fare and we’ve tried to prevent them from --



BARACK OBAMA: -- penetrating that and they try to get that information. There’s a big difference between that and a hacker directly connected with the Chinese government or the Chinese military breaking into Apple’s software systems to see if they can obtain the designs for the latest Apple product. That’s theft. And we can’t tolerate that.

And so we’ve had very blunt conversations about this. They understand, I think, that this can adversely affect the fundamentals of the U.S./China relationship. We don’t consider this a side note in our conversations. We think this is central in part because our economic relationship is going to continue to be premised on the fact that the United States is the world’s innovator. We have the greatest R&D. We have the greatest entrepreneurial culture.

Our value added is at the top of the value chain and if countries like China are stealing that that affects our long-term prosperity in a serious way.

This is an amazing development for someone aware of the history of this issue. President Obama is exactly right concerning the differences between espionage, practiced by all nations since the beginning of time, and massive industrial theft by China against the developed world, which the United States, at least, will not tolerate. I am so pleased that this issue is at the top of the agenda between the US and China and that the President and his team, as well as Congress, are taking it so seriously.

Thursday, June 13, 2013

Pre-Order The Practice of Network Security Monitoring Before Price Hike

When my publisher and I planned and priced my new book The Practice of Network Security Monitoring, we assumed the book would be about 250 pages. As we conclude the copyediting process and put print in layout format, it's clear the book will be well over 300. The current estimate is 328, but I think it could approach 350 pages.

Because of the much larger page count, the publisher and I agreed to reprice the book. The price will rise from the current list of $39.95 for paperback and $31.95 for ebook to $49.95 for paperback and $39.95 for ebook.

However, those prices will not go into effect until next Friday, June 21st. That means if you preorder at the Web site before next Friday, you will get the current lower prices. Furthermore, use preorder code NSM101 to save 30% off list. If you use NSM101 as your discount code it shows No Starch that you got word of this from me.

Those of you who already preordered have already taken advantage of this deal. Thanks for your orders!

We're still on track for publication by July 22, in time for books on hand at my new Network Security Monitoring 101 class in Las Vegas. Seats for the two editions of the class (weekend and weekday) continue to fill.

If you live in Europe or the Middle East or Africa, you may want to attend my new class in Istanbul in September. I hope the protestors and government can manage their differences in time for this great new Black Hat event!

Monday, April 29, 2013

Practice of Network Security Monitoring Table of Contents

Since many of you have asked, I wanted to provide an updated Table of Contents for my upcoming book, The Practice of Network Security Monitoring. The TOC has only solidified in the last day or so. I delayed responding until I completed all of the text, which I did this weekend.

You can preorder the book through No Starch. Please consider using the discount code NSM101 to save 30%.

I'm still on track to publish by July 22, 2013, in time to teach two sessions of my new course, Network Security Monitoring 101, in Las Vegas. I'll be using the new book's themes for inspiration but will likely have to rebuild all the labs.

I expect the book to approach the 350 page mark, exceeding my initial estimates for 256 pages and 7 chapters. Here's the latest Table of Contents.

  • Part I, “Getting Started,” introduces NSM and how to think about sensor placement.
    • Chapter 1, “NSM Rationale,” explains why NSM matters, to help you gain the support needed to deploy NSM in your environment.
    • Chapter 2, “Collecting Network Traffic: Access, Storage, and Management,” addresses the challenges and solutions surrounding physical access to network traffic.

  • Part II, “Security Onion Deployment,” focuses on installing SO on hardware, and configuring SO effectively.
    • Chapter 3, “Stand-alone Deployment,” introduces SO, and explains how to install the software on spare hardware to gain initial NSM capability at low or no cost.
    • Chapter 4, “Distributed Deployment,” extends Chapter 3 to describe how to install a dispersed SO system.
    • Chapter 5, “SO Housekeeping,” discusses maintenance activities for keeping your SO installation running smoothly.

  • Part III, “Tools,” describes key software shipped with SO, and how to use these applications.
    • Chapter 6, “Command Line Packet Analysis Tools,” explains the key features of Tcpdump, Tshark, Dumpcap, and Argus in SO.
    • Chapter 7, “Graphical Packet Analysis Tools,” adds GUI-based software to the mix, describing Wireshark, Xplico, and NetworkMiner.
    • Chapter 8, “Consoles,” shows how NSM suites like Sguil, Squert, Snorby, and ELSA enable detection and response workflows.

  • Part IV, “NSM in Action,” discusses how to use NSM processes and data to detect and respond to intrusions.
    • Chapter 9, “Collection, Analysis, Escalation, and Resolution,” shares my experience building and leading a global Computer Incident Response Team (CIRT).
    • Chapter 10, “Server-Side Compromise,” is the first NSM case study, wherein you’ll learn how to apply NSM principles to identify and validate the compromise of an Internet-facing application.
    • Chapter 11, “Client-Side Compromise,” is the second NSM case study, offering an example of a user being victimized by a client-side attack.
    • Chapter 12, “Extending SO,” covers tools and techniques to expand SO’s capabilities.
    • Chapter 13, “Proxies and Checksums,” concludes the main text by addressing two challenges to conducting NSM.

  • The Conclusion offers a few thoughts on the future of NSM, especially with respect to cloud environments and workflows.
  • Appendix A, “Security Onion Scripts and Configuration,” includes information from SO developer Doug Burks on core SO configuration files and control scripts.

I hope you enjoy the book and consider the new class! If you have comments or questions, please post them here on via @taosecurity.

Sunday, April 21, 2013

Bejtlich Teaching New Class at Black Hat in July

I'm pleased to announce I will teach two sessions of a brand-new two day class at Black Hat USA 2013 this summer. The new class is Network Security Monitoring 101. From the overview:

Is your network safe from intruders? Do you know how to find out? Do you know what to do when you learn the truth? If you are a beginner, and need answers to these questions, Network Security Monitoring 101 (NSM101) is the newest Black Hat course for you.

This vendor-neutral, open source software-friendly, reality-driven two-day event will teach students the investigative mindset not found in classes that focus solely on tools. NSM101 is hands-on, lab-centric, and grounded in the latest strategies and tactics that work against adversaries like organized criminals, opportunistic intruders, and advanced persistent threats.

Best of all, this class is designed *for beginners*: all you need is a desire to learn and a laptop ready to run a few virtual machines.

Instructor Richard Bejtlich has taught over 1,000 Black Hat students since 2002, and this brand new, 101-level course will guide you into the world of Network Security Monitoring.

Black Hat has three remaining price points and deadlines for registration.

  • "Regular" ends 31 May

  • "Late" ends 24 July

  • "Onsite" starts at the conference

Seats are filling -- it pays to register early!

If you have any questions about the class, please leave a comment here or contact me via Twitter at @taosecurity. Thank you.

I'm also talking with Black Hat about teaching at their Istanbul and Seattle events later this year.

Saturday, March 02, 2013

Mandiant APT1 Report: 25 Best Commentaries of the Last 12 Days

Two weeks ago today our team at Mandiant was feverishly preparing the release of our APT1 report.

In the twelve days that followed publication on the evening of Monday the 18th, I've been very pleased by the amount of constructive commentary and related research published online.

In this post I'd like to list those contributions that I believe merit attention, in the event you missed them the first time around.

These sorts of posts are examples of what the security community can do to advance our collective capability to counter digital threats.

Please note I avoided mass media accounts, interviews with Mandiant team members, and most general commentary.

They are listed in no particular order.

  1. Seth Hall (Bro): Watching for the APT1 Intelligence
  2. Jason Wood (SecureIdeas): Reading the Mandiant APT1 Report
  3. Chris Sanders: Making the Mandiant APT1 Report Actionable
  4. Symantec: APT1: Q&A on Attacks by the Comment Crew
  5. Tekdefense (NoVA Infosec): MASTIFF Analysis of APT1
  6. Chort Row (@chort0): Analyzing APT1 with Cuckoobox, Volatility, and Yara
  7. Ron Gula (Tenable): We have Microsoft Tuesday, so how long until we have Indicator Wednesday?
  8. OpenDNS Umbrella Labs:An intimate look at APT1, China’s Cyber-Espionage Threat
  9. Chris Lew (Mandiant): Chinese Advanced Persistent Threats: Corporate Cyber Espionage Processes and Organizations (BSidesSF, slides not online yet)
  10. Adam Segal: Hacking back, signaling, and state-society relations
  11. Snorby Labs: APT Intelligence Update
  12. Wendy Nather: Exercises left to the reader
  13. Brad Shoop (Mandiant): Mandiant’s APT1 Domain/MD5 Intel and Security Onion for Splunk
  14. Brad Shoop (Mandiant): Mandiant’s APT1 Domain/MD5 Intel and Security Onion with ELSA
  15. Kevin Wilcox: NSM With Bro-IDS Part 5: In-house Modules to Leverage Outside Threat Intelligence
  16. Cyb3rsleuth: Chinese Threat Actor Part 5
  17. David Bianco: The Pyramid of Pain
  18. Wesley McGrew: Mapping of Mandiant APT1 malware names to available samples
  19. Russ McRee: Toolsmith: Redline, APT1, and you – we’re all owned
  20. Jaime Blasco ( AlienVault Labs): Yara rules for APT1/Comment Crew malware arsenal
  21. Brandon Dixon: Mandiant APT2 Report Lure
  22. Seculert: Spear-Phishing with Mandiant APT Report
  23. PhishMe: How PhishMe addresses the top attack method cited in Mandiant’s APT1 report
  24. Rich Mogull (Securosis): Why China's Hacking is Different
  25. China Digital Times: Netizens Gather Further Evidence of PLA Hacking

M-Unition (Mandiant) published Netizen Research Bolsters APT1 Attribution.

I'd also like to cite Verizon for their comments and mention of IOCExtractor and Symantec for publishing their indicators via Pastebin after I asked about it.

Thank you to those who took the time to share what you found when analyzing related APT1 data, or when showing how to use APT1 indicators to do detection and response.

Sunday, February 24, 2013

Recovering from Suricata Gone Wild

Recently I tried interacting with one of my lab Security Onion sensors running the Suricata IDS. I found the Sguil server was taking a really long time to offer services on port 7734 TCP. Since I hadn't worked with this lab system in a while, I guessed that there might be too many uncategorized events in the Sguil database. I dusted off an old blog post titled More Snort and Sguil Tuning from 2006 and took a look at the system.

First I stopped the NSM applications on the server.

sudo service nsm stop
Stopping: securityonion
  * stopping: sguil server                                [  OK  ]
Stopping: HIDS
  * stopping: ossec_agent (sguil)                         [  OK  ]
Stopping: Bro
stopping ds61so-eth1-1 ...
stopping proxy ...
stopping manager ...
Stopping: ds61so-eth1
  * stopping: netsniff-ng (full packet data)              [  OK  ]
  * stopping: pcap_agent (sguil)                          [  OK  ]
  * stopping: snort_agent (sguil)                         [  OK  ]
  * stopping: suricata (alert data)                       [  OK  ]
  * stopping: barnyard2 (spooler, unified2 format)        [  OK  ]
  * stopping: prads (sessions/assets)                     [  OK  ]
  * stopping: sancp_agent (sguil)                         [  OK  ]
  * stopping: pads_agent (sguil)                          [  OK  ]
  * stopping: argus                                       [  OK  ]
  * stopping: http_agent (sguil)                      
Next I ran a query to look for the top uncategorized events.
$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1639
Server version: 5.5.29-0ubuntu0.12.04.1 (Ubuntu)

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use securityonion_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
| count   | signature                                                                        |
| 2299160 | SURICATA STREAM Packet with invalid ack                                          |
| 2298505 | SURICATA STREAM ESTABLISHED invalid ack                                          |
| 1777530 | SURICATA STREAM ESTABLISHED packet out of window                                 |
|   38700 | SURICATA STREAM ESTABLISHED retransmission packet before last ack                |
|   24181 | SURICATA STREAM TIMEWAIT ACK with wrong seq                                      |
|    5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management |
|    3160 | SURICATA STREAM Last ACK with wrong seq                                          |
|     753 | ET POLICY Offsite File Backup in Use                                 |
|     637 | SURICATA HTTP unknown error                                                      |
|     626 | SURICATA STREAM SHUTDOWN RST invalid ack                                         |
|     505 | SURICATA STREAM FIN1 FIN with wrong seq                                          |
|     494 | SURICATA HTTP request field too long                                             |
|     448 | ET POLICY PE EXE or DLL Windows file download                                    |
|     315 | ET RBN Known Malvertiser IP (22)                                                 |
|     270 | ET POLICY iTunes User Agent                                                      |
|     266 | SURICATA STREAM CLOSEWAIT ACK out of window                                      |
|     237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                   |
|     219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                      |
|     217 | SURICATA STREAM 3way handshake with ack in wrong dir                             |
|     151 | SURICATA STREAM FIN2 FIN with wrong seq                                          |
20 rows in set (15.24 sec)
Wow, that's a lot of SURICATA STREAM events. I need to categorize them as non-issues to recover the Sguil server.

mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE 'SURICATA STREAM%';
Query OK, 6443375 rows affected, 65535 warnings (3 min 4.89 sec)
Rows matched: 6443375  Changed: 6443375  Warnings: 6443375
Let's see what the database thinks now.
mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
| cnt  | signature                                                                               |
| 5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management        |
|  753 | ET POLICY Offsite File Backup in Use                                        |
|  637 | SURICATA HTTP unknown error                                                             |
|  494 | SURICATA HTTP request field too long                                                    |
|  448 | ET POLICY PE EXE or DLL Windows file download                                           |
|  315 | ET RBN Known Malvertiser IP (22)                                                        |
|  270 | ET POLICY iTunes User Agent                                                             |
|  237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                          |
|  219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                             |
|  133 | ET INFO PDF Using CCITTFax Filter                                                       |
|  106 | ET POLICY Pandora Usage                                                                 |
|   97 | ET CHAT Facebook Chat (buddy list)                                                      |
|   93 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard via GET                     |
|   58 | ET POLICY Internal Host Retrieving External IP via - Possible Infection  |
|   41 | PADS New Asset - ssl TLS 1.0 Client Hello                                               |
|   39 | SURICATA HTTP response header invalid                                                   |
|   39 | ET CURRENT_EVENTS Exploit Kit Delivering Compressed Flash Content to Client             |
|   36 | ET POLICY Python-urllib/ Suspicious User Agent                                          |
|   36 | ET MALWARE Possible Windows executable sent when remote host claims to send a Text File |
|   28 | ET POLICY Http Client Body contains pw= in cleartext                                    |
20 rows in set (0.03 sec)
That's much better.

Before restarting the NSM services, I edit the autocat.conf file to add the following.

This will auto-categorize any SURICATA STREAM alerts as non-issues. I want to keep adding events to the database for testing purposes, but I don't want to see them in the console.

Now I restart the NSM services.

sudo service nsm start
Starting: securityonion
  * starting: sguil server                                                                [  OK  ]
Starting: HIDS
  * starting: ossec_agent (sguil)                                                         [  OK  ]
Starting: Bro
starting manager ...
starting proxy ...
starting ds61so-eth1-1 ...
Starting: ds61so-eth1
  * starting: netsniff-ng (full packet data)                                              [  OK  ]
  * starting: pcap_agent (sguil)                                                          [  OK  ]
  * starting: snort_agent (sguil)                                                         [  OK  ]
  * starting: suricata (alert data)                                                       [  OK  ]
  * starting: barnyard2 (spooler, unified2 format)                                        [  OK  ]
  * starting: prads (sessions/assets)                                                     [  OK  ]
  * starting: pads_agent (sguil)                                                          [  OK  ]
  * starting: sancp_agent (sguil)                                                         [  OK  ]
  * starting: argus                                                                       [  OK  ]
  * starting: http_agent (sguil)                                                          [  OK  ]
  * disk space currently at 22%
I check to see if port 7734 TCP is listening.
sudo netstat -natup | grep 7734
tcp        0      0  *               LISTEN      10729/tclsh
Now the Sguil server is listening. I can connect with a Sguil client, even the 64 bit Windows .exe that I just found this morning. Check it out at

Friday, February 22, 2013

Using Bro to Log SSL Certificates

I remember using an older version of Bro to log SSL certificates extracted from the wire. The version shipped with Security Onion is new and that functionality doesn't appear to be enabled by default. I asked Seth Hall about this capability, and he told me how to get Bro to log all SSL certs that it sees.

Edit /opt/bro/share/bro/site/local.bro to contain the changes as shown below.

diff -u /opt/bro/share/bro/site/local.bro.orig /opt/bro/share/bro/site/local.bro
--- /opt/bro/share/bro/site/local.bro.orig      2013-02-23 01:54:53.291457193 +0000
+++ /opt/bro/share/bro/site/local.bro   2013-02-23 01:55:16.151996423 +0000
@@ -56,6 +56,10 @@
 # This script enables SSL/TLS certificate validation.
 @load protocols/ssl/validate-certs

+# Log certs per Seth
+@load protocols/ssl/extract-certs-pem
+redef SSL::extract_certs_pem = ALL_HOSTS;
 # If you have libGeoIP support built in, do some geographic detections and
 # logging for SSH traffic.
 @load protocols/ssh/geo-data
Restart Bro.
~# broctl

Welcome to BroControl 1.1

Type "help" for help.

[BroControl] > install
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating standalone-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... done.
[BroControl] > status
Name       Type       Host       Status        Pid    Peers  Started            
bro        standalone localhost  running       3042   0      17 Feb 13:22:42
[BroControl] > restart
stopping ...
stopping bro ...
starting ...
starting bro ...
[BroControl] > exit

After restarting you will have a new log for all SSL certs:

ls -al certs-remote.pem
-rw-r--r-- 1 root root 31907 Feb 23 02:05 certs-remote.pem

New certs are appended to the file as Bro sees them. A cert looks like this:

OpenSSL can read them one at a time, e.g.:
openssl x509 -in certs-remote.pem -text -noout
        Version: 3 (0x2)
        Serial Number:
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=Terms of use at (c)06, CN=VeriSign Class 3 Extended Validation SSL CA
            Not Before: Feb 29 00:00:00 2012 GMT
            Not After : Feb 28 23:59:59 2013 GMT
        Subject: Organization/serialNumber=2927442, C=US/postalCode=60603, ST=Illinois, L=Chicago/street=135 S La Salle St, O=Bank of America Corporation, OU=Network Infrastructure,
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
            X509v3 Subject Key Identifier:
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 CRL Distribution Points:

                Full Name:

            X509v3 Certificate Policies:
                Policy: 2.16.840.1.113733.

            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Authority Key Identifier:

            Authority Information Access:
                OCSP - URI:
                CA Issuers - URI:

    Signature Algorithm: sha1WithRSAEncryption
Since each cert has a standard header and footer, I bet someone could write a parser to extract each cert from the certs-remote.pem file to separate files. Thanks a lot Seth!

Monday, February 11, 2013

Practical Network Security Monitoring Book on Schedule

First the good news: my new book Practical Network Security Monitoring is on track, and you can pre-order with a 30% discount using code NSM101.

I'm about 1/3 of the way through writing the book. Since I announced the project last month, I've submitted chapters 1, 2, and 3. They are in various stages of review by No Starch editors and my technical editors. I seem to be writing more than I expected, despite trying to keep the book at an introductory level. I find that I want to communicate the topic sufficiently to make my point, but I try to avoid going too deeply into related areas.

I'm also encountering situations where I have to promise to explain some concepts later, rather than explain everything immediately. I believe once I get the first chapter ironed out with the editor, the rest will be easier to digest. I'm taking a fairly methodical approach (imagine that), so once the foundation in chapter 1 is done the rest is more straightforward.

I'm keeping a fairly aggressive schedule. Basically I have to write a chapter each week, get it to my technical editors, and then spend additional time working with No Starch to get the text legible and ready for print. All of this is happening in parallel in order to have the books in print by Black Hat. That means the text must done by the first week in April. My family is helping me stay on track by giving me time and space to write, especially on the weekends. Thank you!

When working on the examples, I've been very pleased with the performance of VMWare Workstation 9. I have one copy installed on Windows 7, where I write with Word. I have a second copy installed on Ubuntu Server, where it acts like a "VMWare Server." I used to run a real ESXi server on server-class hardware. Now, to save electricity and to more tailor my computer power to my requirements, I run a Shuttle DS61 with a Core i5-3450S 2.80GHz CPU, 16 GB RAM, 750 GB HDD, and two onboard NICs. The two NICs are really awesome in a device this small -- 190(L) x 165(W) x 43(H) mm. With two NICs, I can devote one for management and one for network traffic collection and interpretation. I use a Net Optics Dual Port Aggregator Tap for access to the wire.

I use VMWare Workstation this way. I run a Linux VM on Workstation on my Windows 7 laptop. I connect via Workstation to the Workstation instance on Ubuntu on the DS61. Then I create whatever VMs I need on the DS61. For example, I created a Security Onion server and sensor to test that setup. With 16 GB RAM, I have plenty of RAM for both, plus another VM that I'm running as my "production" Security Onion sensor for the lab network.

Writing is going well, despite the fact that I last wrote a book in 2005. I promised my youngest daughter, who wasn't born until 2006, that this new book is for her. If you have any questions on the writing process, please post them here or ask me on Twitter.

Saturday, January 26, 2013

On Thought Leadership and Non-Technical Relevance

A reader left a comment on my post 2012: The Year I Changed What I Read. He said:

Richard, it's interesting to note that your career has shifted from "pure" technology to more of a thought leadership role where you can leverage your training and interest in history, political science, etc. I wonder if you ever expected to become such a public figure in the whole debate about China when you first started with infosec?

Your career path is an encouraging example for others to follow. Even though I work in technology, I also have a sociology/political science background and I've been wondering how I can leverage those interests, especially as I get older and cheaper/hungrier techies continue to enter the industry.

Thank you for your comment and question. I will try to answer here.

I did not plan to become a "public" figure, and I don't necessarily consider myself exceptionally "public" now. I just reviewed my TaoSecurity news page to see when I first started speaking at conferences. Before joining Foundstone, I spoke at a few events because I believed too few people were discussing incident detection and traffic analysis. Once I joined Foundstone in April 2002 as a member of Kevin Mandia's incident response team, I became a public speaker out of necessity. Kevin and Foundstone expected consultants to speak, teach, and write, in addition to performing consulting duties. I've stayed in that mindset ever since, although I speak, teach, and write on increasingly diverse topics.

I see the "thought leadership" question in two ways. First, I took deliberate actions to get my thoughts to the world. I wrote my books and post to this blog as a way to capture my thinking on a coherent set of subjects. I hope they are useful to others, but I see these as outlets for self-expression.

The second way I think about "thought leadership" involves my work duties. If you look at my press page you will see a jump in activity in 2011, the year I joined Mandiant. In addition to being CSO, I'm also responsible for speaking with the press, industry analysts, policy makers, and some customers and prospects. I enjoy these opportunities because I realize there are a lot of sources for tools but few for methodologies and operational processes. To the extent I can share my recommendations for how to combat intruders and avoid wasting resources or pursuing dead ends, I consider this second form of thought leadership a success.

Finally, let me address the point about leveraging what are traditionally "non-security" skills or interests, namely history and political science. As I've posted and Tweeted earlier, the world is waking up to the fact that the techies and engineers don't have all the answers. Every time you hear someone say that the answer is to build Internet 2, and "get it right," you're listening to an "engineering first" mindset.

I love engineers (my dad is one, I took plenty of engineering in college, I work with engineers, etc.) but their viewpoint is but one of many. Technical knowledge doesn't give anyone a golden ticket to good policy. If we don't engage people who understand lessons of history and policy, we'll continue to lose when facing advanced intruders.

I would argue that a person who knows technology, security, history, and politics is equipped to be very valuable to an organization trying to build a mature security operation, or that seeks to influence policy. Your interests and skills may not align with your current role, so you may need to keep those strengths in mind when looking for a job better aligned with history and politics.

I think the key is to strive to stay relevant in whatever area interests you. If you like non-technical subjects, you've got to stay current with them and develop your thoughts and analysis on those issues the same as you might with technical topics.

Thank you for your comment. I welcome other comments here or on Twitter.

Wednesday, January 16, 2013

How to Win This TCP/IP Book

Last week I wished this blog happy tenth birthday and announced plans for a new book on network security monitoring. I also mentioned a contest involving a book give-away. I finally figured out a good way to select a winner, and it involves your participation in my current writing project!

Thanks to No Starch Press I have a brand-new, shrink-wrapped copy of The TCP/IP Guide, a mammoth 1616 page hardcover book by Charles M. Kozierok.

Here's what you have to do to try to win this book: submit a case study on how network security monitoring helped you detect, respond to, and contain an intrusion in your environment.

You don't have to reveal your organization, but I want to know some general information like the number of users and computers. Readers need to know the sort of environment where NSM worked for you, but I don't want you to reveal your organization (unless you want to).

Tell the reader what happened, what NSM data you used, how you used it, and how you handled the incident. Extra points go to writers who include log excerpts and screen captures.

I will include the submission in my new book, subject to editing by myself and No Starch, for readability and comprehension.

The deadline for submission is 10:00 pm eastern time, Saturday 26 January (sorry for the earlier typo). I managed to extend the deadline a little. Quality trumps quantity here -- I'm not looking for another chapter!

Please submit your entries as plain text in email to taosecurity at gmail dot com. I won't open .doc or .pdf or other files which could contain surprises.

When you take screen captures, save them in high-resolution .tif format without compression. Don't take a capture of command-line information; instead, copy the text into the story. When taking screen captures of GUI tools and the like, don't take a capture of a giant window; resize to something that will be legible on a printed page, witha .

This is an example of a bad screen capture:

This is a good screen capture:

Depending on the quality of any screen captures, I may ask you to resubmit them to meet the publisher's requirements.

If you have any questions, please post them here.

The winner will receive the pictured TCP/IP book. Once my new book arrives, I will ask the publisher to mail you a free copy too.

If I receive one or more good runners-up, I will ask the publisher to send their owners copies of my new book too.

If you have any questions, please submit them as comments here. Good luck!

Tuesday, January 08, 2013

Bejtlich's New Book: Planned for Summer Publication

Nearly ten years after I started writing my first book, the Tao of Network Security Monitoring, I'm pleased to announce that I just signed a contract to write a new book for No Starch titled Network Security Monitoring in Minutes.

From the book proposal:

Network Security Monitoring in Minutes provides the tactics, techniques, and procedures for maximum enterprise defense in a minimum amount of time.

Network Security Monitoring (NSM) is the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. Network Security Monitoring in Minutes teaches information technology and security staff how to leverage powerful NSM tools and concepts immediately.

Using open source software and vendor-neutral methods, the author applies lessons he first began applying to military networks in 1998. After reading this book, the audience will be able to integrate the same winning approaches to better defend his or her company’s data and networks.

Network Security Monitoring in Minutes is an important book because nearly all organizations operate a network. By connecting to the Internet, they expose their intellectual property, trade secrets, critical business processes, personally identifiable information (PII), and other sensitive information to attackers worldwide. Without the network level vigilance provided by this book, organizations will continue to be victimized for months, and in many cases years, before learning they have been breached.

This book consists of the following chapters:

Chapter 1, Network Security Monitoring Rationale, explains why NSM matters and help readers gain the support needed to deploy NSM in their environment.

Chapter 2, Accessing Network Traffic, addresses the challenges and solutions surrounding physical access to network traffic.

Chapter 3, Sensor Deployment and Configuration, introduces Security Onion (SO), and explains how readers can install the software on spare hardware to gain an initial NSM capability at low or no cost.

Chapter 4, Tool Overview, guides the reader through the core SO tool set, focusing on those capabilities most likely to help handle digital intrusions.

Chapter 5, Network Security Monitoring Operations, shares the author’s experience building and leading a global Computer Incident Response Team (CIRT), such that readers can apply those lessons to their own operations.

Chapter 6, Server-Side Compromise, is the first NSM case study, wherein readers will learn how to apply NSM principles to identify and validate a compromise of an Internet-facing application.

Chapter 7, Client-Side Compromise, is the second NSM case study, offering readers an example of a user being victimized by a client-side attack. NSM data will again identify and validate the compromise, prompting efficient incident response.

The Conclusion extends NSM principles beyond the enterprise into hosted and Cloud settings, offering future options for those environments.

The Appendix discusses tools that are not open source, but which may be helpful to those conducting NSM operations.

My goal is to finish this short book (roughly 220 pages) in time for publication at Black Hat this summer. Thank you to Pearson/Addison-Wesley for giving me the flexibility to write this complementary NSM book, and to No Starch for signing me to their publishing house.

Happy 10th Birthday TaoSecurity Blog

Today, 8 January 2013, is the 10th birthday of TaoSecurity Blog!
I wrote my first post on 8 January 2003 while working as an incident response consultant for Foundstone working for Kevin Mandia. Today I am Chief Security Officer at Mandiant, back working for Kevin Mandia. (It's a small world.)
With 2905 posts published over these 10 years, I am still blogging -- but much less. Looking at all 10 years of blogging, I averaged 290 per year, but in the age of Twitter (2009-2012) I averaged only 144 blog posts per year. Last year I wrote 60 times.
Why the drop over the years? First, I "blame" my @taosecurity Twitter account. With over 15,000 followers, easy posting from mobile devices, and greater interactivity, Twitter is an addictive platform. However, I really enjoy Twitter and make the trade-off gladly. It would be nice to become a verified user though, with access to two-factor or two-step authentication.
Second, blogging used to be the primary way I could share my ideas with the community. These days, speaking and writing are a big part of my professional duties. For example, last year news outlets quoted me 55 times. Those citations represent hundreds of hours spent talking to the press, explaining how security works and how to improve our situation. I also wrote for the Mandiant Blog, and spoke or taught at 22 events. At the end of many days, I feel like I'm getting my message out without blogging.
Third, time is precious. I enjoy spending time with my family, or reading, or working out, or learning to play guitar when I'm not working for Mandiant.
However, I still plan to keep blogging in 2013. Twitter's only a 140 character platform, and some days I have the time and inclination to share a few thoughts beyond what I've said or written for work.
To celebrate the blog's 10th birthday, I will be announcing a book giveaway on my @taosecurity Twitter account either today or before the end of the week. Follow me on Twitter for details.
Before finishing I'd like to thank Blogger, now part of Google, for providing me this free platform for the past ten years. Way to go!
In my next post I will share word on an exciting new project. Stay tuned.

Monday, January 07, 2013

Welcome to Network Security Monitoring in the Cloud

I just watched an incredible technical video. If you have about 10 minutes to spare, and want to be amazed, take a look at Snorby Cloud.

I think the video and Web site does an excellent job explaining this new offering, but let me provide a little background.

Many of the readers of this blog are security pros. You're out there trying to defend your organization, not necessarily design, build, and run infrastructure. You still need tools and workflows that accelerate your incident detection and response process though. So, you work as a security admin, system admin, storage admin, database admin... you get the picture. You manage to keep up, but you probably wish you could focus on finding bad guys, as quickly as possible, without taking care of all the *stuff* that you need to do your job.

While many of you are security experts, some are just beginning your journeys. The responsibilities of being an admin of four or more different shades is overwhelming. Furthermore, you don't have the experience, or budget, or support to get the security data and escalation paths needed to defend your network. How can you improve your skills when you're constantly overwhelmed?

Both kinds of users -- senior and junior alike -- are going to find something intriguing about Snorby Cloud. Maybe you've heard of Snorby before, as a Web-based interface to Network Security Monitoring data. Doug Burks packages it with Security Onion (SO), and you can try it via live CD or .iso in a VM. It looks great on my iPad! There's even a mobile version on iTunes.

Snorby Cloud would be cool if it just put the Snorby Web application in the cloud, and managed the administrative side of security infrastructure for you. For example, you'd log into the cloud interface and be greeted by the graphs you remember from traditional Snorby.

However, you have to think of this as a new, better version of Snorby, collecting far more useful data, and making it rapidly available to the analyst. For example, the following shows SMTP logs available in the interface:

You can just as easily access host-based logs for the same victim computer:

As you investigate the incident, you can see who else on your team is working and what they did. You can also chat with them in real time.

I could say a lot more about this new tool, but I think watching the video will convey some of what it can do. My next step is to get the agents running on a test network so I can drive the console myself and become more familiar with it.

Snorby Cloud is a product from Packet Stash. Follow them at @packetstash for updates.

Disclaimer: I'm friends with this team; I hired two of the co-founders into GE-CIRT, and later worked with all three co-founders at Mandiant.