Tuesday, January 08, 2019

Happy 16th Birthday TaoSecurity Blog

Today, 8 January 2019, is TaoSecurity Blog's 16th birthday! This is also my 3,041st blog post.

I wrote my first post on 8 January 2003 while working as an incident response consultant for Foundstone.

Here are a few statistics on the blog. Blogger started providing statistics in May 2010, so these apply to roughly the past 9 years only.

As of today, since May 2010 the blog has nearly 9.4 million all time page views, up from 7.7 million a year ago.

Here are the most popular posts of the last 9 years, as of today:


I'm blogging a bit more recently, with 22 posts in 2018 -- more than my total for 2016 and 2017 combined, but still not half as much as 2015, which saw 55 posts.

Twitter continues to play a role in the way I communicate. Last year @taosecurity had nearly 49,000 followers with less than 18,000 Tweets. Today I have nearly 53,000 followers with 19,000 Tweets.

My rule is generally this: if I start wondering how to fit an idea in 280 characters on Twitter, then a blog post is a better idea. If I start a Twitter "thread," then I really need to write a blog post!

I continue to blog about martial arts and related topics at Rejoining the Tao, which incidentally will be three years old later this month, and is currently 11 posts shy of 100. You can see that during my burnout period I shifted my writing and creativity outside of security.

Thank you to everyone who has been part of this blog's journey since 2003!

Monday, December 31, 2018

Notes on Self-Publishing a Book


In this post I would like to share a few thoughts on self-publishing a book, in case anyone is considering that option.

As I mentioned in my post on burnout, one of my goals was to publish a book on a subject other than cyber security. A friend from my Krav Maga school, Anna Wonsley, learned that I had published several books, and asked if we might collaborate on a book about stretching. The timing was right, so I agreed.

I published my first book with Pearson and Addison-Wesley in 2004, and my last with No Starch in 2013. 14 years is an eternity in the publishing world, and even in the last 5 years the economics and structure of book publishing have changed quite a bit.

To better understand the changes, I had dinner with one of the finest technical authors around, Michael W. Lucas. We met prior to my interest in this book, because I had wondered about publishing books on my own. MWL started in traditional publishing like me, but has since become a full-time author and independent publisher. He explained the pros and cons of going it alone, which I carefully considered.

By the end of 2017, Anna and I were ready to begin work on the book. I believe our first "commits" occurred in December 2017.

For this stretching book project, I knew my strengths included organization, project management, writing to express another person's message, editing, and access to a skilled lead photographer. I learned that my co-author's strengths included subject matter expertise, a willingness to be photographed for the book's many pictures, and friends who would also be willing to be photographed.

None of us was very familiar with the process of transforming a raw manuscript and photos into a finished product. When I had published with Pearson and No Starch, they took care of that process, as well as copy-editing.

Beyond turning manuscript and photos into a book, I also had to identify a publication platform. Early on we decided to self-publish using one of the many newer companies offering that service. We wanted a company that could get our book into Amazon, and possibly physical book stores as well. We did not want to try working with a traditional publisher, as we felt that we could manage most aspects of the publishing process ourselves, and augment with specialized help where needed.

After a lot of research we chose Blurb. One of the most attractive aspects of Blurb was their expert ecosystem. We decided that we would hire one of these experts to handle the interior layout process. We contacted Jennifer Linney, who happened to be local and had experience publishing books to Amazon. We met in person, discussed the project, and agreed to move forward together.

I designed the structure of the book. As a former Air Force officer, I was comfortable with the "rule of threes," and brought some recent writing experience from my abandoned PhD thesis.

I designed the book to have an introduction, the main content, and a conclusion. Within the main content, the book featured an introduction and physical assessment, three main sections, and a conclusion. The three main sections consisted of a fundamental stretching routine, an advanced stretching routine, and a performance enhancement section -- something with Indian clubs, or kettle bells, or another supplement to stretching.

Anna designed all of the stretching routines and provided the vast majority of the content. She decided to focus on three physical problem areas -- tight hips, shoulders/back, and hamstrings. We encouraged the reader to "reach three goals" -- open your hips, expand your shoulders, and touch your toes. Anna designed exercises that worked in a progression through the body, incorporating her expertise as a certified trainer and professional martial arts instructor.

Initially we tried a process whereby she would write section drafts, and I would edit them, all using Google Docs. This did not work as well as we had hoped, and we spent a lot of time stalled in virtual collaboration.

By the spring of 2018 we decided to try meeting in person on a regular basis. Anna would explain her desired content for a section, and we would take draft photographs using iPhones to serve as placeholders and to test the feasibility of real content. We made a lot more progress using these methods, although we stalled again mid-year due to schedule conflicts.

By October our text was ready enough to try taking book-ready photographs. We bought photography lights from Amazon and used my renovated basement game room as a studio. We took pictures over three sessions, with Anna and her friend Josh as subjects. I spent several days editing the photos to prepare for publication, then handed the bundled manuscript and photographs to Jennifer for a light copy-edit and layout during November.

Our goal was to have the book published before the end of the year, and we met that goal. We decided to offer two versions. The first is a "collector's edition" featuring all color photographs, available exclusively via Blurb as Reach Your Goal: Collector's Edition. The second will be available at Amazon in January, and will feature black and white photographs.

While we were able to set the price of the book directly via Blurb, we could basically only suggest a price to Ingram and hence to Amazon. Ingram is the distributor that feeds Amazon and physical book stores. I am curious to see how the book will appear in those retail locations, and how much it will cost readers. We tried to price it competitively with older stretching books of similar size. (Ours is 176 pages with over 200 photographs.)

Without revealing too much of the economic structure, I can say that it's much cheaper to sell directly from Blurb. Their cost structure allows us to price the full color edition competitively. However, one of our goals was to provide our book through Amazon, and to keep the price reasonable we had to sell the black and white edition outside of Blurb.

Overall I am very pleased with the writing process, and exceptionally happy with the book itself. The color edition is gorgeous and the black and white version is awesome too.

The only change I would have made to the writing process would have been to start the in-person collaboration from the beginning. Working together in person accelerated the transfer of ideas to paper and played to our individual strengths of Anna as subject matter expert and me as a writer.

In general, I would not recommend self-publishing if you are not a strong writer. If writing is not your forte, then I highly suggest you work with a traditional publisher, or contract with an editor. I have seen too many self-published books that read terribly. This usually happens when the author is a subject matter expert, but has trouble expressing ideas in written form.

The bottom line is that it's never been easier to make your dream of writing a book come true. There are options for everyone, and you can leverage them to create wonderful products that scale with demand and can really help your audience reach their goals!

If you want to start the new year with better flexibility and fitness, consider taking a look at our book on Blurb! When the Amazon edition is available I will update this post with a link.

Update: Here is the Amazon listing.

Cross-posted from Rejoining the Tao Blog.

Friday, December 21, 2018

Managing Burnout

This is not strictly an information security post, but the topic likely affects a decent proportion of my readership.

Within the last few years I experienced a profound professional "burnout." I've privately mentioned this to colleagues in the industry, and heard similar stories or requests for advice on how to handle burnout.

I want to share my story in the hopes that it helps others in the security scene, either by coping with existing burnout or preparing for a possible burnout.

How did burnout manifest for me? It began with FireEye's acquisition of Mandiant, almost exactly five years ago. 2013 was a big year for Mandiant, starting with the APT1 report in early 2013 and concluding with the acquisition in December.

The prospect of becoming part of a Silicon Valley software company initially seemed exciting, because we would presumably have greater resources to battle intruders. Soon, however, I found myself at odds with FireEye's culture and managerial habits, and I wondered what I was doing inside such a different company.

(It's important to note that the appointment of Kevin Mandia as CEO in June 2016 began a cultural and managerial shift. I give Kevin and his lieutenants credit for helping transform the company since then. Kevin's appointment was too late for me, but I applaud the work he has done over the last few years.)

Starting in late 2014 and progressing in 2015, I became less interested in security. I was aggravated every time I saw the same old topics arise in social or public media. I did not see the point of continuing to debate issues which were never solved. I was demoralized and frustrated.

At this time I was also working on my PhD with King's College London. I had added this stress myself, but I felt like I could manage it. I had earned two major and two minor degrees in four years as an Air Force Academy cadet. Surely I could write a thesis!

Late in 2015 I realized that I needed to balance the very cerebral art of information security with a more physical activity. I took a Krav Maga class the first week of January 2016. It was invigorating and I began a new blog, Rejoining the Tao, that month. I began to consider options outside of informations security.

In early 2016 my wife began considering ways to rejoin the W-2 workforce, after having stayed home with our kids for 12 years. We discussed the possibility of me leaving my W-2 job and taking a primary role with the kids. By mid-2016 she had a new job and I was open to departing FireEye.

By late 2016 I also realized that I was not cut out to be a PhD candidate. Although I had written several books, I did not have the right mindset or attitude to continue writing my thesis. After two years I quit my PhD program. This was the first time I had quit anything significant in my life, and it was the right decision for me. (The Churchill "never, never, never give up" speech is fine advice when defending your nation's existence, but it's stupid advice if you're not happy with the path you're following.)

In March 2017 I posted Bejtlich Moves On, where I said I was leaving FireEye. I would offer security consulting in the short term, and would open a Krav Maga school in the long-term. This was my break with the security community and I was happy to make it. I blogged on security only five more times in 2017.

(Incidentally, one very public metric for my burnout experience can be seen in my blog output. In 2015 I posted 55 articles, but in 2016 I posted only 8, and slightly more, 12, in 2017. This is my 21st post of 2018.)

I basically took a year off from information security. I did some limited consulting, but Mrs B paid the bills, with some support from my book royalties and consulting. This break had a very positive effect on my mental health. I stayed aware of security developments through Twitter, but I refused to speak to reporters and did not entertain job offers.

During this period I decided that I did not want to open a Krav Maga school and quit my school's instructor development program. For the second time, I had quit something I had once considered very important.

I started a new project, though -- writing a book that had nothing to do with information security. I will post about it shortly, as I am finalizing the cover with the layout team this weekend!

By the spring of 2018 I was able to consider returning to security. In May I blogged that I was joining Splunk, but that lasted only two months. I realized I had walked into another cultural and managerial mismatch. Near the end of that period, Seth Hall from Corelight contacted me, and by July 20th I was working there. We kept it quiet until September. I have been very happy at Corelight, finally finding an environment that matches my temperament, values, and interests.

My advice to those of you who have made it this far:

If you're feeling burnout now, you're not alone. It happens. We work in a stressful industry that will take everything that you can give, and then try to take more. It's healthy and beneficial to push back. If you can, take a break, even if it means only a partial break.

Even if you can't take a break, consider integrating non-security activities into your lifestyle -- the more physical, the better. Security is a very cerebral activity, often performed in a sedentary manner. You have a body and taking care of it will make your mind happier too.

If you're not feeling burnout now, I recommend preparing for a possible burnout in the future. In addition to the advice in the previous paragraphs, take steps now to be able to completely step away from security for a defined period. Save a proportion of your income to pay your bills when you're not working in security. I recommend at least a month, but up to six months if you can manage it.

This is good financial advice anyway, in the event you were to lose your job. This is not an emergency fund, though -- this is a planned reprieve from burnout. We are blessed in security to make above-average salaries, so I suggest saving for retirement, saving for layoffs, and saving for burnout.

Finally, it's ok to talk to other people about this. This will likely be a private conversation. I don't see too many people saying "I'm burned out!" on Twitter or in a blog post. I only felt comfortable writing this post months after I returned to regular security work.

I'm very interested in hearing what others have to say on this topic. Replying to my Twitter announcement for the blog post is probably the easiest step. I moderate the comments here and might not get to them in a timely manner.

Tuesday, December 18, 2018

The Origin of the Quote "There Are Two Types of Companies"

While listening to a webcast this morning, I heard the speaker mention

There are two types of companies: those who have been hacked, and those who don’t yet know they have been hacked.

He credited Cisco CEO John Chambers but didn't provide any source.

That didn't sound right to me. I could think of two possible antecedents. so I did some research. I confirmed my memory and would like to present what I found here.

John Chambers did indeed offer the previous quote, in a January 2015 post for the World Economic Forum titled What does the Internet of Everything mean for security? Unfortunately, neither Mr Chambers nor the person who likely wrote the article for him decided to credit the author of this quote.

Before providing proper credit for this quote, we need to decide what the quote actually says. As noted in this October 2015 article by Frank Johnson titled Are there really only “two kinds of enterprises”?, there are really (at least) two versions of this quote:

A popular meme in the information security industry is, “There are only two types of companies: those that know they’ve been compromised, and those that don’t know.”

And the second is like unto it: “There are only two kinds of companies: those that have been hacked, and those that will be.”

We see that the first is a version of what Mr Chambers said. Let's call that 2-KNOW. The second is different. Let's call that 2-BE.

The first version, 2-KNOW, can be easily traced and credited to Dmitri Alperovitch. He stated this proposition as part of the publicity around his Shady RAT report, written while he worked at McAfee. For example, this 3 August 2011 story by Ars Technica, Operation Shady RAT: five-year hack attack hit 14 countries, quotes Dmitri in the following:

So widespread are the attacks that Dmitri Alperovitch, McAfee Vice President of Threat Research, said that the only companies not at risk are those who have nothing worth taking, and that of the world's biggest firms, there are just two kinds: those that know they've been compromised, and those that still haven't realized they've been compromised.

Dmitri used slightly different language in this popular Vanity Fair article from September 2011, titled Enter the Cyber-Dragon:

Dmitri Alperovitch, who discovered Operation Shady rat, draws a stark lesson: “There are only two types of companies—those that know they’ve been compromised, and those that don’t know. If you have anything that may be valuable to a competitor, you will be targeted, and almost certainly compromised.”

No doubt former FBI Director Mueller read this report (and probably spoke with Dmitri). He delivered a speech at RSA on 1 March 2012 that introduced question 2-BE into the lexicon, plus a little more:

For it is no longer a question of “if,” but “when” and “how often.”

I am convinced that there are only two types of companies: those that have been hacked and those that will be. 

And even they are converging into one category: companies that have been hacked and will be hacked again.  

Here we see Mr Mueller morphing Dmitri's quote, 2-KNOW, into the second, 2-BE. He also introduced a third variant -- "companies that have been hacked and will be hacked again." Let's call this version 2-AGAIN.

The very beginning of Mr Mueller's quote is surely a play on Kevin Mandia's long-term commitment to the inevitability of compromise. However, as far as I could find, Kevin did not use the "two companies" language.

One article that mentions version 2-KNOW and Kevin is this December 2014 Ars Technica article titled “Unprecedented” cyberattack no excuse for Sony breach, pros say. However, the article is merely citing other statements by Kevin along with the aphorism of version 2-KNOW.

Finally, there's a fourth version introduced by Mr Mueller's successor, James Comey, as well! In a 6 October 2014 story, FBI Director: China Has Hacked Every Big US Company Mr Comey said:

Speaking to CBS' 60 Minutes, James Comey had the following to say on Chinese hackers: 

There are two kinds of big companies in the United States. There are those who've been hacked by the Chinese and those who don't know they've been hacked by the Chinese.

Let's call this last variant 2-CHINA.

To summarize, there are four versions of the "two companies" quote:

  • 2-KNOW, credited to Dmitri Alperovitch in 2011, says "There are only two types of companies—those that know they’ve been compromised, and those that don’t know."
  • 2-BE, credited to Robert Mueller in 2012, says "[T]here are only two types of companies: those that have been hacked and those that will be."
  • 2-AGAIN, credited to Robert Mueller in 2012, says "[There are only two types of companies:] companies that have been hacked and will be hacked again."
  • 2-CHINA, credited to James Comey in 2014, says "There are two kinds of big companies in the United States. There are those who've been hacked by the Chinese and those who don't know they've been hacked by the Chinese."
Now you know!


Sunday, November 25, 2018

The Origin of the Term Indicators of Compromise (IOCs)

I am an historian. I practice digital security, but I earned a bachelor's of science degree in history from the United States Air Force Academy. (1)

Historians create products by analyzing artifacts, among which the most significant is the written word.

In my last post, I talked about IOCs, or indicators of compromise. Do you know the origin of the term? I thought I did, but I wanted to rely on my historian's methodology to invalidate or confirm my understanding.

I became aware of the term "indicator" as an element of indications and warning (I&W), when I attended Air Force Intelligence Officer's school in 1996-1997. I will return to this shortly, but I did not encounter the term "indicator" in a digital security context until I encountered the work of Kevin Mandia.

In August 2001, shortly after its publication, I read Incident Response: Investigating Computer Crime, by Kevin Mandia, Chris Prosise, and Matt Pepe (Osborne/McGraw-Hill). I was so impressed by this work that I managed to secure a job with their company, Foundstone, by April 2002. I joined the Foundstone incident response team, which was led by Kevin and consisted of Matt Pepe, Keith Jones, Julie Darmstadt, and me.

I Tweeted earlier today that Kevin invented the term "indicator" (in the IR context) in that 2001 edition, but a quick review of the hard copy in my library does not show its usage, at least not prominently. I believe we were using the term in the office but that it had not appeared in the 2001 book. Documentation would seem to confirm that, as Kevin was working on the second edition of the IR book (to which I contributed), and that version, published in 2003, features the term "indicator" in multiple locations.

In fact, the earliest use of the term "indicators of compromise," appearing in print in a digital security context, appears on page 280 in Incident Response & Computer Forensics, 2nd Edition.


From other uses of the term "indicators" in that IR book, you can observe that IOC wasn't a formal, independent concept at this point, in 2003. In the same excerpt above you see "indicators of attack" mentioned.

The first citation of the term "indicators" in the 2003 book shows it is meant as an investigative lead or tip:


Did I just give up my search at this point? Of course not.

If you do time-limited Google searches for "indicators of compromise," after weeding out patent filings that reference later work (from FireEye, in 2013), you might find this document, which concludes with this statement:

Indicators of compromise are from Lynn Fischer, Lynn, "Looking for the Unexpected," Security Awareness Bulletin, 3-96, 1996. Richmond, VA: DoD Security Institute.

Here the context is the compromise of a person with a security clearance.

In the same spirit, the earliest reference to "indicator" in a security-specific, detection-oriented context appears in the patent Method and system for reducing the rate of infection of a communications network by a software worm (6 Dec 2002). Stuart Staniford is the lead author; he was later chief scientist at FireEye, although he left before FireEye acquired Mandiant (and me).

While Kevin, et al were publishing the second edition of their IR book in 2003, I was writing my first book, The Tao of Network Security Monitoring. I began chapter two with a discussion of indicators, inspired by my Air Force intelligence officer training in I&W and Kevin's use of the term at Foundstone.

You can find chapter two in its entirety online. In the chapter I also used the term "indicators of compromise," in the spirit Kevin used it; but again, it was not yet a formal, independent term.

My book was published in 2004, followed by two more in rapid succession.

The term "indicators" didn't really make a splash until 2009, when Mike Cloppert published a series on threat intelligence and the cyber kill chain. The most impactful in my opinion was Security Intelligence: Attacking the Cyber Kill Chain. Mike wrote:


I remember very much enjoying these posts, but the Cyber Kill Chain was the aspect that had the biggest impact on the security community. Mike does not say "IOC" in the post. Where he does say "compromise," he's using it to describe a victimized computer.

The stage is now set for seeing indicators of compromise in a modern context. Drum roll, please!

The first documented appearance of the term indicators of compromise, or IOCs, in the modern context, appears in basically two places simultaneously, with ultimate credit going to the same organziation: Mandiant.

The first Mandiant M-Trends report, published on 25 Jan 2010, provides the following description of IOCs on page 9:


The next day, 26 Jan 2010, Matt Frazier published Combat the APT by Sharing Indicators of Compromise to the Mandiant blog. Matt wrote to introduce an XML-based instantiation of IOCs, which could be read and created using free Mandiant tools.


Note how complicated Matt's IOC example is. It's not a file hash (alone), or a file name (alone), or an IP address, etc. It's a Boolean expression of many elements. You can read in the text that this original IOC definition rejects what some commonly consider "IOCs" to be. Matt wrote:

Historically, compromise data has been exchanged in CSV or PDFs laden with tables of "known bad" malware information - name, size, MD5 hash values and paragraphs of imprecise descriptions... (emphasis added)

On a related note, I looked for early citations of work on defining IOCs, and found a paper by Simson Garfinkel, well-respected forensic analyst. He gave credit to Matt Frazier and Mandiant, writing in 2011:

Frazier (2010) of MANDIANT developed Indicators of Compromise (IOCs), an XML-based language designed to express signatures of malware such as files with a particular MD5 hash value, file length, or the existence of particular registry entries. There is a free editor for manipulating the XML. MANDIANT has a tool that can use these IOCs to scan for malware and the so-called “Advanced Persistent Threat.”

Starting in 2010, the debate was initially about the format for IOCs, and how to produce and consume them. We can see in this written evidence from 2010, however, a definition of indicators of compromise and IOCs that contains all the elements that would be recognized in current usage.

tl;dr Mandiant invented the term indicators of compromise, or IOCs, in 2010, building off the term "indicator," introduced widely in a detection context by Kevin Mandia, no later than his 2003 incident response book.

(1) Yes, a BS, not a BA -- thank you USAFA for 14 mandatory STEM classes.

Saturday, November 24, 2018

Even More on Threat Hunting

In response to my post More on Threat Hunting, Rob Lee asked:

[D]o you consider detection through ID’ing/“matching” TTPs not hunting?

To answer this question, we must begin by clarifying "TTPs." Most readers know TTPs to mean tactics, techniques and procedures, defined by David Bianco in his Pyramid of Pain post as:

How the adversary goes about accomplishing their mission, from reconnaissance all the way through data exfiltration and at every step in between.

In case you've forgotten David's pyramid, it looks like this.


It's important to recognize that the pyramid consists of indicators of compromise (IOCs). David uses the term "indicator" in his original post, but his follow-up post from his time at Sqrrl makes this clear:

There are a wide variety of IoCs ranging from basic file hashes to hacking Tactics, Techniques and Procedures (TTPs). Sqrrl Security Architect, David Bianco, uses a concept called the Pyramid of Pain to categorize IoCs. 

At this point it should be clear that I consider TTPs to be one form of IOC.

In The Practice of Network Security Monitoring, I included the following workflow:

You can see in the second column that I define hunting as "IOC-free analysis." On page 193 of the book I wrote:

Analysis is the process of identifying and validating normal, suspicious, and malicious activity. IOCs expedite this process. Formally, IOCs are manifestations of observable or discernible adversary actions. Informally, IOCs are ways to codify adversary activity so that technical systems can find intruders in digital evidence...

I refer to relying on IOCs to find intruders as IOC-centric analysis, or matching. Analysts match IOCs to evidence to identify suspicious or malicious activity, and then validate their findings.

Matching is not the only way to find intruders. More advanced NSM operations also pursue IOC-free analysis, or hunting. In the mid-2000s, the US Air Force popularized the term hunter-killer in the digital world. Security experts performed friendly force projection on their networks, examining data and sometimes occupying the systems themselves in order to find advanced threats. 

Today, NSM professionals like David Bianco and Aaron Wade promote network “hunting trips,” during which a senior investigator with a novel way to detect intruders guides junior analysts through data and systems looking for signs of the adversary. 

Upon validating the technique (and responding to any enemy actions), the hunters incorporate the new detection method into a CIRT’s IOC-centric operations. (emphasis added)

Let's consider Chris Sanders' blog post titled Threat Hunting for HTTP User Agents as an example of my definition of hunting. 

I will build a "hunting profile" via excerpts (in italics) from his post:

Assumption: "Attackers frequently use HTTP to facilitate malicious network communication."

Hypothesis: If I find an unusual user agent string in HTTP traffic, I may have discovered an attacker.

Question: “Did any system on my network communicate over HTTP using a suspicious or unknown user agent?”

Method: "This question can be answered with a simple aggregation wherein the user agent field in all HTTP traffic for a set time is analyzed. I’ve done this using Sqrrl Query Language here:

SELECT COUNT(*),user_agent FROM HTTPProxy GROUP BY user_agent ORDER BY COUNT(*) ASC LIMIT 20

This query selects the user_agent field from the HTTPProxy data source and groups and counts all unique entries for that field. The results are sorted by the count, with the least frequent occurrences at the top."

Results: Chris offers advice on how to interpret the various user agent strings produced by the query.

This is the critical part: Chris did not say "look for *this user agent*. He offered the reader an assumption, a hypothesis, a question, and a method. It is up to the defender to investigate the results. This, for me, is true hunting.

If Chris had instead referred users to this list of malware user agents (for example) and said look for "Mazilla/4.0", then I consider that manual (human) matching. If I created a Snort or Suricata rule to look for that user agent, then I consider that automated (machine) matching.

This is where my threat hunting definition likely diverges from modern practice. Analyst Z sees the results of Chris' hunt and thinks "Chris found user agent XXXX to be malicious, so I should go look for it." Analyst Z queries his or her data and does or does not find evidence of user agent XXXX.

I do not consider analyst Z's actions to be hunting. I consider it matching. There is nothing wrong with this. In fact, one of the purposes of hunting is to provide new inputs to the matching process, so that future hunting trips can explore new assumptions, hypotheses, questions, and methods, and let the machines do the matching on IOCs already found to be suggestive of adversary activity. This is why I wrote in my 2013 book "Upon validating the technique (and responding to any enemy actions), the hunters incorporate the new detection method into a CIRT’s IOC-centric operations."

The term "hunting" is a victim of its own success, with emotional baggage. We defenders have finally found a way to make "blue team" work appealing to the wider security community. Vendors love this new way to market their products. "If you're not hunting, are you doing anything useful?" one might ask.

Compared to "I'm threat hunting!" (insert chest beating), the alternative, "I'm matching!" (womp womp), seems sad. 

Nevertheless, we must remember that threat hunting methodologies were invented to find adversary activity for which there were no IOCs. Hunting was IOC-free analysis because we didn't know what to look for. Once you know what to look for, you are matching. Both forms of detection require analysis to validate adversary activity, of course. Let's not forget that.

I'm also very thankful, however it's defined or packaged, that people are excited to search for adversary activity in their environment, whether via matching or hunting. It's a big step from the mindset of 10 years ago, which had a "prevention works" milieu.

tl;dr Because TTPs are a form of IOC, then detection via matching IOCs is a form of matching, and not hunting.

Friday, November 23, 2018

More on Threat Hunting

Earlier this week hellor00t asked via Twitter:

Where would you place your security researchers/hunt team?

I replied:

For me, "hunt" is just a form of detection. I don't see the need to build a "hunt" team. IR teams detect intruders using two major modes: matching and hunting. Junior people spend more time matching. Senior people spend more time hunting. Both can and should do both functions.

This inspired Rob Lee to blog a response, from which I extract his core argument:

[Hunting] really isn’t, to me, about detecting threats...

Hunting is a hypothesis-led approach to testing your environment for threats. The purpose, to me, is not in finding threats but in determining what gaps you have in your ability to detect and respond to them...

In short, hunting, to me, is a way to assess your security (people, process, and technology) against threats while extending your automation footprint to better be prepared in the future. Or simply stated, it’s incident response without the incident that’s done with a purpose and contributes something. 

As background for my answer, I recommend my March 2017 post The Origin of Threat Hunting, which cites my article "Become a Hunter," published in the July-August 2011 issue of Information Security Magazine. I wrote it in the spring of 2011, when I was director of incident response for GE-CIRT.

For the term "hunting," I give credit to briefers from the Air Force and NSA who, in the mid-2000s briefed "hunter-killer" missions to the Red Team/Blue Team Symposium at the Johns Hopkins University Applied Physics Lab in Laurel, MD.

As a comment to that post, Tony Sager, who ran NSA VAO at the time I was briefed at ReBl, described hunting thus:

[Hunting] was an active and sustained search for Attackers...

For us, "Hunt" meant a very planned and sustained search, taking advantage of the existing infrastructure of Red/Blue Teams and COMSEC Monitoring, as well as intelligence information to guide the search. 

For the practice of hunting, as I experienced it, I give credit to our GE-CIRT incident handlers -- David Bianco,  Ken Bradley, Tim Crothers, Tyler Hudak, Bamm Visscher, and Aaron Wade -- who took junior analysts on "hunting trips," starting in 2008-2009.

It is very clear, to me, that hunting has always been associated with detecting an adversary, not "determining what gaps you have in your ability to detect and respond to them," as characterized by Rob.

For me, Rob is describing the job of an enterprise visibility architect, which I described in a 2007 post:

[W]e are stuck with numerous platforms, operating systems, applications, and data (POAD) for which we have zero visibility. 

I suggest that enterprises consider hiring or assigning a new role -- Enterprise Visibility Architect. The role of the EVA is to identify visibility deficiencies in existing and future POAD and design solutions to instrument these resources.

A primary reason to hire an enterprise visibility architect is to build visibility in, which I described in several posts, including this one from 2009 titled Build Visibility In. As a proponent of the "monitor first" school, I will always agree that it is important to identify and address visibility gaps.

So where do we go from here?

Tony Sager, as one of my wise men, offers sage advice at the conclusion of his comment:

"Hunt" emerged as part of a unifying mission model for my Group in the Information Assurance Directorate at NSA (the defensive mission) in the mid-late 2000's. But it was also a way to unify the relationship between IA and the SIGINT mission - intelligence as the driver for Hunting. The marketplace, of course, has now brought its own meaning to the term, but I just wanted to share some history. 

In my younger days I might have expressed much more energy and emotion when encountering a different viewpoint. At this point in my career, I'm more comfortable with other points of view, so long as they do not result in harm, or a waste of my taxpayer dollars, or other clearly negative consequences. I also appreciate the kind words Rob offered toward my point of view.

tl;dr I believe the definition and practice of hunting has always been tied to adversaries, and that Rob describes the work of an enterprise visibility architect when he focuses on visibility gaps rather than adversary activity.

Update 1: If in the course of conducting a hunt you identify a visibility or resistance deficiency, that is indeed beneficial. The benefit, however, is derivative. You hunt to find adversaries. Identifying gaps is secondary although welcome.

The same would be true of hunting and discovering misconfigured systems, or previously unidentified assets, or unpatched software, or any of the other myriad facts on the ground that manifest when one applies Clausewitz's directed telescope towards their computing environment.