Tuesday, September 18, 2018

Firewalls and the Need for Speed

I was looking for resources on campus network design and found these slides (pdf) from a 2011 Network Startup Resource Center presentation. These two caught my attention:



This bothered me, so I Tweeted about it.

This started some discussion, and prompted me to see what NSRC suggests for architecture these days. You can find the latest, from April 2018, here. Here is the bottom line for their suggested architecture:






What do you think of this architecture?

My Tweet has attracted some attention from the high speed network researcher community, some of whom assume I must be a junior security apprentice who equates "firewall" with "security." Long-time blog readers will laugh at that, like I did. So what was my problem with the original recommendation, and what problems do I have (if any) with the 2018 version?

First, let's be clear that I have always differentiated between visibility and control. A firewall is a poor visibility tool, but it is a control tool. It controls inbound or outbound activity according to its ability to perform in-line traffic inspection. This inline inspection comes at a cost, which is the major concern of those responding to my Tweet.

Notice how the presentation author thinks about firewalls. In the slides above, from the 2018 version, he says "firewalls don't protect users from getting viruses" because "clicked links while browsing" and "email attachments" are "both encrypted and firewalls won't help." Therefore, "since firewalls don't really protect users from viruses, let's focus on protecting critical server assets," because "some campuses can't develop the political backing to remove firewalls for the majority of the campus."

The author is arguing that firewalls are an inbound control mechanism, and they are ill-suited for the most prevalent threat vectors for users, in his opinion: "viruses," delivered via email attachment, or "clicked links."

Mail administrators can protect users from many malicious attachments. Desktop anti-virus can protect users from many malicious downloads delivered via "clicked links." If that is your worldview, of course firewalls are not important.

His argument for firewalls protecting servers is, implicitly, that servers may offer services that should not be exposed to the Internet. Rather than disabling those services, or limiting access via identity or local address restrictions, he says a firewall can provide that inbound control.

These arguments completely miss the point that firewalls are, in my opinion, more effective as an outbound control mechanism. For example, a firewall helps restrict adversary access to his victims when they reach outbound to establish post-exploitation command and control. This relies on the firewall identifying the attempted C2 as being malicious. To the extent intruders encrypt their C2 (and sites fail to inspect it) or use covert mechanisms (e.g., C2 over Twitter), firewalls will be less effective.

The previous argument assumes admins rely on the firewall to identify and block malicious outbound activity. Admins might alternatively identify the activity themselves, and direct the firewall to block outbound activity from designated compromised assets or to designated adversary infrastructure.

As some Twitter responders said, it's possible to do some or all of this without using a stateful firewall. I'm aware of the cool tricks one can play with routing to control traffic. Ken Meyers and I wrote about some of these approaches in 2005 in my book Extrusion Detection. See chapter 5, "Layer 3 Network Access Control."

Implementing these non-firewall-based security choices requries a high degree of diligence, which requires visibility. I did not see this emphasized in the NSRC presentation. For example:


These are fine goals, but I don't equate "manageability" with visibility or security. I don't think "problems and viruses" captures the magnitude of the threat to research networks.

The core of the reaction to my original Tweet is that I don't appreciate the need for speed in research networks. I understand that. However, I can't understand the requirement for "full bandwidth, un-filtered access to the Internet." That is a recipe for disaster.

On the other hand, if you define partner specific networks, and allow essentially site-to-site connectivity with exquisite network security monitoring methods and operations, then I do not have a problem with eliminating firewalls from the architecture. I do have a problem with unrestricted access to adversary infrastructure.

I understand that security doesn't exist to serve itself. Security exists to enable an organizational mission. Security must be a partner in network architecture design. It would be better to emphasize enhance monitoring for the networks discussed above, and think carefully about enabling speed without restrictions. The NSRC resources on the science DMZ merit consideration in this case.

Tuesday, September 11, 2018

Twenty Years of Network Security Monitoring: From the AFCERT to Corelight

I am really fired up to join Corelight. I’ve had to keep my involvement with the team a secret since officially starting on July 20th. Why was I so excited about this company? Let me step backwards to help explain my present situation, and forecast the future.

Twenty years ago this month I joined the Air Force Computer Emergency Response Team (AFCERT) at then-Kelly Air Force Base, located in hot but lovely San Antonio, Texas. I was a brand new captain who thought he knew about computers and hacking based on experiences from my teenage years and more recent information operations and traditional intelligence work within the Air Intelligence Agency. I was desperate to join any part of the then-five-year-old Information Warfare Center (AFIWC) because I sensed it was the most exciting unit on “Security Hill.”

I had misjudged my presumed level of “hacking” knowledge, but I was not mistaken about the exciting life of an AFCERT intrusion detector! I quickly learned the tenets of network security monitoring, enabled by the custom software watching and logging network traffic at every Air Force base. I soon heard there were three organizations that intruders knew to be wary of in the late 1990s: the Fort, i.e. the National Security Agency; the Air Force, thanks to our Automated Security Incident Measurement (ASIM) operation; and the University of California, Berkeley, because of a professor named Vern Paxson and his Bro network security monitoring software.

When I wrote my first book in 2003-2004, The Tao of Network Security Monitoring, I enlisted the help of Christopher Jay Manders to write about Bro 0.8. Bro had the reputation of being very powerful but difficult to stand up. In 2007 I decided to try installing Bro myself, thanks to the introduction of the “brolite” scripts shipped with Bro 1.2.1. That made Bro easier to use, but I didn’t do much analysis with it until I attended the 2009 Bro hands-on workshop. There I met Vern, Robin Sommer, Seth Hall, Christian Kreibich, and other Bro users and developers. I was lost most of the class, saved only by my knowledge of standard Unix command line tools like sed, awk, and grep! I was able to integrate Bro traffic analysis and logs into my TCP/IP Weapons School 2.0 class, and subsequent versions, which I taught mainly to Black Hat students. By the time I wrote my last book, The Practice of Network Security Monitoring, in 2013, I was heavily relying on Bro logs to demonstrate many sorts of network activity, thanks to the high-fidelity nature of Bro data.

In July of this year, Seth Hall emailed to ask if I might be interested in keynoting the upcoming Bro users conference in Washington, D.C., on October 10-12. I was in a bad mood due to being unhappy with the job I had at that time, and I told him I was useless as a keynote speaker. I followed up with another message shortly after, explained my depressed mindset, and asked how he liked working at Corelight. That led to interviews with the Corelight team and a job offer. The opportunity to work with people who really understood the need for network security monitoring, and were writing the world’s most powerful software to generate NSM data, was so appealing! Now that I’m on the team, I can share how I view Corelight’s contribution to the security challenges we face.

For me, Corelight solves the problems I encountered all those years ago when I first looked at Bro. The Corelight embodiment of Bro is ready to go when you deploy it. It’s developed and maintained by the people who write the code. Furthermore, Bro is front and center, not buried behind someone else’s logo. Why buy this amazing capability from another company when you can work with those who actually conceptualize, develop, and publish the code?

It’s also not just Bro, but it’s Bro at ridiculous speeds, ingesting and making sense of complex network traffic. We regularly encounter open source Bro users who spend weeks or months struggling to get their open source deployments to run at the speeds they need, typically in the tens or hundreds of Gbps. Corelight’s offering is optimized at the hardware level to deliver the highest performance, and our team works with customers who want to push Bro to the even greater levels. 

Finally, working at Corelight gives me the chance to take NSM in many exciting new directions. For years we NSM practitioners have worried about challenges to network-centric approaches, such as encryption, cloud environments, and alert fatigue. At Corelight we are working on answers for all of these, beyond the usual approaches — SSL termination, cloud gateways, and SIEM/SOAR solutions. We will have more to say about this in the future, I’m happy to say!

What challenges do you hope Corelight can solve? Leave a comment or let me know via Twitter to @corelight_inc or @taosecurity.

Sunday, July 22, 2018

Defining Counterintelligence

I've written about counterintelligence (CI) before, but I realized today that some of my writing, and the writing of others, may be confused as to exactly what CI means.

The authoritative place to find an American definition for CI is the United States National Counterintelligence and Security Center. I am more familiar with the old name of this organization, the  Office of the National Counterintelligence Executive (ONCIX).

The 2016 National Counterintelligence Strategy cites Executive Order 12333 (as amended) for its definition of CI:

Counterintelligence – Information gathered and activities conducted to identify, deceive,
exploit, disrupt, or protect against espionage, other intelligence activities, sabotage, or assassinations conducted for or on behalf of foreign powers, organizations, or persons, or their agents, or international terrorist organizations or activities. (emphasis added)

The strict interpretation of this definition is countering foreign nation state intelligence activities, such as those conducted by China's Ministry of State Security (MSS), the Foreign Intelligence Service of the Russian Federation (SVR RF), Iran's Ministry of Intelligence, or the military intelligence services of those countries and others.

In other words, counterintelligence is countering foreign intelligence. The focus is on the party doing the bad things, and less on what the bad thing is.

The definition, however, is loose enough to encompass others; "organizations," "persons," and "international terrorist organizations" are in scope, according to the definition. This is just about everyone, although criminals are explicitly not mentioned.

The definition is also slightly unbounded by moving beyond "espionage, or other intelligence activities," to include "sabotage, or assassinations." In those cases, the assumptions is that foreign intelligence agencies and their proxies are the parties likely to be conducting sabotage or assassinations. In the course of their CI work, paying attention to foreign intelligence agents, the CI team may encounter plans for activities beyond collection.

The bottom line for this post is a cautionary message. It's not appropriate to call all intelligence activities "counterintelligence." It's more appropriate to call countering adversary intelligence activities counterintelligence.

You may use similar or the same approaches as counterintelligence agents when performing your cyber threat intelligence function. For example, you may recruit a source inside a carding forum, or you may plant your own source in a carding forum. This is similar to turning a foreign intelligence agent, or inserting your own agent in a foreign intelligence service. However, activities directing against a carding forum are not counterintelligence. Activities directing against a foreign intelligence service are counterintelligence.

The nature and target of your intelligence activities are what determine if it is counterintelligence, not necessarily the methods you use. Again, this is in keeping with the stricter definition, and not becoming a victim of scope creep.


Thursday, June 28, 2018

Why Do SOCs Look Like This?

When you hear the word "SOC," or the phrase "security operations center," what image comes to mind? Do you think of analyst sitting at desks, all facing forward, towards giant screens? Why is this?

The following image is from the outstanding movie Apollo 13, a docudrama about the challenged 1970 mission to the moon.


It's a screen capture from the go for launch sequence. It shows mission control in Houston, Texas. If you'd like to see video of the actual center from 1970, check out This Is Mission Control.

Mission control looks remarkably like a SOC, doesn't it? When builders of computer security operations centers imagined what their "mission control" rooms would look like, perhaps they had Houston in mind?

Or perhaps they thought of the 1983 movie War Games?


Reality was way more boring however:


I visited NORAD under Cheyenne Mountain in 1989, I believe, when visiting the Air Force Academy as a high school senior. I can confirm it did not look like the movie depiction!

Let's return to mission control. Look at the resources available to personnel manning the mission control room. The big screens depict two main forms of data: telemetry and video of the rocket. What about the individual screens, where people sit? They are largely customized. Each station presents data or buttons specific to the role of the person sitting there. Listen to Ed Harris' character calling out the stations: booster, retro, vital, etc. For example:


This is one of the key differences between mission control and any modern computerized operations center. In the 1960s and 1970s, workstations (literally, places where people worked) had to be customized. They lacked the technology to have generic workstations where customization was done via screen, keyboard, and mouse. They also lacked the ability to display video on demand, and relied on large television screens. Personnel with specific functions sat at specific locations, because that was literally the only way they could perform their jobs.

With the advent of modern computing, every workstation is instantly customizable. There is no need to specialize. Anyone can sit anywhere, assuming computers allow one's workspace to follow their logon. In fact, modern computing allows a user to sit in spaces outside of their office. A modern mission control could be distributed.

With that in mind, what does the current version of mission control look like? Here is a picture of the modern Johnson Space Center's mission control room.



It looks similar to the 1960s-1970s version, except it's dominated by screens, keyboards, and mice.

What strikes me about every image of a "SOC" that I've ever seen is that no one is looking at the big screens. They are almost always deployed for an audience. No one in an operational role looks at them.

There are exceptions. Check out the Arizona Department of Transportation operations center.


Their "big screen" is a composite of 24 smaller screens showing traffic and roadways. No one is looking at the screen, but that sort of display is perfect for the human eye.

It's a variant of Edward Tufte's "small multiple" idea. There is no text. The eye can discern if there is a lot of traffic, or little traffic, or an accident pretty easily. It's likely more for the benefit of an audience, but it works decently well.

Compare those screens to what one is likely to encounter in a cyber SOC. In addition to a "pew pew" map and a "spinning globe of doom," it will likely look like this, from R3 Cybersecurity:


The big screens are a waste of time. No one is standing near them. No one sitting at their workstations can read what the screens show. They are purely for an audience, who can't discern what they show either.

The bottom line for this post is that if you're going to build a "SOC," don't build it based on what you've seen in the movies, or in other industries, or what a consultancy recommends. Spend some time determining your SOC's purpose, and let the workflow drive the physical setting. You may determine you don't even need a "SOC," either physically or logically, based on maturing understandings of a SOC's mission. That's a topic for a future post!

Monday, June 25, 2018

Bejtlich on the APT1 Report: No Hack Back

Before reading the rest of this post, I suggest reading Mandiant/FireEye's statement Doing Our Part -- Without Hacking Back.

I would like to add my own color to this situation.

First, at no time when I worked for Mandiant or FireEye, or afterwards, was there ever a notion that we would hack into adversary systems. During my six year tenure, we were publicly and privately a "no hack back" company. I never heard anyone talk about hack back operations. No one ever intimated we had imagery of APT1 actors taken with their own laptop cameras. No one even said that would be a good idea.

Second, I would never have testified or written, repeatedly, about our company's stance on not hacking back if I knew we secretly did otherwise. I have quit jobs because I had fundamental disagreements with company policy or practice. I worked for Mandiant from 2011 through the end of 2013, when FireEye acquired Mandiant, and stayed until last year (2017). I never considered quitting Mandiant or FireEye due to a disconnect between public statements and private conduct.

Third, I was personally involved with briefings to the press, in public and in private, concerning the APT1 report. I provided the voiceover for a 5 minute YouTube video called APT1: Exposing One of China's Cyber Espionage Units. That video was one of the most sensitive, if not the most sensitive, aspects of releasing the report. We showed the world how we could intercept adversary communications and reconstruct it. There was internal debate about whether we should do that. We decided to cover the practice in the report, as Christopher Glyer Tweeted:


In none of these briefings to the press did we show pictures or video from adversary laptops. We did show the video that we published to YouTube.

Fourth, I privately contacted former Mandiant personnel with whom I worked during the time of the APT1 report creation and distribution. Their reaction to Mr Sanger's allegations ranged from "I've never heard of that" to "completely false." I asked former Mandiant colleagues, like myself, in the event that current Mandiant or FireEye employees were told not to talk to outsiders about the case.

What do I think happened here? I agree with the theory that Mr Sanger misinterpreted the reconstructed RDP sessions for some sort of "camera access." I have no idea about the "bros" or "leather jackets" comments!

In the spirit of full disclosure, prior to publication, Mr Sanger tried to reach me to discuss his book via email. I was sick and told him I had to pass. Ellen Nakashima also contacted me; I believe she was doing research for the book. She asked a few questions about the origin of the term APT, which I answered. I do not have the book so I do not know if I am cited, or if my message was included.

The bottom line is that Mandiant and FireEye did not conduct any hack back for the APT1 report.

Update: Some of you wondered about Ellen's role. I confirmed last night that she was working on her own project.

Tuesday, May 15, 2018

Bejtlich Joining Splunk


Since posting Bejtlich Moves On I've been rebalancing work, family, and personal life. I invested in my martial arts interests, helped more with home duties, and consulted through TaoSecurity.

Today I'm pleased to announce that, effective Monday May 21st 2018, I'm joining the Splunk team. I will be Senior Director for Security and Intelligence Operations, reporting to our CISO, Joel Fulton. I will help build teams to perform detection and monitoring operations, digital forensics and incident response, and threat intelligence. I remain in the northern Virginia area and will align with the Splunk presence in Tyson's Corner.

I'm very excited by this opportunity for four reasons. First, the areas for which I will be responsible are my favorite aspects of security. Long-time blog readers know I'm happiest detecting and responding to intruders! Second, I already know several people at the company, one of whom began this journey by Tweeting about opportunities at Splunk! These colleagues are top notch, and I was similarly impressed by the people I met during my interviews in San Francisco and San Jose.

Third, I respect Splunk as a company. I first used the products over ten years ago, and when I tried them again recently they worked spectacularly, as I expected. Fourth, my new role allows me to be a leader in the areas I know well, like enterprise defense and digital operational art, while building understanding in areas I want to learn, like cloud technologies, DevOps, and security outside enterprise constraints.

I'll have more to say about my role and team soon. Right now I can share that this job focuses on defending the Splunk enterprise and its customers. I do not expect to spend a lot of time in sales cycles. I will likely host visitors in the Tyson's areas from time to time. I do not plan to speak as much with the press as I did at Mandiant and FireEye. I'm pleased to return to operational defense, rather than advise on geopolitical strategy.

If this news interests you, please check our open job listings in information technology. As a company we continue to grow, and I'm thrilled to see what happens next!

Monday, May 07, 2018

Trying Splunk Cloud

I first used Splunk over ten years ago, but the first time I blogged about it was in 2008. I described how to install Splunk on Ubuntu 8.04. Today I decided to try the Splunk Cloud.

Splunk Cloud is the company's hosted Splunk offering, residing in Amazon Web Services (AWS). You can register for a 15 day free trial of Splunk Cloud that will index 5 GB per day.

If you would like to follow along, you will need a computer with a Web browser to interact with Splunk Cloud. (There may be ways to interact via API, but I do not cover that here.)

I will collect logs from a virtual machine running Debian 9, inside Oracle VirtualBox.

First I registered for the free Splunk Cloud trial online.

After I had a Splunk Cloud instance running, I consulted the documentation for Forward data to Splunk Cloud from Linux. I am running a "self-serviced" instance and not a "managed instance," i.e., I am the administrator in this situation.

I learned that I needed to install a software package called the Splunk Universal Forwarder on my Linux VM.

I downloaded a 64 bit Linux 2.6+ kernel .deb file to the /home/Downloads directory on the Linux VM.

richard@debian:~$ cd Downloads/

richard@debian:~/Downloads$ ls

splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-amd64.deb

With elevation permissions I created a directory for the .deb, changed into the directory, and installed the .deb using dpkg.

richard@debian:~/Downloads$ sudo bash
[sudo] password for richard: 

root@debian:/home/richard/Downloads# mkdir /opt/splunkforwarder

root@debian:/home/richard/Downloads# mv splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-amd64.deb /opt/splunkforwarder/

root@debian:/home/richard/Downloads# cd /opt/splunkforwarder/

root@debian:/opt/splunkforwarder# ls

splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-amd64.deb

root@debian:/opt/splunkforwarder# dpkg -i splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-amd64.deb 

Selecting previously unselected package splunkforwarder.
(Reading database ... 141030 files and directories currently installed.)
Preparing to unpack splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-amd64.deb ...
Unpacking splunkforwarder (7.1.0) ...
Setting up splunkforwarder (7.1.0) ...
complete

root@debian:/opt/splunkforwarder# ls
bin        license-eula.txt
copyright.txt  openssl
etc        README-splunk.txt
ftr        share
include        splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-amd64.deb
lib        splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest

Next I changed into the bin directory, ran the splunk binary, and accepted the EULA.

root@debian:/opt/splunkforwarder# cd bin/

root@debian:/opt/splunkforwarder/bin# ls

btool   copyright.txt   openssl slim   splunkmon
btprobe   genRootCA.sh   pid_check.sh splunk   srm
bzip2   genSignedServerCert.sh  scripts splunkd
classify  genWebCert.sh   setSplunkEnv splunkdj

root@debian:/opt/splunkforwarder/bin# ./splunk start

SPLUNK SOFTWARE LICENSE AGREEMENT

THIS SPLUNK SOFTWARE LICENSE AGREEMENT ("AGREEMENT") GOVERNS THE LICENSING,
INSTALLATION AND USE OF SPLUNK SOFTWARE. BY DOWNLOADING AND/OR INSTALLING SPLUNK
SOFTWARE: (A) YOU ARE INDICATING THAT YOU HAVE READ AND UNDERSTAND THIS

...

Splunk Software License Agreement 04.24.2018

Do you agree with this license? [y/n]: y

Now I had to set an administrator password for this Universal Forwarder instance. I will refer to it as "mypassword" in the examples that follow although Splunk does not echo it to the screen below.

This appears to be your first time running this version of Splunk.

An Admin password must be set before installation proceeds.
Password must contain at least:
   * 8 total printable ASCII character(s).
Please enter a new password: 
Please confirm new password: 

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Done

With that done, I had to return to the Splunk Cloud Web site, and click the link to "Download Universal Forwarder Credentials" to download a splunkclouduf.spl file. As noted in the documentation, splunkclouduf.spl is a "credentials file, which contains a custom certificate for your Splunk Cloud deployment. The universal forwarder credentials are different from the credentials that you use to log into Splunk Cloud."

After downloading the splunkclouduf.spl file, I installed it. Note I pass "admin" as the user and "mypassword" as the password here. After installing I restart the universal forwarder.

root@debian:/opt/splunkforwarder/bin# ./splunk install app /home/richard/Downloads/splunkclouduf.spl -auth admin:mypassword

App '/home/richard/Downloads/splunkclouduf.spl' installed 

root@debian:/opt/splunkforwarder/bin# ./splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
.......
Stopping splunk helpers...

Done.

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Done

It's time to take the final steps to get data into Splunk Cloud. I need to forwarder management in the Splunk Cloud Web site. Observe the input-prd-p-XXXX.cloud.splunk.com in the command. You obtain this (mine is masked with XXXX) from the URL for your Splunk Cloud deployment, e.g., https://prd-p-XXXX.cloud.splunk.com. Note that you have to add "input-" before the fully qualified domain name used by the Splunk Cloud instance.

root@debian:/opt/splunkforwarder/bin# ./splunk set deploy-poll input-prd-p-XXXX.cloud.splunk.com:8089

Your session is invalid.  Please login.
Splunk username: admin
Password: 
Configuration updated.

Once again I restart the universal forwarder. I'm not sure if I could have done all these restarts at the end.

root@debian:/opt/splunkforwarder/bin# ./splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
.......
Stopping splunk helpers...

Done.

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Done

Finally I need to tell the universal forwarder to watch some logs on this Linux system. I tell it to monitor the /var/log directory and restart one more time.

root@debian:/opt/splunkforwarder/bin# ./splunk add monitor /var/log
Your session is invalid.  Please login.
Splunk username: admin
Password: 
Added monitor of '/var/log'.

root@debian:/opt/splunkforwarder/bin# ./splunk restart

Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
...............
Stopping splunk helpers...

Done.

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Done

At this point I return to the Splunk Cloud Web interface and click the "search" feature. I see Splunk is indexing some data.


I run a search for "host=debian" and find my logs.


Not too bad! Have you tried Splunk Cloud? What do you think? Leave me a comment below.

Update: I installed the Universal Forwarder on FreeBSD 11.1 using the method above (except with a FreeBSD .tgz) and everything seems to be working!

Monday, February 26, 2018

Importing Pcap into Security Onion

Within the last week, Doug Burks of Security Onion (SO) added a new script that revolutionizes the use case for his amazing open source network security monitoring platform.

I have always used SO in a live production mode, meaning I deploy a SO sensor sniffing a live network interface. As the multitude of SO components observe network traffic, they generate, store, and display various forms of NSM data for use by analysts.

The problem with this model is that it could not be used for processing stored network traffic. If one simply replayed the traffic from a .pcap file, the new traffic would be assigned contemporary timestamps by the various tools observing the traffic.

While all of the NSM tools in SO have the independent capability to read stored .pcap files, there was no unified way to integrate their output into the SO platform.

Therefore, for years, there has not been a way to import .pcap files into SO -- until last week!

Here is how I tested the new so-import-pcap script. First, I made sure I was running Security Onion Elastic Stack Release Candidate 2 (14.04.5.8 ISO) or later. Next I downloaded the script using wget from https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-import-pcap.

I continued as follows:

richard@so1:~$ sudo cp so-import-pcap /usr/sbin/

richard@so1:~$ sudo chmod 755 /usr/sbin/so-import-pcap

I tried running the script against two of the sample files packaged with SO, but ran into issues with both.

richard@so1:~$ sudo so-import-pcap /opt/samples/10k.pcap

so-import-pcap

Please wait while...
...creating temp pcap for processing.
mergecap: Error reading /opt/samples/10k.pcap: The file appears to be damaged or corrupt
(pcap: File has 263718464-byte packet, bigger than maximum of 262144)
Error while merging!

I checked the file with capinfos.

richard@so1:~$ capinfos /opt/samples/10k.pcap
capinfos: An error occurred after reading 17046 packets from "/opt/samples/10k.pcap": The file appears to be damaged or corrupt.
(pcap: File has 263718464-byte packet, bigger than maximum of 262144)

Capinfos confirmed the problem. Let's try another!

richard@so1:~$ sudo so-import-pcap /opt/samples/zeus-sample-1.pcap

so-import-pcap

Please wait while...
...creating temp pcap for processing.
mergecap: Error reading /opt/samples/zeus-sample-1.pcap: The file appears to be damaged or corrupt
(pcap: File has 1984391168-byte packet, bigger than maximum of 262144)
Error while merging!

Another bad file. Trying a third!

richard@so1:~$ sudo so-import-pcap /opt/samples/evidence03.pcap

so-import-pcap

Please wait while...
...creating temp pcap for processing.
...setting sguild debug to 2 and restarting sguild.
...configuring syslog-ng to pick up sguild logs.
...disabling syslog output in barnyard.
...configuring logstash to parse sguild logs (this may take a few minutes, but should only need to be done once)...done.
...stopping curator.
...disabling curator.
...stopping ossec_agent.
...disabling ossec_agent.
...stopping Bro sniffing process.
...disabling Bro sniffing process.
...stopping IDS sniffing process.
...disabling IDS sniffing process.
...stopping netsniff-ng.
...disabling netsniff-ng.
...adjusting CapMe to allow pcaps up to 50 years old.
...analyzing traffic with Snort.
...analyzing traffic with Bro.
...writing /nsm/sensor_data/so1-eth1/dailylogs/2009-12-28/snort.log.1261958400

Import complete!

You can use this hyperlink to view data in the time range of your import:
https://localhost/app/kibana#/dashboard/94b52620-342a-11e7-9d52-4f090484f59e?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2009-12-28T00:00:00.000Z',mode:absolute,to:'2009-12-29T00:00:00.000Z'))

or you can manually set your Time Range to be:
From: 2009-12-28    To: 2009-12-29


Incidentally here is the capinfos output for this trace.

richard@so1:~$ capinfos /opt/samples/evidence03.pcap
File name:           /opt/samples/evidence03.pcap
File type:           Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
Packet size limit:   file hdr: 65535 bytes
Number of packets:   1778
File size:           1537 kB
Data size:           1508 kB
Capture duration:    171 seconds
Start time:          Mon Dec 28 04:08:01 2009
End time:            Mon Dec 28 04:10:52 2009
Data byte rate:      8814 bytes/s
Data bit rate:       70 kbps
Average packet size: 848.57 bytes
Average packet rate: 10 packets/sec
SHA1:                34e5369c8151cf11a48732fed82f690c79d2b253
RIPEMD160:           afb2a911b4b3e38bc2967a9129f0a11639ebe97f
MD5:                 f8a01fbe84ef960d7cbd793e0c52a6c9
Strict time order:   True

That worked! Now to see what I can find in the SO interface.

I accessed the Kibana application and changed the timeframe to include those in the trace.


Here's another screenshot. Again I had to adjust for the proper time range.


Very cool! However, I did not find any IDS alerts. This made me wonder if there was a problem with alert processing. I decided to run the script on a new .pcap:

richard@so1:~$ sudo so-import-pcap /opt/samples/emerging-all.pcap

so-import-pcap

Please wait while...
...creating temp pcap for processing.
...analyzing traffic with Snort.
...analyzing traffic with Bro.
...writing /nsm/sensor_data/so1-eth1/dailylogs/2010-01-27/snort.log.1264550400

Import complete!

You can use this hyperlink to view data in the time range of your import:
https://localhost/app/kibana#/dashboard/94b52620-342a-11e7-9d52-4f090484f59e?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2010-01-27T00:00:00.000Z',mode:absolute,to:'2010-01-28T00:00:00.000Z'))

or you can manually set your Time Range to be:
From: 2010-01-27    To: 2010-01-28

When I searched the interface for NIDS alerts (after adjusting the time range), I found results:


The alerts show up in Sguil, too!



This is a wonderful development for the Security Onion community. Being able to import .pcap files and analyze them with the standard SO tools and processes, while preserving timestamps, makes SO a viable network forensics platform.

This thread in the mailing list is covering the new script.

I suggest running on an evaluation system, probably in a virtual machine. I did all my testing on Virtual Box. Check it out! 

Monday, January 22, 2018

Lies and More Lies

Following the release of the Spectre and Meltdown CPU attacks, the security community wondered if other researchers would find related speculative attack problems. When the following appeared, we were concerned:

"Skyfall and Solace

More vulnerabilities in modern computers.

Following the recent release of the Meltdown and Spectre vulnerabilities, CVE-2017-5175, CVE-2017-5753 and CVE-2017-5754, there has been considerable speculation as to whether all the issues described can be fully mitigated. 

Skyfall and Solace are two speculative attacks based on the work highlighted by Meltdown and Spectre.

Full details are still under embargo and will be published soon when chip manufacturers and Operating System vendors have prepared patches.

Watch this space..."

It turns out this was a hoax. The latest version of the site says, in part:

"With little more than a couple of quickly registered domain names, thousands of people were hooked...

Skyfall

The idea here was to suggest a link to Intel's Skylake processor.

Solace

The idea here was to suggest a link to the Solaris operating system.

Copy the styling of the original Meltdown and Spectre sites and add a couple of favicons based loosely on the Intel and Solaris logos and I was nearly done.

The final step was to add on https, because if a site's got an SSL certificate it must be legitimate, and the bait was set."

The problem with this "explanation" is that it wasn't just a logo, domain name and SSL certificate. The "security professional" who created this site outright lied, as shown at the top of this post. Don't fall for his false narrative.

I'm not naming names or linking to the sites here, because the person responsible already thinks he's too clever.

Tuesday, January 16, 2018

Addressing Innumeracy in Reporting

Anyone involved in cybersecurity reporting needs a strong sense of numeracy, or mathematical literacy. I see two sorts of examples of innumeracy repeatedly in the media.

The first involves the time value of money. Recently CNN claimed Amazon CEO Jeff Bezos was the "richest person in history" and Recode said Bezos was "now worth more than Bill Gates ever was." Thankfully both Richard Steinnon and Noah Kirsch recognized the foolishness of these reports, correctly noting that Bezos would only rank number 17 on a list where wealth was adjusted for inflation.

This failure to recognize the time value of money is pervasive. Just today I heard the host of a podcast claim that the 1998 Jackie Chan movie Rush Hour was "the top grossing martial arts film of all time." According to Box Office Mojo, Rush Hour earned $244,386,864 worldwide. Adjusting for inflation, in 2017 dollars that's $367,509,865.67 -- impressive!

For comparison, I researched the box office returns for Bruce Lee's Enter the Dragon. Box Office Mojo lacked data, but I found a 2017 article stating his 1973 movie earned "$25 million in the U.S. and $90 million worldwide, excluding Hong Kong." If I adjust the worldwide figure of $90 million for inflation, in 2017 dollars that's $496,864,864.86 -- making Enter the Dragon easily more successful than Rush Hour.

If you're wondering about Crouching Tiger, Hidden Dragon, that 2000 movie earned $213,525,736 worldwide. That movie earned less than Rush Hour, and arrived two years later, so it's not worth doing the inflation math.

The take-away is that any time you are comparing dollars from different time periods, you must adjust for inflation to have your comparisons have any meaning whatsoever.

Chart by @CanadianFlags
The second sort of innumeracy I'd like to highlight today also involves money, but in a slightly different way. This involves changes in values over time.

For example, a company may grow revenue from 2015 to 2016, with 2015 revenue being $100,000 and 2016 being $200,000. That's a 100% gain.

If the company grows another $100,000 from 2016 to 2017, from $200,000 to $300,000, the growth rate has declined to 50%. To have maintained a 100% growth rate, the company needed to make $400,000 in 2016.

That same $100,000 dollar increase isn't so great when compared to the new base value.

We see the same dynamic at play when tracking the growth of individual stocks or market indices over time.

CNN wrote a story about the 1,000 point rise in the Dow Jones Industrial Average over a period of 7 days, from 25,000 to 26,000. One person Tweeted the chart at the above right, asking "is that healthy?" My answer -- you need a proper chart!

My second reaction was "that's a jump, but it's only (1-(25000/26000)) = 3.8%. Yes, 3.8% in 7 days is a lot, but that doesn't even rate in the top 20 one-day percentage gains or losses over the life of the index.

If the DJIA gained 1,000 points in 7 days 5 years ago, when the market was at 13,649, a rise to 14,649 would be a 6.8% gain. 20 years ago the market was roughly 3,310, so a 1,000 point rise to 4,310 would be a massive 23.2% gain.

A better way to depict the growth in the DJIA would be to use a logarithmic chart. The charts below show a linear version on the top and a logarithmic version below it.



Using barcharts.com, I drew the last 30 years of the DJIA at the top using a linear Y axis, meaning there is equal distance between 2,000 and 4,000, 4,000 and 6,000, and so on. The blue line shows the slope of the growth.

I then drew the same period using a logarithmic Y axis, meaning the percentage gains from one line to another are equal. For example, a 100% increase from 1,000 to 2,000 occupies the same distance as the 100% increase from 5,000 to 10,000. The green line shows the slope of the growth.

I put the blue and green lines on both charts to permit comparison of the slopes. As you can see, the growth, when properly indicated using a log chart and the green line, is less than the exaggerations introduced by the linear chart blue line.

There is indeed an upturn recently in the log chart, but the growth is probably on trend over time.

While we're talking about the market, let's take one minute to smack down the old trope that "what comes up, must come down." There is no "law of gravity" in investing, at least for the US market, as a whole.

The best example I have seen of the reality of the situation is this 2017 article titled The Dow’s tumultuous 120-year history, in one chart. Here is the chart:

Chart by Chris Kacher, managing director of MoKa Investors

What an amazing story. The title of the article should not be gloomy. It should be triumphant. Despite two World Wars, a Cold War, wars in Korea, Vietnam, the Middle East, and elsewhere, assassinations of world leaders, market depressions and recessions, and so on, the trend line is up, and up in a big way. While the DJIA doesn't represent the entire US market, it captures enough of it to be representative. This is why I do not bet against the US market over the long term. (And yes I recognize that the market and the economy are different.)

Individual companies may disappear, and the DJIA has indeed been changed many times over the years. However, those changes were made so that the index roughly reflected the makeup of the economy. Is it perfect? No. Does it capture the overall directional trend line since 1896? Yes.

Please keep in mind these two sorts of innumeracy -- the time value of money, and the importance of percentage changes over time -- when dealing with numbers and time.

Sunday, January 14, 2018

Remembering When APT Became Public

Last week I Tweeted the following on the 8th anniversary of Google's blog post about its compromise by Chinese threat actors:

This intrusion made the term APT mainstream. I was the first to associate it with Aurora, in this post 

https://taosecurity.blogspot.com/2010/01/google-v-china.html

My first APT post was a careful reference in 2007, when we all feared being accused of "leaking classified" re China: 

https://taosecurity.blogspot.com/2007/10/air-force-cyberspace-report.html

I should have added the term "publicly" to my original Tweet. There were consultants with years of APT experience involved in the Google incident response, and they recognized the work of APT17 at that company and others. Those consultants honored their NDAs and have stayed quiet.

I wrote my original Tweet as a reminder that "APT" was not a popular, recognized term until the Google announcement on 12 January 2010. In my Google v China blog post I wrote:

Welcome to the party, Google. You can use the term "advanced persistent threat" (APT) if you want to give this adversary its proper name.

I also Tweeted a similar statement on the same day:

This is horrifying: http://bit.ly/7x7vVW Google admits intellectual property theft from China; it's called Advanced Persistent Threat, GOOG

I made the explicit link of China and APT because no one had done that publicly.

This slide from a 2011 briefing I did in Hawaii captures a few historical points:


The Google incident was a watershed, for reasons I blogged on 16 January 2010. I remember the SANS DFIR 2008 event as effectively "APTCon," but beyond Mandiant, Northrup Grumman, and NetWitness, no one was really talking publicly about the APT until after Google.

As I noted in the July 2009 blog post, You Down With APT? (ugh):

Aside from Northrup Grumman, Mandiant, and a few vendors (like NetWitness, one of the full capture vendors out there) mentioning APT, there's not much else available. A Google search for "advanced persistent threat" -netwitness -mandiant -Northrop yields 34 results (prior to this blog post). (emphasis added)

Today that search yields 244,000 results.

I would argue we're "past APT." APT was the buzzword for RSA and other vendor-centric events from, say, 2011-2015, with 2013 being the peak following Mandiant's APT1 report.

The threat hasn't disappeared, but it has changed. I wrote my Tweet to mark a milestone and to note that I played a small part in it.

All my APT posts here are reachable by this APT tag. Also see my 2010 article for Information Security Magazine titled What APT Is, and What It Isn't.