TaoSecurity: 03/01/2007

Saturday, March 31, 2007

Help Johnny Long Go to Uganda

Long-time readers of my blog know I severely limit the number of non-technical stories I write here. I've probably written less than a dozen in over four years. This one definitely deserves to be posted, however.

I shook hands with Johnny Long at ShmooCon last week, but we didn't get a chance to chat. If you don't know Johnny Long, you haven't paid attention to the scene during the last few years! In short, Johnny invented Google hacking, and he's one of the nicest guys you could meet at a security conference.

Today I received an email from Johnny stating that he and his wife Jen are flying to Uganda in May to do missionary work. He's working for AIDS Orphans Education Trust. In his usual low-key manner, he's asking for help. He didn't specifically ask people outside of his email addressees to help, but I figure there are a lot of people who could contribute a few dollars to help defray the costs he and his wife must bear to fly and live in Uganda.

His trip is going to cost $4200 and I can guarantee not a penny will be wasted. How often do you get a chance to personally assist someone you know? Johnny has decided to crawl out of his digital shell and try to make a difference in the real world. If you want to join me in helping Johnny and his wife, send a contribution via PayPal to johnny [at] ihackstuff [dot] com.

Thank you for your time.

Friday, March 30, 2007

Full Content Monitoring as a Wiretap

I received the following question today:

When installing Sguil, what legal battles have you fought/won about full packet capture and its vulnerability to open records requests from outside parties? I am getting concerns, from various management, regarding the legal ramifications of the installation of a system similar to Sguil in the state government arena. Do you have any advice for easing their worries? I know how important full data capture is to investigating incidents, and I consider it of paramount importance to the security of our state that we do so. Are there any legal precedents that can be cited?

Before I say anything else it is important to realize I am not a lawyer, I don't play one on YouTube, and I recommend you consult your lawyer rather than listen to anything I might say.

With that out of the way, I have written about wiretaps a few times before. Let me get these generic wiretapping issues out of the way before addressing the question specifically.

Understanding Legal Issues of Network Monitoring

Excellent Coverage of Wiretapping Issues at News.com

Two Great Wiretapping Articles

The pertinent Federal law is 18 U.S.C. §2511.

A great place to look for commentary and precedents on digital security issues is Orin Kerr's Computer Crime Case Updates. This search for wiretap may or may not be helpful.

Finally, for recent commentary by a lawyer (but not your lawyer), I recommend Sysadmins, Network Managers, and Wiretap Law (.pdf slides) by Alex Muentz. These notes from his LISA 2006 talk are helpful too.

I think the key element of the question originally posed was full packet capture and its vulnerability to open records requests from outside parties. It sounds like the question asker is worried about discoverability of full content data. I touched on this briefly in The Revolution Will Be Monitored.

My answer to this problem is what I would consider both practical and technically limiting: do not store more full content data than you need. For any modern production network, capturing and storing days or weeks of full content traffic can be an expensive proposition. For example, in one client location I have about 200 GB of space available for full content storage. That space allows me to save a little more than 10 days of full content, even with fairly draconian BPFs limiting what is stored. If for some reason I needed to produce that data to management or attorneys, I could only provide the last 10 days of information. If the event in question occured prior to that period, I just don't have it.

I do know of some locations that operate massive storage area networks to save TBs of full content. I do not advocate that for anyone but the most specialized of clients. I do recommend collecting the amount of full content (if possible, legally and technically) that works for your investigative window. For example, if you have a requirement to review your alert and session data such that you are never more than 5 days past an event of interest, you might want to save 7 days of full content. From an investigation point of view, more is always better. From a practical point of view, it might be too costly.

Remember that any network data collection should be considered a wiretap. Full content is the form of network data that most resembles a wiretap.

With respect to session data, I recommend saving as much of that as possible. In practical terms it comes down to the amount of space you're willing to devote to database files. At the same client I am collecting as many sessions as I can, without filters. 30 days of such session data is producing about 20 GB of uncompressed MySQL table files. As you can see I can store many more days of session data as compared to full content data. That means much more session data is discoverable. I might choose to limit storage of that session data to meet whatever guidance corporate legal counsel might provide.

Session data is like pen register/trap and trace data, because it does reveal content. I still treat it like a wiretap but it probably does not meet the same standards.

Event data, i.e. IDS alerts, take so little space as to not require any real storage consideration (compared to full content and session data). Therefore, the primary limiting factor is legal and policy, not technical.

I think anyone who really wants a better answer would do well to check our Prof Kerr's list, and potentially ask him. Alex Muentz would be another good resource.

Richard Bejtlich is training in a city near you! Seats in classes at the Sys Admin Magazine Conference, Techno Security 2007, SANSFIRE 2007, and Black Hat Las Vegas are filling fast -- register today.

Threat Deterrence, Mitigation, and Elimination

A comment on my last post prompted me to answer here. My thesis is this: a significant portion, if not the majority, of security in the analog world is based on threat deterrence, mitigation, and elimination. Security in the analog world is not based on eliminating or applying countermeasures for vulnerabilities. A vulnerability-centric approach is too costly, inconvenient, and static to be effective.

Consider the Metro subway in DC, pictured above. There are absolutely zero physical barriers between the platform and the trains. If evil attacker Evelyn were so inclined, she could easily push a waiting passenger off the platform into the path of an arriving train, maiming or killing the person instantly.

Why does this not happen (regularly)? Evelyn is presumably a rational actor, and she is deterred by vigilante justice and the power of the legal system. If she killed a Metro passenger in the state of Virginia she would probably be executed herself, or at the very least spend the rest of her life in prison. Hopefully they are few people like Evelyn in the world, but would more Metro passengers be murdered if there were no attribution or apprehension of the killers?

How do you think the Metro board would react to such an incident?

Build barriers to limit the potential for passengers to land in front of moving trains

Screen passengers as they enter Metro stations

Mandate trains to crawl within reach of waiting passengers

Add Metro police to watch for suspicious individuals

Add cameras to watch all Metro stations

Lobby Congress to increase penalties

My ranking is intentional. 1 would never happen; it is simply too costly when weighed against the risks. 2 would be impossible to implement in any meaningful fashion and would provoke a public backlash. 3 might happen for a brief period, but it would be abandoned because it would slow the number of trains carrying passengers. 4 might happen for a brief period as well, but the costs of additional personal make it an unlikely permanent solution; it's also ineffective unless the police is right next to a likely incident. 5 and 6 could happen, but they are only helpful for deterrence -- which is not prevention.

Earlier I said Evelyn is a rational actor, so she could presumably be deterred. She could also be mitigated or eliminated. Imagine if Evelyn's action was a ritual associated with gang membership. Authorities could identify and potentially restrict gang members from entering the Metro. (Difficult? Of course. This is why deterrence is a better option.) Authorities could also infiltrate and/or destroy the gang.

Irrational actors cannot be deterred. They may be mitigated and/or eliminated.

Forces of nature cannot be deterred either. Depending on their scope they may be mitigated, but they probably cannot be eliminated. Evelyn's house cannot be built for a reasonable amount of money to withstand a Category V hurricane. Such a force of nature cannot be deterred or eliminated. Given a large enough budget Evelyn's house could be built to survive such a force, so mitigation is an option. Insurance is usually how threats like hurricanes are mitigated, however.

Everyone approaches this problem via the lens of their experience and capabilities. Coders think they can code their way out of this problem. Architects think they can design their way out. I am mainly an operator and, in some ways, an historian. I have seen in my own work that prevention eventually fails, and by learning about the past I have seen the same. In December 2005 I wrote an article called Engineering Disasters for a magazine, and in the coming weeks a second article with more lessons for digital security engineers will be published in a different venue.

I obviously favor whatever cost-effective, practical trade-offs (not solutions) we can implement to limit the risks facing digital assets. I am not saying we should roll over and die, hoping the authorities will catch the bad guys and prevent future crimes. Nevertheless, the most pressing problem in digital security is attribution and apprehension of those perpetrating crimes involving information resources. Until we take the steps necessary to address that problem, no amount of technical vulnerability remediation is going to matter.

Thursday, March 29, 2007

Remember that TJX Is a Victim

Eight years ago this week news sources buzzed about the Melissa virus. How times change! Vulnerabilities and exposures are being monetized with astonishing efficiency these days. 1999 seems so quaint, doesn't it?

With the release of TJX's 10-K to the SEC all news sources are discussing the theft of over 45 million credit cards from TJX computers. I skimmed the 10-K but didn't find details on the root cause. I hope this information is revealed in one of the lawsuits facing TJX. Information on what happened is the only good that can come from this disaster.

It's important to remember that TJX is a victim, just as its customers are victims. The real bad guys here are the criminals who compromised TJX resources and stole sensitive information. TJX employees may be found guilty of criminal negligence, but that doesn't remove the fact that an unauthorized party attacked TJX and stole sensitive information. Unfortunately I believe the amount of effort directed at apprehending the offenders will be dwarfed by the resources directed at TJX. That will leave those intruders and others like them to continue preying on other weak holders of valuable information.

Update: At least US credit card holders don't have it as bad as our friends in the UK.

VMware Server 1.0.2 on Ubuntu 6.10

Previously I documented installing VMware Workstation 6 Beta on my Thinkpad x60s. I decided to uninstall Workstation and install VMware Server 1.0.2. I should have used the vmware-uninstall.pl script but even without using it directly I managed to remove the old Workstation installation without real trouble.

Running Server on Ubuntu 6.10 (desktop) required me to add a few packages. I found Martti Kuparinen's installation guide very helpful. I had to add the following packages to ensure a smooth Server installation.


sudo apt-get install xinetd
sudo apt-get install libX11-dev
sudo apt-get install xlibs-dev

I did not have to install linux-kernel-headers.

I was really impressed that Martti provided a patch for two scripts that did not work correctly out of the box. When I applied the patch I was able to start VMware's Web server and access it via my browser.


richard@neely:/tmp$ wget http://users.piuha.net/martti/comp/ubuntu/httpd.vmware.diff
--13:52:24--  http://users.piuha.net/martti/comp/ubuntu/httpd.vmware.diff
           => `httpd.vmware.diff'
Resolving users.piuha.net... 193.234.218.130
Connecting to users.piuha.net|193.234.218.130|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,973 (2.9K) [text/plain]

100%[====================================>] 2,973         --.--K/s             

13:52:25 (1.81 MB/s) - `httpd.vmware.diff' saved [2973/2973]

richard@neely:/tmp$ cd /
richard@neely:/$ sudo patch -b -p0 < /tmp/httpd.vmware.diff 
Password:
patching file /etc/init.d/httpd.vmware
patching file /usr/lib/vmware-mui/src/lib/httpd.vmware
richard@neely:/$ sudo netstat -natup | grep vm
tcp        0      0 0.0.0.0:8333            0.0.0.0:*
               LISTEN     5205/httpd.vmware   
tcp        0      0 0.0.0.0:8222            0.0.0.0:*
               LISTEN     5205/httpd.vmware

Thanks to this guide I made this addition to /etc/xinetd.d/vmware-authd so the vmware console on port 902 TCP didn't listen on all interfaces:


    bind            = 127.0.0.1

To prevent the Web server from starting at boot and potentially listening on a hostile network, I removed the x bit from the script in /etc/init.d so it would not be started at boot. I can start it manually.


richard@neely:~$ sudo chmod -x /etc/init.d/httpd.vmware
richard@neely:~$ sudo sh /etc/init.d/httpd.vmware start
   Starting httpd.vmware:                                              done

I noticed while installing the packages the suggestion to run apt-get autoremove, so I did once everything was installed.


richard@neely:~$ sudo apt-get autoremove
Password:
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libnl1-pre6 network-manager libnm-util0 dhcdbd
The following packages will be REMOVED:
  dhcdbd libnl1-pre6 libnm-util0 network-manager
0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
Need to get 0B of archives.
After unpacking 1217kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 115360 files and directories currently installed.)
Removing network-manager ...
 * Stopping NetworkManager daemon                                        [ ok ] 
 * Stopping NetworkManager dispatcher                                    [ ok ] 
Removing dhcdbd ...
Removing libnl1-pre6 ...
Removing libnm-util0 ...

I have VMware Server running well on Ubuntu now.

Wednesday, March 28, 2007

Mesh vs Chain

When Matasano Chargen suggested reading Nate Lawson's blog, I immediately added it to my Bloglines collection. Today I read Building a Mesh Vs a Chain and Mesh Approach vs Defense-in-Depth. Nate's basic premise is this:

When explaining the desired properties of a security system, I often use the metaphor of a mesh versus a chain. A mesh implies many interdependent checks, protection measures, and stopgaps. A chain implies a long sequence of independent checks, each assuming or relying on the results of the others.

With a mesh, it’s clear that if you cut one or more links, your security still holds. With a chain, any time a single link is cut, the whole chain fails.

He explains why mesh != defense-in-depth:

A commenter suggested by email that the mesh concept in my previous post is very similar to defense-in-depth. While they are similar, there are some critical differences that are especially important when you apply them to software protection.

Defense-in-depth comes from military history where a defender would build a series of positions and then fall back each time the enemy advanced forward through the first positions. This works in security as well. For instance, a web server may be run in a restricted chroot environment so that if the web server is compromised, damage is limited to the files in the restricted directory, not the whole system.

The mesh model, on the other hand, involves a series of interlocking checks and enforcement mechanisms. There is nothing to fall back to because all the defenses are active at the same time, mutually reinforcing each other. This concept is less common than defense-in-depth for network security use due to the difficulty of incorporating it into system designs. However, it is extremely common in cryptography.

I suggest reading both posts for more information. I found this design idea very interesting, but I agree that implementing it outside of cryptography seems difficult. It would be neat to devise more mesh-based systems.

Security Operations Fundamentals

Last year I last wrote:

Marcus [Ranum] noted that the security industry is just like the diet industry. People who want to lose weight know they should eat less, eat good food, and exercise regularly. Instead, they constantly seek the latest dieting fad, pill, plan, or program -- and wonder why they don't get the results they want!

You might be wondering about the digital security equivalent to eating less, eating good food, and exercising regularly. Addressing that subject adequately would take more than this blog post, but I want to share the steps I use as a consultant when encountering a new client's enterprise.

You'll notice that these steps fit nicely within Mike Rothman's Pragmatic CSO construct. These are a little more specific and focused because I am not acting as a Chief Security Officer when I work as a consultant.

Instrument sample ingress/egress points. What, monitor first? That's exactly right. Start collecting NSM data immediately (at least session, preferably alert, full content, session, and statistical). It's going to take time to progress through the rest of the steps that follow. While working on the next steps your network forensics appliance can be capturing data to be analyzed later.

Understand business operations. Replace business with whatever term makes you more comfortable if you are a .gov, .mil, .edu, etc. You've got to know the purpose of the organization before you can understand the data it needs to do its job. This requires interviewing people who know this, preferably business owners and managers.

Identify and prioritize business data. Once you understand the purpose of the organization, you should determine the data it needs to function. Not all data is equal, so perform a relative ranking to determine the most important down to least important. This work must be done with the cooperation of the businesses; it cannot be security- or consultant-driven.

Identify and prioritize systems processing business data. By systems I mean an entire assemblage for processing data, not individual computers. Systems include payroll processing, engineering and development, finance projections, etc. Prioritize these systems as you did the data they carry. Hopefully these two sets of rankings will match, but perhaps not.

Identify and prioritize resources comprising systems. Here we start dealing with individual servers, clients, and infrastructure. For example, the database containing payroll data is probably more important than the Web server offering access to clients. Here tech people are more important than managers because tech people build and maintain these devices.

Define policy, profile resources, and identify violations. Steps 2-5 have gotten you to the point where you should have a good understanding of the business and its components. If you have a policy, review it to ensure it makes sense given the process thus far. If you haven't yet defined a policy for the use of your information resources, do so now.

Next, profile how those resources behave to determine if they are supporting business operations or if they are acting suspiciously or maliciously. I recommend taking a passive, traffic-centric approach. This method has near-zero business impact, and, if executed properly, can be done without alerting anyone insider or outside the company acting maliciously. Here you use the data you started collecting in step 1.

Implement short term incident containment, investigation, and remediation. I have yet to encounter an enterprise that doesn't immediately find a hot-button item in step 6. Put out those fires and score some early wins before moving on.

Plan and execute instrumentation improvements. Based on step 7, you'll realize you want visibility across the entire enterprise. Increase the number of sensors to cover all of the areas you want. This step encompasses improved host-centric logging and other visibility intitiatives.

Plan and execute infrastructure improvements. You'll probably decide to implement components of my Defensible Network Architecture to take a more proactive stance towards defending the network. You may be able to reconfigure existing processes, products, and people to act in a more secure manner. You may need to design, buy, or train those elements.

Plan and execute server improvements. Here you decide what, if any, changes should be made to the resources offering business data to users, customers, partners, and the like. Maybe you want to encrypt data at rest as well as in motion. Maybe you decide to abandon an old Web framework for a new one... and so on.

Plan and execute user platform improvements. This step changes the gear users rely upon, so it's the last step. Users are most likely to resist that which they can immediately see, so tread carefully. Improvements here involve OS upgrades or changes, moves to thin clients, removal or upgrades of software, and similar issues.

Measure results and return to step 1. I recommend using metrics like those I described here. Measure Days since last compromise of type X, System-days compromised, Time for a pen testing team of [low/high] skill with [internal/external] access to obtain unauthorized [unstealthy/stealthy] access to a specified asset using [public/custom] tools and [complete/zero] target knowledge, and so on.

You may notice steps 8-11 reflect my TaoSecurity Pyramid of Trust. That is no accident.

It is also important to realize that steps 8-11 are based on data collected in step 1 and analyzed in step 6. Enterprise security improvements should not be driven by the newest products or concept. Improvements should be driven by understanding the enterprise and specifically the network. Otherwise, you are playing soccer goal security by making assumptions and not judgements.

Only when you understand what is happening in the enterprise should you consider changing it. Only when you realize existing processes, products, and/or people are deficient should you consider changes or additions. Think in terms of what problem am I trying to solve, not what new process, product, or person is now available.

Tuesday, March 27, 2007

Ayoi on the Importance of NSM Data

At my ShmooCon talk I provided a series of case studies showing the importance of Network Security Monitoring data. The idea was to ask how it would be possible to determine if an IDS alert represented a real problem if high-quality data didn't exist. Alert management is not security investigation, and unfortunately most products and processes implement the former while the latter is truly needed.

I noticed that Ayoi in Malaysia posted a series of blog stories showing his investigative methodology using NSM data and Sguil (Not Only Alert Data parts I, II, and III). These posts demonstrate several alerts and compare data available via an alert management tool like BASE versus a security investigation tool like Sguil. I am glad to see these sorts of stories because they show how people in the trenches do their jobs.

I have yet to meet an analyst -- someone responsible for finding intrusions -- who rejects my methods or the need for collecting NSM data. Almost everyone who argues against these methods is not directly responsible for the technical aspects of personally detecting and responding to intrusions.

Monday, March 26, 2007

SANS Software Security Institute

Today I attended a free three-plus-hour seminar offered by the new SANS Software Security Institute. This is part of SANS dedicated to software security. I recommend reading their press release (.pdf) for the full scoop, but basically SANS is introducing a Secure Programming Skills Assessement, additional training (eventually), and a certification path. Other people will summarize the program, so I'd like to share a few thoughts from the speakers at today's event.

Michael Sutton from SPI Dynamics said that the idea of assembling a team of security people to address enterprise vulnerabilities worked (more or less) for network and infrastructure security because the team could (more or less) introduce elements or alter the environment sufficiently to improve their security posture. The same approach is not working and will not work for application security because addressing the problem requires altering code. Because code is owned by developers, the security team can't directly change it. This is an important point for those who think they can just turn their CSIRT loose on the software security problem in the same way they attacked network security.

Michael also said no security is trustworthy until trusted. (He actually said "trusted." There's a difference. Anyone can "trust" software. The question is whether it is worthy of trust, i.e., "trustworthy.")

Alan Paller made a few comments. He said we have 1.5 million programmers in the world, so training all of them probably isn't an option. He said SANS is working with Tipping Point to create a "Programmer's @Risk" newsletter like the existing vulnerabilities @Risk newsletter. Alan repeated a recommendation made my John Pescatore that organizations should run security tests against bids as well as upon acceptance.

Alan noted that software testing should be considered a part of a "building permit" (pre-development) and a second "occupancy permit" (deployment in the enterprise). Alan also said PCI is the only worthwhile security standard. Others just require writing about security, while PCI requires a modicum of doing security. (Mark Curphey disagrees!)

Jim Routh of DTCC said it's important for developers to recognize that security flaws are software defects, and not the security team's problem! His team of 450 inhouse developers uses three stages of testing: 1) white box for developers; 2) black box for integrators; and 3) third party for deployment.

Mike Willburn from the FBI said FISMA C&A results in "well-documented" systems that score well on report cards but are "full of holes." Bravo.

Andrew Wing from Teranet said he doesn't let an inhouse project progress to user acceptance training unless it scores a certain rank using an automated software security assessment tool.

Jack Danahy from Ounce Labs stressed the importance of contract language for procuring. The OWASP Legal Project also offers sample language. Alan stressed the need to build security into contracts, rather than relying on the vague concept of "negligence" when security isn't explicitly included in a contract.

Michael Weider from Watchfire said he fears user-supplied content will be the next exploitation vector. I shuddered at the horror of MySpace and the like.

Steve Christey mentioned SAMATE (Software Assurance Metrics And Tool Evaluation).

That's what I can document given the time I have. Thanks to SANS for their leadership in this endeavor.

Manipulating Packet Captures

While capturing traffic at Hack or Halo I realized the timestamps on the packets were off by one hour. Apparently I didn't patch this infrequently used Hacom box for the recent DST change.

I captured traffic using Sguil's log_packets.sh script, which uses Snort to write a new full content trace every hour. For the first round of the contest, the script produced two traces. I combined them using Mergecap, bundled with Wireshark.


richard@neely:/var/tmp/shmoocon2007$ mergecap -w shmoocon_hack_rd1.pcap 
 snort.log.1174770982 snort.log.1174773600

The Capinfos program accompanying Wireshark summarizes the new trace:


richard@neely:/var/tmp/shmoocon2007$ capinfos shmoocon_hack_rd1.pcap
File name: shmoocon_hack_rd1.pcap
File type: Wireshark/tcpdump/... - libpcap
Number of packets: 719534 
File size: 155340234 bytes
Data size: 143827666 bytes
Capture duration: 4587.056482 seconds
Start time: Sat Mar 24 17:17:41 2007
End time: Sat Mar 24 18:34:08 2007
Data rate: 31355.11 bytes/s
Data rate: 250840.89 bits/s
Average packet size: 199.89 bytes

I decided to alter the timestamps using Editcap, also packaged with Wireshark.


richard@neely:/var/tmp/shmoocon2007$ editcap -t 3600 shmoocon_hack_rd1.pcap 
 shmoocon_hack_rd1_timeadj.pcap

Now the timestamps are correct.


richard@neely:/var/tmp/shmoocon2007$ capinfos shmoocon_hack_rd1_timeadj.pcap
File name: shmoocon_hack_rd1_timeadj.pcap
File type: Wireshark/tcpdump/... - libpcap
Number of packets: 719534 
File size: 155340234 bytes
Data size: 143827666 bytes
Capture duration: 4587.056482 seconds
Start time: Sat Mar 24 18:17:41 2007
End time: Sat Mar 24 19:34:08 2007
Data rate: 31355.11 bytes/s
Data rate: 250840.89 bits/s
Average packet size: 199.89 bytes

I'm getting these traces to Shmoo now so they can be shared.

Sunday, March 25, 2007

ShmooCon 2007 Wrap-Up

ShmooCon 2007 ended today. Only four talks occurred today (Sunday), and only two of them (Mike Rash, Rob King/Rohlt Dhamankar) really interested me. Therefore, I went to church with my family this morning and took lead on watching the kids afterwards. I plan to watch those two interesting talks once they are released as video downloads. (It takes me 1 1/2 - 2 hours each way into and out of DC via driving and Metro, so I would have spent more time on the road than listening to speakers.)

I also left right after Bruce Potter's introductory comments on Friday afternoon. If it hadn't been for the NoVA Sec meeting I scheduled Friday at 1230, I probably would have only attended Saturday's sessions. I heard Avi Rubin's 7 pm keynote was good, and I would have liked to watch Johnny Long's talk. Otherwise I thought spending time with my family was more important.

That leaves Saturday. I spent the whole day at ShmooCon, from the first talk to the end of Hack or Halo. I began the day with Ofir Arkin from Insightix. (I actually spent about half an hour chatting with Ofir Friday afternoon, which was cool. I also spent time Friday speaking with several people I recognized.) Ofir demonstrated that just about all Network Admission Control concepts and implementations are broken. He only covered about half his material, but I left wondering who would bother spending thousands or millions on NAC when it doesn't seem to work and is fighting the last war anyway.

Ofir emphasized that knowledge of the enterprise is the key to network defense. He pointed out that NAC products which provide a shared medium quarantine area are exactly where an intruder wants his machine to be delivered. Once in that area he can attack the weakest, non-compliant systems on the same subnet or VLAN used by the quarantine. Using PVLANs an avoid this problem, but only if not subject to VLAN hopping attacks. Ofir questioned whether per-port security is ever feasible, especially in an age of increasing use of VMs.

One basic take-away for me was this: if I find myself on a network requiring NAC, do the following.

Find the nearest printer.

Unplug the network cable.

Connect the network cable from the printer to a hub, and connect the hub to the network port.

Connect my laptop to the hub.

Sniff printer's MAC address and IP address.

Disconnect the printer.

Assign the printer's MAC and IP address to my laptop, and access the network.

While this will not work everywhere, it's probably going to work in enough places to make NAC a questionable prospect for physical defense. Hosts connecting via VPN are another issue.

After Ofir spoke I saw Joel Wilbanks, Matt Fisher, and Mike Murphy talk about incident response when Web applications are attacked. They made the point that Web app incidents don't usually leave artifacts (think files on the hard drive) on the victim. Web app forensics becomes a log analysis exercise. If no logs exist (Web, database, OS, etc.), you're hosed. They recommended populating database tables with honeytokens and writing custom IDS signatures to alert on the presence of those tokens in network traffic.

During their presentation several attendees questioned the role of SSL for inbound connections. The speakers recommending terminating SSL at an accelerator, and passing clear text by an IDS before sending it to the Web server or re-encrypting it. At least one of the attendees was shocked -- shocked -- to consider passing "sensitive" data in the clear like that. I have never understood this argument. The question is simple: do you care to know what is being carried in SSL, or do you not care? If you do care (and you should), architect your enterprise so you have visibility into what's happening. If you don't care, tell me so I can avoid doing business with you.

As far as SSL is concerned, I consider inbound SSL a solved problem. Outbound SSL, as might be used for a command and control channel, is not solved -- unless you want to break SSL and teach users to accept a man-in-the-middle attack scenario. I worry about outbound SSL, not inbound.

I had lunch with Joe Stewart, so in some sense I didn't really miss his talk. He was nice enough to share his thoughts with me on his next Sandnet and other projects.

My talk happened at 1300. This means I missed Billy Hoffman release Jikto, so I plan to download his talk (and Joe's) when available. I was really pleased by the outcome. The room was totally filled and people were standing outside the room listening. Thanks to everyone who attended. I wish we had more time for questions, so feel free to leave a comment here or email if you have unanswered issues.

After my talk I listened to Raven talk about backbone security. She is fuzzing key routing protocols (RIP, OSPF, EIGRP, BGP, etc.) by mainly attacking open source implementations. She just got a Cisco 2600 series router so IOS is her next target. If she is getting results doing this work in her spare time sitting in airports, you can only imagine what funded, dedicated teams are doing with budgets for equipment and manpower.

I spent the next hour chatting with familiar faces in the area near the talks. Marty McKeay was there, along with Mike Rash, Jamie Butler, and Bret Padres and Ovie Carroll from the CyberSpeak Podcast. (Sorry I couldn't get back to you guys in time!)

At 1600 I squeezed into Dan Kaminsky's talk. Before he started I had a chance to chat briefly with Mike Poor and Ed Skoudis from Intel Guardians. Mike and Marc Sachs (who I saw independently) were not happy with my TCP options analysis. Oh well!

I felt bad for Dan. The poor guy showed remarkable resolve trying to speak, despite an attendee who felt compelled to interrupt every fifth sentence. Dan had to dodge plenty of Shmoo balls while explaining slides with way too many words on them. I think Dan's research is way outside the realm of what most security people do, but probably perfect for a paper at USENIX.

I stayed in the same room to listen to Josh Wright and Mike Kershaw talk about LORCON. As their Web page states: LORON is "a generic library for injecting 802.11 frames, capable of injection via multiple driver frameworks, without forcing modification of the application code." Basically, if you write a wireless packet injector, you should use LORCON. Don't write something for a specific wireless driver -- let LORCON handle that for you. I was really impressed, especially since I had never seen Mike (author of Kismet) and Josh (lots of tools, cool research) in person. In addition to LORCON they mentioned this WiFi frame injection patch for Wireshark.

When their talk was done I headed over to the Hack or Halo room. I set up my Hacom Lex Twister on a SPAN port (argh, yes, I forgot a tap) and captured the traffic from the Hack contest. I monitored it live with Sguil, which was fun.

Overall, I was again impressed by the organization and manpower demonstrated by ShmooCon. I was less impressed by the overall slate of talks, but I think the quality of attendees compensated for that. The first ShmooCon in 2005 attracted about 350 people. The second had about 800. This year nearly 1200 people attended. I was very thankful to attend and speak and I look forward to at least attending next year.

Update: I forgot to ask -- if you liked my talk, please send feedback to feedback [at] shmoocon [dot] org. Thank you!

Saturday, March 24, 2007

Blogging from ShmooCon Hack or Halo

So much from my lousy camera phone. That's my best attempt to show Sguil monitoring traffic at the ShmooCon Hack or Halo contest. I plan to share the network traffic from the hacking contest when I get the opportunity. Thanks to WXS and the ShmooCon crew for letting my attach a sensor to the network.

Friday, March 23, 2007

Taking the Fight to the Enemy

ShmooCon started today. ShmooCon leader Bruce Potter finished his opening remarks by challenging the audience to find anyone outside of the security community who cares about security. I decided to take his idea seriously and I thought about it on the Metro ride home.

It occurred to me that the digital security community fixates on vulnerabilities because that is the only aspect of the Risk Equation we can influence. Lines of business control assets, so we can't decrease risk by making assets less valuable. (That doesn't even make sense.) We do not have the power or authority to remove threats, so we can't decrease risk by lowering the attacks against our assets. (Threat mitigation is the domain of law enforcement and the military.) We can only address vulnerabilities, but unless we develop the asset ourselves we're stuck with whatever security the vendor provided.

I would like to hear if anyone can imagine another realm of human endeavor where the asset owner or agent is forced to defend his own interests, without help from law enforcement or the military. The example can be historical, fictional, or contemporary. I'm reminded of Wells Fargo stagecoaches being robbed as they crossed the West, forcing WF to hire private guards with guns to defend company assets in transit. As a fictional example, Sherlock Holmes didn't work for Scotland Yard; victims hired the Great Detective to solve crimes that the authorities were too slow or unwilling to handle.

As I've said many times before, we are wasting a lot of time and money trying to "secure" systems when we should be removing threats. I thought of this again last night while watching Chris Hansen work with law enforcement to take more child predators off the streets. Imagine if I didn't have law enforcement deterring and jailing criminals like that. I'd have to wrap my kids in some sort of personal tank when I send them to school, and they'd still probably end up in harm's way. That's the situation we face on the Internet. There's no amount of bars over windows, high fences, or other defenses that will stop determined intruders. Removing or deterring the intruders is history's lesson.

This FCW article has the right idea:

The best defense against cyberattacks on U.S. military, civil and commercial networks is to go on the offensive, said Marine Gen. James Cartwright, commander of the Strategic Command (Stratcom), said March 21 in testimony to the House Armed Services Committee.

“History teaches us that a purely defensive posture poses significant risks,” Cartwright told the committee. He added that if “we apply the principle of warfare to the cyberdomain, as we do to sea, air and land, we realize the defense of the nation is better served by capabilities enabling us to take the fight to our adversaries, when necessary, to deter actions detrimental to our interests...”

The Stratcom commander told the committee that the United States is under widespread, daily attacks in cyberspace. He added that the country lacks dominance in the cyberdomain and that it could become “increasingly vulnerable if we do not fundamentally change how we view this battle space.”

Put me in, coach. I'm ready to play, today.

Wireless Ubuntu on Thinkpad x60s

I'm used to doing everything manually when running wireless FreeBSD on older laptops. Running Ubuntu has shielded me from some of the command-line configuration I used to perform on FreeBSD. Linux uses different commands for certain tasks. My new laptop also has a different chipset from my old laptop, so I wanted to see if I could get Kismet working on it.

If I want to find wireless networks via the command line I use this command.


richard@neely:~$ sudo iwlist eth1 scan
eth1      Scan completed :
          Cell 01 - Address: 00:13:10:65:2F:AD
                    ESSID:"shaolin"
                    Protocol:IEEE 802.11bg
                    Mode:Master
                    Channel:1
                    Encryption key:on
                    Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 6 Mb/s; 9 Mb/s
                              11 Mb/s; 12 Mb/s; 18 Mb/s; 24 Mb/s; 36 Mb/s
                              48 Mb/s; 54 Mb/s
                    Quality=76/100  Signal level=-58 dBm  Noise level=-58 dBm
                    Extra: Last beacon: 68ms ago
...truncated...

If I want to associate with that WAP using WEP I use this command.


richard@neely:~$ sudo iwconfig eth1 essid shaolin channel 1 key KEYDIGITS

I am associated now.


richard@neely:~$ iwconfig eth1
eth1      IEEE 802.11g  ESSID:"shaolin"  
          Mode:Managed  Frequency:2.412 GHz  Access Point: 00:13:10:65:2F:AD   
          Bit Rate:54 Mb/s   Tx-Power:15 dBm   
          Retry limit:15   RTS thr:off   Fragment thr:off
          Power Management:off
          Link Quality=76/100  Signal level=-58 dBm  Noise level=-59 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:2909   Missed beacon:0

I can grab an IP via DHCP.


richard@neely:~$ sudo dhclient eth1
Internet Systems Consortium DHCP Client V3.0.4
Copyright 2004-2006 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth1/00:13:02:4c:30:2d
Sending on   LPF/eth1/00:13:02:4c:30:2d
Sending on   Socket/fallback
DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 8
DHCPOFFER from 192.168.2.1
DHCPREQUEST on eth1 to 255.255.255.255 port 67
DHCPACK from 192.168.2.1
bound to 192.168.2.103 -- renewal in 42728 seconds.

Here is ifconfig output.


richard@neely:~$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:13:02:4C:30:2D  
          inet addr:192.168.2.103  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::213:2ff:fe4c:302d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4984 errors:19 dropped:2928 overruns:0 frame:0
          TX packets:239 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5491350 (5.2 MiB)  TX bytes:188020 (183.6 KiB)
          Interrupt:74 Base address:0xc000 Memory:edf00000-edf00fff

I can check my gateway.


richard@neely:~$ netstat -nr -4
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
172.16.250.0    0.0.0.0         255.255.255.0   U         0 0          0 vmnet8
172.16.207.0    0.0.0.0         255.255.255.0   U         0 0          0 vmnet1
0.0.0.0         192.168.2.1     0.0.0.0         UG        0 0          0 eth1

I can change my IP from DHCP to static.


richard@neely:~$ sudo killall dhclient
richard@neely:~$ sudo ifconfig eth1 inet 192.168.2.8 netmask 255.255.255.0 up
richard@neely:~$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:13:02:4C:30:2D  
          inet addr:192.168.2.8  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::213:2ff:fe4c:302d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5625 errors:20 dropped:2929 overruns:0 frame:0
          TX packets:245 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5492954 (5.2 MiB)  TX bytes:192494 (187.9 KiB)
          Interrupt:74 Base address:0xc000 Memory:edf00000-edf00
ichard@neely:~$ sudo route add default gw 192.168.2.1
richard@neely:~$ netstat -nr -4
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
172.16.250.0    0.0.0.0         255.255.255.0   U         0 0          0 vmnet8
172.16.207.0    0.0.0.0         255.255.255.0   U         0 0          0 vmnet1
0.0.0.0         192.168.2.1     0.0.0.0         UG        0 0          0 eth1

Here are the changes I made to enable Kismet after checking my wireless card.


richard@neely:~$ sudo lshw -businfo | grep eth1
pci@03:00.0     eth1       network     PRO/Wireless 3945ABG Network Connection

richard@neely:~$ diff -u /etc/kismet/kismet.conf.orig /etc/kismet/kismet.conf
--- /etc/kismet/kismet.conf.orig        2007-03-23 09:53:28.000000000 -0400
+++ /etc/kismet/kismet.conf     2007-03-23 09:56:00.000000000 -0400
@@ -7,10 +7,10 @@
 version=2005.06.R1
 
 # Name of server (Purely for organizational purposes)
-servername=Kismet
+servername=neely
 
 # User to setid to (should be your normal user)
-#suiduser=your_user_here
+suiduser=richard
 
 # Sources are defined as:
 # source=sourcetype,interface,name[,initialchannel]
@@ -19,7 +19,7 @@
 # The initial channel is optional, if hopping is not enabled it can be used
 # to set the channel the interface listens on.
 # YOU MUST CHANGE THIS TO BE THE SOURCE YOU WANT TO USE
-source=none,none,addme
+source=ipw3945,eth1,addme

Kismet works fine. When operating eth1 is in monitor mode.


richard@neely:~$ iwconfig eth1
eth1      unassociated  ESSID:"shaolin"  
          Mode:Monitor  Frequency=2.412 GHz  Access Point: 00:13:10:65:2F:AD   
          Bit Rate:0 kb/s   Tx-Power:16 dBm   
          Retry limit:15   RTS thr:off   Fragment thr:off
          Power Management:off
          Link Quality:0  Signal level:0  Noise level:0
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0

When Kismet exits I'm able to cleanly use my original connection.

Thursday, March 22, 2007

Committing Changes to CVS

In my last post I set up CVS so I could upload my Sguil scripts. I decided I would document how I make changes to those scripts and commit them to CVS.

First I needed to check out a copy of the scripts. I made a dev directory and will now use that for all future development.


richard@macmini:~$ export CVS_RSH=ssh
richard@macmini:~$ mkdir dev
richard@macmini:~$ cd dev
richard@macmini:~/dev$ cvs -z3 \
> -d:ext:taosecurity@taosecurity.cvs.sf.net:/cvsroot/taosecurity checkout -P \
> taosecurity_sguil_scripts
taosecurity@taosecurity.cvs.sf.net's password: 
cvs checkout: Updating taosecurity_sguil_scripts
U taosecurity_sguil_scripts/README
U taosecurity_sguil_scripts/sancp
U taosecurity_sguil_scripts/sguil_client_install.sh
U taosecurity_sguil_scripts/sguil_database_install_pt1.sh
U taosecurity_sguil_scripts/sguil_database_install_pt2.sh
U taosecurity_sguil_scripts/sguil_sensor_install.sh
U taosecurity_sguil_scripts/sguil_sensor_install_patch.sh
U taosecurity_sguil_scripts/sguil_server_install.sh
U taosecurity_sguil_scripts/sguild_adduser.sh
U taosecurity_sguil_scripts/snort
U taosecurity_sguil_scripts/snort_src_install.sh
richard@macmini:~/dev$ cd taosecurity_sguil_scripts/
richard@macmini:~/dev/taosecurity_sguil_scripts$ ls
CVS                      sguild_adduser.sh              sguil_sensor_install.sh
README                   sguil_database_install_pt1.sh  sguil_server_install.sh
sancp                    sguil_database_install_pt2.sh  snort
sguil_client_install.sh  sguil_sensor_install_patch.sh  snort_src_install.sh

With the scripts checked out I edit the 'snort' script and commit it.


richard@macmini:~/dev/taosecurity_sguil_scripts$ cvs commit -m \
> "Update for Snort 2.6.1.2." snort
taosecurity@taosecurity.cvs.sf.net's password: 
Checking in snort;
/cvsroot/taosecurity/taosecurity_sguil_scripts/snort,v  <--  snort
new revision: 1.2; previous revision: 1.1
done

When done I realize I need to change two other scripts, so I change them and commit them at the same time.


richard@macmini:~/dev/taosecurity_sguil_scripts$ cvs commit -m \
> "Update for Snort 2.6.1.3." snort_src_install.sh README 
taosecurity@taosecurity.cvs.sf.net's password: 
Checking in snort_src_install.sh;
/cvsroot/taosecurity/taosecurity_sguil_scripts/snort_src_install.sh,v  <--  snort_src_install.sh
new revision: 1.2; previous revision: 1.1
done
Checking in README;
/cvsroot/taosecurity/taosecurity_sguil_scripts/README,v  <--  README
new revision: 1.2; previous revision: 1.1
done
richard@macmini:~/dev/taosecurity_sguil_scripts$

That's it. The changes are visible immediately via Web-based CVS.

TaoSecurity CVS at Sourceforge

For a while I've maintained a set of fairly lame scripts for automating installation of certain Sguil components on FreeBSD. These scripts have previously been posted as .tar.gz archives in various places. Today I decided to make use of the TaoSecurity Sourceforge site I created a few months back. From now on you can access my scripts via CVS at that site.

My CVS experience is minimal, although I posted some notes from Sguil a few years ago.

I wanted to document how I set this up, because it was not intuitive. Thanks to Bamm for helping me via IRC. I also found this doc and this how-to helpful.

I decided to maintain my local repository on macmini. I wanted to experiment with a local repository before committing anything to Sourceforge. Here I set up that local repository. I have my scripts in a directory called taosecurity_sguil_scripts. I'm going to call the CVS module taosecurity_sguil_scripts too.


richard@macmini:~$ mkdir cvsroot
richard@macmini:~$ cvs -d /home/richard/cvsroot init
richard@macmini:~$ cd taosecurity_sguil_scripts

I wanted my scripts to have lines in them indicating the version number. Bamm pointed me towards this keyword list, which resulted in me adding the following:


richard@macmini:~/taosecurity_sguil_scripts$ cat sguild_adduser.sh 
#!/bin/sh
#
# $Id$ #
#
SGUIL=sguil-0.6.1
LD_LIBRARY_PATH=/usr/local/lib/mysql
export LD_LIBRARY_PATH
cd /usr/local/src/$SGUIL/server/
./sguild -c sguild.conf -u sguild.users -adduser sguil
cp sguild.users /usr/local/etc/nsm/
chown sguil:sguil /usr/local/etc/nsm/sguild.users

That # $Id$ # will be transformed into what I want later.

Now I check the scripts into the local repository.


richard@macmini:~/taosecurity_sguil_scripts$ cvs -d /home/richard/cvsroot/ \
> import -m "Initial import." taosecurity_sguil_scripts TaoSecurity start
N taosecurity_sguil_scripts/sguil_sensor_install.sh
N taosecurity_sguil_scripts/snort_src_install.sh
N taosecurity_sguil_scripts/sguil_server_install.sh
N taosecurity_sguil_scripts/sguil_database_install_pt2.sh
N taosecurity_sguil_scripts/sguil_database_install_pt1.sh
N taosecurity_sguil_scripts/README
N taosecurity_sguil_scripts/sguild_adduser.sh
N taosecurity_sguil_scripts/sguil_client_install.sh
N taosecurity_sguil_scripts/sguil_sensor_install_patch.sh
N taosecurity_sguil_scripts/snort
N taosecurity_sguil_scripts/sancp

No conflicts created by this import

To experiment with checking scripts out of the local repository, I make a wc_tss working copy directory and try retrieving the files.


richard@macmini:~/taosecurity_sguil_scripts$ cd 
richard@macmini:~$ mkdir wc_tss
richard@macmini:~$ cd wc_tss/
richard@macmini:~/wc_tss$ cvs -d /home/richard/cvsroot/ checkout \
> taosecurity_sguil_scripts
cvs checkout: Updating taosecurity_sguil_scripts
U taosecurity_sguil_scripts/README
U taosecurity_sguil_scripts/sancp
U taosecurity_sguil_scripts/sguil_client_install.sh
U taosecurity_sguil_scripts/sguil_database_install_pt1.sh
U taosecurity_sguil_scripts/sguil_database_install_pt2.sh
U taosecurity_sguil_scripts/sguil_sensor_install.sh
U taosecurity_sguil_scripts/sguil_sensor_install_patch.sh
U taosecurity_sguil_scripts/sguil_server_install.sh
U taosecurity_sguil_scripts/sguild_adduser.sh
U taosecurity_sguil_scripts/snort
U taosecurity_sguil_scripts/snort_src_install.sh
richard@macmini:~/wc_tss$ ls
taosecurity_sguil_scripts
richard@macmini:~/wc_tss$ cd taosecurity_sguil_scripts/
richard@macmini:~/wc_tss/taosecurity_sguil_scripts$ ls
CVS                      sguild_adduser.sh              sguil_sensor_install.sh
README                   sguil_database_install_pt1.sh  sguil_server_install.sh
sancp                    sguil_database_install_pt2.sh  snort
sguil_client_install.sh  sguil_sensor_install_patch.sh  snort_src_install.s

Great, that worked. Let's see if sguild_adduser.sh has the Id I expect.


richard@macmini:~/wc_tss/taosecurity_sguil_scripts$ cat sguild_adduser.sh
#!/bin/sh
#
# $Id: sguild_adduser.sh,v 1.1.1.1 2007/03/22 16:24:55 richard Exp $ #
#
SGUIL=sguil-0.6.1
LD_LIBRARY_PATH=/usr/local/lib/mysql
export LD_LIBRARY_PATH
cd /usr/local/src/$SGUIL/server/
./sguild -c sguild.conf -u sguild.users -adduser sguil
cp sguild.users /usr/local/etc/nsm/
chown sguil:sguil /usr/local/etc/nsm/sguild.users

Awesome. I think I'm ready to upload the scripts to Sourceforge.


richard@macmini: export CVS_RSH=ssh
richard@macmini:~$ cd taosecurity_sguil_scripts
richard@macmini:~/taosecurity_sguil_scripts$ cvs \
> -d:ext:taosecurity@taosecurity.cvs.sf.net:/cvsroot/taosecurity import -m \
> "Initial import." taosecurity_sguil_scripts TaoSecurity start
The authenticity of host 'taosecurity.cvs.sf.net (66.35.250.90)' can't be
 established.
RSA key fingerprint is 13:f1:65:c3:6c:b7:7e:a5:f0:f3:f5:19:f4:42:9c:4a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'taosecurity.cvs.sf.net,66.35.250.90' (RSA)
 to the list of known hosts.
taosecurity@taosecurity.cvs.sf.net's password: 
N taosecurity_sguil_scripts/sguil_sensor_install.sh
N taosecurity_sguil_scripts/snort
N taosecurity_sguil_scripts/sancp
N taosecurity_sguil_scripts/sguil_client_install.sh
N taosecurity_sguil_scripts/sguil_server_install.sh
N taosecurity_sguil_scripts/sguil_database_install_pt1.sh
N taosecurity_sguil_scripts/sguil_database_install_pt2.sh
N taosecurity_sguil_scripts/snort_src_install.sh
N taosecurity_sguil_scripts/sguil_sensor_install_patch.sh
N taosecurity_sguil_scripts/sguild_adduser.sh
N taosecurity_sguil_scripts/README

No conflicts created by this import

That worked. I can now browse CVS and see my files.

To test checking them out, I go to another machine and try the following.


richard@neely:/tmp$ cvs \
> -d:pserver:anonymous@taosecurity.cvs.sourceforge.net:/cvsroot/taosecurity \
> login
Logging in to 
 :pserver:anonymous@taosecurity.cvs.sourceforge.net:2401/cvsroot/taosecurity
CVS password: 
cvs login: CVS password file /home/richard/.cvspass does not exist
 - creating a new file
richard@neely:/tmp$ cvs \
> -d:pserver:anonymous@taosecurity.cvs.sourceforge.net:/cvsroot/taosecurity \
> checkout taosecurity_sguil_scripts
cvs checkout: Updating taosecurity_sguil_scripts
U taosecurity_sguil_scripts/README
U taosecurity_sguil_scripts/sancp
U taosecurity_sguil_scripts/sguil_client_install.sh
U taosecurity_sguil_scripts/sguil_database_install_pt1.sh
U taosecurity_sguil_scripts/sguil_database_install_pt2.sh
U taosecurity_sguil_scripts/sguil_sensor_install.sh
U taosecurity_sguil_scripts/sguil_sensor_install_patch.sh
U taosecurity_sguil_scripts/sguil_server_install.sh
U taosecurity_sguil_scripts/sguild_adduser.sh
U taosecurity_sguil_scripts/snort
U taosecurity_sguil_scripts/snort_src_install.sh

That worked too. So, from now on, if you'd like to get my FreeBSD Sguil installation scripts, please retrieve them from CVS.

Gconcat on FreeBSD

The last time I wanted to combine two smaller drives into a single virtual drive on FreeBSD I used Gvinum. Ceri Davies posted a helpful comment indicating I should try using gconcat(8). I did that today and thanks to an insightful piece of advice from Robert Watson, I got it working.

This is what the drive looked like.


shuttle01# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad5s1a    496M     36M    420M     8%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/ad5s1f    989M     22K    910M     0%    /home
/dev/ad5s1h     54G    4.0K     50G     0%    /nsm1
/dev/ad7s1d    361G    4.0K    332G     0%    /nsm2
/dev/ad5s1g    989M     12K    910M     0%    /tmp
/dev/ad5s1d    1.9G    531M    1.3G    29%    /usr
/dev/ad5s1e    4.8G    1.6M    4.5G     0%    /var

I want to create /dev/concat/nsm. However, if I try to do that while /nsm1 and /nsm2 are mounted I'll get errors like this:


shuttle01# gconcat label -v nsm ad5s1h ad7s1d
Can't store metadata on ad5s1h: Operation not permitted.

So, unmount /nsm1 and /nsm2 before continuing. Next:


shuttle01# umount /nsm1
shuttle01# umount /nsm2
shuttle01# gconcat label -v nsm ad5s1h ad7s1d
Metadata value stored on ad5s1h.
Metadata value stored on ad7s1d.
Done.
shuttle01# ls /dev/concat/nsm
/dev/concat/nsm
shuttle01# newfs /dev/concat/nsm
/dev/concat/nsm: 438631.6MB (898317520 sectors) block size 16384, fragment size 2048
        using 2387 cylinder groups of 183.77MB, 11761 blks, 23552 inodes.
super-block backups (for fsck -b #) at:
 160, 376512, 752864, 1129216, 1505568, 1881920, 2258272, 2634624, 3010976,
...truncated...

Now I can create /nsm and mount it.


shuttle01# mkdir /nsm
shuttle01# mount /dev/concat/nsm /nsm
shuttle01# df -h
Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/ad5s1a        496M     36M    420M     8%    /
devfs              1.0K    1.0K      0B   100%    /dev
/dev/ad5s1f        989M     22K    910M     0%    /home
/dev/ad5s1g        989M     12K    910M     0%    /tmp
/dev/ad5s1d        1.9G    531M    1.3G    29%    /usr
/dev/ad5s1e        4.8G    1.6M    4.5G     0%    /var
/dev/concat/nsm    415G    4.0K    382G     0%    /nsm
shuttle01# touch /nsm/test

Cool. If I don't want to use it again:


shuttle01# umount /nsm
shuttle01# gconcat stop nsm
shuttle01# gconcat unload
shuttle01# ls /dev/concat

To use it again:

shuttle01# gconcat load nsm
shuttle01# ls /dev/concat/nsm
/dev/concat/nsm
shuttle01# mount /dev/concat/nsm /nsm
shuttle01# ls /nsm
.snap   test

Remember to enable gconcat in /boot/loader.conf:


shuttle01# cat /boot/loader.conf 
geom_concat_load="YES"

Remember to edit /etc/fstab. The original looked like this:


# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ad5s1b             none            swap    sw              0       0
/dev/ad5s1a             /               ufs     rw              1       1
/dev/ad5s1f             /home           ufs     rw              2       2
/dev/ad5s1h             /nsm1           ufs     rw              2       2
/dev/ad7s1d             /nsm2           ufs     rw              2       2
/dev/ad5s1g             /tmp            ufs     rw              2       2
/dev/ad5s1d             /usr            ufs     rw              2       2
/dev/ad5s1e             /var            ufs     rw              2       2
/dev/acd0               /cdrom          cd9660  ro,noauto       0       0

The new one looks like this:


# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ad5s1b             none            swap    sw              0       0
/dev/ad5s1a             /               ufs     rw              1       1
/dev/ad5s1f             /home           ufs     rw              2       2
#/dev/ad5s1h            /nsm1           ufs     rw              2       2
#/dev/ad7s1d            /nsm2           ufs     rw              2       2
/dev/ad5s1g             /tmp            ufs     rw              2       2
/dev/ad5s1d             /usr            ufs     rw              2       2
/dev/ad5s1e             /var            ufs     rw              2       2
/dev/concat/nsm         /nsm            ufs     rw              2       2
/dev/acd0               /cdrom          cd9660  ro,noauto       0       0

I'm deploying this system in a RAID 0 configuration because it's a Shuttle with two HDDs. I'm not worrying about recovering from a disaster with this box. It's going to help with the ShmooCon network. If I had three or more drives I'd consider using gstripe(8).

Recovering from Corrupted MySQL Database

Today one of my clients ran into a problem with his Sguil installation. The server hosting his Sguil MySQL database experienced a crash, as shown by dmesg on reboot:


Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
WARNING: /home was not properly dismounted
WARNING: /nsm was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted

The original error message said:


ERROR: loaderd: mysqlexec/db server: Incorrect key file for table 
'./sguildb/sancp_sensor_20070322.MYI'; try to repair it

If the sensor crashed while SANCP data was loading, it would make sense that sancp_sensor_20070322.MYI was corrupted.

When trying to restart sguild, the following error appeared:


[user@sensor ~]$ ./sguild_start.sh 
pid(3119)  Loading access list: ./sguild.access
pid(3119)  Sensor access list set to ALLOW ANY.
pid(3119)  Client access list set to ALLOW ANY.
pid(3119)  Adding AutoCat Rule: 
pid(3119)  Adding AutoCat Rule: ||ANY||ANY||ANY||ANY||ANY||ANY||tag:
 Tagged Packet||1
pid(3119)  Email Configuration:
pid(3119)    Config file: ./sguild.email
pid(3119)    Enabled: No
pid(3119)  Connecting to localhost on 3306 as user
pid(3119)  MySQL Version: version 5.0.27
pid(3119)  SguilDB Version: 0.11
pid(3119)  Creating event MERGE table.
pid(3119)  Creating tcphdr MERGE table.
pid(3119)  Creating udphdr MERGE table.
pid(3119)  Creating icmphdr MERGE table.
pid(3119)  Creating data MERGE table.
ERROR: loaderd: You appear to be using an old version of the
 sguil database schema that does not support the MERGE sancp
 table. Please see the CHANGES document for more information
.
SGUILD: Exiting...

That doesn't look good.

Whenever I encounter a database problem, I first run mysqlcheck (with the database running) like so:


[user@sensor ~]$ mysqlcheck -r sguildb -p
Enter password: 
sguildb.data
note     : The storage engine for the table doesn't support repair
sguildb.data_sensor_20070215                       OK
sguildb.data_sensor_20070216                       OK
...edited...
sguildb.sancp_sensor_20070215                      OK
sguildb.sancp_sensor_20070216                      OK
...edited...
sguildb.sancp_sensor_20070320                      OK
sguildb.sancp_sensor_20070321                      OK
sguildb.sensor                                     OK
...truncated...

Note sguildb.sancp_sensor_20070322 isn't listed.

I stopped MySQL and then ran myisamchk, which showed the following:


sensor:/var/db/mysql/sguildb# myisamchk *.MYI
...edited...
Checking MyISAM file: sancp_sensor_20070322.MYI
Data records:    2687   Deleted blocks:       0
myisamchk: warning: Table is marked as crashed
myisamchk: warning: 1 client is using or hasn't closed the table properly
- check file-size
myisamchk: error: Size of indexfile is: 303104          Should be: 306176
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
myisamchk: error: Found key at page 295936 that points to
 record outside datafile
MyISAM-table 'sancp_sensor_20070322.MYI' is corrupted
Fix it using switch "-r" or "-o"
...truncated...

I fixed it like this:


sensor:/var/db/mysql/sguildb# myisamchk -r sancp_sensor_20070322.MYI
- recovering (with sort) MyISAM-table 'sancp_sensor_20070322.MYI'
Data records: 2687
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
Data records: 2678

Next I restarted the database and re-ran my sguild startup script. Everything returned to normal as I had hoped.

This is another example of the idea that anyone who uses detection systems long enough eventually becomes a database admin!

Wednesday, March 21, 2007

Backscatter Detected

Recently I posted a conclusion to my backscatter investigation, where people were reporting backscatter from SYN and other DoS attacks to SANS ISC. When you monitor your own cable modem it's not common to see this sort of traffic unless you go explicitly looking for it. Today however I saw the following using Sguil.


Count:2 Event#1.204541 2007-03-20 18:04:19
BLEEDING-EDGE DROP Known Bot C&C Server Traffic (group 1)
69.143.202.28 -> 193.109.122.67
IPVer=4 hlen=5 tos=0 dlen=40 ID=2644 flags=2 offset=0 ttl=64 chksum=58655
Protocol: 6 sport=1024 -> dport=6667

Seq=2934031229 Ack=0 Off=5 Res=0 Flags=*****R** Win=0 urp=54297 chksum=0
Payload:
None.

What got my attention at first was an alert indicating my host (possibly some box behind my NAT gateway) appeared to initiate a connection to port 6667 TCP (IRC) on an IP that was a "Known Bot" IP for command and control. Looking at the packet data in the alert and seeing the RST flag, I guessed this wasn't a problem. Still, I wanted to know why one of my boxes would send a RST. Could that be some sort of phone-home method designed to elude uber-packet monkeys? (Conspiracy theory hat on!)

I decided to query for session data to see if I could find any other sessions involving 193.109.122.67 (ede.nl.eu.undernet.org):


------------------------------------------------------------------------
Sensor:cel433 Session ID:5044069116374886360
Start Time:2007-03-20 18:04:19 End Time:2007-03-20 18:04:19
193.109.122.67:6667 -> 69.143.202.28:1024
Source Packets:1 Bytes:0
Dest Packets:1 Bytes:0
------------------------------------------------------------------------
Sensor:cel433 Session ID:5044103368738397945
Start Time:2007-03-20 20:17:14 End Time:2007-03-20 20:17:14
193.109.122.67:6667 -> 69.143.202.28:3072
Source Packets:1 Bytes:0
Dest Packets:1 Bytes:0

All I found were the two sessions indicated by the aggregated Snort alerts. Notice the session data shows the source and destination each sent one packet. Interesting. One of the nice aspects of SANCP is that it can report a summary of the TCP flags seen during a session. That information is shown below.

This shows the source sent a SYN ACK and the dest sent a RST. The RST caused the alert. The SYN ACK triggered the RST.

Finally we can look at the full content:


14:04:19.731091 IP 193.109.122.67.6667 > 69.143.202.28.1024:
 S 4257997451:4257997451(0) ack 2934031229 win 2048 
14:04:19.731429 IP 69.143.202.28.1024 > 193.109.122.67.6667:
 R 2934031229:2934031229(0) win 0

This doesn't tell us anything we didn't already know, but it's nice to see exactly what happened on the wire. Remember that if we weren't collecting independent, content-neutral, non-alert-triggered full content data, we wouldn't have the first SYN ACK packet. That is the key to knowing this is a backscatter event. Some unknown party spoofed my IP, 69.143.202.28, and SYN flooded 192.109.122.67 on port 6667 TCP. The SYN ACK was sent back to me, and my NAT gateway replied with a RST. Simple -- and no conspiracy. Hat off.

When Lawsuits Attack

I haven't said anything about the intrusions affecting TJX until now because I haven't felt the need to contribute to this company's woes. Today I read TJX Faces Suit from Shareholder:

The Arkansas Carpenters Pension Fund owns 4,500 shares of TJX stock, and TJX denied its request to access documents outlining the company's IT security measures and its response to the data breach.

The shareholder filed the lawsuit in Delaware's Court of Chancery Monday afternoon under a law permitting shareholders to sue for access to corporate documents in certain cases, The Associated Press reported. The pension fund wants the records to see whether TJX's board has been doing its job in overseeing the company's handling of customer data, the news agency said.

Imagine having your security measures and incident response procedures laid bare for everyone to see. (It's possible there might not be anything to review!) How would your policies and procedures fare?

The following sounds like many incidents I've investigated.

The TJX breach was worse than first thought, TJX officials recently admitted. The company initially believed that attackers had access to its network between May 2006 and January 2007. However, the ongoing investigation has turned up evidence that the thieves also were inside the network several other times, beginning in July 2005.

Originally the company was compromised for nine months, but now the scope could reach almost a year prior. The question is whether this is evidence of compromise by another group or the same group. In either case the company's security posture looks terrible.

The sad part about this sort of incident is that most if not all of the preventative systems TJX might have applied are worthless for response and forensics. I'm guessing TJX is relying on host-centric forensics like analysis of MAC times of files on artifacts on victim servers to scope the incident. I bet TJX is paying hundreds of thousands of dollars in investigative consulting right now, beyond the damage to their brand and other technical and financial recovery costs.

Hopefully these lawsuits will shed some light on TJX's security practices so other companies can learn from their mistakes. This is the sort of incident that my future National Digital Security Board would do well to investigate and report.

Ubiquitous Monitoring on the Horizon

In January I wrote The Revolution Will Be Monitored. Today I read Careful, the Boss Is Watching:

Recently, software vendor Ascentive LLC installed its new BeAware employee monitoring application on all the PCs at one of its new corporate clients. The corporation notified its employees that their Web surfing habits -- as well as their email, instant messaging, and application usage -- were now being monitored and recorded.

"Internet usage at the corporation dropped by 90 percent almost overnight," recalls Adam Schran, CEO of Ascentive. "As soon as employees knew they were being monitored, they changed their behavior."

Wow, what a bandwidth saver. Who needs to upgrade the T-3 when you actually take measures to enforce your stated security policy? The story continues:

While tools for tracking employee network usage have been available for years, emerging products such as BeAware take monitoring to a whole new level. The new BeAware 6.7 lets managers track workers' activity not only on the network or in the browser, but also in email, chatrooms, applications, and shared files. And at any unannounced moment, a manager can capture an employee's screen, read it, and even record it for posterity.

Such exhaustive monitoring may seem a bit draconian to the uninitiated, but analysts and vendors all say the use of such "Big Brother" software can make a drastic impact on productivity and security. In a recent study by AOL and Salary.com, 44.7 percent of workers cited personal Internet use as their top distraction at work. A Gallup poll conducted in 2005 indicated that the average employee spends more than 75 minutes a day using office computers for non-business purposes.

Once employees know their activities are being monitored, however, their personal computer use is quickly curtailed, Schran observes.

This reminds me of an event that happened when I was working the night shift at the AFCERT in 1999. We had witnessed a rash of attacks against vulnerable Microsoft Front Page installations. Around 2 or 3 am I noticed someone altering the Web site of an Air Force base in Florida. Looking at the source IP it looked like it might belong to someone who worked on base. I managed to tie a home telephone number to the IP and I called, asking if so-and-so was currently modifying the af.mil Web site. I remember a surprised lady answering the phone and asking, "So you can see what I'm doing right now?"

I have never been a fan of monitoring network traffic to reduce what .mil and .gov call "fraud, waste, and abuse." You won't read recommendations for using Network Security Monitoring to intercept questionable Web surfing, for example. However, this story is another data point for my prediction that we are moving to a workplace where everything is monitored, all the time.

If you try to implement this sort of activity, you better be sure to have an ironclad policy and support from your legal staff. I would call this level of invasion of privacy a wiretap.

Wine on Ubuntu

I'm finding more reasons to like running Ubuntu on the desktop. Two of my favorite Windows applications are MWSnap (a simple screen capture tool) and Irfanview (a simple image viewer and editor). (Gimp fans, please spare me your comments. I can't stand that program. It's a bulldozer when all I need is a garden shovel.)

I poked around looking for native Linux programs that might suit my needs, but then I thought "What about using Wine to run the Windows binaries on Linux?" I'd never used Wine before, but it was only an 'apt-get install wine' away from appearing on my Ubuntu laptop.

I first tried Irfanview, but I ran into the same issues as described here. After creating /home/richard/wine and putting mfc42.dll there with installation binaries for Irfanview and MWSnap, I was able to run Wine in that directory and install both programs.

Wine ended up creating the following directory structure.


richard@neely:~/.wine/drive_c/Program Files$ ls -al
total 5
drwxr-xr-x 5 richard richard 1024 2007-03-21 11:46 .
drwxr-xr-x 4 richard richard 1024 2007-03-21 11:42 ..
drwxr-xr-x 2 richard richard 1024 2007-03-21 11:42 Common Files
drwxr-xr-x 5 richard richard 1024 2007-03-21 11:43 IrfanView
drwxr-xr-x 3 richard richard 1024 2007-03-21 11:46 MWSnap

Running each program requires something like this:


richard@neely:~$ wine .wine/drive_c/Program\ Files/IrfanView/i_view32.exe 
richard@neely:~$ wine .wine/drive_c/Program\ Files/MWSnap/MWSnap.exe

Overall I am really pleased to see this working so well.

ShmooCon Talk

If you're attending ShmooCon this weekend you may have seen I am scheduled to speak at the same time as security ninjas Joe Stewart and Billy Hoffman. It's bad enough that people have to choose between Joe and Billy, my involvement as a third talk aside.

Joe and I asked the ShmooCon organizers if it might be possible to switch me to another slot, since I would like to see Joe's talk too. Based on feedback from many of you, you also want to see Joe's talk. Unfortunately, the ShmooCon organizers did not find a way to change the schedule.

This is really bad because Billy is releasing Jikto at ShmooCon, so choosing between Joe and Billy is another lousy decision.

Given the feedback from you I've heard, I'm considering my options. They are:

Talk at 1300 as scheduled.

Give up my slot and volunteer to speak at 1200 Saturday during lunch.

Give up my slot and volunteer to speak after the keynote Friday night.

Other ideas?

What are your thoughts on this? Is it worth making waves in order to deal with this situation? Thank you.

Update: Thanks for your public and private feedback. I'll just appear at my talk and make the best of it!

Tuesday, March 20, 2007

Proactive vs Reactive Security

Whenever I hear someone talk about the merits of "proactive" security vs "reactive" security I will politely nod, but you may notice a tightening of my jaw. I can't stand these sorts of comparisons. When I hear people praise proactive measures they're usually talking about "stopping attacks" rather than "watching them." Since a good portion of my technical life is spent cleaning up the messes left by people who put faith in preventing intrusions, I am a little jaded. Before I go any further, believe me, I would much rather not have intrusions occur at all. I would much rather prevent than detect and respond to intrusions. The fact of the matter is that intrusions still happen and that proactive measures aren't always that great. In fact, sometimes so-called proactive measures are worse than reactive or passive ones. How can that be?

Kelly Jackson Higgins' latest article Grab Fingerprint, Then Attack provides an example. She writes the following:

First you determine if an IDS/IPS is sitting at the perimeter, and then "fingerprint" it to find out the brand of the device, says the hacker also known as Mark Loveless, security architect for Vernier Networks. By probing the devices, "You can extrapolate what brand of IPS is blocking them and use that to plan your attack."

Different IDS/IPS products block different threats, so an attacker can use those characteristics to gather enough intelligence to pinpoint the brand name, he says. And it's not hard to distinguish an IDS from an IPS: If you can access XYZ before the attack, but not after, it's an IPS. And if there are delays in blocking your traffic, it could be an admin reading the IDS logs, Loveless says.

This concept is as old as dirt, dating all the way back to fingerprinting firewalls. However, it illustrates my point very well. A "proactive" device like an IPS would block traffic it deems malicious. An intruder smart enough to want to identify and evade said IPS could do so using test traffic, then launch an attack that sails through the IPS -- which at that point is ignorant and ineffective. The only reason the intruder could accomplish this task is that the "proactive" nature of the IPS revealed its operation, thereby providing intelligence to the intruder. In aggregate security has been degraded by a "proactive" device.

Contrast that scenario with that of the lowly, "reactive," passive network forensics appliance. All it does is record what it sees. It doesn't stop anything. It's so quiet no one knows it is there -- including the intruder. Of course it isn't blocking anything, but it is providing Network Security Monitoring data. Properly configured and used it can act as a sort of intrusion detection system as well. In aggregate security has been improved by a "reactive" or passive device.

I hope this post has challenged the convention wisdom in the same way that my diatribes against mandatory anti-virus installation may have done. I think one way to overcome the problems caused by the active device is to complement it with the passive one, but most organizations emphasize "prevention" over all else and discard detection and response.

Security Bloggers Network

I noticed security ninja and fellow former Foundstoner Mark Curphey mentioned me in a post on his departure from the Security Bloggers Network. You may wonder why I never joined SBN. When I was asked to join in December, I politely declined. I saw no benefit to myself or my readers to joining some kind of meta-feed hosted by Feedburner. Not joining SBN was probably as popular as my personal LinkedIn policy since it means I exercise some discretion regarding the parties with whom I associate.

My personal version of SBN is my Bloglines subscription. Anything I care to read is there. I probably only pay attention to 2/3 to 3/4 of the feeds on that list, so in some cases the lesser-noticed feeds are acting like bookmarks. (In other words, some of my friends may have blogs but I don't necessarily care that they trimmed their cat's toenails last weekend.)

I think it's healthy to have discussions about the state of our "security community." Debate is one of the features of a vibrant community. If no one read this blog I would still use it mainly as a way to record how I build or configure systems, or how I think about certain issues. (Prior to writing any book or major article I review my blog posts to refresh my memory on certain issues.) If you find such content helpful, great! If not, no problem!

Programming and Digital Security

I received the following question recently, so I thought I would anonymize the person asking the question but post my response publicly.

I have a question regarding programming languages and their relation to computer security research. I would appreciate your input on the following. In order for one to be able to "contribute" to security research, do you feel it is necessary for one to become familiar with programming languages?

I am fascinated by computer security and have read several books about stages of attack, malware, and defenses but have not read any books containing any code as I do not understand it. I therefore feel as if I am of no use if I cannot write tools or examine exploits on my own.

I would again really appreciate your input on this, and if you recommend learning programming languages, do you believe one can get away with knowing just one or do you feel an understanding of several is necessary (and if so, which one[s] would you suggest)?

These are great questions. I struggle with them as well because I am not a programmer. When I was a kid in 1980 I programmed my Timex Sinclair using BASIC. I remember looking at books on coding in Assembly for my Commodore 64 several years later so I could try writing cool "demo" programs to impress my BBS buddies. In high school I basically abandoned those sorts of pursuits and only used my C-64 for writing papers, much the same as I did in college from 1990-1994 with my first PC. At USAFA I took the mandatory programming course for freshmen, which basically involved me writing programs in PASCAL and then helping classmates get their versions to compile. (Talk about zero concepts of security!) I did manage to install an Ethernet NIC in my 486SX and play the first game of Doom on FalconNet at USAFA. I didn't own a computer for most of 1996 and early 1997, but by 1998 I was back online and doing my first hands-on security work at the Air Force CERT.

The Air Force formally trained me as an intelligence officer, so I've always been an analyst and operator. Since my first hands-on security work required inspecting network traffic, I've always been deeply involved with TCP/IP. My first jobs also held me responsible for detecting and responding to intrusions. I've enjoyed staying within defensive roles, although I've poked my head out to do "offensive" work occasionally.

I've managed to build a career without being a programmer, but I'm not satisfied with my level of skill or knowledge. If you review my terribly stalled reading list and my Amazon.com Wish List you'll see plenty of books on programming.

I think programming is important for several reasons.

The development community is becoming security-aware. The big pushes I see for improvement lie with groups like US-CERT's Build Security In. Most of the interesting books being published cover software security of one sort or the other. If you want to really participate in this work you need to understand programming.

Programmers can build tools to solve their problems. I have many ideas for cool tools but no ability to execute on them. Because I didn't study programming earlier I am stuck with learning a language to code my tool, then writing the tool itself. My lack of progress over the last several years indicates how tough it is to overcome this hurdle when one's primary work doesn't require programming. I end up using basic scripting capabilities to get other people's tools to solve some of my problems, which is sub-optimal.

Programmers can deeply understand security-related code. Even if I can't code my own tools, it would be extremely helpful to be able to reverse engineer other people's tools or malware, see how they solve certain problems, discover flaws in open source code, and perform other code-centric functions. This is probably the most realistic place for me to apply some beginning programming knowledge. Understanding what someone else wrote can be easier than starting to write a program from scratch.

To directly answer your question, I don't think you need to be a programmer to "contribute," but if you want to be a researcher I think it would be extremely beneficial. I am not a researcher; I am an operator. Programming would still be helpful but it's not critical.

You are definitely not "of no use" because you can't program. I hope my background demonstrates you can be "useful" while not being a programmer.

Regarding languages, here are my two cents.

Assembly: If you want to understand shellcode, I recommend learning some Assembly.

C: If you want to know write or understand operating systems and many security tools, learn C.

Perl: Perl seems to be a prerequisite for many security jobs and a lot of custom code is written in Perl. I would prefer to avoid it but I think some familiarity with Perl is helpful.

Python and/or Ruby: These two newer languages are very popular. Jose Nazario, for example, writes many cool Python tools. Metasploit 2.x was written in Perl but 3.x is written in Ruby.

Lua: I only learned about Lua recently, but it's apparently got a role in Snort 3.x and a researcher friend showed me how he uses it in his work.

I hope this answered your questions. I hope those of you with backgrounds or skills similar to mine will take heart, if you find yourselves doubting your worth compared to your programmer peers!

Monday, March 19, 2007

Bejtlich Teaching in Krakow, Poland at CONFidence 2007

I'm happy to announce I will be speaking on the Self-Defeating Network on Sunday 13 May 2007 in Krakow, Poland at CONFidence 2007. I am looking forward to speaking at a conference where no one else thinks my name is especially odd or difficult to pronounce! (Bejtlich is an Eastern European name with roots in present-day Poland, Germany, and probably the Czech Republic.) Please register while the lower rates are still in effect. Thank you.

Bejtlich Teaching at Sys Admin Magazine Conference in Baltimore

I will be teaching two half-day tutorials for the Sys Admin Technical Conference on Monday 7 May 2007 in Baltimore, MD. I'll spend the morning teaching Network Incident Response and the afternoon teaching Network Forensics. Early Bird Pricing for SA Tech 2007 ends 30 March 2007, after which the price will escalate by $250. Please register before the seats fill. Thank you.

Bejtlich at AusCERT and Secure Agility/Sydney

I'm pleased to announce I will be speaking and training in Australia in May 2007. First, I will attend the AusCERT Asia Pacific Information Technology Security Conference in Gold Coast, Australia. According to the schedule I'll be discussing the Self-Defeating Network at 1420 on Wednesday 23 May 2007. The following day I'll present half-day tutorials on Network Incident Response and Network Forensics. Registration is open now.

The day after my AusCERT tutorials I will be joining friends at Secure Agility to teach Network Security Monitoring in Sydney, Australia on Friday 25 May 2007. If you'd like to attend this class please review the class page and return the registration form to me before the class fills. Thanks to Christian Heinrich for coordinating my visit to Sydney. Secure Agility will be handling collecting class fees, and I'll post more information when that aspect of the event is finalized. Thank you.

Symantec Internet Security Threat Report Volume XI

The latest Symantec Internet Security Threat Report has been published. Again it's free with no registration required. I recommend reading the summary. Nothing really jumped out at me with this report, but it's good background data if you need to cite the state of digital security for a report.

NSM and Intrusion Detection Differences

We had a good discussion this morning in the #snort-gui channel on irc.freenode.net. I was on my usual soap box complaining that no commercial tools provide all of the data I need to implement Network Security Monitoring, while developers and employees of a certain well-known intrusion detection system didn't understand why their product didn't meet my needs.

Sguil author Bamm Visscher cut through the argument with a very astute summary. He basically said that IDS developers want "Immaculate Detection" while NSM practitioners want "Immaculate Collection." (Now you understand the image of the Immaculate Reception above.)

Bamm is exactly right. From my experience I know that no detection product is 100% accurate, and that even good alerts require investigation to see what is happening and what else might be happening. IDS developers are rightly trying to improve the quality of their products, but many people interpret their avoidance of NSM collection as a sign it isn't necessary. In other words, detection can be so good that you never need to investigate. I know some IDS developers don't agree with this misplaced notion but they argue it's too expensive to collect the sorts of data I advocate. I argue that it's too expensive (in terms of damage to the enterprise) not to collect that NSM data.

I think we will see commercial solutions during the next 1-3 years that will give me the NSM data I need to detect and respond to intrusions. Already network forensic appliance vendors are publishing APIs that can be called by IDS/IPS/SIM/SEM/SIEM/etc. products for access to network traffic collected independently of any alerting mechanism. This is a great development and I can't wait to see this sort of arrangement in production.

Nail in the TCP Options Coffin

I just listened to the relevant part of a recent SANS Webcast that mentioned my response to their conspiracy theory on SYN ACK and other packets with weird TCP options.

At first all I wanted to do with this post was link to Michal Zalewski's Museum of Broken Packets and say that SANS ISC is wasting time on a non-issue. Then I started reading some of the MOBP entries. I nearly fell out of my chair when I read this.


Exhibit 7: DoS tool changes into DoS exploit

Internet Protocol Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
Total Length: 48
Identification: 0x6fbb
Flags: 0x04 (DF)
Fragment offset: 0
Time to live: 127
Protocol: TCP (0x06)
Header checksum: 0x63b6 (correct)
Source: 64.190.25.48 (64.190.25.48)
Destination: XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX)

Transmission Control Protocol, Src Port: 1113 (1113), Dst Port: 490 (490),
 Seq: 269484601, Ack: 0
Source port: 1113 (1113)
Destination port: 490 (490)
Sequence number: 269484601
Header length: 28 bytes
Flags: 0x0002 (SYN)
Window size: 16384
Checksum: 0x023b (correct)
Options: (8 bytes)

Maximum segment size: 1460 bytes
NOP
Maximum segment size (option length = 4 bytes says option goes past end of options)

0000 XX XX XX XX XX XX XX XX XX XX XX XX 08 00 45 00 ..............E.
0010 00 30 6f bb 40 00 7f 06 63 b6 40 be 19 30 XX XX .0o.@...c.@.....
0020 XX XX 04 59 01 ea 10 10 02 39 00 00 00 00 70 02 ...Y.....9....p.
0030 40 00 02 3b 00 00 02 04 05 b4 01 02 04 03 .. .. @..;............

Yes, those last eight bytes are exactly the same as the TCP options in the SANS packet on their slide.

Michal explains:

This one comes from Andy Brown, who "does security" for a large web portal. It is a nice example of how simple error in DoS tool turned it into potential DoS exploit, and that conspiration theory is not always the best answer :). The packet you see above is generated by a DoS SYN tool called Juno-z. This tool was used against their site. As Andy describes, the fun is in the TCP options: 0x020405B4 MSS=1460, 0x01020403 NOP, MSS=0x03??

Normally the NOP option (0x01) is used to pad options so they lie on 4-byte boundaries, causing the actual MSS value to lie conveniently on a 16-bit boundary. Here, it's not - this NOP causes MSS value to end past the end of this packet. Might cause some dumb stack to have a bus/alignment fetch error because the MSS is shifted. Also, it could cause a dumb stack to read past the end of the packet causing a segmentation fault. Turns out, this was all unintentional - I talked to the author, and it was a coding glitch :-)

Here is juno-z. Notice dre mentioned juno-z and bang.c as tools that produce malformed packets. Apparently the real victims in this DoS attack repeated the bad TCP options they saw.

So, between my previous post and this one, I've identified the real DoS victim (compton.ameri.ca), the botnet C&C ordering a related attack, (thanks to the ShadowServer project), and the actual tool used to conduct the attack. I think it's time for SANS ISC to give up on this one. You are not seeing scanning, you are not seeing OS identification, you are not seeing another grand conspiracy whose secret lies in TCP options. You're seeing traffic caused by SYN floods.

Intrusion Detection RFCs

It's been three years since I think I blogged on this topic, but I noticed three RFCs on intrusion detection were published this month:

RFC 4765: The Intrusion Detection Message Exchange Format (IDMEF)

RFC 4766: Intrusion Detection Message Exchange Requirements

RFC 4767: The Intrusion Detection Exchange Protocol (IDXP)

Is anyone using these? I think Prelude does, but how about commercial products?

Instrumentation is the Next Internet Explorer

I read Rik Farrow's Musings(.pdf) in the latest USENIX ;login: and noticed this section:

[Rik read] an amazing paper by Chad Verbowski... of Microsoft Research... Flight Data Recorder (.pdf) (FDR) (say, haven’t I heard of another similarly named software project?) has the goal of capturing conﬁguration and ﬁle changes from Microsoft systems and will be shipped with Windows Vista.

Using a time window of only 6 ms, FDR captures all changes to system conﬁguration–related registry entries and ﬁles, saves the log locally, then cleverly compresses it, without losing any interesting data, before uploading the compressed logs to a server. The goal was to capture data from thousands of servers while using less than 1% of network bandwidth, with a less than 20 MB/day logﬁle per system that can be analyzed in 3 seconds.

Sounds unbelievable, but FDR manages to compress each event into an average of 0.7 of a byte. The motivation for this clever work was the discovery that 33% of system outages were related to conﬁguration changes, so tracking those changes was key to system reliability.

This made me remember a comment in the Joanna Rutkowska interview I didn't previously highlight:

"The scary part is that once an attacker [gets] into the system, we can't reliably read system memory, neither using software-based, nor hardware-based, methods. That means we can't answer the question of whether the system is clean or not," she says.

In other words, we have a problem with instrumentation. Microsoft is working on better instrumentation as demonstrated by FDR, but it's not what Joanna means or needs. We might be able to get better instrumentation if the OS is running within a hypervisor that monitors all aspects of the OS.

How does this relate to Internet Explorer? The Web browser was one of the first major components moved into the OS, once Microsoft saw how powerful Netscape was. Since then Microsoft has continued to move other features into the OS or into the suites of applications Microsoft provides, with anti-virus, host-based IPS and PC health the latest initiatives. I think we'll see greater moves towards instrumentation of the OS for security purposes, but the question will be whether non-Microsoft people and/or products will be able to access these offerings.

If these instrumentation readings are available, then we'll see vendors like Guidance Software make use of them for forensics purposes. Alternatively Microsoft might move deeper into the security space and sell its own forensics software. Many of the great Sysinternals tools which are now part of Microsoft have forensic and security purposes. That could be one sign of the future.

Criminal Intent

InfoSecSellout (who lives up to his name by posting Google Adwords on his blog while criticizing security consultants for being sellouts) posted a great story -- Conversations with a Blackhat. I'd like to highlight a few of his comments.

I recently had the opportunity to share some drinks and interesting conversation with a group of self confessed "bad guys"... These guys make a living in various ways but it seems that spamming, phishing, and carding are their main source of income and of course, once they are done with a zero day exploit they will go ahead and make a few dollars selling it to the highest bidder...

[T]hese guys, and most like them, are against Full Disclosure. Their reason for this is also very obvious, Full Disclosure takes away their opportunity to use exploits that they have been previously able to use...

[They said] "there should be laws to prevent people from releasing exploit code on the Internet." This is obviously a self serving statement as the people already breaking the current laws are not going to care about breaking some new ones and this would take care of their problem with Full Disclosure...

[They also said] "A law like this would take the information away from the actual smart people and leave the retards who feel that having alphabet soup behind their name gives them some sort of self and professional worth watching the vault."

All of this rings true to me, and I can't find a plausible way to argue against it. (It's similar to the argument over firearms in the US. If possessing guns is illegal, only criminals will have guns.) If possessing exploits is illegal, only criminals will have exploits. There will be no way to test if vendor patches work. There will be no way to test if new code possesses older vulnerabilities, or if "security products" (e.g., host IPS and/or AV) introduce previously patched vulnerabilities. There will be no way to understand how exploits work to detect and/or prevent them. Criminals would roam freely while the world sits dumb and blind. (We're not that far away from that situation now, but it would be worse in a world without free information on exploits.)

I always like reporting on real threats, because without these insights we're playing soccer-goal security.

For a timely example of the full disclosure debate, check out Chris Shiflett's publication of a vulnerability in Amazon.com's infrastructure. If you need to understand Cross Site Request Forgeries I recommend this post.

Friday, March 16, 2007

USB to Serial Adapter

My new laptop doesn't have a serial port. This is one of the great tragedies of modern laptops in my opinion. At least for situations where I want to connect to the serial port on a server or system with a serial port, I can use a USB to serial adapter like this adapter I bought at NewEgg. I tested it just now by connecting to the serial port on my old FreeBSD laptop.

First I enabled the serial port in /etc/ttys


#ttyd0   "/usr/libexec/getty std.9600"   dialup  off secure
ttyd0   "/usr/libexec/getty std.9600"   dialup  on secure

Then I restarted init to activate it.


# kill -HUP 1

On my Ubuntu laptop I attached the USB to serial adapter to a null modem and a gender changer, then connected it to the FreeBSD laptop serial port.

I installed cu(1) on Ubuntu


# apt-get install cu

then checked dmesg output to ensure I had a device to which I could connect.


[17213831.716000] usb 1-1: new full speed USB device using uhci_hcd and address 2
[17213831.876000] usb 1-1: configuration #1 chosen from 1 choice
[17213831.984000] usbcore: registered new driver usbserial
[17213831.984000] drivers/usb/serial/usb-serial.c: USB Serial support registered for generic
[17213831.984000] usbcore: registered new driver usbserial_generic
[17213831.984000] drivers/usb/serial/usb-serial.c: USB Serial Driver core
[17213831.988000] drivers/usb/serial/usb-serial.c: USB Serial support registered for pl2303
[17213831.992000] pl2303 1-1:1.0: pl2303 converter detected
[17213831.992000] usb 1-1: pl2303 converter now attached to ttyUSB0
[17213831.992000] usbcore: registered new driver pl2303
[17213831.992000] drivers/usb/serial/pl2303.c: Prolific PL2303 USB to serial adaptor driver

Now that I saw /dev/ttyUSB0 was enabled, I connected to it.


richard@neely:~$ cu -l /dev/ttyUSB0
Connected.

FreeBSD/i386 (orr.taosecurity.com) (ttyd0)

login: richard
Password:
Last login: Fri Mar 16 11:57:49 on ttyv1
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.

FreeBSD 6.1-SECURITY (GENERIC) #0: Wed Feb 14 15:33:28 UTC 2007

Welcome to FreeBSD!

That's it. If I needed to set a different speed I'd use the -s switch. For example, if 9600 above was 19200 in /etc/ttys, I'd use syntax like


richard@neely:~$ cu -l /dev/ttyUSB0 -s 19200

Now I know I can rely on this USB to serial adapter when I visit servers in the data center.

Way to Go Joanna

I briefly met Joanna Rutkowska at Black Hat Federal 2006 when she spoke about rootkits. Today I saw she was interviewed by Dark Reading and said the following:

Still, she worries that security technology and research is too prevention-oriented and doesn't emphasize detection enough. "The whole industry is focusing on prevention, and we have all those anti-exploitation technologies, which are very helpful indeed. But I'm so surprised that no one cares about detection," she says. "Every time there's prevention, there is some bypass method" created.

Without detection, there's no way to know if an attacker has grabbed administrative access to a machine, she says. And if you can't see that an attacker has infiltrated the system, nothing in that system will be "reliable" anymore. "The scary part is that once an attacker [gets] into the system, we can't reliably read system memory, neither using software-based, nor hardware-based, methods. That means we can't answer the question of whether the system is clean or not," she says. (emphasis added)

Wow. I am so pleased to read someone of Johanna's caliber stressing the need for detection. I have been working on slides for ShmooCon and I plan to talk about this very subject, and you probably know I've been saying for years that prevention eventually fails. Her comment about reliability of evidence relates to my TaoSecurity Pyramid of Trust, where I mentioned Johanna with respect to her techniques to defeat memory capture.

Incident Response Clarifications

Recently I posted Five Thoughts on Incident Response. Based on the comments and some blog responses I wanted to clarify what I originally posted. The first three items seemed to attract the most attention so I'll only address those.

Anti-Virus is (or should not be) an incident response tool. The emphasis here is on response. I agree that AV is often an incident detection tool, and ideally an incident avoidance tool. However, if you think AV is going to help recover from a totally compromised system, you are probably going to be upset by the results.

Your default incident recovery strategy should be to rebuild from scratch. The emphasis here is on recovery. I am not saying your default incident response strategy should be to rebuild from scratch. Your default response strategy should be to investigate to determine how the victim was compromised, what aspects of Confidentiality/Integrity/Availability were violated, and so on. I agree that any response which begins with re-imaging the victim is a recipe for failure.

SPAN ports should not be the default traffic access option. I'm standing by this one. The only time SPAN ports are superior to taps is the situation where intra-switch traffic needs to be seen. Otherwise, spend a few dollars and get a product designed to grant reliable traffic access. I'm talking about professional ways to perform incident response here. Hardware is the easiest thing to gain budget for in any organization. It's easier to buy a piece of hardware than it is to send a person to training, or hire new help, or bring in outside consultants, or any other activity that could benefit a security shop.

I appreciate the other recommendations I've seen. These are only a few big thoughts which struck me based on recent engagements. I have over a dozen recommendations in my Network Security Operations class and I think I cover similar material in Extrusion Detection.

Thursday, March 15, 2007

Cisco's Secure Routers Are Winning

Infonetics Research published a synopsis of a recent report they authored. I'd like to highlight a few points.

Growth in the network security appliance and software market is expected to slow to the single-digits after 2007 as content security gateways and NAC products begin to infringe on network security product budgets.

“The most important appliance category to watch over the next year is secure routers. Sales were up 25% in 2006 and this year will pass $1 billion in worldwide sales, representing a significant portion of the overall network security market..."

Secure routers account for 29% of the total integrated security appliance market in 2006 and will continue to increase their share of the market through at least 2010...

Cisco continues to lead the overall network security market, with 38% worldwide revenue share in 2006, posting growth in all network security market segments tracked by Infonetics

Juniper and Check Point are tied for second, each with 9% worldwide revenue in 2006...

Infonetics’ network security report provides worldwide and regional market size and forecasts and worldwide market share for integrated security appliances in 6 price categories, secure routers, SSL VPN gateways, VPN and firewall software, and host- and network-based IDS/IPS products. Companies tracked include 3Com/TippingPoint, AEP, Alcatel-Lucent, Array, Aventail, Check Point, Cisco, Citrix, CA, D-Link, Enterasys, F5, Fortinet, Juniper, McAfee, NETASQ, Nokia, Nortel, Secure Computing, SonicWALL, Symantec, WatchGuard, ZyXEL, and others.

I wrote last year that all network security functions will end up in the switch. This Infonetics story is talking about "secure routers," which I assume are devices like Cisco's Integrated Services Routers. That idea is consistent with my vision for the "security switch."

The revenue share numbers are interesting; Cisco dominates, and when you add in Juniper (where the wheels are apparently coming off) you've got 56% of the market accounted for.

It's important for technical folks to understand that the people with budgets who ultimately procure equipment are not looking for the best technical solution. They are looking for something that is good enough. This is called satisficing. Decision-makers want to spend the least amount of money necessary to get them to a level they believe is as good as their peer group. Given these conditions, buying a Cisco ISR that advertises "Routing, Security and VPN, Voice, Wireless [and] Optimization of network bandwidth and applications" makes perfect sense.

If you're a vendor that makes a network traffic inspection and/or manipulation product, you're going to either end up on someone's switch or be a niche player. I see two functions that will not end up on the switch:

Network forensic appliances: ISRs are not going to incorporate a multi-TB storage array and the associated analysis software required for network forensics.

SIM/SEM/SIEM/Network Security Management suites: Just because all of your gear will end up belonging to Cisco doesn't mean you'll want to use something like Cisco MARS to make sense of it. It's likely you'll buy into the Self-Defeating Network but I see room for vendors like Q1 Labs to survive and maybe thrive.

Please note I am not talking about all security here. I am talking about devices that inspect and act upon network traffic.

Wednesday, March 14, 2007

Someone Likes This Blog

I just received an email from an editor at IT Security saying "Congratulations, you have been included in ITSecurity.com's list of the top 59 most influential security experts of 2007." I guess I didn't upset the person who originally emailed me about his proposed choices asking me to comment on the names listed. In my reply I said I was not comfortable participating in the creation of such a list, but it appears I am still included anyway -- mainly for this blog:

Richard Bejtlich, President and CEO of TaoSecurity, has written several books on network security, including specific topics like internal network intrusion and digital forensics. In his book, “Hacking Exposed,” Bejtlich was the first to publish the term “network security monitoring.” He blogs about network security, naturally, with a penchant for including all the code and computer feedback, which transforms his blog posts into helpful how-to guides.

That's a nice write-up, although a four-page case study in Hacking Exposed, 4th Ed hardly qualifies it as "my book." I did popularize the term NSM but it's based on Todd Heberlein's Network Security Monitor paper/code that became the Air Force's ASIM sensor.

In any case, thanks for the mention. Welcome new readers. :)

Update: Another list, by someone in the scene. Thanks for the mention. :) The definitive discussion on this topic appears at Matasano.

Five Thoughts on Incident Response

Speaking of incidents, I thought it might be interesting to share a few brief observations based on incidents I've worked recently. Please remember this is a blog post. If you expect thorough explanations of these points with footnotes, historical references, arguments to the contrary expertly swept aside, etc., please wait for a future book! :)

Anti-Virus is (or should not be) an incident response tool. I am baffled when I see machines compromised, and the owners think a magic signature from their AV vendor is going to save the day. In this day and age intruders who gain kernel level control of a host often disable AV and will not give up the fight so easily. My second point relates to this one.

Your default incident recovery strategy should be to rebuild from scratch. By scratch I mean reinstallation from original trusted media and re-installation of applications and data.

Today, in 2007, I am still comfortable saying that existing hardware can usually be trusted, without evidence to the contrary, as a platform for reinstallation. This is one year after I saw John Heasman discuss PCI rootkits (.pdf). I was lucky enough to spend a few hours chatting with John and fellow NGS Software guru David Litchfield after John's talk on firmware rootkits (.pdf). John's talks indicate that the day is coming when even hardware that hosted a compromised OS will eventually not be trustworthy.

One day I will advise clients to treat an incident zone as if a total physical loss has occurred and new platforms have to be available for hosting a reinstallation. If you doubt me now, wait for the post in a few years where I link back to this point. In brief, treat an incident like a disaster, not a nuisance. Otherwise, you will be perpetually compromised.

SPAN ports should not be the default traffic access option. I cannot tell you how much time, effort, and frustration has accompanied the use (or attempted use) of SPAN ports in incident response situations.
- "The SPAN port is already used."
- "The SPAN port can't do that." (although it probably can, the network engineer either doesn't know how to set it up or doesn't want it configured to help the security team)
- "Do you see anything? No? Try now. No? Try now. No?"
- "You only see half the traffic? Wait, try this. Now you see double? Ok, try now."
For Pete's sake, buy a tap, put it in the proper place, and stand back while the packets are collected properly.

A Linux live CD is not a substitute for a real network security monitoring platform. Upon realizing that Cisco MARS is not an incident response solution, I was desperate to collect some form of useful network-centric data at one client site. In a last-ditch attempt to salvage a bad situation my on-site colleague deployed a Network Security Toolkit live CD on top of a box previously running Linux natively. I was able to SSH into it, mount the local filesystem, and start writing packets to the hard drive using Tshark's ring buffer. This is absolutely making the best out of a mess, which is standard incident response behavior.

I would ask anyone who turns to a live CD for their monitoring needs to avoid the temptation to think Snort on a live CD on spare, old hardware is anything like Snort on properly sized, configured, deployed hardware. Furthermore, Snort != monitoring. Live CDs are fine for assessment work but they are nearly worthless for packet capture. Needless to say I was able to talk my colleague through a FreeBSD installation and was soon collecting data in a somewhat better environment.

When you are compromised, you are probably not facing a zero-day exploit unique to you and not capable of being prevented. When you are compromised you're most likely suffering from some fairly modern variant of attack code that nevertheless contains exploits dating back to 2002. For some reason people seem to feel better if they think the incident is caused by some uber elite intruder who saved up his killer 0-day just for their enterprise. In reality someone probably connected an infected laptop physically to the network, or via VPN, and found a way to get a worm or other malware to the segment of the enterprise running "production" machines that never get patched.

Do you have any IR stories or lessons to share? Please post them as comments or write on your blog, then post a link here as a comment. Thank you.

Richard Bejtlich is training in a city near you! Seats in classes at the Sys Admin Magazine Conference, Techno Security 2007, SANSFIRE 2007, and Black Hat Las Vegas are filling fast -- register today

Keith Jones on Incident Preparation

My friend Keith Jones published a second article, titled Inevitable Corporate Incidents, and How to Bite Them Back! Keith discusses ways to prepare for the incidents that will happen. I cover similar material in my Network Incident Response portion of my classes.

Tuesday, March 13, 2007

Preview of IPv6 Problems

I recommend reading this advisory from Core Security:

The OpenBSD kernel contains a memory corruption vulnerability in the code that handles IPv6 packets. Exploitation of this vulnerability can result in:

1) Remote execution of arbitrary code at the kernel level on the vulnerable systems (complete system compromise), or;

2) Remote denial of service attacks against vulnerable systems (system crash due to a kernel panic)

The issue can be triggered by sending a specially crafted IPv6 fragmented packet...

OpenBSD systems using default installations are vulnerable because the default pre-compiled kernel binary (GENERIC) has IPv6 enabled and OpenBSD's firewall does not filter inbound IPv6 packets in its default configuration...

[I]n order to exploit a vulnerable system an attacker needs to be able to inject fragmented IPv6 packets on the target system's local network. This requires direct physical/logical access to the target's local network -- in which case the attacking system does not need to have a working IPv6 stack -- or the ability to route or tunnel IPv6 packets to the target from a remote network.

I'm not posting this story to criticize OpenBSD. I'd like to use it as a preview of problems we're going to see in all operating systems as security researchers (of the above- and underground variety) scrutinize IPv6 stacks. TCP/IP stack vulnerabilities can be a real problem because the main defense is patching. Sometimes you can filter odd packets before they hit vulnerable stacks, but what do you do if your filtering device is also vulnerable? There are no "unnecessary services" to disable, unless you choose not to run IPv6.

For my previous thoughts on IPv6 I recommend reading this post. From what I hear managers and CIOs in .gov, .mil, and elsewhere mostly think IPv6 brings "security" and other goodies; they are clearly not clued in to the problems on the horizon.

Update: Thanks to Shirkdog for prompting me to see if FreeBSD shares the same code. You can use Robert Watson's Kernel Cross Reference to see a comparison of OpenBSD HEAD and FreeBSD RELENG6. I'll leave it to the experts to decide if the problem exists in FreeBSD too. I'm worried because the BSDs all use the same KAME IPv6 code.

Richard Bejtlich is training in a city near you! Seats in classes at the Sys Admin Magazine Conference, Techno Security 2007, SANSFIRE 2007, and Black Hat Las Vegas are filling fast -- register today

Friday, March 09, 2007

Reviews of FISMA and Wireshark Posted

Yes, you are reading that title correctly. After four months of inactivity I managed to read and review two new books. The first is FISMA by Laura Taylor. From my four star review:

I am no fan of the FISMA law. I've posted several stories on my blog explaining why I think FISMA is a waste of taxpayer money. Laura Taylor's FISMA Certification and Accreditation Handbook, however, is a good book if you are unfortunate enough to be tasked with performing FISMA work.

The second is Wireshark & Ethereal Network Protocol Analyzer Toolkit by Angela Orebaugh. From my four star review:

Despite the new title, Wireshark & Ethereal Protocol Analyzer Toolkit (WEPAT) is a second edition of Ethereal Packet Sniffing (EPS). I reviewed that book almost three years ago, in May 2004. WEPAT has replaced all of the earlier screen captures with Wireshark replacements. Unfortunately, WEPAT is largely a repeat of EPS, really only featuring a new wireless chapter. If you own EPS, you don't need to upgrade. If you don't own EPS but want to learn how to use Wireshark, I recommend buying WEPAT.

I'm still reading and plan to continue posting reviews going forward.

Sourcefire Is Now FIRE

With the appearance of Sourcefire as FIRE on the NASDAQ, I'd like to congratulate Marty Roesch and friends for bringing their company to the public market. I can't think of another company where one can chat with the CTO and founder in IRC.

Several of you have asked for my thoughts on this development. I posted Thoughts on Sourcefire IPO in October and I don't see anything that changes those opinions. Since then, I've been working with several customers, including one who brought me to a Sourcefire sales demo. At that demo, and in meetings with other customers, the ability for a detection product to act like a Security Event Management / Security Information-Incident Management (SEM/SIM) solution repeatedly arose. Sourcefire's products can feed a SEM/SIM but their Defense Center is not a SEM/SIM.

This is a big hurdle for Sourcefire. I don't see customers buying a Sourcefire intrusion sensor, and RNA, and a Defense Center, and then paying more money for a SEM/SIM. Instead I see customers adding an IDS module to their router or switch and feeding everything into MARS. (You know how much I love MARS, so this is not something I want to see happen. It's just what is happening.)

I think Q1 Labs has the right idea, even though I don't have hands-on time with their gear (yet). Products which are a SEM/SIM and a network management platform are going to be one of the few network-centric security products to not be collapsed into switches. (Network forensic appliances, due to their storage requirements, will also not collapse into switches.) If Sourcefire moves up the food chain into the Q1 Labs model, then I think they have a future as an independent security vendor. If they concentrate on their IDS/IPS solution they will eventually be purchased by a bigger security company like Cisco or a competitor.

Wednesday, March 07, 2007

Bejtlich Teaching at Black Hat Las Vegas 2007 Training

I'll be teaching two sessions of TCP/IP Weapons School, Black Hat Edition at Black Hat in Las Vegas, 28-29 July and 30-31 July 2007. This is my first time teaching independently at Black Hat, although I did teach with Foundstone in 2002 and 2003. TCP/IP Weapons School is usually a four-day class (if I cover layers 2-7), so for Black Hat I am teaching the best material in a two-day format, repeating the class twice for those who have a conflict with another class.

Please register early. Seats are already being filled.

Remember I am also organizing a meeting of Single Digit Security Service Providers, and I will make details available as they develop.

Bejtlich Teaching at SANSFIRE 2007

I'll be teaching a special one-day course, Enterprise Network Instrumentation, at SANSFIRE 2007 in Washington, DC on 25 July 2007. ENI is a one-day course designed to teach all methods of network traffic access. If you have a network you need to monitor, ENI will teach you what equipment is available (hubs, switch SPAN ports, taps, bypass switches, matrix switches, and so on) and how to use it effectively. Everyone else assumes network instrumentation is a given. ENI teaches the reality and provides practical solutions.

Please register while there are still seats available. Thank you.

Snort Report 4 Posted

The fourth Snort Report -- Installing Snort 2.7 Beta 1 and Snort CVS -- has been posted. In the article I show how to install the latest Snort beta from an archive and how to operate a patched version by installing Snort from CVS.

If you missed the earlier editions they are linked at the top of the list on my company research page.

Cigital Starts Blogging

Gary McGraw from Cigital told me his company has started a blog called Justice League. (Cue Ted Knight saying "Meanwhile, at the Hall of Justice...") Gary wrote on of my favorite books of 2006 so I expect to see interesting comments on his company's new blog. Good luck!

Tuesday, March 06, 2007

Nothing to See Here

Recently I noticed a posting at the SANS Internet Storm Center titled Deformed TCP Options - Got Packets? The story featured packets like the following:


07:11:45.781421 IP (tos 0x0, ttl 113, id 9433, offset 0, flags [DF],
 proto: TCP (6), length: 48)
 129.250.128.21.1256 > www.xxx.yyy.zzz.1229: S, cksum 0x5ed4 (correct), 
 2627126762:2627126762(0) ack 257795 6091 win 1460 

0x0000: 4500 0030 24d9 4000 7106 4944 81fa 8015 E..0$.@.q.ID....
0x0010: wwxx XXYY 04e8 04cd 9c96 c5ea 99a8 7cfb ...{..........|.
0x0020: 7012 05b4 5ed4 0000 0204 05b4 0102 0403 p...^...........


07:11:51.517325 IP (tos 0x0, ttl 113, id 21659, offset 0, flags [DF], 
 proto: TCP (6), length: 48)
 129.250.128.21.1252 > www.xxx.yyy.zzz.1070: S, cksum 0xa40c (correct), 
 1381904945:1381904945(0) ack 2301854615 win 1460 

0x0000: 4500 0030 549b 4000 7106 764c 81fa 8015 E..0T.@.q.vL....
0x0010: wwxx XXYY 04e4 042e 525e 3231 8933 8397 ........R^21.3..
0x0020: 7012 05b4 a40c 0000 0204 05b4 0102 0403 p...............

In English, 129.250.128.21 is sending SYN ACK packets to the obfuscated IP addresses. 129.250.128.21 is sending SYN ACK packets because it is replying to SYN packets from the obfuscated IP. I knew right away what was happening. Some people called it "backscatter." When I wrote my first paper on the subject in 1999 I used the exceptionally exciting term "third party effects," thereby consigning my research to some vague corner of the Internet. In brief, 129.250.128.21 was being SYN flooded; it is a victim, not a perpetrator.

The ISC handlers wrote:

Generally the source ports are 80, 6667, 6666, and 443. However there are also a number of packets with random source ports between 1024 and about 1300. Ports 1024-1300 are also used as the target port. Looking at the Dshield data for ports 6666 and 6667 there was a small peak for these ports as source ports on the 27/28 Feb.

So we are seeing multiple hosts sending SYN,ACK with truncated option to targets (sometimes both scan the same IP) from known ports. The response is typically a RST, or RST,ACK. The window size is also often changed in the response.

What it looks like is a mapping/fingerprinting exercise using the target stacks' response to bad options. But at this stage the end goal is unclear.

I thought this was not right, and I told the handlers as much. They didn't believe me, so I asked them if they thought it might be a good idea to step out of the cave of packet conspiracies and ask the owner of 129.250.128.21 what was happening. (This was the strategy I followed in my original paper.) They decided not to pursue this non-technical investigative method, so I emailed hostmaster [at] ameri [dot] ca based on the resolution for the IP: compton.ameri.ca.

Since then I've been exchanging emails with Brad Dreisbach. He wrote:

i have been getting tcp syn attacked for about 3 weeks now. i have re-installed the OS on the host just to be safe, but im fairly sure my systems are secure. i have also taken measures with my upstream, whom i also work for, to migitate the attack. some stuff is still getting through but at this point im just waiting for the attackers to give up...

There you go -- Brad is being attacked. 129.250.128.21 is the victim. 129.250.128.21 is not doing some kind of reconnaissance.

When I asked Brad if he had captured any traffic related to his ongoing denial of service attack, he replied:

this pcap file doesnt appear to be a tcp syn attack, but some other form of tcp attack. whoever is doing this is changing their stategy though. the attack has changed a few times over its duration. they even attacked me via ipv6 for a few hours.

Here's part of the traffic he sent:


2007-02-22 17:40:20.844503 IP (tos 0x0, ttl 119, id 5698, offset 0,
 flags [DF], proto: TCP (6), length: 40)
 68.102.133.89.3072 > 129.250.128.21.80: .,
 cksum 0xd7b5 (correct), 0:0(0) ack 0 win 0
        0x0000:  4500 0028 1642 4000 7706 21bf 4466 8559
        0x0010:  81fa 8015 0c00 0050 0000 0000 0000 0000
        0x0020:  5010 0000 d7b5 0000

2007-02-22 17:40:20.855549 IP (tos 0x0, ttl 120, id 48455, offset 0,
 flags [DF], proto: TCP (6), length: 40)
 72.207.227.118.3072 > 129.250.128.21.80: .,
 cksum 0x752f (correct), 0:0(0) ack 0 win 0
        0x0000:  4500 0028 bd47 4000 7806 1733 48cf e376
        0x0010:  81fa 8015 0c00 0050 0000 0000 0000 0000
        0x0020:  5010 0000 752f 0000

2007-02-22 17:40:20.948933 IP (tos 0x0, ttl 118, id 3193, offset 0,
 flags [DF], proto: TCP (6), length: 40)
 71.237.133.30.3072 > 129.250.128.21.80: .,
 cksum 0xd469 (correct), 0:0(0) ack 0 win 0
        0x0000:  4500 0028 0c79 4000 7606 293c 47ed 851e
        0x0010:  81fa 8015 0c00 0050 0000 0000 0000 0000
        0x0020:  5010 0000 d469 0000

This looks like an ACK flood to port 80 TCP. The victim will reply with RST packets (not RST ACK) if it can.

I think ISC got thrown off the trail because SANS has a history of seeing global conspiracies in weird packet patterns. This dates back almost ten years and is documented in my first book. The unfortunate result of this attitude is hundreds, if not thousands, of other security analysts have "learned" to see these patterns as malicious too. ISC also spent too much time concentrating on "broken TCP options" on the backscatter traffic. They did not accept my observation that victims under DoS attack tend to respond slowly, weirdly, and eventually not at all when they are flooded.

If you decide to post a nasty reply here because you love ISC and SANS, please keep in mind you're only seeing the public side of the story. ISC and I exchanged very polite emails, but I am not going to disclose the additional justifications of the ISC evil packet theory here because those emails were private.

I think ISC does a great service to the security community, but I would like to see SANS officially abandon its adherence to this misguided view of the world. I am not trying to pick a fight with SANS. All I want is for security analysts to know this pattern is often seen as "malicious" by many when it is utterly benign -- at least as far as the backscatter recipient is concerned. Remember the site sending the SYN ACK, RST ACK, or RST traffic is responding to a SYN flood (to an open or closed port, respectively) or to an ACK flood.

The moral of the story is that it is not a good idea to assume the world is out to get you when a stray packet arrives at your doorstep. A second lesson is that it sometimes make sense to investigate issues by doing human analysis rather than peering at the world through the eyes of Tcpdump/Ethereal/Wireshark. A third lesson is that it makes little sense to interpret traffic as being malicious if the probablility of the "scanner" learning anything remotely useful is extremely low.

Update: One of my friends noted that the Virginia Tech DShield has 129.250.128.21 listed as one of their "Top 10 Most Wanted" "attackers". This demonstrates the trouble with automated reporting systems.

Update 2: Be sure to read the exciting conclusion here.

Saturday, March 03, 2007

Full Content

Thanks to this story I learned of the latest 2006 FISMA Report. If you want a summary of the findings, read the story. Here I'd like to highlight an amazing paragraph on page 14.

B. Incident Detection

Agencies must be able to quickly detect and respond to incidents. During the next year, OMB will work with federal agencies to increase the exchange of packet level (full content) information regarding incidents which have penetrated an agency’s perimeter. Sharing this data will enable more effective analysis of attacks targeting multiple Federal agencies, and may enable more timely responses to new threats. The sharing of intrusion data will also improve the knowledge base of analysts in Federal agencies. (emphasis added)

I have a feeling the person who wrote that part of the report has read Tao or another one of my works.

I am detecting a trend. People are starting to realize they cannot understand or even detect incidents without having facts to analyze. Most security products provide inferences in the form of alerts; the product makes a judgement on what it's seen. Alerts are helpful but never sufficient. Analysts are driven to investigate NSM data in the form of facts; amateurs are satisfied with managing inferences in the form of alerts.

Full content data is the best form of network-centric fact since it completely represents a conversation. Session data is another excellent form of network-centric fact, but it sacrifices some granularity. Statistical data is a third form of network-centric fact, but it is least helpful because so much detail has been lost.

In an attempt to head off a blizzard of complaints, note I say "network-centric." As I've said many times elsewhere, sometimes a single accurate log statement like "File X containing Y was transferred between hosts A and B at time C over an encrypted channel using protocol Z" is more helpful than a million packets. However, sometimes the only data you have is that which you can gather passively and independently. I call that self-reliant Network Security Monitoring.

Expect to hear more on this topic at my ShmooCon talk. (Why oh why did they schedule me against Joe Stewart? I really looked forward to seeing his talk. Argh.)

I am not alone in these thoughts. Please read this blog post by Tate Hansen. I'd reproduce the whole thing here since I like all of it.

Thursday, March 01, 2007

Printing to Lexmark from Ubuntu

I'm pleased to report I was able to use the printer configuration menu in Ubuntu to print to the Lexmark C530dn shared via SMB on a Windows XP SP2 host. All I needed to do was install the C520 CUPS driver and I was able to print a .pdf launched from within a Gmail attachment. I'm very pleased, but I don't think this setup supports dual-sided printing.

External Monitor on X60s

I decided today to test support for external monitors on my Thinkpad X60s. I figured it was a good idea to make sure it worked before trying to deliver a presentation. It turns out I could not use FN+F7 to send the display to an external monitor. However, I did find a document that recommended making the following modification to xorg.conf. It sends display to the LCD and the VGA out simultaneously.


richard@neely:/etc/X11$ diff -u xorg.conf.orig xorg.conf
--- xorg.conf.orig      2007-03-01 10:43:11.000000000 -0500
+++ xorg.conf   2007-03-01 10:37:47.000000000 -0500
@@ -103,6 +103,8 @@
        Identifier      "Intel Corporation Mobile Integrated Graphics Controller"
        Driver          "i810"
        BusID           "PCI:0:2:0"
+Option   "Clone" "yes"
+Option    "MonitorLayout" "CRT,LFP"   
 EndSection
 
 Section "Monitor"

Now when I attach a monitor to the VGA port, the laptop can be seen on it.

Security Mentoring

In January I reviewed Mike Rothman's Pragmatic CSO. Related to that book I saw my name mentioned in a post by Cutaway. He writes:

I am, however, more concerned about Mike’s approach to young security professionals. “Buy my book, it is a good approach for dealing with executive management” is not, in my honest opinion, an effective way of approaching our next generation. Sure, he has made the information available to the public, but security professionals are pummeled with literature almost on a daily basis. His book might be on the list of top purchases but where is the actual teacher to help with the interpretation to assist with the evolution of the concepts within an individual?

I understand Cutaway's concerns, but I think his request is unrealistic. I have plenty of experience with mentorship, starting as a cadet and continuing during my officer years in the Air Force. In my experience it is difficult for the mentoree to obtain mentorship (of any type) even when mentorship is a job requirement for the supervisor. In fact, my last commander asked me for job advice when I was leaving, rather than try to convince me to stay!

I wholeheartedly support Mike Rothman's recommendations for people to read his book. Does anyone think technical authors write books to make money? Almost no one can make a living being a technical author, unless you have very modest needs, no family, and have multiple books in print simultaneously and constantly.

So why write? Technical authors (and many others) write to share their ideas. One of the main reasons I wrote Tao was the desire to not have to repeat the same material whenever I trained a new analyst. Instead I could say "read my book, and then we'll talk." I think writing a good book has the capability to do far more good for the community at large than a one-on-one relationship. Books certainly scale better than people.

Speaking of people, those who you would probably want as mentors are most likely the busiest people you'll ever meet. Mike is running his own company. I'm running my own company. I regularly receive emails from students and others asking for help with their PhD topics and other issues. If I have the time to help I usually respond in the form of a blog post or a CC to a mailing list so what I say can be shared.

If you really want a human mentor I recommend joining a security association like ISSA, hanging out in an IRC channel with people you respect, and/or joining a company or organization to work for someone you want to emulate. I've done all three at various stages of my career.

Saturday, March 31, 2007

Friday, March 30, 2007

Thursday, March 29, 2007

Wednesday, March 28, 2007

Tuesday, March 27, 2007

Monday, March 26, 2007

Sunday, March 25, 2007

Saturday, March 24, 2007

Friday, March 23, 2007

Thursday, March 22, 2007

Wednesday, March 21, 2007

Tuesday, March 20, 2007

Monday, March 19, 2007

Friday, March 16, 2007

Thursday, March 15, 2007

Wednesday, March 14, 2007

Tuesday, March 13, 2007

Friday, March 09, 2007

Wednesday, March 07, 2007

Tuesday, March 06, 2007

Saturday, March 03, 2007

Thursday, March 01, 2007

About Me

twitter.com/taosecurity

SANS Incident Detection Summit

TCP/IP Weapons School 2.x at Black Hat DC 2010 31 Jan - 01 Feb

Books by Richard Bejtlich

Association of Former Information Warriors

Blog Archive

Hakin9 Magazine

BSD Magazine

Google Analytics