Wednesday, November 24, 2010

Trying Ubuntu 10.10 in AWS Free Usage Tier

After trying 60 Free Minutes with Ubuntu 10.10 in Amazon EC2 yesterday, I decided to take the next step and try the AWS Free Usage Tier. This blog post by Jay Andrew Allen titled Getting Started (for Free!) with Amazon Elastic Cloud Computing (EC2) helped me.

One important caveat applies: this activity will not be completely free. The AMI chose uses a 15 GB filesystem, and the terms of the free usage stipulate no more than a 10 GB filesystem. I'll pay $0.50 per month for the privilege of using a prebuilt Ubuntu AMI. Since I'm an AMI n00b, I decided to pay the $0.50. At some point when I am comfortable creating or trusting 10 GB AMIs, maybe I'll switch.

  1. First I visited http://aws.amazon.com/ec2/ and signed up for Amazon EC2. At Amazon Web Services Sign In, I chose to "Identity Verification by Telephone." When I completed sign up I received three emails: 1) Amazon Virtual Private Cloud Sign-Up Confirmation; 2) Amazon Elastic Compute Cloud Sign-Up Confirmation; and 3) Amazon Simple Notification Service Sign-Up Confirmation.

  2. Next I visited the AWS Management Console at https://console.aws.amazon.com/ec2/home. In Getting Started, I choose Launch Instance. I had to decide what sort of virtual machine I wanted to run. I decided to try a 64 bit Ubuntu 10.10 Amazon Machine Image (AMI) I found mentioned at http://uec-images.ubuntu.com/releases/maverick/release/ and at http://alestic.com/. I selected an AMI available at Amazon's us-east-1 facility, identified as ami-548c783d. This AMI uses Amazon's Elastic Block Store (EBS) so that changes persist.

  3. Under Instance Details, I chose:

    Number of Instances: 1
    Availability Zone: No Preference
    Instance Type: Micro (t1.micro, 613 MB)

  4. Under Select Launch Instances, I chose:

    Kernel ID: Use Default
    RAM Disk ID: Use Default
    No Monitoring
    No User Data
    No Tags

  5. Next I had to Create and Download Key Pair. That produced a file called taosecuritykey.pem which we'll use later.

  6. I chose

    Security Groups: Default

  7. When I reviewed my choices I saw:

    AMI: Ubuntu AMI ID ami-548c783d (x86_64)
    Name:
    Description:
    Number of Instances: 1
    VPC Subnet:
    Availability Zone: No Preference
    Instance Type: Micro (t1.micro)
    Instance Class: On Demand
    Number of Instances: 1
    Availability Zone: No Preference
    Instance Class: On Demand
    Maximum Price:
    Request Valid From:
    Availability Zone Group:
    Request Valid Until:
    Launch Group:
    Persistent Request:
    Placement Group:
    Strategy:
    Monitoring: Disabled
    Bursting:
    Kernel ID: Use Default
    RAM Disk ID: Use Default
    IP Address:
    User Data:
    Key Pair Name: taosecuritykey
    Security Group(s): default

  8. Finally I launched Launched the instance and visited the Instances Page.

  9. In order to SSH to my AMI I had to add "SSH" to my Security Group and I decided to add my own IP address (with /32 netmask) as the IP allowed to traverse the firewall.

  10. To SSH to the system I had to find the hostname in the EC2 Instance listing at the bottom of the page, e.g., ec2-obfuscated.compute-1.amazonaws.com. I also had to set permissions on my .pem so I could use it with SSH:


    richard@neely:~$ mv taosecuritykey.pem .ssh/
    richard@neely:~$ chmod 400 .ssh/taosecuritykey.pem

  11. Then I connected to the AMI:

    richard@neely:~$ ssh -v -i .ssh/taosecuritykey.pem \
    ubuntu@ec2-obfuscated.compute-1.amazonaws.com

    Linux domU-12-31-39-14-F9-0C 2.6.35-22-virtual #33-Ubuntu SMP
    Sun Sep 19 21:05:42 UTC 2010 x86_64 GNU/Linux

    Ubuntu 10.10

    Welcome to Ubuntu!
    * Documentation: https://help.ubuntu.com/

    System information as of Wed Nov 24 20:36:24 UTC 2010

    System load: 0.0 Processes: 60
    Usage of /: 4.4% of 14.76GB Users logged in: 0
    Memory usage: 6% IP address for eth0: 10.206.250.250
    Swap usage: 0%

    Graph this data and manage this system at https://landscape.canonical.com/
    ---------------------------------------------------------------------
    At the moment, only the core of the system is installed. To tune the
    system to your needs, you can choose to install one or more
    predefined collections of software by running the following
    command:

    sudo tasksel --section server
    ---------------------------------------------------------------------

    The programs included with the Ubuntu system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.

    Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
    applicable law.

    To run a command as administrator (user "root"), use "sudo ".
    See "man sudo_root" for details.

    ubuntu@domU-12-31-39-14-F9-0C:~$


At this point my system was working, so I poked around a little.

ubuntu@domU-12-31-39-14-F9-0C:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 15G 665M 14G 5% /
none 290M 108K 290M 1% /dev
none 297M 0 297M 0% /dev/shm
none 297M 48K 297M 1% /var/run
none 297M 0 297M 0% /var/lock

ubuntu@domU-12-31-39-14-F9-0C:~$ sudo netstat -natup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 468/sshd
tcp 0 48 10.206.250.250:22 98.218.35.11:57655 ESTABLISHED 577/sshd: ubuntu [p
tcp6 0 0 :::22 :::* LISTEN 468/sshd
udp 0 0 0.0.0.0:68 0.0.0.0:* 387/dhclient3

ubuntu@domU-12-31-39-14-F9-0C:~$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 12:31:39:14:f9:0c
inet addr:10.206.250.250 Bcast:10.206.251.255 Mask:255.255.254.0
inet6 addr: fe80::1031:39ff:fe14:f90c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:429 errors:0 dropped:0 overruns:0 frame:0
TX packets:337 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:67019 (67.0 KB) TX bytes:49777 (49.7 KB)
Interrupt:9

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

ubuntu@domU-12-31-39-14-F9-0C:~$ sudo lft -D eth0 www.bejtlich.net

Tracing __________________________________.

TTL LFT trace to vhost.identityvector.com (205.186.148.46):80/tcp
1 10.206.248.3 0.8ms
2 216.182.232.236 0.5ms
3 216.182.232.64 0.4ms
** [neglected] no reply packets received from TTLs 4 through 6
7 dca-edge-18.inet.qwest.net (65.120.78.57) 2.1ms
8 dcp-brdr-03.inet.qwest.net (205.171.251.110) 4.9ms
** [neglected] no reply packets received from TTL 9
10 216.88.34.170 3.7ms
11 cr02-1-1.iad1.net2ez.com (65.97.48.206) 9.7ms
12 65.97.50.26 4.2ms
13 static-70-32-64-246.mtsvc.net (70.32.64.246) 4.2ms
14 vzd052.mediatemple.net (205.186.147.5) 3.7ms
15 [target] vhost.identityvector.com (205.186.148.46):80 4.1ms

I decided to update the AMI using apt.

$ sudo apt-get update
$ sudo apt-get upgrade

After reboot

ubuntu@domU-12-31-39-14-F9-0C:~$ uname -a
Linux domU-12-31-39-14-F9-0C 2.6.35-22-virtual #35-Ubuntu
SMP Sat Oct 16 23:19:29 UTC 2010 x86_64 GNU/Linux

I decided to try sending email from the system:

ubuntu@domU-12-31-39-14-F9-0C:~$ sudo apt-get install exim4-daemon-light
...edited...
ubuntu@domU-12-31-39-14-F9-0C:~$ sudo dpkg-reconfigure exim4-config
* Stopping MTA for restart [ OK ]
* Restarting MTA [ OK ]

ubuntu@domU-12-31-39-14-F9-0C:~$ echo "test mail 1557" | mailx -v -s "test mail 1557" richard@bejtlich.net
LOG: MAIN
<= ubuntu@domu-12-31-39-14-f9-0c.compute-1.amazonaws.com U=ubuntu P=local S=489
ubuntu@domU-12-31-39-14-F9-0C:~$ delivering 1PLMPR-0000eu-4P
R: dnslookup for richard@bejtlich.net
T: remote_smtp for richard@bejtlich.net
Connecting to ASPMX.L.GOOGLE.COM [74.125.93.27]:25 ... connected
SMTP<< 220 mx.google.com ESMTP g35si18125523qcs.170
SMTP>> EHLO domU-12-31-39-14-F9-0C.compute-1.internal
SMTP<< 250-mx.google.com at your service, [174.129.106.239]
250-SIZE 35651584
250-8BITMIME
250 ENHANCEDSTATUSCODES
SMTP>> MAIL FROM: SIZE=1523
SMTP<< 250 2.1.0 OK g35si18125523qcs.170
SMTP>> RCPT TO:
SMTP<< 250 2.1.5 OK g35si18125523qcs.170
SMTP>> DATA
SMTP<< 354 Go ahead g35si18125523qcs.170
SMTP>> writing message and terminating "."
SMTP<< 250 2.0.0 OK 1290632265 g35si18125523qcs.170
SMTP>> QUIT
LOG: MAIN
=> richard@bejtlich.net R=dnslookup T=remote_smtp H=ASPMX.L.GOOGLE.COM [74.125.93.27]
LOG: MAIN
Completed

I also decided to try an IPv6 tunnel client:
ubuntu@domU-12-31-39-14-F9-0C:~$ sudo apt-get install miredo

ubuntu@domU-12-31-39-14-F9-0C:~$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 12:31:39:14:f9:0c
inet addr:10.206.250.250 Bcast:10.206.251.255 Mask:255.255.254.0
inet6 addr: fe80::1031:39ff:fe14:f90c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5025 errors:0 dropped:0 overruns:0 frame:0
TX packets:2849 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2717010 (2.7 MB) TX bytes:1308113 (1.3 MB)
Interrupt:9

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

teredo Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet6 addr: 2001:0:53aa:64c:102c:3760:517e:9510/32 Scope:Global
inet6 addr: fe80::ffff:ffff:ffff/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1280 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B) TX bytes:144 (144.0 B)

ubuntu@domU-12-31-39-14-F9-0C:~$ host ipv6.google.com
ipv6.google.com is an alias for ipv6.l.google.com.
ipv6.l.google.com has IPv6 address 2001:4860:800f::68

ubuntu@domU-12-31-39-14-F9-0C:~$ ping6 2001:4860:800f::68
PING 2001:4860:800f::68(2001:4860:800f::68) 56 data bytes
64 bytes from 2001:4860:800f::68: icmp_seq=1 ttl=59 time=3.70 ms
64 bytes from 2001:4860:800f::68: icmp_seq=2 ttl=59 time=3.97 ms
64 bytes from 2001:4860:800f::68: icmp_seq=3 ttl=59 time=4.73 ms
^C
--- 2001:4860:800f::68 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.707/4.140/4.736/0.435 ms

I did that all under an hour, so before the first hour finished I shut down the AMI.

The next time I want to use it, I'll visit the console, start it, and SSH. I don't have any real plans for this AMI besides experimentation, for now. I'll probably keep my eye on this ec2ubuntu Google Group too.

Tuesday, November 23, 2010

60 Free Minutes with Ubuntu 10.10 in Amazon EC2

I decided to try Ubuntu in the Cloud because 1) I had a few minutes this afternoon and 2) it's free. If you follow the directions on their Web site you'll have access to an Ubuntu 10.10 server for 60 minutes, hosted by Amazon Elastic Compute Cloud (Amazon EC2). It's really simple, so easy a caveman could do it. (Ouch.)

  1. First make sure you have a public-private SSH key pair.


    richard@neely:~$ ssh-keygen -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/richard/.ssh/id_rsa):
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /home/richard/.ssh/id_rsa.
    Your public key has been saved in /home/richard/.ssh/id_rsa.pub.
    The key fingerprint is:
    c6:e0:9c:84:74:3d:2d:09:b3:a2:e5:97:7b:63:59:da richard@neely
    The key's randomart image is:
    +--[ RSA 2048]----+
    | . +o o |
    | . o o= . |
    | + + o |
    | + = = |
    | . . * S . |
    | . o = |
    | . * E |
    | o . |
    | |
    +-----------------+

  2. Next visit www.launchpad.net and create and account.

  3. Visit the editsshkeys page created for your account (like https://launchpad.net/~taosecurity/+editsshkeys for me) and paste the content of your public SSH key into the window.

  4. Now it's time for https://10.cloud.ubuntu.com/. I read:

    Try Ubuntu 10.10 Server in Amazon EC2, entirely on our dime!

    All you need is an SSH client, and an SSH public key associated with your Launchpad.net account, and we will launch an Ubuntu Server instance in Amazon EC2 for you.

    We will give you the hostname and you can SSH directly to the instance with your public SSH key on file in Launchpad. You will have full sudo (root) access, so take it for an hour-long joyride, install applications, configure services, test your programs, and evaluate the overall experience. We will terminate and clean up the instance automatically within an hour.


    I selected Ubuntu Server (10.10) with WordPress for fun.

  5. WAIT while the server is provisioned. It takes a few minutes but the Web site keeps refreshing to keep you informed.

  6. When done, SSH to the server us user ubuntu. Be ready to enter your SSH keyphrase.

    richard@neely:~$ ssh ubuntu@184.72.80.52
    The authenticity of host '184.72.80.52 (184.72.80.52)' can't be established.
    RSA key fingerprint is 56:df:06:bf:30:c6:d6:26:76:2f:f1:6f:51:97:86:70.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added '184.72.80.52' (RSA) to the list of known hosts.
    Linux ip-10-212-127-243 2.6.35-22-virtual #33-Ubuntu SMP Sun Sep 19 23:54:13 UTC 2010 i686 GNU/Linux
    Ubuntu 10.10
    Hello taosecurity, welcome to the Cloud!
    This instance will terminate around Tue Nov 23 21:37:00 UTC 2010"

    Welcome to Ubuntu!
    * Documentation: https://help.ubuntu.com/

    System information as of Tue Nov 23 20:42:00 UTC 2010

    System load: 0.35 Processes: 76
    Usage of /: 7.0% of 9.84GB Users logged in: 0
    Memory usage: 17% IP address for eth0: 10.212.127.243
    Swap usage: 0% IP address for eth0:0: 184.72.80.52

    Graph this data and manage this system at https://landscape.canonical.com/
    ---------------------------------------------------------------------
    At the moment, only the core of the system is installed. To tune the
    system to your needs, you can choose to install one or more
    predefined collections of software by running the following
    command:

    sudo tasksel --section server

  7. At this point I had a fully functional server with Wordpress installed. I played with the server to create a first post.


  8. I also tested how quickly I could add software. WOW.

    sudo apt-get install ubuntu-desktop
    ...edited...
    Fetched 429MB in 28s (15.2MB/s)


  9. I started a second SSH session to tunnel the X protocol and started Firefox:


  10. From another server I scanned the EC2 instance to see what services are exposed:

    tao001:~# nmap -sV 184.72.80.52

    Starting Nmap 4.62 ( http://nmap.org ) at 2010-11-23 15:56 EST
    Interesting ports on ec2-184-72-80-52.compute-1.amazonaws.com (184.72.80.52):
    Not shown: 1710 closed ports
    PORT STATE SERVICE VERSION
    22/tcp open ssh (protocol 2.0)
    25/tcp open smtp Postfix smtpd
    80/tcp open http Apache httpd 2.2.16 ((Ubuntu))
    5901/tcp open vnc VNC (protocol 3.8)
    6001/tcp open X11 (access denied)
    1 service unrecognized despite returning data.
    If you know the service/version, please submit the following fingerprint at
    http://www.insecure.org/cgi-bin/servicefp-submit.cgi :
    SF-Port22-TCP:V=4.62%I=7%D=11/23%Time=4CEC2A95%P=x86_64-unknown-linux-gnu%
    SF:r(NULL,27,"SSH-2\.0-OpenSSH_5\.5p1\x20Debian-4ubuntu4\r\n");
    Service Info: Host: ec2-184-72-80-52.compute-1.amazonaws.com

    Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
    Nmap done: 1 IP address (1 host up) scanned in 7.457 seconds

  11. I ran Tshark to capture traffic and created a capture with this protocol distribution:

    richard@neely:~$ tshark -q -r tshark.pcap -z io,phs
    can't open file /home/richard//tmpssl/Renegotiating_TLS_20091104_pub/caps/apache22_wget_DHE/server.key

    ===================================================================
    Protocol Hierarchy Statistics
    Filter: frame

    frame frames:3764 bytes:424367
    eth frames:3764 bytes:424367
    ip frames:3750 bytes:422885
    udp frames:177 bytes:120953
    dns frames:80 bytes:8271
    ntp frames:24 bytes:2160
    data frames:70 bytes:105980
    dcerpc frames:3 bytes:4542
    icmp frames:17 bytes:1710
    tcp frames:3556 bytes:300222
    http frames:54 bytes:100166
    data-text-lines frames:10 bytes:17428
    media frames:1 bytes:818
    image-jfif frames:1 bytes:4434
    png frames:1 bytes:1194
    xml frames:2 bytes:1430
    unreassembled frames:1 bytes:2962
    smtp frames:14 bytes:3392
    imf frames:1 bytes:561
    tcp.segments frames:1 bytes:116
    http frames:1 bytes:116
    ssh frames:1 bytes:105
    ipv6 frames:14 bytes:1482
    udp frames:14 bytes:1482
    dns frames:14 bytes:1482
    ===================================================================


Near the end of my hour I got this warning in the shell:

Broadcast Message from root@ip-10-212-127-243
(somewhere) at 21:17 ...

You have about 10 minutes before instance termination

So, I logged out and that was it!

I suggest everyone give this a try, especially if you've never spun up an EC2 instance. Next I'd like to try the AWS Free Usage Tier.

Thanks to Ubuntu and Amazon EC2 for making this such an easy process.

My only concern is this: how easy would it be to spin up free VMs like this for nefarious means?

Monday, November 22, 2010

Stop Killing Innovation

I hear and read a lot about how IT is supposed to innovate to enable "the business." Anytime I see "IT" in one part of a sentence and "the business" in another, a little part of me dies. Somewhere there is a Nirvana where "thought leaders" understand that there is no business without IT, that IT is as part of the business as the sales person or factory worker or janitor, and that IT would be better off not constantly justifying its existence to "the business." But I digress.

I want to address the "innovation" issue in this post. CIO magazine recently published an interview with Vinnie Mirchandani titled Taking Business Risks With Your IT Budget. I liked what Mr Mirchandani had to say, although I'm going to omit his multiple references to "cloud." Instead, consider how he sees innovation in IT:

More [CIOs] want to be [innovators], but organizations don’t let them...

In the 1980s, we talked about IT as a competitive advantage... In the 1990s, we didn’t hear much of that at all, and IT started reporting to CFOs. In the early 2000s, the CFO made IT a compliance function for auditing and security.

We’ve beaten the innovation out of CIOs at many companies. We want them to be risk mitigators, not innovators. People are afraid to be associated with any failure. They buy IT from vendors that are safe choices. They know they’re overspending, yet they do it anyway...


Mr Mirchandani doesn't say this, but he could have also mentioned that many managers expect CIOs to be "productivity engines," meaning they inherently shrink their budget every year. This drives cost reduction as the primary goal for an IT shop -- not innovation. It's like expecting the business development team to concentrate on decreasing the amount of money spent per new customer acquired, while not caring so much on the quantity or quality of the new customers -- if any!

So what to do?

The best thing they could do is get out from under the CFO. Go to your CEO and say, “I want to report to you.” Make sure the CFO doesn’t stand in the way. Some CIOs will get fired for doing that. Others will get a chance...

Cost pressure isn't limited to those who only report to the CFO, but he doesn't address that issue.

The shocking thing about corporate IT is that without realizing it, 85 percent to 90 percent of the IT spend is with a vendor, including outsourcers and the staff you buy from them...

When you’re spending 90 percent of your money with a vendor, you have only a sliver left for [internal] talent — yet it’s with your own internal talent that you can innovate. There’s very little left for CIOs to innovate with.

The more progressive CIOs are saying they’ve overdone it with outsourcing and are starting to hire their own enterprise architects and business analysts and other strategic resources.


To me this is the crux of the issue. Businesses cannot outsource innovation. Businesses can crush innovation pretty easily though.

I found one comment he made about the cloud to be very interesting:

CIOs resist it. It’s not secure, they say. It’s not always available. CIOs say cloud vendors go down too often.

I know CIOs who haven’t run a full disaster-recovery drill for years and turn around and say that the cloud isn’t production-ready.


So, my message to readers is this: if cost-out, five nines uptime, outsourced workforces, and other failed strategies are your goal, forget innovation. If you want innovation to thrive, try considering the alternatives.

Thursday, November 18, 2010

The Problem Is with Gmail

In my last post I lamented a problem with Sendmail on FreeBSD. I was trying to troubleshoot a problem sending email from FreeBSD's periodic scripts to Gmail. I've determined that, as crazy as this sounds, Gmail is broken. (Some of you are probably not surprised. If you want to skip the drama and see the bottom line, scroll to the bottom of the post.)

Let me start my case by showing network transcripts of one successful "periodic" email and one unsuccessful "periodic" email. I'm not going to change any email addresses in this post.

The following email is delivered successfully. Computer vm.taosecurity.com sits behind NAT so the public IP is 73.128.35.11. The entries prior to the SMTP transactions (e.g. 074.125.091.027.00025-073.128.035.011.57184: and similar) were added by Tcpflow, which I used to render the transcript manually.

074.125.091.027.00025-073.128.035.011.57184: 220 mx.google.com ESMTP my6si2476635qcb.101

073.128.035.011.57184-074.125.091.027.00025: EHLO vm.taosecurity.com

074.125.091.027.00025-073.128.035.011.57184: 250-mx.google.com at your service, [73.128.35.11]
250-SIZE 35651584
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING

073.128.035.011.57184-074.125.091.027.00025: MAIL From:<analyst@vm.taosecurity.com> SIZE=917

074.125.091.027.00025-073.128.035.011.57184: 250 2.1.0 OK my6si2476635qcb.101

073.128.035.011.57184-074.125.091.027.00025: RCPT To:<taosecurity@gmail.com>
DATA

074.125.091.027.00025-073.128.035.011.57184: 250 2.1.5 OK my6si2476635qcb.101
354 Go ahead my6si2476635qcb.101

073.128.035.011.57184-074.125.091.027.00025: Received: from vm.taosecurity.com (localhost [127.0.0.1])
.by vm.taosecurity.com (8.14.4/8.14.4) with ESMTP id oAJ66xa2021306
.for <root@vm.taosecurity.com>; Fri, 19 Nov 2010 01:06:59 -0500 (EST)
.(envelope-from analyst@vm.taosecurity.com)
Received: (from root@localhost)
.by vm.taosecurity.com (8.14.4/8.14.4/Submit) id oAJ66xF4021296
.for root; Fri, 19 Nov 2010 01:06:59 -0500 (EST)
.(envelope-from analyst)
Date: Fri, 19 Nov 2010 01:06:59 -0500 (EST)
From: analyst <analyst@vm.taosecurity.com>
Message-Id: <201011190606.oAJ66xF4021296@vm.taosecurity.com>
To: root@vm.taosecurity.com
Subject: vm.taosecurity.com security run output

Checking setuid files and devices:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

vm.taosecurity.com login failures:

vm.taosecurity.com refused connections:

-- End of security output --

073.128.035.011.57184-074.125.091.027.00025: .

074.125.091.027.00025-073.128.035.011.57184: 250 2.0.0 OK 1290128829 my6si2476635qcb.101

073.128.035.011.57184-074.125.091.027.00025: QUIT

074.125.091.027.00025-073.128.035.011.57184: 221 2.0.0 closing connection my6si2476635qcb.101

The following email fails to be delivered. Computer r200b has the public IP address 73.128.35.11 as shown. Again the lines are prepended by Tcpflow headers.

074.125.091.027.00025-073.128.035.011.19228: 220 mx.google.com ESMTP f23si2500736qcq.34

073.128.035.011.19228-074.125.091.027.00025: EHLO r200b.taosecurity.com

074.125.091.027.00025-073.128.035.011.19228: 250-mx.google.com at your service, [73.128.35.11]
250-SIZE 35651584
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING

073.128.035.011.19228-074.125.091.027.00025: MAIL From:<richard@r200b.taosecurity.com> SIZE=1658

074.125.091.027.00025-073.128.035.011.19228: 250 2.1.0 OK f23si2500736qcq.34

073.128.035.011.19228-074.125.091.027.00025: RCPT To:<taosecurity@gmail.com>
DATA

074.125.091.027.00025-073.128.035.011.19228: 250 2.1.5 OK f23si2500736qcq.34
354 Go ahead f23si2500736qcq.34

073.128.035.011.19228-074.125.091.027.00025: Received: from r200b.taosecurity.com (localhost [127.0.0.1])
.by r200b.taosecurity.com (8.14.4/8.14.4) with ESMTP id oAJ17UwM063291
.for <root@r200b.taosecurity.com>; Thu, 18 Nov 2010 20:07:30 -0500 (EST)
.(envelope-from richard@r200b.taosecurity.com)
Received: (from root@localhost)
.by r200b.taosecurity.com (8.14.4/8.14.4/Submit) id oAJ17UKs063248
.for root; Thu, 18 Nov 2010 20:07:30 -0500 (EST)
.(envelope-from richard)
Date: Thu, 18 Nov 2010 20:07:30 -0500 (EST)
From: Richard Bejtlich <richard@r200b.taosecurity.com>
Message-Id: <201011190107.oAJ17UKs063248@r200b.taosecurity.com>
To: root@r200b.taosecurity.com
Subject: r200b.taosecurity.com security run output

Checking setuid files and devices:

Checking for uids of 0:

root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

r200b.taosecurity.com kernel log messages:
+++ /tmp/security.QW4ZT9Yc.2010-11-18 20:07:29.000000000 -0500

+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled

r200b.taosecurity.com login failures:

Nov 17 07:51:58 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.
Nov 17 07:52:02 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.

r200b.taosecurity.com refused connections:

Checking for a current audit database:

Database created: Thu Nov 18 19:05:00 EST 2010

Checking for packages with security vulnerabilities:

0 problem(s) in your installed packages found.

-- End of security output --

073.128.035.011.19228-074.125.091.027.00025: .

074.125.091.027.00025-073.128.035.011.19228: 550-5.7.1 [73.128.35.11] The IP you're using to send mail is not authorized to
550-5.7.1 send email directly to our servers. Please use the SMTP relay at your
550-5.7.1 service provider instead. Learn more at
550 5.7.1 http://mail.google.com/support/bin/answer.py?answer=10336 f23si2500736qcq.34

073.128.035.011.19228-074.125.091.027.00025: QUIT

Darn. As you can see, Gmail claims "The IP you're using to send mail is not authorized to send email directly to our servers." Is that true? Didn't I just send email from the same IP address, as far as Gmail was concerned?

There is basically no difference between these emails, other than the contents of the security reports in each. (Hint, hint.)

I can prove the Gmail error message is bogus.

Let's start by showing both computers can send email to Gmail. If I don't send email using the periodic scripts, I can send email to Gmail from both systems successfully.

First, the message from host vm succeeds (and I saw it in my Inbox).

vm# mail -v -s "From vm" taosecurity@gmail.com
Test from vm.
.
EOT
taosecurity@gmail.com... Connecting to [127.0.0.1] via relay...
220 vm.taosecurity.com ESMTP Sendmail 8.14.4/8.14.4; Fri, 19 Nov 2010 01:31:20 -0500 (EST)
>>> EHLO vm.taosecurity.com
250-vm.taosecurity.com Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> MAIL From:<analyst@vm.taosecurity.com> SIZE=58
250 2.1.0 <analyst@vm.taosecurity.com>... Sender ok
>>> RCPT To:<taosecurity@gmail.com>
>>> DATA
250 2.1.5 <taosecurity@gmail.com>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 oAJ6VKaj021400 Message accepted for delivery
taosecurity@gmail.com... Sent (oAJ6VKaj021400 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 vm.taosecurity.com closing connection

vm# grep oAJ6VKaj021400 /var/log/maillog

Nov 19 01:31:20 vm sm-mta[21400]: oAJ6VKaj021400: from=<analyst@vm.taosecurity.com>,
size=393, class=0, nrcpts=1, msgid=<201011190631.oAJ6VKlp021399@vm.taosecurity.com>,
proto=ESMTP, daemon=Daemon0, relay=localhost [127.0.0.1]

Nov 19 01:31:20 vm sendmail[21399]: oAJ6VKlp021399: to=taosecurity@gmail.com, ctladdr=analyst
(1001/1001), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30058, relay=[127.0.0.1]
[127.0.0.1], dsn=2.0.0, stat=Sent (oAJ6VKaj021400 Message accepted for delivery)

Nov 19 01:31:21 vm sm-mta[21402]: oAJ6VKaj021400: to=<taosecurity@gmail.com>,
ctladdr=<analyst@vm.taosecurity.com> (1001/1001), delay=00:00:01, xdelay=00:00:01,
mailer=esmtp, pri=30393, relay=gmail-smtp-in.l.google.com. [74.125.91.27], dsn=2.0.0, stat=Sent
(OK 1290130290 g35si2521350qcs.118)

Second, the message from r200b succeeds (and I saw it in my Inbox).

r200b:/root# mail -v -s "From r200b" taosecurity@gmail.com
Test from r200b.
.
EOT
taosecurity@gmail.com... Connecting to [127.0.0.1] via relay...
220 r200b.taosecurity.com ESMTP Sendmail 8.14.4/8.14.4; Thu, 18 Nov 2010 20:31:01 -0500 (EST)
>>> EHLO r200b.taosecurity.com
250-r200b.taosecurity.com Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> MAIL From:<richard@r200b.taosecurity.com> SIZE=64
250 2.1.0 <richard@r200b.taosecurity.com>... Sender ok
>>> RCPT To:<taosecurity@gmail.com>
>>> DATA
250 2.1.5 <taosecurity@gmail.com>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 oAJ1V1Xx063384 Message accepted for delivery
taosecurity@gmail.com... Sent (oAJ1V1Xx063384 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 r200b.taosecurity.com closing connection

r200b:/root# grep oAJ1V1Xx063384 /var/log/maillog

Nov 18 20:31:01 r200b sm-mta[63384]: oAJ1V1Xx063384: from=<
richard@r200b.taosecurity.com>, size=417, class=0, nrcpts=1, msgid=<
201011190131.oAJ1V1SP063383@r200b.taosecurity.com>, proto=ESMTP, daemon=Daemon0,
relay=localhost [127.0.0.1]

Nov 18 20:31:01 r200b sendmail[63383]: oAJ1V1SP063383: to=taosecurity@gmail.com, ctladdr=richard
(1001/1001), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30064, relay=[127.0.0.1]
[127.0.0.1], dsn=2.0.0, stat=Sent (oAJ1V1Xx063384 Message accepted for delivery)

Nov 18 20:31:02 r200b sm-mta[63386]: oAJ1V1Xx063384: to=<taosecurity@gmail.com>,
ctladdr=<richard@r200b.taosecurity.com> (1001/1001), delay=00:00:01, xdelay=00:00:01,
mailer=esmtp, pri=30417, relay=gmail-smtp-in.l.google.com. [74.125.91.27], dsn=2.0.0, stat=Sent
(OK 1290130252 m5si2493978qcu.183)

As you can see, both computers, vm and r200b, can send email fine to Gmail.

Now this will blow your mind. What happens when I manually send an email with the content of the periodic email that Gmail refused to accept from r200b?

Let's send it from vm, which so far has had no trouble talking to Gmail under any circumstances, whether sending manual email or its own periodic output.

vm# mail -v -s "From vm, fake periodic output for blog" taosecurity@gmail.com
Checking setuid files and devices:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

r200b.taosecurity.com kernel log messages:
+++ /tmp/security.QW4ZT9Yc.2010-11-18 20:07:29.000000000 -0500
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled

r200b.taosecurity.com login failures:
Nov 17 07:51:58 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.
Nov 17 07:52:02 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.

r200b.taosecurity.com refused connections:

Checking for a current audit database:

Database created: Thu Nov 18 19:05:00 EST 2010

Checking for packages with security vulnerabilities:

0 problem(s) in your installed packages found.

-- End of security output --
.
EOT
taosecurity@gmail.com... Connecting to [127.0.0.1] via relay...
220 vm.taosecurity.com ESMTP Sendmail 8.14.4/8.14.4; Fri, 19 Nov 2010 02:03:17 -0500 (EST)
>>> EHLO vm.taosecurity.com
250-vm.taosecurity.com Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> MAIL From:<analyst@vm.taosecurity.com> SIZE=1026
250 2.1.0 <analyst@vm.taosecurity.com>... Sender ok
>>> RCPT To:<taosecurity@gmail.com>
>>> DATA
250 2.1.5 <taosecurity@gmail.com>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 oAJ73HIk021517 Message accepted for delivery
taosecurity@gmail.com... Sent (oAJ73HIk021517 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 vm.taosecurity.com closing connection

vm# grep oAJ73HIk021517 /var/log/maillog

Nov 19 02:03:17 vm sm-mta[21517]: oAJ73HIk021517: from=<analyst@vm.taosecurity.com>,
size=1361, class=0, nrcpts=1, msgid=<201011190703.oAJ73G8n021516@vm.taosecurity.com>,
proto=ESMTP, daemon=Daemon0, relay=localhost [127.0.0.1]

Nov 19 02:03:17 vm sendmail[21516]: oAJ73G8n021516: to=taosecurity@gmail.com, ctladdr=analyst
(1001/1001), delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=31026, relay=[127.0.0.1]
[127.0.0.1], dsn=2.0.0, stat=Sent (oAJ73HIk021517 Message accepted for delivery)

Nov 19 02:03:18 vm sm-mta[21519]: oAJ73HIk021517: to=<taosecurity@gmail.com>,
ctladdr=<analyst@vm.taosecurity.com> (1001/1001), delay=00:00:01, xdelay=00:00:01,
mailer=esmtp, pri=31361, relay=gmail-smtp-in.l.google.com. [74.125.91.27], dsn=5.0.0,
stat=Service unavailable

Nov 19 02:03:18 vm sm-mta[21519]: oAJ73HIk021517: oAJ73IIk021519: DSN: Service unavailable

What's up with that, Gmail? If I sniff the traffic I can see Gmail refuse it again:

074.125.091.027.00025-073.128.035.011.58727: 220 mx.google.com ESMTP o12si2579217qcs.143

073.128.035.011.58727-074.125.091.027.00025: EHLO vm.taosecurity.com

074.125.091.027.00025-073.128.035.011.58727: 250-mx.google.com at your service, [73.128.35.11]
250-SIZE 35651584
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING

073.128.035.011.58727-074.125.091.027.00025: MAIL From:<analyst@vm.taosecurity.com> SIZE=1361

073.128.035.011.58727-074.125.091.027.00025: MAIL From:<analyst@vm.taosecurity.com> SIZE=1361

074.125.091.027.00025-073.128.035.011.58727: 250 2.1.0 OK o12si2579217qcs.143

073.128.035.011.58727-074.125.091.027.00025: RCPT To:<taosecurity@gmail.com>
DATA

074.125.091.027.00025-073.128.035.011.58727: 250 2.1.5 OK o12si2579217qcs.143
354 Go ahead o12si2579217qcs.143

073.128.035.011.58727-074.125.091.027.00025: Received: from vm.taosecurity.com (localhost [127.0.0.1])
.by vm.taosecurity.com (8.14.4/8.14.4) with ESMTP id oAJ73HIk021517
.for <taosecurity@gmail.com>; Fri, 19 Nov 2010 02:03:17 -0500 (EST)
.(envelope-from analyst@vm.taosecurity.com)
Received: (from root@localhost)
.by vm.taosecurity.com (8.14.4/8.14.4/Submit) id oAJ73G8n021516
.for taosecurity@gmail.com; Fri, 19 Nov 2010 02:03:16 -0500 (EST)
.(envelope-from analyst)
Date: Fri, 19 Nov 2010 02:03:16 -0500 (EST)
From: analyst <analyst@vm.taosecurity.com>
Message-Id: <201011190703.oAJ73G8n021516@vm.taosecurity.com>
To: taosecurity@gmail.com
Subject: From vm, fake periodic output for blog

Checking setuid files and devices:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

r200b.taosecurity.com kernel log messages:
+++ /tmp/security.QW4ZT9Yc.2010-11-18 20:07:29.000000000 -0500
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled

r200b.taosecurity.com login failures:
Nov 17 07:51:58 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.
Nov 17 07:52:02 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.

r200b.taosecurity.com refused connections

Checking for a current audit database:

Database created: Thu Nov 18 19:05:00 EST 2010

Checking for packages with security vulnerabilities:

0 problem(s) in your installed packages found.

-- End of security output --

073.128.035.011.58727-074.125.091.027.00025: .

074.125.091.027.00025-073.128.035.011.58727: 550-5.7.1 [73.128.35.11] The IP you're using to send mail is not authorized to
550-5.7.1 send email directly to our servers. Please use the SMTP relay at your
550-5.7.1 service provider instead. Learn more at
550 5.7.1 http://mail.google.com/support/bin/answer.py?answer=10336 o12si2579217qcs.143

073.128.035.011.58727-074.125.091.027.00025: QUIT

The transcript ends with the bogus "The IP you're using to send mail is not authorized to send email directly to our servers." message. So what's the bottom line?

Gmail appears to be filtering email based on content, providing a bogus "The IP you're using to send mail is not authorized to send email directly to our servers." message.

Does anyone have another explanation? I would love to hear it. Thank you.

Incidentally, I am considering workarounds that WOULD use my ISP's SMTP server and hopefully avoid this problem. Also, I don't expect to see this issue using the Gmail Web interface. It must be a filter Gmail applies when users talk to their SMTP servers directly.

FreeBSD Sendmail Problem

Thanks for the help with my script issue recently. I was wondering if anyone has seen this problem with Sendmail? I aliased root to "taosecurity at gmail dot com" as shown below. (I used the real email address on the computer.) This is a fresh install of FreeBSD 8.1.

$ uname -a
FreeBSD vm.taosecurity.com 8.1-RELEASE FreeBSD 8.1-RELEASE #0: \
Mon Jul 19 02:55:53 UTC 2010 \
root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386

vm# diff -u /etc/aliases /etc/aliases.orig
--- /etc/aliases 2010-11-18 10:30:37.000000000 -0500
+++ /etc/aliases.orig 2010-11-18 10:30:26.000000000 -0500
@@ -18,7 +18,6 @@
# root's email from here.

# root: me@my.domain
-root: taosecurity at gmail dot com

# Basic system aliases -- these MUST be present
MAILER-DAEMON: postmaster
vm# newaliases
/etc/mail/aliases: 28 aliases, longest 21 bytes, 300 bytes total

My /etc/mail and /var/spool directories are pristine from the factory"

vm# ls -al /etc/mail
total 300
drwxr-xr-x 2 root wheel 512 Oct 31 11:28 .
drwxr-xr-x 20 root wheel 2048 Nov 18 10:30 ..
-rw-r--r-- 1 root wheel 6818 Jul 18 22:25 Makefile
-rw-r--r-- 1 root wheel 2905 Jul 18 22:25 README
-rw-r--r-- 1 root wheel 634 Jul 18 22:25 access.sample
-rw-r--r-- 1 root wheel 1695 Nov 18 10:30 aliases
-rw-r----- 1 root wheel 65536 Nov 18 10:30 aliases.db
-rw-r--r-- 1 root wheel 58276 Jul 18 22:25 freebsd.cf
-rw-r--r-- 1 root wheel 4118 Jul 18 22:25 freebsd.mc
-r--r--r-- 1 root wheel 40751 Jul 18 22:25 freebsd.submit.cf
-r--r--r-- 1 root wheel 901 Jul 18 22:25 freebsd.submit.mc
-r--r--r-- 1 root wheel 5657 Jul 18 22:25 helpfile
-rw-r--r-- 1 root wheel 409 Jul 18 22:25 mailer.conf
-rw-r--r-- 1 root wheel 253 Jul 18 22:25 mailertable.sample
-rw-r--r-- 1 root wheel 58276 Jul 18 22:25 sendmail.cf
-r--r--r-- 1 root wheel 40751 Jul 18 22:25 submit.cf
-rw-r--r-- 1 root wheel 582 Jul 18 22:25 virtusertable.sample

vm# ls -al /var/spool
total 16
drwxr-xr-x 8 root wheel 512 Jul 18 22:23 .
drwxr-xr-x 23 root wheel 512 Nov 12 11:45 ..
drwxrwx--- 2 smmsp smmsp 512 Nov 18 10:00 clientmqueue
drwxrwxr-x 2 uucp dialer 512 Nov 12 16:45 lock
drwxr-xr-x 2 root daemon 512 Jul 18 22:23 lpd
drwxr-xr-x 2 root daemon 512 Nov 18 10:31 mqueue
drwx------ 2 root daemon 512 Jul 18 22:23 opielocks
drwxr-xr-x 3 root daemon 512 Jul 18 22:23 output

I can send email when testing as root (email addr "obfuscated"):

vm# date | sendmail -v -Am postmaster
postmaster... aliased to root
root... aliased to taosecurity at gmail dot com
taosecurity at gmail dot com... Connecting to gmail-smtp-in.l.google.com. via esmtp...
220 mx.google.com ESMTP n10si1312258qcu.1
>>> EHLO vm.taosecurity.com
250-mx.google.com at your service, [98.218.35.11]
250-SIZE 35651584
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING
>>> MAIL From: SIZE=29
250 2.1.0 OK n10si1312258qcu.1
>>> RCPT To:
>>> DATA
250 2.1.5 OK n10si1312258qcu.1
354 Go ahead n10si1312258qcu.1
>>> .
250 2.0.0 OK 1290094272 n10si1312258qcu.1
taosecurity at gmail dot com... Sent (OK 1290094272 n10si1312258qcu.1)
Closing connection to gmail-smtp-in.l.google.com.
>>> QUIT
221 2.0.0 closing connection n10si1312258qcu.1

That worked. However, I cannot send email as a user:

$ date | sendmail -v -Am postmaster
postmaster... aliased to root
root... aliased to taosecurity at gmail.com
collect: Cannot write ./dfoAIFVDIG019327 (bfcommit, uid=1001, gid=25): Permission denied
queueup: cannot create queue file ./qfoAIFVDIG019327, euid=1001, fd=-1, fp=0x0: Permission denied

Behavior is the same on FreeBSD 7.3 with a fresh install.

I did a ton of research and usually found references to incorrect permissions, etc. In fact, in this post I got the idea to check directories using mtree:

r200a# mtree -p /var -e -U -f /etc/mtree/BSD.var.dist
run changed
permissions expected 0755 found 0777 modified
r200a# mtree -p /var -e -U -f /etc/mtree/BSD.sendmail.dist
./var missing (created)
./var/spool missing (created)
./var/spool/clientmqueue missing (created)

Computer r200a was another FreeBSD system where I tried to fix this problem. However, these changes made no difference.

Any ideas? Thank you.

Update: The reason I investigated this activity was I found errors like this in /var/log/messages on another FreeBSD system, r200b:

Nov 13 03:01:11 r200b sm-mta[40505]: oAD81AUR040505: Losing ./qfoAD81AUR040505: savemail panic
Nov 13 03:01:11 r200b sm-mta[40505]: oAD81AUR040505: SYSERR(root): savemail: cannot save rejected email anywhere

As you can see, whatever was trying to send email using sm-mta was failing.

Sunday, November 14, 2010

Thanks for Help with Startup Scripts

Thanks to @sevanjaniyan and @cperciva for helping with my FreeBSD startup script issue. By removing the ${barnyard2_flags} argument from the command_args section I was able to start barnyard2 properly:

root 45842 54.9 0.5 18572 11116 ?? Ss 7:15PM 0:00.00
/usr/local/bin/barnyard2 -D -U -d /nsm/r200a -f snort.unified2
-c /usr/local/etc/nsm/barnyard2.conf

In other words, the script has this now:

. /etc/rc.subr

name="barnyard2"
load_rc_config $name
rcvar=`set_rcvar`
# set some defaults
: ${barnyard2_enable="NO"}
: ${barnyard2_conf="/usr/local/etc/barnyard2.conf"}
: ${barnyard2_flags="-D"}

command="/usr/local/bin/barnyard2"
command_args="-c ${barnyard2_conf}"

run_rc_command "$1"

I made changes to some other startup scripts and needed to commit them via Git. I did it this way.

richard@macmini:~/taosecurity_freebsd_sguil$ git status
# On branch master
# Changes to be committed:
# (use "git reset HEAD ..." to unstage)
#
# new file: pcap_agent
# new file: sancp_agent
# new file: sguild
# new file: snort_agent
#

richard@macmini:~/taosecurity_freebsd_sguil$ git add pcap_agent sancp_agent sguild snort_agent

richard@macmini:~/taosecurity_freebsd_sguil$ git commit -am "Added new startup scripts."
Created commit 296687e: Added new startup scripts.
4 files changed, 145 insertions(+), 0 deletions(-)
create mode 100755 pcap_agent
create mode 100755 sancp_agent
create mode 100755 sguild
create mode 100755 snort_agent

richard@macmini:~/taosecurity_freebsd_sguil$ git push origin master
taosecurity@taosecurity.git.sourceforge.net's password:
Counting objects: 7, done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.89 KiB, done.
Total 6 (delta 3), reused 0 (delta 0)
To ssh://taosecurity@taosecurity.git.sourceforge.net/gitroot/taosecurity/taosecurity
9cad54a..296687e master -> master

Thanks again for your help!

Now I'm watching commits to https://github.com/firnsy/barnyard2 to see if Barnyard2 is updated to work with the new Snort event types that kills it.

Saturday, November 13, 2010

Calling FreeBSD Startup Script Experts

Has anyone encountered this situation? I've found several startup scripts on FreeBSD that result in duplicate arguments passed during startup. For example:

vm# uname -a
FreeBSD vm.taosecurity.com 7.3-RELEASE FreeBSD 7.3-RELEASE #0:
Sun Mar 21 06:15:01 UTC 2010
root@walker.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386

vm# pkg_info
sancp-1.6.1_3 A network connection profiler

vm# cat /etc/rc.conf

# -- sysinstall generated deltas -- # Fri Nov 12 16:36:42 2010
# Created: Fri Nov 12 16:36:42 2010
# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
defaultrouter="10.10.1.1"
hostname="vm.taosecurity.com"
ifconfig_em0="inet 10.10.1.13 netmask 255.255.255.0"
sshd_enable="YES"
sancp_enable="YES"
sancp_interface="em0"

vm# cat /usr/local/etc/rc.d/sancp
#!/bin/sh
#

# PROVIDE: sancp
# REQUIRE: DAEMON
# BEFORE: LOGIN
# KEYWORD: shutdown

# Add the following lines to /etc/rc.conf to enable sancp:
# sancp_enable (bool): Set to YES to enable sancp
# Default: NO
# sancp_flags (str): Extra flags passed to sancp
# Default: -D
# sancp_conf (str): Sancp configuration file
# Default: /usr/local/etc/sancp.conf
# sancp_interface (str): Default: none - MUST BE SET
#
...edited, all comments...

. /etc/rc.subr

name="sancp"
rcvar=`set_rcvar`

command="/usr/local/bin/sancp"

start_precmd=start_precmd

start_precmd()
{
if [ -z "${sancp_interface}" ]; then
err 1 "sancp_interface must set."
fi
}

# set some defaults
load_rc_config $name

: ${sancp_enable="NO"}
: ${sancp_flags="-D"}
: ${sancp_conf="/usr/local/etc/sancp.conf"}
: ${sancp_interface=""}

command_args="${sancp_flags} -c ${sancp_conf} -i ${sancp_interface}"

run_rc_command "$1"

Now look what happens when I start sancp:

vm# /usr/local/etc/rc.d/sancp start
Starting sancp.
(4287) sancp daemonized successfully!

vm# ps -auxww | grep sancp
root 4287 0.0 0.9 4420 2264 ?? Ss 9:53PM 0:00.00
/usr/local/bin/sancp -D -D -c /usr/local/etc/sancp.conf -i em0

That's right, TWO instances of "-D".

I think it has something to do with this, extracted from sh -x output when starting the script another time:


+ _rc_conf_loaded=true
+ [ -f /etc/rc.conf.d/sancp ]
+ : YES
+ : -D
+ : /usr/local/etc/sancp.conf
+ : em0
+ command_args=-D -c /usr/local/etc/sancp.conf -i em0
+ run_rc_command start
+ _return=0
+ rc_arg=start
+ [ -z sancp ]
+ shift 1
+ rc_extra_args=
+ _rc_prefix=
+ eval _override_command=$sancp_program
+ _override_command=
+ command=/usr/local/bin/sancp
+ _keywords=start stop restart rcvar
+ rc_pid=
+ _pidcmd=
+ _procname=/usr/local/bin/sancp
+ [ -n /usr/local/bin/sancp ]
+ [ -n ]
+ _pidcmd=rc_pid=$(check_process /usr/local/bin/sancp )
+ [ -n rc_pid=$(check_process /usr/local/bin/sancp ) ]
+ _keywords=start stop restart rcvar status poll
+ [ -z start ]
+ [ -n ]
+ eval rc_flags=$sancp_flags
+ rc_flags=-D
...edited...
+ echo Starting sancp.
Starting sancp.
+ [ -n ]
+ _doit=/usr/local/bin/sancp -D -D -c /usr/local/etc/sancp.conf -i em0
+ [ -n ]
+ [ -n ]
+ _run_rc_doit /usr/local/bin/sancp -D -D -c /usr/local/etc/sancp.conf -i em0
+ debug run_rc_command: doit: /usr/local/bin/sancp -D -D -c /usr/local/etc/sancp.conf -i em0
+ eval /usr/local/bin/sancp -D -D -c /usr/local/etc/sancp.conf -i em0
+ /usr/local/bin/sancp -D -D -c /usr/local/etc/sancp.conf -i em0
(4075) sancp daemonized successfully!

That "rc_flags=-D" looks suspicious to me.

So what, two instances of -D, you might think. The problem is with more complicated scripts I'm seeing lots of command line arguments duplicated. It's not "clean" and I want to know what this is happening.

Incidentally, I get similar behavior on FreeBSD 8.1. I tried 7.3 here to see if there was a difference.

Any ideas? I've been looking at /etc/rc.subr to see if I can figure out how _run_rc_doit gets built.

For reference, the /usr/local/etc/sancp.conf file is stock:

vm# grep -v ^# /usr/local/etc/sancp.conf

# snort pcap filter format # description
var ip 8 # ether proto 0x0800 # ip traffic
var arp 1544 # ether proto 0x0806 # arp traffic
var loopback 144 # ether proto 0x9000 # Loopback: used to test ethernet interfaces
var 802.3 1024 # ether proto 0x0004 # IEEE 802.3 traffic

var pixfw1 10.10.10.1
var pixfw2 10.10.10.2
var webserver1 10.10.11.24
var webserver2 10.10.11.25
var webserver3 10.10.11.26
var dnsserver1 10.10.11.27
var dnsserver2 10.10.11.28
var mailserver1 10.10.11.29
var mailserver2 10.10.11.30
var proxyserver 10.10.11.30
var ntpserver 210.121.2.64

var icmp 1
var tcp 6
var udp 17

var http 80
var https 443
var ssh 22
var telnet 23
var irc_ports 6665-6667
var dns 53
var highports 1024-65535

known_ports tcp http,https,ssh,telnet,irc_ports,dns
known_ports udp dns

default realtime=log

default stats=log

default pcap=log

default limit=0

default timeout=120

default tcplag=0 # after a tcp connection would normally be considered closed

default rid=0

default status=0

default node=2

default strip-80211=enable

ip any any icmp any any, realtime=pass, pcap=pass, status=1, rid=23, timeout=1500 # test rule

arp any any any any any, ignore # ignore arp traffic
loopback any any any any any, ignore # ignore local ethernet loopback test packets
802.3 any any any any any, ignore # ignore IEEE 802.3 traffic on the switch

ip pixfw1 pixfw2 105 0 0, pcap pass, realtime=pass, status=100, rid=1 #2003-12-14 18:21:53

ip pixfw1 ntpserver 17 123 123, realtime=pass, status=200, rid=2 #2003-12-14 18:21:53
ip pixfw2 ntpserver 17 123 123, realtime=pass, status=200, rid=3 #2003-12-14 18:21:53

ip pixfw2 any tcp highports 80, realtime=pass, status=201, rid=4 #2003-12-14 18:21:53
ip pixfw2 any udp highports 443, realtime=pass, status=202, rid=6 #2003-12-14 18:21:53
ip pixfw2 any udp highports 53, realtime=pass, status=203, rid=5 #2003-12-14 18:21:53

ip proxyserver any tcp highports any, realtime=pass, status=299, rid=8 #2003-12-14 18:21:53

ip any webserver1 6 any 80, realtime=pass, status=301, rid=9 #2003-12-14 19:19:27
ip any webserver1 6 any 443, realtime=pass, status=302, rid=10 #2003-12-14 19:19:27

ip any webserver2 6 any 80, realtime=pass, status=301, rid=11 #2003-12-14 19:19:27
ip any webserver2 6 any 443, realtime=pass, status=302, rid=12 #2003-12-14 19:19:27

ip any webserver3 6 any 80, realtime=pass, status=301, rid=13 #2003-12-14 19:19:27
ip any webserver3 6 any 443, realtime=pass, status=302, rid=14 #2003-12-14 19:19:27

ip any dnsserver1 17 any 53, realtime=pass, status=303, rid=15 #2003-12-14 19:19:27
ip any dnsserver2 17 any 53, realtime=pass, status=303, rid=16 #2003-12-14 19:19:27

ip any mailserver1 6 any 25, realtime=pass, status=304, rid=17 #2003-12-14 19:19:27
ip mailserver1 any 6 any 25, realtime=pass, status=204, rid=18 #2003-12-14 19:19:27

ip any mailserver2 6 any 25, realtime=pass, status=304, rid=19 #2003-12-14 19:19:27
ip mailserver2 any 6 any 25, realtime=pass, status=204, rid=20 #2003-12-14 19:19:27


Wednesday, November 10, 2010

Two New Tools in Snort

No sooner do I get Snort 2.9.0.1 running than something breaks. However, thanks to Niels Horn I know a little more about two new tools included with Snort.

First is u2spewfoo, which reads Unified2 output files and outputs them as text.

[sguil@r200a /nsm/r200a]$ u2spewfoo snort.unified2.1289360307 | head -20

(Event)
sensor id: 0 event id: 1 event second: 1289360859 event microsecond: 881345
sig id: 2011032 gen id: 1 revision: 4 classification: 3
priority: 2 ip source: 192.168.2.107 ip destination: 172.16.2.1
src port: 44597 dest port: 3128 protocol: 6 impact_flag: 0 blocked: 0

Packet
sensor id: 0 event id: 1 event second: 1289360859
packet second: 1289360859 packet microsecond: 881345
linktype: 1 packet_length: 1168
00 15 17 0B | 7D 4C 00 13 | 10 65 2F AC | 08 00 45 00
04 82 C2 E3 | 40 00 3F 06 | 03 6E C0 A8 | 02 6B AC 10
02 01 AE 35 | 0C 38 73 6F | 02 7F 12 37 | D9 A8 80 18
03 EA 6D 85 | 00 00 01 01 | 08 0A 01 2A | 34 44 75 11
33 8C 41 46 | 69 72 73 74 | 25 32 43 25 | 32 30 49 25
32 30 74 65 | 73 74 65 64 | 25 32 30 6D | 79 25 32 30
6F 6C 64 25 | 32 30 73 63 | 72 69 70 74 | 73 25 32 30
6F 6E 25 32 | 30 46 72 65 | 65 42 53 44 | 25 32 30 37
2E 78 25 32 | 43 25 32 30 | 61 6E 64 25 | 32 30 6E 6F

I guess that's good for troubleshooting. It feels a little like 1999!

The second tool is u2boat, which transforms the pcap data in a Unified2 output file into a normal pcap file.

[sguil@r200a /nsm/r200a]$ u2boat snort.unified2.1289360307
Usage: u2boat [-t type]
[sguil@r200a /nsm/r200a]$ u2boat snort.unified2.1289360307 snort.unified2.1289360307.pcap
Defaulting to pcap output.
[sguil@r200a /nsm/r200a]$ file snort.unified2.1289360307.pcap
snort.unified2.1289360307.pcap: tcpdump capture file (little-endian)
- version 2.4 (Ethernet, capture length 65535)
[sguil@r200a /nsm/r200a]$ tcpdump -n -r snort.unified2.1289360307.pcap
reading from file snort.unified2.1289360307.pcap, link-type EN10MB (Ethernet)
22:47:39.881345 IP 192.168.2.107.44597 > 172.16.2.1.3128: Flags [P.],
ack 305650088, win 1002, options [nop,nop,TS val 19543108 ecr 1964061580], length 1102

So those are great, but fortunately unless I fix Barnyard2 or a fix is committed, Barnyard2 is going to die when it encounters record types from Snort that Barnyard2 doesn't recognize, e.g.:

r200a# barnyard2 -U -d /nsm/r200a -f snort.unified2 -c /usr/local/etc/nsm/barnyard2.conf
Running in Continuous mode

--== Initializing Barnyard2 ==--
Initializing Input Plugins!
Initializing Output Plugins!
Parsing config file "/usr/local/etc/nsm/barnyard2.conf"
Log directory = /var/log/barnyard2
sguil: sensor name = r200a
sguil: agent port = 7735
sguil: Connected to localhost on 7735.

--== Initialization Complete ==--

______ -*> Barnyard2 <*-
/ ,,_ \ Version 2.1.8 (Build 251)
|o" )~| By the SecurixLive.com Team: http://www.securixlive.com/about.php
+ '''' + (C) Copyright 2008-2010 SecurixLive.

Snort by Martin Roesch & The Snort Team: http://www.snort.org/team.html
(C) Copyright 1998-2007 Sourcefire Inc., et al.

Using waldo file '/nsm/r200a/waldo':
spool directory = /nsm/r200a
spool filebase = snort.unified2
time_stamp = 1289360307
record_idx = 4
Opened spool file '/nsm/r200a/snort.unified2.1289360307'
ERROR: Unknown record type read: 110
Fatal Error, Quitting..

The good news is the alerts will continue to be logged to disk, and can be processed once Barnyard2 can read them.

Tuesday, November 09, 2010

Using Git with FreeBSD Sguil Scripts

Before today I never committed anything using Git. Previously I used CVS, but never got around to trying something more modern like SVN. However, I know several developers at work use Git, so I figured I would try committing my FreeBSD Sguil scripts (lame as they are) to Git at Sourceforge. This would allow me to keep track of changes and get the code out of my own repository for sharing and safekeeping.

I started by cleaning up the directory where I kept the scripts.

After following the instructions to enable Git, I took these actions.


richard@macmini:~/taosecurity_freebsd_sguil$ git init
Initialized empty Git repository in /home/richard/taosecurity_freebsd_sguil/.git/

richard@macmini:~/taosecurity_freebsd_sguil$ git config user.name "Richard Bejtlich"

richard@macmini:~/taosecurity_freebsd_sguil$ git config user.email \
"taosecurity@users.sourceforge.net"

richard@macmini:~/taosecurity_freebsd_sguil$ git remote add origin \
ssh://taosecurity@taosecurity.git.sourceforge.net/gitroot/taosecurity/taosecurity

richard@macmini:~/taosecurity_freebsd_sguil$ git config branch.master.remote origin

richard@macmini:~/taosecurity_freebsd_sguil$ git config branch.master.merge refs/head/master

richard@macmini:~/taosecurity_freebsd_sguil$ git push origin master

taosecurity@taosecurity.git.sourceforge.net's password:
error: src refspec master does not match any.
fatal: The remote end hung up unexpectedly
error: failed to push some refs to 'ssh://taosecurity@taosecurity.git.sourceforge.net/gitroot \
/taosecurity/taosecurity'

That was unfortunate. I didn't see that error in the Sourceforge guide, but after checking here I found that trying to add all the files might be the right step.


richard@macmini:~/taosecurity_freebsd_sguil$ git add *

richard@macmini:~/taosecurity_freebsd_sguil$ git commit -am "Message"

Created initial commit bd18669: Message
28 files changed, 1400 insertions(+), 0 deletions(-)
create mode 100755 README
create mode 100644 SguildLoaderd.tcl.patch
create mode 100644 SguildMysqlMerge.tcl.patch
create mode 100755 barnyard2
create mode 100644 barnyard2.conf
create mode 100644 barnyard2.conf.patch
create mode 100644 log_packets.sh.crontab
create mode 100644 log_packets.sh.patch
create mode 100644 pcap_agent.conf.patch
create mode 100755 prep_platform.sh
create mode 100644 rc-adds.txt
create mode 100755 rc-conf.sh
create mode 100755 sancp
create mode 100644 sancp.conf.patch
create mode 100644 sancp_agent.conf.patch
create mode 100644 sensor_agent.conf.patch
create mode 100755 sguil_database_install_pt1.sh
create mode 100755 sguil_database_install_pt2.sh
create mode 100755 sguil_sensor_install.sh
create mode 100755 sguil_sensor_install_patch.sh
create mode 100644 sguil_sensor_users.txt
create mode 100755 sguil_server_install.sh
create mode 100644 sguild.conf.patch
create mode 100755 sguild_adduser.sh
create mode 100755 snort
create mode 100644 snort.conf.patch
create mode 100644 snort_agent.conf.patch
create mode 100755 snort_src_install.sh

richard@macmini:~/taosecurity_freebsd_sguil$ git push origin master
taosecurity@taosecurity.git.sourceforge.net's password:

Counting objects: 30, done.
Compressing objects: 100% (29/29), done.
Writing objects: 100% (30/30), 17.31 KiB, done.
Total 30 (delta 4), reused 0 (delta 0)
To ssh://taosecurity@taosecurity.git.sourceforge.net/gitroot/taosecurity/taosecurity
* [new branch] master -> master

That did it. I found that if I didn't make a change but tried to note one, nothing happened (as expected).


richard@macmini:~/taosecurity_freebsd_sguil$ git commit -am "Commit scripts using Git"
# On branch master
nothing to commit (working directory clean)

richard@macmini:~/taosecurity_freebsd_sguil$ git push origin
mastertaosecurity@taosecurity.git.sourceforge.net's password:
Everything up-to-date

Next I made some fixes and committed those.

richard@macmini:~/taosecurity_freebsd_sguil$ vi README
richard@macmini:~/taosecurity_freebsd_sguil$ git commit -am "Modify README to reflect changing ExtNet."
Created commit 2ef21f3: Modify README to reflect changing ExtNet.
1 files changed, 3 insertions(+), 1 deletions(-)

richard@macmini:~/taosecurity_freebsd_sguil$ git push origin mastertaosecurity@taosecurity.git.sourceforge.net's password:
Counting objects: 5, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 413 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
To ssh://taosecurity@taosecurity.git.sourceforge.net/gitroot/taosecurity/taosecurity
bd18669..2ef21f3 master -> master

Checking out files is pretty easy, assuming Git is installed.

richard@neely:~$ mkdir gittest

richard@neely:~$ cd gittest

richard@neely:~/gittest$ git clone git://taosecurity.git.sourceforge.net/gitroot/taosecurity/taosecurity

Initialized empty Git repository in /home/richard/gittest/taosecurity/.git/
remote: Counting objects: 30, done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 30 (delta 4), reused 0 (delta 0)
Receiving objects: 100% (30/30), 17.25 KiB, done.
Resolving deltas: 100% (4/4), done.

richard@neely:~/gittest$ cd taosecurity

richard@neely:~/gittest/taosecurity$ ls

barnyard2 sguild_adduser.sh
barnyard2.conf sguil_database_install_pt1.sh
barnyard2.conf.patch sguil_database_install_pt2.sh
log_packets.sh.crontab sguild.conf.patch
log_packets.sh.patch SguildLoaderd.tcl.patch
pcap_agent.conf.patch SguildMysqlMerge.tcl.patch
prep_platform.sh sguil_sensor_install_patch.sh
rc-adds.txt sguil_sensor_install.sh
rc-conf.sh sguil_sensor_users.txt
README sguil_server_install.sh
sancp snort
sancp_agent.conf.patch snort_agent.conf.patch
sancp.conf.patch snort.conf.patch
sensor_agent.conf.patch snort_src_install.sh

So, now my scripts are available for me to add changes and for anyone who might be interested to retrieve them.

Updates to Sguil on FreeBSD Scripts

Early last year I posted Notes on Installing Sguil Using FreeBSD 7.1 Packages where I examined using the various FreeBSD ports for Sguil. In that post I showed that a lot of work was required to deploy Sguil, even if you used the ports or packages. Previously I've written about a set of scripts I maintain for deploying Sguil platforms in my lab. I decided to take a look at those scripts and update them for a modern environment, since a lot has happened in the almost two years since I last used the scripts.

First, I tested my old scripts on FreeBSD 7.x, and now 8.x is common. Second, Snort 2.9.0.1 is available, and with it the new DAQ mechanism for accessing network traffic. Third, Barnyard has been deprecated in favor of Barnyard2, thanks to the guys at the NSMNow project. There have been a lot of changes with rules and other areas. I also wanted to try running a 64 bit environment on a Dell R200 as my primary lab sensor. Finally, I decided to switch from using CVS at Sourceforge to Git at Sourceforge. I'll explain that in a separate post.

The end result of my work is available now at http://taosecurity.git.sourceforge.net. Please remember that these scripts are basically a way for me to document how I installed certain versions of various NSM applications on a specific FreeBSD platform. There's no error checking, and no support available. Basically, if you want to see how I deploy all of the non-client parts of Sguil on FreeBSD 8.1, feel free to check out the scripts.

One aspect of this that might be helpful is that by reading the scripts you can follow how to go from a basic FreeBSD installation to a completely functioning, all-in-one (minus the client) Sguil platform.

Monday, November 01, 2010

Collage: Defeating Censorship [aka Security] with User-Generated Content

The Economist article Anti-censorship: Hidden truths; A new way of beating the web’s censors brought a system called "Collage" to my attention. Collage, a project by Sam Burnett, Nick Feamster, and Santosh Vempala, described this way on its project site:

We have developed Collage, which allows users to exchange messages through hidden channels in sites that host user-generated content.

Collage has two components: a message vector layer for embedding content in cover traffic; and a rendezvous mechanism to allow parties to publish and retrieve messages in the cover traffic.

Collage uses user-generated content (e.g., photo-sharing sites) as “drop sites” for hidden messages.

To send a message, a user embeds it into cover traffic and posts the content on some site, where receivers retrieve this content using a sequence of tasks.

Collage makes it difficult for a censor to monitor or block these messages by exploiting the sheer number of sites where users can exchange messages and the variety of ways that a message can be hidden. Our evaluation of Collage shows that the performance overhead is acceptable for sending small messages (e.g., Web articles, email).

Applications use Collage to send and receive messages, by hiding these messages inside user-generated cover content (e.g., images, tweets, etc.) and publishing them on user-generated content hosts like Flickr or Twitter. At the receiver, Collage fetches the cover content from content hosts and decodes the message. By hiding data inside user-generated content as they traverse the network, Collage escapes detection by censors.


Freedom FTW, right? Let's rewrite this description from the point of view I care more about:

We have developed Collage, which allows intruders to exchange messages through hidden channels in sites that host user-generated content.

Collage has two components: a message vector layer for embedding content in cover traffic that will fly past your proxies and other filtering mechanisms; and a rendezvous mechanism to allow parties to publish and retrieve messages in the cover traffic.

Collage uses user-generated content (e.g., photo-sharing sites) as “drop sites” for hidden messages, like command and control traffic, or stolen data.

To send a message, a user embeds it into cover traffic and posts the content on some site, where receivers retrieve this content using a sequence of tasks that defenders will not recognize as malicious.

Collage makes it difficult for incident detection and response teams to monitor or block these messages by exploiting the sheer number of sites where users can exchange messages and the variety of ways that a message can be hidden. Our evaluation of Collage shows that the performance overhead is acceptable for sending small messages (e.g., Web articles, email), perfect for command and control instructions.

Malware or backdoors use Collage to send and receive messages, by hiding these messages inside user-generated cover content (e.g., images, tweets, etc.) and publishing them on user-generated content hosts like Flickr or Twitter that are not blocked by reputation systems, which some security vendors think solve the world's problems. At the receiver, Collage fetches the cover content from content hosts and decodes the message. By hiding data inside user-generated content as they traverse the network, Collage escapes detection by organizations trying to protect their data.


I wonder if I'm not the only one thinking this way?