Thursday, November 18, 2010

The Problem Is with Gmail

In my last post I lamented a problem with Sendmail on FreeBSD. I was trying to troubleshoot a problem sending email from FreeBSD's periodic scripts to Gmail. I've determined that, as crazy as this sounds, Gmail is broken. (Some of you are probably not surprised. If you want to skip the drama and see the bottom line, scroll to the bottom of the post.)

Let me start my case by showing network transcripts of one successful "periodic" email and one unsuccessful "periodic" email. I'm not going to change any email addresses in this post.

The following email is delivered successfully. Computer vm.taosecurity.com sits behind NAT so the public IP is 73.128.35.11. The entries prior to the SMTP transactions (e.g. 074.125.091.027.00025-073.128.035.011.57184: and similar) were added by Tcpflow, which I used to render the transcript manually.

074.125.091.027.00025-073.128.035.011.57184: 220 mx.google.com ESMTP my6si2476635qcb.101

073.128.035.011.57184-074.125.091.027.00025: EHLO vm.taosecurity.com

074.125.091.027.00025-073.128.035.011.57184: 250-mx.google.com at your service, [73.128.35.11]
250-SIZE 35651584
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING

073.128.035.011.57184-074.125.091.027.00025: MAIL From:<analyst@vm.taosecurity.com> SIZE=917

074.125.091.027.00025-073.128.035.011.57184: 250 2.1.0 OK my6si2476635qcb.101

073.128.035.011.57184-074.125.091.027.00025: RCPT To:<taosecurity@gmail.com>
DATA

074.125.091.027.00025-073.128.035.011.57184: 250 2.1.5 OK my6si2476635qcb.101
354 Go ahead my6si2476635qcb.101

073.128.035.011.57184-074.125.091.027.00025: Received: from vm.taosecurity.com (localhost [127.0.0.1])
.by vm.taosecurity.com (8.14.4/8.14.4) with ESMTP id oAJ66xa2021306
.for <root@vm.taosecurity.com>; Fri, 19 Nov 2010 01:06:59 -0500 (EST)
.(envelope-from analyst@vm.taosecurity.com)
Received: (from root@localhost)
.by vm.taosecurity.com (8.14.4/8.14.4/Submit) id oAJ66xF4021296
.for root; Fri, 19 Nov 2010 01:06:59 -0500 (EST)
.(envelope-from analyst)
Date: Fri, 19 Nov 2010 01:06:59 -0500 (EST)
From: analyst <analyst@vm.taosecurity.com>
Message-Id: <201011190606.oAJ66xF4021296@vm.taosecurity.com>
To: root@vm.taosecurity.com
Subject: vm.taosecurity.com security run output

Checking setuid files and devices:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

vm.taosecurity.com login failures:

vm.taosecurity.com refused connections:

-- End of security output --

073.128.035.011.57184-074.125.091.027.00025: .

074.125.091.027.00025-073.128.035.011.57184: 250 2.0.0 OK 1290128829 my6si2476635qcb.101

073.128.035.011.57184-074.125.091.027.00025: QUIT

074.125.091.027.00025-073.128.035.011.57184: 221 2.0.0 closing connection my6si2476635qcb.101

The following email fails to be delivered. Computer r200b has the public IP address 73.128.35.11 as shown. Again the lines are prepended by Tcpflow headers.

074.125.091.027.00025-073.128.035.011.19228: 220 mx.google.com ESMTP f23si2500736qcq.34

073.128.035.011.19228-074.125.091.027.00025: EHLO r200b.taosecurity.com

074.125.091.027.00025-073.128.035.011.19228: 250-mx.google.com at your service, [73.128.35.11]
250-SIZE 35651584
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING

073.128.035.011.19228-074.125.091.027.00025: MAIL From:<richard@r200b.taosecurity.com> SIZE=1658

074.125.091.027.00025-073.128.035.011.19228: 250 2.1.0 OK f23si2500736qcq.34

073.128.035.011.19228-074.125.091.027.00025: RCPT To:<taosecurity@gmail.com>
DATA

074.125.091.027.00025-073.128.035.011.19228: 250 2.1.5 OK f23si2500736qcq.34
354 Go ahead f23si2500736qcq.34

073.128.035.011.19228-074.125.091.027.00025: Received: from r200b.taosecurity.com (localhost [127.0.0.1])
.by r200b.taosecurity.com (8.14.4/8.14.4) with ESMTP id oAJ17UwM063291
.for <root@r200b.taosecurity.com>; Thu, 18 Nov 2010 20:07:30 -0500 (EST)
.(envelope-from richard@r200b.taosecurity.com)
Received: (from root@localhost)
.by r200b.taosecurity.com (8.14.4/8.14.4/Submit) id oAJ17UKs063248
.for root; Thu, 18 Nov 2010 20:07:30 -0500 (EST)
.(envelope-from richard)
Date: Thu, 18 Nov 2010 20:07:30 -0500 (EST)
From: Richard Bejtlich <richard@r200b.taosecurity.com>
Message-Id: <201011190107.oAJ17UKs063248@r200b.taosecurity.com>
To: root@r200b.taosecurity.com
Subject: r200b.taosecurity.com security run output

Checking setuid files and devices:

Checking for uids of 0:

root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

r200b.taosecurity.com kernel log messages:
+++ /tmp/security.QW4ZT9Yc.2010-11-18 20:07:29.000000000 -0500

+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled

r200b.taosecurity.com login failures:

Nov 17 07:51:58 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.
Nov 17 07:52:02 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.

r200b.taosecurity.com refused connections:

Checking for a current audit database:

Database created: Thu Nov 18 19:05:00 EST 2010

Checking for packages with security vulnerabilities:

0 problem(s) in your installed packages found.

-- End of security output --

073.128.035.011.19228-074.125.091.027.00025: .

074.125.091.027.00025-073.128.035.011.19228: 550-5.7.1 [73.128.35.11] The IP you're using to send mail is not authorized to
550-5.7.1 send email directly to our servers. Please use the SMTP relay at your
550-5.7.1 service provider instead. Learn more at
550 5.7.1 http://mail.google.com/support/bin/answer.py?answer=10336 f23si2500736qcq.34

073.128.035.011.19228-074.125.091.027.00025: QUIT

Darn. As you can see, Gmail claims "The IP you're using to send mail is not authorized to send email directly to our servers." Is that true? Didn't I just send email from the same IP address, as far as Gmail was concerned?

There is basically no difference between these emails, other than the contents of the security reports in each. (Hint, hint.)

I can prove the Gmail error message is bogus.

Let's start by showing both computers can send email to Gmail. If I don't send email using the periodic scripts, I can send email to Gmail from both systems successfully.

First, the message from host vm succeeds (and I saw it in my Inbox).

vm# mail -v -s "From vm" taosecurity@gmail.com
Test from vm.
.
EOT
taosecurity@gmail.com... Connecting to [127.0.0.1] via relay...
220 vm.taosecurity.com ESMTP Sendmail 8.14.4/8.14.4; Fri, 19 Nov 2010 01:31:20 -0500 (EST)
>>> EHLO vm.taosecurity.com
250-vm.taosecurity.com Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> MAIL From:<analyst@vm.taosecurity.com> SIZE=58
250 2.1.0 <analyst@vm.taosecurity.com>... Sender ok
>>> RCPT To:<taosecurity@gmail.com>
>>> DATA
250 2.1.5 <taosecurity@gmail.com>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 oAJ6VKaj021400 Message accepted for delivery
taosecurity@gmail.com... Sent (oAJ6VKaj021400 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 vm.taosecurity.com closing connection

vm# grep oAJ6VKaj021400 /var/log/maillog

Nov 19 01:31:20 vm sm-mta[21400]: oAJ6VKaj021400: from=<analyst@vm.taosecurity.com>,
size=393, class=0, nrcpts=1, msgid=<201011190631.oAJ6VKlp021399@vm.taosecurity.com>,
proto=ESMTP, daemon=Daemon0, relay=localhost [127.0.0.1]

Nov 19 01:31:20 vm sendmail[21399]: oAJ6VKlp021399: to=taosecurity@gmail.com, ctladdr=analyst
(1001/1001), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30058, relay=[127.0.0.1]
[127.0.0.1], dsn=2.0.0, stat=Sent (oAJ6VKaj021400 Message accepted for delivery)

Nov 19 01:31:21 vm sm-mta[21402]: oAJ6VKaj021400: to=<taosecurity@gmail.com>,
ctladdr=<analyst@vm.taosecurity.com> (1001/1001), delay=00:00:01, xdelay=00:00:01,
mailer=esmtp, pri=30393, relay=gmail-smtp-in.l.google.com. [74.125.91.27], dsn=2.0.0, stat=Sent
(OK 1290130290 g35si2521350qcs.118)

Second, the message from r200b succeeds (and I saw it in my Inbox).

r200b:/root# mail -v -s "From r200b" taosecurity@gmail.com
Test from r200b.
.
EOT
taosecurity@gmail.com... Connecting to [127.0.0.1] via relay...
220 r200b.taosecurity.com ESMTP Sendmail 8.14.4/8.14.4; Thu, 18 Nov 2010 20:31:01 -0500 (EST)
>>> EHLO r200b.taosecurity.com
250-r200b.taosecurity.com Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> MAIL From:<richard@r200b.taosecurity.com> SIZE=64
250 2.1.0 <richard@r200b.taosecurity.com>... Sender ok
>>> RCPT To:<taosecurity@gmail.com>
>>> DATA
250 2.1.5 <taosecurity@gmail.com>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 oAJ1V1Xx063384 Message accepted for delivery
taosecurity@gmail.com... Sent (oAJ1V1Xx063384 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 r200b.taosecurity.com closing connection

r200b:/root# grep oAJ1V1Xx063384 /var/log/maillog

Nov 18 20:31:01 r200b sm-mta[63384]: oAJ1V1Xx063384: from=<
richard@r200b.taosecurity.com>, size=417, class=0, nrcpts=1, msgid=<
201011190131.oAJ1V1SP063383@r200b.taosecurity.com>, proto=ESMTP, daemon=Daemon0,
relay=localhost [127.0.0.1]

Nov 18 20:31:01 r200b sendmail[63383]: oAJ1V1SP063383: to=taosecurity@gmail.com, ctladdr=richard
(1001/1001), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30064, relay=[127.0.0.1]
[127.0.0.1], dsn=2.0.0, stat=Sent (oAJ1V1Xx063384 Message accepted for delivery)

Nov 18 20:31:02 r200b sm-mta[63386]: oAJ1V1Xx063384: to=<taosecurity@gmail.com>,
ctladdr=<richard@r200b.taosecurity.com> (1001/1001), delay=00:00:01, xdelay=00:00:01,
mailer=esmtp, pri=30417, relay=gmail-smtp-in.l.google.com. [74.125.91.27], dsn=2.0.0, stat=Sent
(OK 1290130252 m5si2493978qcu.183)

As you can see, both computers, vm and r200b, can send email fine to Gmail.

Now this will blow your mind. What happens when I manually send an email with the content of the periodic email that Gmail refused to accept from r200b?

Let's send it from vm, which so far has had no trouble talking to Gmail under any circumstances, whether sending manual email or its own periodic output.

vm# mail -v -s "From vm, fake periodic output for blog" taosecurity@gmail.com
Checking setuid files and devices:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

r200b.taosecurity.com kernel log messages:
+++ /tmp/security.QW4ZT9Yc.2010-11-18 20:07:29.000000000 -0500
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled

r200b.taosecurity.com login failures:
Nov 17 07:51:58 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.
Nov 17 07:52:02 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.

r200b.taosecurity.com refused connections:

Checking for a current audit database:

Database created: Thu Nov 18 19:05:00 EST 2010

Checking for packages with security vulnerabilities:

0 problem(s) in your installed packages found.

-- End of security output --
.
EOT
taosecurity@gmail.com... Connecting to [127.0.0.1] via relay...
220 vm.taosecurity.com ESMTP Sendmail 8.14.4/8.14.4; Fri, 19 Nov 2010 02:03:17 -0500 (EST)
>>> EHLO vm.taosecurity.com
250-vm.taosecurity.com Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> MAIL From:<analyst@vm.taosecurity.com> SIZE=1026
250 2.1.0 <analyst@vm.taosecurity.com>... Sender ok
>>> RCPT To:<taosecurity@gmail.com>
>>> DATA
250 2.1.5 <taosecurity@gmail.com>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 oAJ73HIk021517 Message accepted for delivery
taosecurity@gmail.com... Sent (oAJ73HIk021517 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 vm.taosecurity.com closing connection

vm# grep oAJ73HIk021517 /var/log/maillog

Nov 19 02:03:17 vm sm-mta[21517]: oAJ73HIk021517: from=<analyst@vm.taosecurity.com>,
size=1361, class=0, nrcpts=1, msgid=<201011190703.oAJ73G8n021516@vm.taosecurity.com>,
proto=ESMTP, daemon=Daemon0, relay=localhost [127.0.0.1]

Nov 19 02:03:17 vm sendmail[21516]: oAJ73G8n021516: to=taosecurity@gmail.com, ctladdr=analyst
(1001/1001), delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=31026, relay=[127.0.0.1]
[127.0.0.1], dsn=2.0.0, stat=Sent (oAJ73HIk021517 Message accepted for delivery)

Nov 19 02:03:18 vm sm-mta[21519]: oAJ73HIk021517: to=<taosecurity@gmail.com>,
ctladdr=<analyst@vm.taosecurity.com> (1001/1001), delay=00:00:01, xdelay=00:00:01,
mailer=esmtp, pri=31361, relay=gmail-smtp-in.l.google.com. [74.125.91.27], dsn=5.0.0,
stat=Service unavailable

Nov 19 02:03:18 vm sm-mta[21519]: oAJ73HIk021517: oAJ73IIk021519: DSN: Service unavailable

What's up with that, Gmail? If I sniff the traffic I can see Gmail refuse it again:

074.125.091.027.00025-073.128.035.011.58727: 220 mx.google.com ESMTP o12si2579217qcs.143

073.128.035.011.58727-074.125.091.027.00025: EHLO vm.taosecurity.com

074.125.091.027.00025-073.128.035.011.58727: 250-mx.google.com at your service, [73.128.35.11]
250-SIZE 35651584
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING

073.128.035.011.58727-074.125.091.027.00025: MAIL From:<analyst@vm.taosecurity.com> SIZE=1361

073.128.035.011.58727-074.125.091.027.00025: MAIL From:<analyst@vm.taosecurity.com> SIZE=1361

074.125.091.027.00025-073.128.035.011.58727: 250 2.1.0 OK o12si2579217qcs.143

073.128.035.011.58727-074.125.091.027.00025: RCPT To:<taosecurity@gmail.com>
DATA

074.125.091.027.00025-073.128.035.011.58727: 250 2.1.5 OK o12si2579217qcs.143
354 Go ahead o12si2579217qcs.143

073.128.035.011.58727-074.125.091.027.00025: Received: from vm.taosecurity.com (localhost [127.0.0.1])
.by vm.taosecurity.com (8.14.4/8.14.4) with ESMTP id oAJ73HIk021517
.for <taosecurity@gmail.com>; Fri, 19 Nov 2010 02:03:17 -0500 (EST)
.(envelope-from analyst@vm.taosecurity.com)
Received: (from root@localhost)
.by vm.taosecurity.com (8.14.4/8.14.4/Submit) id oAJ73G8n021516
.for taosecurity@gmail.com; Fri, 19 Nov 2010 02:03:16 -0500 (EST)
.(envelope-from analyst)
Date: Fri, 19 Nov 2010 02:03:16 -0500 (EST)
From: analyst <analyst@vm.taosecurity.com>
Message-Id: <201011190703.oAJ73G8n021516@vm.taosecurity.com>
To: taosecurity@gmail.com
Subject: From vm, fake periodic output for blog

Checking setuid files and devices:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

r200b.taosecurity.com kernel log messages:
+++ /tmp/security.QW4ZT9Yc.2010-11-18 20:07:29.000000000 -0500
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled

r200b.taosecurity.com login failures:
Nov 17 07:51:58 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.
Nov 17 07:52:02 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.

r200b.taosecurity.com refused connections

Checking for a current audit database:

Database created: Thu Nov 18 19:05:00 EST 2010

Checking for packages with security vulnerabilities:

0 problem(s) in your installed packages found.

-- End of security output --

073.128.035.011.58727-074.125.091.027.00025: .

074.125.091.027.00025-073.128.035.011.58727: 550-5.7.1 [73.128.35.11] The IP you're using to send mail is not authorized to
550-5.7.1 send email directly to our servers. Please use the SMTP relay at your
550-5.7.1 service provider instead. Learn more at
550 5.7.1 http://mail.google.com/support/bin/answer.py?answer=10336 o12si2579217qcs.143

073.128.035.011.58727-074.125.091.027.00025: QUIT

The transcript ends with the bogus "The IP you're using to send mail is not authorized to send email directly to our servers." message. So what's the bottom line?

Gmail appears to be filtering email based on content, providing a bogus "The IP you're using to send mail is not authorized to send email directly to our servers." message.

Does anyone have another explanation? I would love to hear it. Thank you.

Incidentally, I am considering workarounds that WOULD use my ISP's SMTP server and hopefully avoid this problem. Also, I don't expect to see this issue using the Gmail Web interface. It must be a filter Gmail applies when users talk to their SMTP servers directly.

21 comments:

davehull said...

Very strange. I was able to cut and paste the contents of your failing message and email it to my gmail account with no issues, so it may not be purely content filtering.

Meller said...

Richard,

I was also not able to replicate your failure. One thing i did notice though that your VM is future dating stuff for the 0500 timezone.

From your logs...

"Date: Fri, 19 Nov 2010 02:03:16 -0500 (EST)"

Maybe fix you datetime?

TOoSmOotH said...

I think your issue is here:

073.128.035.011.58727-074.125.091.027.00025: Received: from vm.taosecurity.com (localhost [127.0.0.1])

For it to work properly that needs to look like vm.taosecurity.com (vm.taosecurity.com [73.128.35.11])

Make sure you have vm.taosecurity.com in your local-host-names config.

Phil H said...

I would bet one or both of the following are causing the error:
- Google is blocking mail from known "home" class IP addresses. I couldn't find 73.128.35.11 in any RBL that I know of, but I'm sure that Google has their own.
- Google *may* be using SPF or TKIP to determine who can and cannot send to their servers. Each technology has its own problems, so I highly doubt they are rejecting solely on this one, but it's possible they have a defense-in-depth implementation, where an IP in the "home" RBL requires more extensive approval to send.

Anonymous said...

I have seen something similar with godaddy. They filter based on "links" which can be any domain or ip. Try taking the ip address out of your message and see if it still fails.

In the GoDaddy case, they blocked any email that references a server in the PBL, which is supposed to be used for "end-user IP address ranges which should not be delivering unauthenticated SMTP email to any Internet mail server."

Richard Bejtlich said...

Phil -- sorry, I obfuscated my source IP. I don't see how that would block me, since I WAS able to send email to Gmail -- not just one with the content of that particular periodic email. In fact, I'm getting email from periodic on other computers in the same network. Weird!

Meller and Toosmooth, great ideas -- I fixed the time (odd!) and added vm.taosecurity.com to /etc/mail/local-host-names:

vm# cat /etc/mail/local-host-names
vm.taosecurity.com

vm# echo '$=w' | sendmail -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
< [10.10.1.13]
vm
[IPv6:::1]
[IPv6:fe80::1]
localhost.taosecurity.com
localhost
[127.0.0.1]
[localhost.taosecurity.com]
vm.taosecurity.com

I then reran the test with the following transcript showing the same failed result:

074.125.091.027.00025-073.128.035.011.62818: 220 mx.google.com ESMTP dn12si2766596qcb.160

073.128.035.011.62818-074.125.091.027.00025: EHLO vm.taosecurity.com

074.125.091.027.00025-073.128.035.011.62818: 250-mx.google.com at your service, [73.128.35.11]
250-SIZE 35651584
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING

073.128.035.011.62818-074.125.091.027.00025: MAIL From:<analyst@vm.taosecurity.com> SIZE=1387

074.125.091.027.00025-073.128.035.011.62818: 250 2.1.0 OK dn12si2766596qcb.160

073.128.035.011.62818-074.125.091.027.00025: RCPT To:<taosecurity@gmail.com>
DATA

074.125.091.027.00025-073.128.035.011.62818: 250 2.1.5 OK dn12si2766596qcb.160
354 Go ahead dn12si2766596qcb.160

073.128.035.011.62818-074.125.091.027.00025: Received: from vm.taosecurity.com (localhost [127.0.0.1])
.by vm.taosecurity.com (8.14.4/8.14.4) with ESMTP id oAJ3a5Vd022002
.for <taosecurity@gmail.com>; Thu, 18 Nov 2010 22:36:05 -0500 (EST)
.(envelope-from analyst@vm.taosecurity.com)
Received: (from root@localhost)
.by vm.taosecurity.com (8.14.4/8.14.4/Submit) id oAJ3a52a022001
.for taosecurity@gmail.com; Thu, 18 Nov 2010 22:36:05 -0500 (EST)
.(envelope-from analyst)
Date: Thu, 18 Nov 2010 22:36:05 -0500 (EST)
From: analyst <analyst@vm.taosecurity.com>
Message-Id: <201011190336.oAJ3a52a022001@vm.taosecurity.com>
To: taosecurity@gmail.com
Subject: From vm, fake periodic output for blog, fixed time, local-host-names

Checking setuid files and devices:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

r200b.taosecurity.com kernel log messages:
+++ /tmp/security.QW4ZT9Yc.2010-11-18 20:07:29.000000000 -0500
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled
+bge0: promiscuous mode disabled
+bge0: promiscuous mode enabled

r200b.taosecurity.com login failures:
Nov 17 07:51:58 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.
Nov 17 07:52:02 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.

r200b.taosecurity
073.128.035.011.62818-074.125.091.027.00025: .com refused connections

Checking for a current audit database:

Database created: Thu Nov 18 19:05:00 EST 2010

Checking for packages with security vulnerabilities:

0 problem(s) in your installed packages found.

-- End of security output --

073.128.035.011.62818-074.125.091.027.00025: .

074.125.091.027.00025-073.128.035.011.62818: 550-5.7.1 [73.128.35.11] The IP you're using to send mail is not authorized to
550-5.7.1 send email directly to our servers. Please use the SMTP relay at your
550-5.7.1 service provider instead. Learn more at
550 5.7.1 http://mail.google.com/support/bin/answer.py?answer=10336 dn12si2766596qcb.160

073.128.035.011.62818-074.125.091.027.00025: QUIT

Richard Bejtlich said...

Phil -- sorry, I obfuscated my source IP. I don't see how that would block me, since I WAS able to send email to Gmail -- not just one with the content of that particular periodic email. In fact, I'm getting email from periodic on other computers in the same network. Weird!

Meller and Toosmooth, great ideas -- I fixed the time (odd!) and added vm.taosecurity.com to /etc/mail/local-host-names:

vm# cat /etc/mail/local-host-names
vm.taosecurity.com

vm# echo '$=w' | sendmail -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
< [10.10.1.13]
vm
[IPv6:::1]
[IPv6:fe80::1]
localhost.taosecurity.com
localhost
[127.0.0.1]
[localhost.taosecurity.com]
vm.taosecurity.com

I then reran the test with the following transcript showing the same failed result (added as an update since it's too much for a comment.)

gerald said...

Incidentally I've gotten two gmails with jpg displayed what it wasn't set to display any images??

this poses a security problem.
I'd like to know how they are doing it?

Gerald Anthro

Richard Bejtlich said...

Sorry about the duplicate comments -- Blogger complained that the large comment was too large, but posted it anyway.

Anonymous, interesting idea about the IP addresses; I tried replacing them with REMOVED but the email was still blocked.

Incidentally, here's an example of a periodic message that was NOT blocked (it arrived on its own from computer vm at 10 pm eastern):

Checking setuid files and devices:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

vm.taosecurity.com login failures:

vm.taosecurity.com refused connections:

-- End of security output --

I can also manually get it through by sending that content alone.

bamm said...

Try adding MASQUERADE_AS(`taosecurity.com') to your sendmail.mc and rebuild.

Joe said...

Could it be that a reverse lookup is failing and thus gmail rejects your mail?

Felipe said...

For me:

$ host r200b.taosecurity.com
Host r200b.taosecurity.com not found: 3(NXDOMAIN)

Where do you see that host r200b has 73.128.35.11? Make sure you have a valid PTR record.

Richard Bejtlich said...

Bamm, thanks -- I'll try that when I get a chance.

Felipe -- I obfuscated my IP address.

Joe, I was able to send email to Gmail from both systems, just not a particular output from "periodic." I don't think it's a reverse lookup problem.

Anonymous said...

I'm confused about the error
Nov 17 07:51:58 r200b sshd[53170]: error: connect_to 73.128.35.11 port 80: failed.


Also - you definitely need a valid PTR for the machine, which when looked up, should match the address.
Check on this.

And is the HELO field a valid domain name? The RFC's state it does not have to be a valid hostname, but I found all valid mail systems do this, while spam messages do not. So I reject any mail with a bogus HELO value.

And I can't access the URL google provided as an error. This might provide a clue. You must have munged it.

- Bruce

Anonymous said...

Try adding SPF TXT record for the domain you are using to send mail.

The Ubiquitous Mr. Lovegroove said...

Google uses SPF. Your mail from: are different in both cases. vm.taosecurity.com is accepted while r200b.taosecurity.com is not.

Add SPF records and see it magicall work.

John said...

I had different but possibly not unrelated problems sending to the US Army a few years back. Very, very long story short- it was a problem with a machine in a cluster. It had sorta kinda dropped out of the cluster somehow. The reason you can send from some machines sometimes and not others later has to do somehow with affinity.

Check ACK sequences. I was able to determine that the machines were different when I showed that the ACK sequences from 2 different messages sent at roughly the same time had to come from two different machines.

Headache to figure out technically
HUGE headache to get someone to listen.

Richard Bejtlich said...

Thanks again for comments, but I think some of you are missing an important point:

1. I can send email to Gmail from vm.taosecurity.com without making any changes to anything.

2. When I try to send email to Gmail from vm.taosecurity.com, but the content is the periodic email content, Gmail rejects it.

The best explanation I can imagine is that sending from a residential cable modem is a "borderline" accept for Gmail, but something in the periodic email content pushes it over the edge and prompts an error message (which doesn't describe the real problem in any case because I CAN send email from that IP).

Does that make sense?

pstoklosa said...

Can you strip the email down, almost line by line, to see if you can narrow it down to a particular line that's triggering the rejection?

Chris said...

You're hitting the postini filters. Postini uses a weighted scoring system, and your email, because it's coming from a non-SPF, residential source, is borderline. Add in some secret message-body hurestic sauce, and your score drops below the allowed for gmail delivery.

You can test this hypothesis by sending mail to a commercial postini customer - the scoring details are added to the SMTP header. That way, you can see what changes in the score from one email to the other.

Anonymous said...

Have you tried installing the sSMTP port and seeing if this problem still persists with the daily run output?