A common question about moving applications to EC2 is what to do about sending email. Web applications occasionally send email, to their administrators (“Help! I’m running out of disk space!”) or to their users (“Here is your forgotten password”), and this needs to work when the application is hosted within EC2 also. Unfortunately, it is not simple to send email reliably from within the cloud. Here are some tips about sending reliable email from EC2 instances.
What is the Problem?
One word: spam. Email sent from servers within EC2 is marked as spam by most ISPs and email providers. The cloud offers a very scalable way for anyone to launch email-sending machines, on-demand. Spammers know this. So do the anti-spammers. Luckily (for the anti-spammers), existing anti-spam safeguards are effective against attempts to send spam from within EC2. Unluckily (for you), these same anti-spam measures make your legitimate (I hope!) emails get flagged as spam.
The anti-spam safeguard that trips up your application’s emails is called reverse DNS lookup. This lookup checks that the name and IP address of the mailserver sending the message match the name and IP address specified in DNS records. Here’s how it works:
- A machine at
mail.yahoo.comreceives a message from your mail server, whose IP address is (let’s say) 1.2.3.4. mail.yahoo.comlooks in DNS for a PTR record for the IP address 1.2.3.4. If none exists, the mail is not delivered. Otherwise, the PTR record specifies the FQDN of the machine that should have the address 1.2.3.4. Let’s say this is specified asmail.mydomain.com.mail.yahoo.comlooks in DNS for the IP address of the host specified by the PTR lookup:mail.mydomain.com. If the IP address does not match the address from which the mail was received (1.2.3.4 in our case), the message is not delivered.
The above is a bit of a simplification, but it is enough to explain the difficulty in sending email from within EC2. The problem lies in step 2, when the receiving mail server tries to find the PTR record for the IP address from which the mail was sent. The public IP addresses used by EC2 belong to Amazon, and the DNS lookups for these addresses will therefore look in Amazon’s DNS records. Amazon does not have PTR records pointing to your mailserver’s FQDN in their DNS, so the reverse DNS lookup fails at step 2, and your email message is not delivered.
Amazon, unlike traditional hosting companies, does not have a service that allows you to set a reverse DNS entry (a PTR record) within their DNS records. If they did offer such a service it would solve the reverse DNS lookup problem, and enable your EC2 application to send emails. Of course, such a service would also allow spammers to easily use EC2 for sending spam, so it is clear that Amazon cannot offer a service to allow custom PTR records without instituting some effective safeguards against abuse.
Update 30 October 2009: Amazon now has a new email policy in which outbound SMTP traffic is blocked (beyond miniscule usage). In order to be able to send email directly from EC2 you also need to provision an Elastic IP address for your instance and submit the following form. In return, Amazon will work to keep that Elastic IP of of the common anti-spam lists. More information here. Note that the comments in this article relating to reverse DNS are still applicable: some email providers do not check the PTR records, and Amazon’s new program will help email get through to those providers – but not all.
Update 25 January 2010: Amazon has announced a new private beta where they will set PTR records for your Elastic IP address. To participate in the private beta, join the EC2 developer forums and send a private message to ian@aws.
Update 23 March 2010: Reverse DNS for Elastic IPs is now officially supported!
Sending Email from EC2: What Works
In any case, let us explore how you can send email from within EC2. Here are a number of cases that work:
- One case that works is sending email to email accounts that are hosted on the same EC2 instance, using sendmail as the mail transfer agent (MTA). This can be appropriate for a corporate setting, where your recipients can be told to configure their email readers accordingly. To do this, set up sendmail on your EC2 instance, then set up a POP3 or IMAP server on the same instance, and then set up your users’ email readers to fetch email from those accounts. Be sure to configure the sendmail and POP/IMAP services to accept connections only from machines you trust. (You’ll probably want to use an Elastic IP and an EBS drive also, to ensure that access to those email boxes can be recovered if the instance goes down.)This case does not have the reverse DNS lookup problem because sendmail will deliver the messages to the account on the local machine directly, bypassing the reverse DNS lookup.One limitation of this approach is that the users must learn to check this “special” email account. Some email readers make this easy by allowing more than one server-side account to be checked.
A more significant limitation to this approach is that it will not work for sending mail to recipients who have accounts elsewhere. The reverse DNS lookup anti-spam measure described above will prevent those messages from getting through.
- Sending email from EC2 to an address whose email delivery you control can work also. The application sends email from a known email account (say, “myAppAdminEvents@mydomain.com”) to your application administrators’ email accounts in your domain. In this approach you set up an SMTP server on your EC2 instance. You configure the SMTP installation on EC2 to relay mail directly to your domain’s mail servers. And you need to make sure that the anti-spam tools running on your domain’s mail servers (and in the user’s email reader) are configured to whitelist messages from the known email account.
- You can use third-party SMTP services to send emails from your EC2 instance. There are many third-party email services available, including the free-but-limited Google Apps Email, and also including paid-but-less-limited solutions. A detailed discussion of how to set up your email to use a third-party SMTP server is beyond the scope of this article (but here’s a good one). However, a useful strategy is to run an SMTP server locally on your EC2 instance which is configured to relay the mail to the third-party server. This relaying setup allows your application to benefit from reduced connection time when sending messages, and relays the messages in a separate process.There are many third-party SMTP service providers available, each with their own pricing structure and Terms & Conditions.
Google Apps Email’s limitations are an interesting subject, and I plan another blog article on strategies for dealing with these limitations.
The above three methods of sending email from your EC2 instances each have their pros and cons. If you’re only delivering messages to internal recipients, use the first or the second method. If you are also sending mails to arbitrary addresses, go for a third-party SMTP provider.