Tag: ec2

Sending Email from EC2

A common question about moving applications to EC2 is what to do about sending email. Web applications occasionally send email, to their administrators (“Help! I’m running out of disk space!”) or to their users (“Here is your forgotten password”), and this needs to work when the application is hosted within EC2 also. Unfortunately, it is not simple to send email reliably from within the cloud. Here are some tips about sending reliable email from EC2 instances.

What is the Problem?

One word: spam. Email sent from servers within EC2 is marked as spam by most ISPs and email providers. The cloud offers a very scalable way for anyone to launch email-sending machines, on-demand. Spammers know this. So do the anti-spammers. Luckily (for the anti-spammers), existing anti-spam safeguards are effective against attempts to send spam from within EC2. Unluckily (for you), these same anti-spam measures make your legitimate (I hope!) emails get flagged as spam.

The anti-spam safeguard that trips up your application’s emails is called reverse DNS lookup. This lookup checks that the name and IP address of the mailserver sending the message match the name and IP address specified in DNS records. Here’s how it works:

A machine at mail.yahoo.com receives a message from your mail server, whose IP address is (let’s say) 1.2.3.4.
mail.yahoo.com looks in DNS for a PTR record for the IP address 1.2.3.4. If none exists, the mail is not delivered. Otherwise, the PTR record specifies the FQDN of the machine that should have the address 1.2.3.4. Let’s say this is specified as mail.mydomain.com.
mail.yahoo.com looks in DNS for the IP address of the host specified by the PTR lookup: mail.mydomain.com. If the IP address does not match the address from which the mail was received (1.2.3.4 in our case), the message is not delivered.

The above is a bit of a simplification, but it is enough to explain the difficulty in sending email from within EC2. The problem lies in step 2, when the receiving mail server tries to find the PTR record for the IP address from which the mail was sent. The public IP addresses used by EC2 belong to Amazon, and the DNS lookups for these addresses will therefore look in Amazon’s DNS records. Amazon does not have PTR records pointing to your mailserver’s FQDN in their DNS, so the reverse DNS lookup fails at step 2, and your email message is not delivered.

Amazon, unlike traditional hosting companies, does not have a service that allows you to set a reverse DNS entry (a PTR record) within their DNS records. If they did offer such a service it would solve the reverse DNS lookup problem, and enable your EC2 application to send emails. Of course, such a service would also allow spammers to easily use EC2 for sending spam, so it is clear that Amazon cannot offer a service to allow custom PTR records without instituting some effective safeguards against abuse.

Update 30 October 2009: Amazon now has a new email policy in which outbound SMTP traffic is blocked (beyond miniscule usage). In order to be able to send email directly from EC2 you also need to provision an Elastic IP address for your instance and submit the following form. In return, Amazon will work to keep that Elastic IP of of the common anti-spam lists. More information here. Note that the comments in this article relating to reverse DNS are still applicable: some email providers do not check the PTR records, and Amazon’s new program will help email get through to those providers – but not all.

Update 25 January 2010: Amazon has announced a new private beta where they will set PTR records for your Elastic IP address. To participate in the private beta, join the EC2 developer forums and send a private message to ian@aws.

Update 23 March 2010: Reverse DNS for Elastic IPs is now officially supported!

Sending Email from EC2: What Works

In any case, let us explore how you can send email from within EC2. Here are a number of cases that work:

One case that works is sending email to email accounts that are hosted on the same EC2 instance, using sendmail as the mail transfer agent (MTA). This can be appropriate for a corporate setting, where your recipients can be told to configure their email readers accordingly. To do this, set up sendmail on your EC2 instance, then set up a POP3 or IMAP server on the same instance, and then set up your users’ email readers to fetch email from those accounts. Be sure to configure the sendmail and POP/IMAP services to accept connections only from machines you trust. (You’ll probably want to use an Elastic IP and an EBS drive also, to ensure that access to those email boxes can be recovered if the instance goes down.)This case does not have the reverse DNS lookup problem because sendmail will deliver the messages to the account on the local machine directly, bypassing the reverse DNS lookup.One limitation of this approach is that the users must learn to check this “special” email account. Some email readers make this easy by allowing more than one server-side account to be checked.
A more significant limitation to this approach is that it will not work for sending mail to recipients who have accounts elsewhere. The reverse DNS lookup anti-spam measure described above will prevent those messages from getting through.
Sending email from EC2 to an address whose email delivery you control can work also. The application sends email from a known email account (say, “myAppAdminEvents@mydomain.com”) to your application administrators’ email accounts in your domain. In this approach you set up an SMTP server on your EC2 instance. You configure the SMTP installation on EC2 to relay mail directly to your domain’s mail servers. And you need to make sure that the anti-spam tools running on your domain’s mail servers (and in the user’s email reader) are configured to whitelist messages from the known email account.
You can use third-party SMTP services to send emails from your EC2 instance. There are many third-party email services available, including the free-but-limited Google Apps Email, and also including paid-but-less-limited solutions. A detailed discussion of how to set up your email to use a third-party SMTP server is beyond the scope of this article (but here’s a good one). However, a useful strategy is to run an SMTP server locally on your EC2 instance which is configured to relay the mail to the third-party server. This relaying setup allows your application to benefit from reduced connection time when sending messages, and relays the messages in a separate process.There are many third-party SMTP service providers available, each with their own pricing structure and Terms & Conditions.
Google Apps Email’s limitations are an interesting subject, and I plan another blog article on strategies for dealing with these limitations.

The above three methods of sending email from your EC2 instances each have their pros and cons. If you’re only delivering messages to internal recipients, use the first or the second method. If you are also sending mails to arbitrary addresses, go for a third-party SMTP provider.

Tags ec2, email, PTR, reverse dns, smtp, spam

Cloud Developer Tips

EC2 Reserved Instance Billing Gotcha

Post author By Shlomo Swidler
Post date July 1, 2009
No Comments on EC2 Reserved Instance Billing Gotcha

So you purchased an EC2 Reserved Instance and you’re looking to see the reduced hourly rate take effect immediately.

Don’t.

Unlike all the other resources in EC2, which you can use immediately, reserved instances are not actually available until the credit card charge for your account is processed.

You’ll need to wait until ~~the next billing period begins~~ the charge clears your credit card before the reserved instance is actually available and the discounted hourly rate takes effect. Until your credit card charge goes through, your instances will continue to be billed at the regular hourly rates.

It’s a classic gotcha, but it’s in the EC2 FAQ.

Update: According to AWS folks, the charge for the reservation is posted to your credit card soon after you order the reserved instance, instead of waiting for the next billing period. Here is the AWS Forum thread where AWS says so.

Tags ec2, reserved instance

Cloud Developer Tips

Tagging EC2 Instances Using Security Groups

Post author By Shlomo Swidler
Post date June 30, 2009
12 Comments on Tagging EC2 Instances Using Security Groups

Update September 2010: AWS has added support for tagging instances, volumes, snapshots, and many other EC2 resource types via the API and the AWS Management Console. The techniques described here are no longer necessary.

So you have a bunch of instances running in Amazon EC2 and you want to be able to tell them apart easily. The way to do this is to “tag” each instance with a descriptive tag, such as “database”, “web server”, “memcached”, etc. The easy, human-friendly way to tag your instances is to use ElasticFox, and the robust, programming-friendly way is to use Security Groups.

Tagging Instances in ElasticFox

The ElasticFox Firefox extension is a great UI for managing your EC2 resources. ElasticFox supports adding tags for instances (as well as tagging EBS volumes, AMIs, and Elastic IPs). These tags are stored in internal Firefox preferences.

Let’s tag an instance in ElasticFox:

1. Here is the ElasticFox extension. Note how it shows that I am running one EC2 instance.
Notice the “Tag” column, which is empty.

2. Right-click on the entry for the instance we want to tag:
Now, add the tag in the dialog that pops up:
And the result:

Tagging instances in ElasticFox is great if you are the only person managing EC2 instances, and if you only use ElasticFox from a single machine. Because the tags are stored in your local browser, they are only visible to you. But, if you need your tags to be visible to other people, or on multiple browsers/computers, tagging in ElasticFox will not help because the tags are not stored in the cloud, only in the local browser. Other browsers / users / computers will not see the tags that you supply using ElasticFox. And, if you need programmatic access (via the EC2 API or command-line tools) to the tags, ElasticFox tags will not help.

Update 15 July 2009: Check out my article about how to copy ElasticFox settings (including tags) between browsers.

Tagging Instances Using Security Groups

Amazon Security Groups can also be used as a tagging mechanism for EC2 instances. The basic idea is to create a custom security group named, for example, “#database” or “#web server”, and launch the instance(s) in that custom security group in addition to the other security groups you want. The custom security group is defined without any permissions so it will have no effect on any instances in the group. This works without affecting the “real” security because the effective security permissions for an instance are the union (the sum) of all the permissions on the security groups the instance belongs to, and the “tag” security group grants zero permissions: x + 0 = x.

Using security groups to tag instances is slightly more cumbersome because you must create the custom-named security group in a separate step before launching instances in it. However, tagging via security groups is a more robust solution for the following reasons:

Security groups are stored inside the cloud. When you add an instance to a security group, you do not need to “remember” anything on the client-side: you can change browsers or change computers and your assigned security groups will still be visible every time you look.
Security groups are visible via the API and the command-line tools. This mean you can write code, in bash/PHP/Java/python/etc., that reads and manipulates the instance “tags” (really security groups). In fact, if you choose a convention for naming your security groups that are really tags, you can programmatically distinguish between the “real” security group (such as “default”) and the “tag” security group. I use the prefix “# ” for naming the security groups that function as tags.

However, all this extra flexibility comes with a price: you cannot add an instance to a security group after it is running; you can only specify an instance’s security groups at launch time. If you use security groups to tag instances, your must specify all your security group tags when you launch your instance(s).

You can use ElasticFox to create tag security groups and to specify the security groups when you launch instances. There is one gotcha: when creating the tag security group, make sure you choose “I will authorize protocols for this group as needed” – this creates the group without adding any permissions.

Instead of walking through that process in ElasticFox, here is an example of using the command-line tools to manage tag security groups.

Managing Tag Security Groups from the Command-Line

To create a security group on the command-line, use the ec2-add-group command:

ec2-add-group "#web server" -d "tag web server"
Output:

GROUP #web server tag web server

This creates a security group called #web server with the description tag web server.

To launch an instance with this tag, use the ec2-run-instances command:

ec2-run-instances ami-12345678 --instance-type m1.small --key gsg-keypair --group default --group "#web server"
Output:

RESERVATION r-27e29b4e 614123456789 #web server,default INSTANCE i-031b376a ami-12345678 pending gsg-keypair 0 m1.small 2009-06-30T11:36:30+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc monitoring-disabled

This launches a single instance of ami-12345678, in any availability zone in the US region (I got us-east-1c), using the gsg-keypair, and having the security groups default and #web server. In practice, you will substitute the AMI id you want to use and the name of your keypair. You may also want to add other arguments, so check out the docs or try ec2-run-instances --help for more command-line options. You can see in the output above that the security groups are default and #web server.

Once the instance starts running, you can use the ec2-describe-instances command to see all your running instances and the security groups they belong to:

ec2-describe-instances
Output:

RESERVATION r-27e29b4e 614123456789 #web server,default INSTANCE i-031b376a ami-12345678 ec2-75-101-177-214.compute-1.amazonaws.com ip-10-250-18-244.ec2.internal running gsg-keypair 0 m1.small 2009-06-30T11:36:30+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc monitoring-disabled

Note the #web server security group in the output. This is the tag security group for this instance.

You can parse the output of the above command using shell utilities (awk, perl, cut, etc.) if you want to programmatically discover what instances belong to which groups.

Update 24 September 2009: AWS now supports the ability to add a long description to EBS snapshots.

Tags aws, ec2, elasticfox, security groups, tagging, tagging instances