Categories
Cloud Developer Tips

AWS Auto-Scaling and ELB with Reliable Root Domain Handling

Update May 2011: Now that AWS Route 53 can be used to allow an ELB to host a domain zone apex, the technique described here is no longer necessary. Cool, but not necessary.

Someone really has to implement this. I’ve had this draft sitting around ever since AWS announced support for improved CloudWatch alerts and AutoScaling policies (August 2010), but I haven’t yet turned it into a clear set of commands to follow. If you do, please comment.

Background

You want an auto-scaled, load-balanced pool of web servers to host your site at example.com. Unfortunately it’s not so simple, because AWS Elastic Load Balancer can’t be used to host a domain apex (AKA a root domain). One of the longest threads on the AWS Developer Forum discusses this limitation: because ELB utilizes DNS CNAMEs, which are not legal for root domain entries, ELB does not support root domains.

An often-suggested workaround is to use an instance with an Elastic IP address to host the root domain, via standard static DNS, with the web server redirecting all root domain requests to the subdomain (www) served by the ELB. There are four drawbacks to this approach:

  1. The instance with the Elastic IP address is liable to be terminated by auto-scaling, leaving requests to the root domain unanswered.
  2. The instance with the Elastic IP address might fail unnaturally, again leaving requests to the root domain unanswered.
  3. Even when traffic is very low, we need at least two instances running: the one handling the root domain outside the auto-scaled ELB group (due to issue #1) and the one inside the auto-scaled ELB group (to handle the actual traffic hitting the ELB-managed subdomain).
  4. The redirect adds additional latency to requests hitting the root domain.

While we can’t do anything about the fourth issue, what follows is a technique to handle the first three issues.

The Idea

The idea is built on these principles:

  • The instance with the Elastic IP is outside the auto-scaled group so it will not be terminated by auto-scaling.
  • The instance with the Elastic IP is managed using AWS tools to ensure the root domain service is automatically recovered if the instance dies unexpectedly.
  • The auto-scaling group can scale back to zero size, so only a single instance is required to serve low traffic volumes.

How do we put these together?

Here’s how:

  1. Create an AMI for your web server. The AMI will need some special boot-time hooks, which are described below in italics. The web server should be set up to redirect root domain traffic to the subdomain that you’ll want to associate with the ELB, and to serve the subdomain normally.
  2. Create an ELB for the site’s subdomain with a meaningful Health Check (e.g. a URL that exercises representative areas of the application).
  3. Create an AutoScaling group with min=1 and max=1 instances of that AMI. This AutoScaling group will benefit from the default health checks that such groups have, and if EC2 reports the instance is degraded it will be replaced. The LaunchConfiguration for this AutoScaling group should specify user-data that indicates this instance is the “root domain” instance. Upon booting, the instance will notice this flag in the user data, associate the Elastic IP address with itself, an add itself to the ELB.
    Note: At this point, we have a reliably-hosted single instance hosting the root domain and the subdomain.
  4. Create a second AutoScaling group (the “ELB AutoScaling group”) that uses the same AMI, with min=0 instances – the max can be anything you want it to – and set it up to use the ELB’s Health Check. The LaunchConfiguration for this group should not contain the abovementioned special flag – these are not root domain instances.
  5. Create an Alarm that looks at the CPUUtilization across all instances of the AMI, and connect it to the “scale up” and “scale down” Policies for the ELB AutoScaling group.

That is the basic idea. The result will be:

  • The root domain is hosted on an instance that redirects to the ELB subdomain. This instance is managed by a standalone Auto Scaling group that will replace the instance if it becomes degraded. This instance is also a member of the ELB, so it serves the subdomain traffic as well.
  • A second AutoScaling group manages the “overflow” traffic, measured by the CPUUtilization of all the running instances of the AMI.

TODO

Here are the missing pieces:

  1. A script that can be run as a boot-time hook that checks the user-data for a special flag. When this flag is detected, the script associates the root domain’s Elastic IP address (which should be specified in the user-data) and adds the instance to the ELB (whose name is also specified in the user-data). This will likely require AWS Credentials to be placed on the instance – perhaps in the user-data itself (be sure you understand the security implications of this) as well as a library such as boto or the AWS SDK to perform the AWS API calls.
  2. The explicit step-by-step instructions for carrying out steps 1 through 5 above using the relevant AWS command-line tools.

Do you have these missing pieces? If so, please comment.

9 replies on “AWS Auto-Scaling and ELB with Reliable Root Domain Handling”

I thought about something similar, but decided it was too much overhead. We have dozens to hundreds of different websites active at any time and having 2x autoscaling groups for that is too onerous.

Instead, I created a special ‘redirect only’ server AMI that has its own elastic IP and auto scaling group (min/max size 1). The redirect server redirects any request for blah.com to http://www.blah.com.

@jsd,

My goal in outlining this technique is to encourage people to publish the detailed step-by-step instructions, which can then be automated. This would reduce the overhead significantly.

For your use case hosting hundreds of sites this solution won’t work: you would hit AWS limits on the number of AutoScaling groups per account. And you can benefit from economy of scale by hosting all the redirect services on the same instance. Of course, as you probably know, then all root domains go down together when that instance has a problem.

Thanks for the bootstrap. Here’s how to automatically assign the IP on boot (linux):

Security first, create yourself a new account via IAM. We’ll call this the “redirector account”. Use IAM to lock this account down to only have ec2-associate-address permissions. The IAM Policy Generator is your friend.

Next, you’ll set up a certificate for the redirector account. You can do this like so (stolen from https://forums.aws.amazon.com/thread.jspa?threadID=51862):

$ openssl version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
$ openssl genrsa 1024 > creds/fitch_nate/pk.pem
$ openssl req -new -x509 -nodes -sha1 -days 730 -key creds/fitch_nate/pk.pem > creds/fitch_nate/cert.pem
(For this exmaple, I just chose all defaults)

Now associate to an IAM user:
$ IAMCli/iam-useraddcert -u nate -f creds/fitch_nate/cert.pem

Set up the EC2 command line tool:
$ export EC2_CERT=./creds/fitch_nate/cert.pem
$ export EC2_PRIVATE_KEY=./creds/fitch_nate/pk.pem

Be sure to chmod 0600 these credentials. The redirector account is now set up. You now need to write a boot script to assign the elastic IP. As of this writing, the Amazon 32-bit Linux instance allows you a /etc/rc.local file which is called as the last element of the boot sequence. Be lazy and put these two commands in here:

export INSTANCE_ID=`curl ‘http://169.254.169.254/latest/meta-data/instance-id’`
ec2-associate-address -K /root/.ec2/creds/webredirector.pk.pem -C /root/.ec2/creds/webredirector.cert.pem -i $INSTANCE_ID your_elastic_ip_address

This will query a special local webserver to obtain the instance id, then pass this instance id into the ec2-associate-address command. Set up your autoscaling group ala http://docs.amazonwebservices.com/AutoScaling/latest/GettingStartedGuide/ and you’re set.

As a standing benchmark, a micro instance is processing over 500 requests/second at 50% cpu usage. Simple apache configuration. YRMV. Cheers.

hi, this doesn’t need an implementation anymore. you can achieve this quite easily with route53 since may 2011.

The Route53 solution doesn’t work for companies that host sites for customers but don’t control the DNS for those sites.

Leave a Reply to Servers are not balanced across multiple availability zones | Newvem Cancel reply

Your email address will not be published.