Cloud Developer Tips

Copying ElasticFox Tags from One Browser to Another

The ElasticFox Firefox extension allows you to tag EC2 instances, EBS volumes, EBS snapshots, Elastic IPs, and AMIs. ElasticFox’s tags are stored locally within Firefox, so if you use ElasticFox from more than one browser your tags from one browser are not visible in any other browser. In this article I demonstrate how to copy tags from one ElasticFox installation to another.

Where Are ElasticFox Tags Stored?

My article about Tagging EC2 Instances Using Security Groups shows a brief walkthrough of using ElasticFox to tag instances. Under the hood, ElasticFox stores tags in the prefs.js file within your Firefox Profile directory. But instead of poking around in there directly, take a look at Firefox’s built-in interface to these preferences. It’s the about:config screen, and you get there by typing about:config into the address bar:

The Firefox about:config screen

Click “I’ll be careful, I promise!” Up comes the about:config screen, with all sorts of interesting-looking settings. We’re interested in the ElasticFox tags only, so type ec2ui.*tag in the Filter field, and the result looks like this:

Viola! The ElasticFox tags are stored here.

Viola! Those are the ElasticFox tags that I’ve set up.

Copying ElasticFox Tags to Another Browser Manually

You can copy ElasticFox tags from the about:config screen, via the clipboard, into a text file. First, open the about:config screen and use the Filter as described above to see the relevant settings. Create an empty text file and copy each setting (right-click on the entry and choose “Copy”) and paste it into the text file. Add each setting you want to transfer into the text file. You can email the text file to yourself (or put it on your USB key) and then use the text file to help you change the values of these settings in the target browser’s about:config screen.

Once copied setting looks like this:;({'':''})

The setting’s name is and its value is ({'':''}), separated from the name by a semicolon (;). For each setting you want to transfer to the target browser, select the value (after the semicolon) from the text file and copy it to the clipboard. Then, in the target browser’s about:config screen, right-click on the corresponding setting entry and choose “Modify”, and paste in the new value.

All that selecting and copying and pasting is annoying, and you might make a mistake when entering the values in the target browser. I feel your pain. There must be a better way.

Copying ElasticFox Tags to Another Browser with OPIE

OPIE is a Firefox extension that allows you to import and export preference settings. The good thing about it is that it can be used to export only some settings – and we only want to export the ElasticFox settings. Unfortunately, it does not support ElasticFox!

Please help lobby the OPIE developer to add ElasticFox support by posting in this forum thread – OPIE support would make copying ElasticFox tags (and indeed, entire ElasticFox setups) between browsers much easier.

Copying ElasticFox Tags to Another Browser with Shell Scripts

I’ve put together two shell scripts that can be used to directly export ElasticFox settings from the Firefox prefs.js file and to directly import them back in. These scripts really export and import all your ElasticFox settings, not just tags. Read on for directions on how to use these scripts, which work on Linux, Mac OS X, and Solaris, for Firefox 3.x and above.

A word of caution before you export and import via prefs.js directly: make sure Firefox is not running when you export or import. Firefox overwrites the prefs.js file upon exiting, and reads it upon starting up. In order to make sure all your ElasticFox tags (and all other settings, too) are in the export, Firefox must be closed before exporting. Likewise, in order to make sure Firefox picks up your imported settings you need to make sure it is closed before importing.

How to Export ElasticFox Settings

Here is a handy script to export ElasticFox preferences from Firefox. You can download it, or copy it from here and save it to your local machine:

#! /bin/bash

# the base name of the extension preferences
# that are to be imported
# ec2ui is the extension base name of ElasticFox

# backslash-escape special characters as you
# would in a regular expression
# the following command will only escape
# periods (".") but that should suffice
# for firefox extensions
ESCAPED_PREF_BASE=$(echo $PREF_BASE | sed -e "s/\./\\\\./g")

# export desired prefs
egrep "^user_pref\(\"$ESCAPED_PREF_BASE\." prefs.js

Once you save the script locally, make it executable:

chmod +x

The script expects to be run in the same directory as the prefs.js file, but it’s most convenient to keep this script in your home directory and create a soft-link there to the prefs.js file. On my machine, the command to make a soft-link to the prefs.js file is this:

ln -s /Users/shlomo/Library/Application\ Support/Firefox/Profiles/f0psntn6.default/prefs.js prefs.js

You’ll need to specify the location of your Firefox profile directory (instead of mine).

Remember to exit Firefox. Now you’re ready to run the export script.

./ > savedPrefs

This will create a file called savedPrefs containing the ElasticFox settings. Move this file to the target browser’s machine, where we will import it.

How to Import ElasticFox Settings

Here is a script to import to import ElasticFox settings into Firefox. Download it, or copy it from here and save it to your local machine:

#! /bin/bash


# the base name of the extension preferences
# that are to be imported
# ec2ui is the extension base name of ElasticFox

if [ -z "$1" ]
  echo "Please provide the filename of the file containing the prefs to import."
# backslash-escape any  characters as you would
# in a regular expression
# this command only handles periods
# but that should suffice for firefox extensions
ESCAPED_PREF_BASE=$(echo $PREF_BASE | sed -e "s/\./\\\\./g")

# backup the existing prefs
cp prefs.js prefs.js.old

# get the file header
egrep -v "^user_pref\(" prefs.js > $OUTFILE

# add all other prefs
egrep "^user_pref\(" prefs.js | egrep -v "^user_pref\(\"$ESCAPED_PREF_BASE\." > $WORKFILE

# add desired prefs

# sort all prefs into output

# clean up

# uncomment this when you trust this script
# to replace your prefs automatically
#cp $OUTFILE prefs.js

After you save the script locally, make it executable:

chmod +x

As with the export script above, the import script expects to be run in the same directory as the Firefox prefs.js file, so make a soft-link to the prefs.js file in the same directory as the script (see above).

Once you’ve made sure that Firefox is not running, you’re ready to run the import script:

./ savedPrefs

This will create a file called newPrefs.js in the same directory. newPrefs.js will contain the imported ElasticFox settings from the savedPrefs file, merged with the other already-existing Firefox settings from prefs.js. The script will also create a backup copy of prefs.js called prefs.js.old. All you need to do is to replace your Firefox’s prefs.js with newPrefs.js, as follows:

cp newPrefs.js prefs.js
rm newPrefs.js

Now, restart Firefox. It should contain the ElasticFox settings from the source browser. If anything doesn’t work, simply exit Firefox and restore the previous prefs.js:

cp prefs.js.old prefs.js

Once you are confident that the script works consistently, you can uncomment the last few lines of the script and let it automatically replace Firefox’s prefs.js file by itself – then you do not need to perform that step manually.

This method of copying ElasticFox settings using shell scripts is relatively easy, but not perfect. OPIE would be a friendlier way, and would support Windows as well (so please lobby the developer to support ElasticFox). If you’re looking for your ElasticFox tags to be available via the EC2 APIs, check out my article on using security groups as a way to tag instances (but unfortunately not Elastic IPs, EBS volumes and snapshots, or AMIs).

Cloud Developer Tips

Boot EC2 Instances from EBS

EBS offers the ability to have a “virtual disk drive” whose lifetime is independent of any EC2 instance. You can attach EBS drives to any running instance, and any changes made to the drive will persist even after the instance is terminated. You can also set up an EBS drive to be the root filesystem for an instance – giving you the benefits of an always-on instance but without paying for it when it’s not in use – and this article shows you how to do that. I also explore how to estimate the cost savings you can achieve using this solution.

I’m not the first person to think of this idea: credit goes to AWS forum users rickdane for starting the discussion and N. Martin and Troy Volin for posts in this AWS forum thread that describe most of the heavy-lifting – this solution is largely based on their work.

Update December 2009: With the introduction of native support for boot-from-EBS AMIs, the techniques in this article are largely unnecessary. But they’re still pretty cool.

Note: this article assumes you are comfortable in the linux shell and familiar with the EC2 AMI tools and EC2 API tools.

Why Boot EC2 Instances from EBS?

My company runs applications in EC2, and we test new application features in EC2 (after testing them locally) before we deploy them to our production environment. At first, each time we had a new feature to test out, we would construct a testing environment as follows:

  1. Launch a new instance of our base AMI (based on an Alestic Ubuntu Hardy Server AMI, which I highly recommend, and customized with our own basic application server stack).
  2. Make the necessary environmental changes (e.g. adding monitoring services) to the test instance.
  3. Deploy the the application (the one with new features we need to test) to the test instance.
  4. Deploy the test database from S3 to the test instance.
  5. Update the test database with any schema or data changes necessary.
  6. Test and debug the application.
  7. Save the updated test database to S3 for later use.
  8. Terminate the test instance.

This process worked great at first (even if it was a manual process). Our application is a WAR and a bunch of jar files, so they were easily uploaded to the test instance. We stored our test database in S3 as a gzipped MySQL mysqldump file, and it was easy to import into / export from MySQL. That is, until we got to the point where our test database was big enough for it to take a long time – over an hour – to reconstitute. At that point it became very annoying to bring up the test environment.

The last thing you want, as a developer, is a test environment that is difficult or annoying to set up. You want testing to be easy, quick, and inexpensive (otherwise you will start looking for shortcuts to avoid testing, which is not good for quality). Once our test environment began to require almost an hour just to set up, we realized it was no longer serving our needs. I began researching a setup that:

  • Is ready-to-go within a short time (less than five minutes)
  • Contains the most-recent environment (tools, application, and database) already
  • Only costs money when it is being used

Basically, we wanted the benefits of a server that is always-on but without paying for it when we weren’t using it.

That’s the “why”. Read on for the “what” and the “how”.

Ingredients and Tools

The solution consists of two pieces:

  • An AMI that can boot from an EBS drive (the “boot AMI”)
  • An EBS drive that contains the bootable linux stack and, later, anything else you add (the “bootable EBS volume”)

The main tool used to put the pieces together is the linux utility pivot_root. This program can only be run by the /sbin/init process (the startup process itself, pid 1), and it “swaps” the root filesystem out from under the OS and replaces it with a directory you specify. This will be the root directory of the EBS drive.

EBS volumes are attached to specific devices (/dev/sdb through /dev/sdp), and you’ll need to choose what attach device you want the boot AMI to use. We could, theoretically, build the boot AMI to read the attach device name from the user-data provided at launch time. But this would require bringing up networking in order to download the user-data to the instance, which is complicated. Instead of doing this we will hardcode the attach device into the boot AMI. You will need to remember what device you chose in order to know where to attach the EBS drive when you launch an instance of the boot AMI. A good way to remember the chosen attach device is to name the boot AMI accordingly, for example boot-from-EBS-to-dev-sdp-32bit-20090624.

As part of choosing the attach device you’ll need to know the mknod minor number associated with the device. mknod minor numbers begin at 0 (for /dev/sda) and progress in increments of 16 (so /dev/sdb is number 16, /dev/sdc is number 32, etc.) until /dev/sdp (which is number 240). mknod minor numbers are detailed in the Documentation/devices.txt file in the linux kernel source code. In the procedure below I use /dev/sdp (“p” for “pivot_root”), with mknod minor number 240.

The EC2 AMI tools are useful in preparing the bootable EBS drive: they can create an image file from an existing instance. These tools will be used to create a temporary image containing the bootable linux stack copied from the running instance, and this temporary image will then be copied to the EBS drive to make it bootable.

How to Set it Up

Here is an outline of the setup process:

  1. Set up a bootable EBS volume.
  2. Set up the boot AMI.

The detailed setup instructions follow.

Setting up a bootable EBS volume:

  1. Launch an instance of an AMI you like. I use the Alestic Ubuntu Hardy Server AMI.
  2. Create an EBS volume in the same availability zone as the instance you launched. This volume should be large enough to fit everything you plan to put on the root partition (later, not now). It can even exceed 10GB: unlike the “real” root filesystem on EC2 instances which is limited to 10GB, the bootable EBS volume is not limited to this size.
  3. Once the EBS volume is created, attach it to the instance on /dev/sdp.
  4. SSH into the instance as root and format the EBS volume:
    mkfs.ext3 /dev/sdp
  5. Mount the EBS volume to the instance’s filesystem:
    mkdir /ebs && mount /dev/sdp /ebs
  6. Make an image bundle containing the root filesystem:
    mkdir /mnt/prImage
    ec2-bundle-vol -c cert -k key -u user -e /ebs -r i386 -d /mnt/prImage

    These arguments are the same as you would use to bundle an AMI – please see the Developer Guide: Bundling a Unix or Linux AMI for details. The certificate and private key credentials should be copied to the instance in the /mnt partition so they don’t get bundled into the image and copied to the bootable EBS volume. If you’re creating a 64-bit image you’ll need to substitute -r x86_64 instead.
  7. Copy the contents of the image to the EBS drive:
    mount -o loop /mnt/prImage/image /mnt/img-mnt
    rsync -a /mnt/img-mnt/ /ebs/
    umount /mnt/img-mnt
  8. Prepare the EBS volume for being the target of pivot_root. First, edit /ebs/etc/fstab, changing the entry for / (the root filesystem) from /dev/sda1 to /dev/sdp. Next,
    mkdir /ebs/old-root
    rm /ebs/root/.ssh/authorized_keys
    umount /ebs
    rmdir /ebs
  9. Detach the EBS volume from your instance.

Your bootable EBS volume is now ready. You might want to take a snapshot of it. Don’t terminate the EC2 instance yet, because it will be used to create the boot AMI.

Setting up the boot AMI:

  1. Create the mount point where the bootable EBS will be mounted:
    mkdir /new-root
  2. Replace the original /sbin/init with a new one. First, rename the original:
    mv /sbin/init /sbin/init.old
    Then, copy this file into /sbin/init, as follows:
    curl -o /sbin/init -L
    As mentioned above, if you choose an attach device different than /dev/sdp you should edit the new /sbin/init file to assign DEVNO the corresponding mknod minor number.
    Finally, make it executable:
    chmod 755 /sbin/init
  3. Clean up the current SSH public key that was used to login to the instance:
    rm /root/.ssh/authorized_keys
    Not so fast! Once you perform this step you will no longer be able to SSH into the instance. Instead, you can move the SSH authorized_keys file out of the way, as follows:
    mv /root/.ssh/authorized_keys /mnt
  4. Bundle, upload, and register the AMI. Give the bundle a name that indicates the processor architecture and the device on which it expects the bootable EBS volume to be attached. For example, boot-from-EBS-to-dev-sdp-32bit-20090624.
  5. If you opted to move the SSH authorized_keys out of the way in Step 3, restore SSH access to the instance:
    mv /mnt/authorized_keys /root/.ssh/authorized_keys

The boot AMI is now ready to be launched. If you are feeling brave you can terminate the instance. Otherwise, you can leave it running and use it to troubleshoot problems with the launch process – but hopefully this won’t be necessary.

How to Launch an Instance

Once the bootable EBS volume and boot AMI are set up, launch instances that boot from the EBS volume as follows:

  1. Launch an instance of the boot AMI.
  2. Attach the bootable EBS volume to the chosen device (/dev/sdp in the above instructions).

The boot AMI will wait until a volume is attached to the chosen device, pivot_root to the EBS volume, and then continue its boot sequence from the EBS volume.

Some troubleshooting tips:

  • To troubleshoot problems launching the instance you should look at the console output from the instance. The console output might not get updated consistently. If the console output is still empty a few minutes after launching the instance and attaching the EBS volume, try rebooting the instance, which will force the console to refresh.
  • If you need to attach the bootable EBS volume to another instance (not the boot AMI), attach it to /dev/sdp and mount it as follows:
    mkdir /ebs && mount /dev/sdp /ebs
  • The boot AMI includes a hook that allows you to add a program to be executed before it performs a pivot_root to the EBS drive. This avoids the need to re-bundle the boot AMI if you need to change the startup process. The hook looks for a file called / on the EBS drive and executes the file if it can.
  • Booting from an EBS drive seems to take slightly longer than booting a regular EC2 instance. Don’t be surprised if it takes up to five minutes until you can get an SSH connection to the instance.
  • You don’t need to re-attach the EBS volume when you reboot the instance – it stays attached.

Usage Tips

  • Create a file in the root of each bootable EBS volume labeling the purpose of the volume, such as /bootable-32bit-appserver. This helps when you mount the EBS drive to an instance as a “regular” volume: the label indicates what the purpose of the volume is, and helps you distinguish it from other attached EBS drives. This is good practice even for non-bootable EBS volumes. See below for a tip on how to track the purpose of EBS volumes without looking at their contents.
  • Once you get the boot AMI running properly with the bootable EBS volume, shut it down and take a snapshot of the volume before you make any other changes to it. This snapshot is your “master bootable volume” snapshot, containing only the minimum setup necessary to boot.
  • After creating a “master bootable volume” snapshot you can (and should!) launch the boot AMI again (remembering to attach the bootable EBS volume) and customize the instance any way you want: add your application stack, your database, your tools, anything else. These changes will persist on the bootable EBS volume even after the instance has terminated. This is the main motivation for the bootable EBS volume solution!
  • You can create multiple bootable EBS volumes from the “master bootable volume” snapshot and customize them each. See below for a tip on how to keep track of the purpose of each EBS volume.
  • The setup instructions above can be used for creating either a 32-bit or a 64-bit boot AMI and matching bootable EBS volume. Because you can’t run an AMI for one architecture on an EC2 instance of the other architecture, you’ll need to create two separate boot AMIs and two separate bootable EBS volumes if you plan to run on both 32-bit and 64-bit EC2 instances. And, because the bootable EBS volumes created by this procedure will contain a 32-bit or 64-bit specific linux stack, be sure to attach the corresponding bootable EBS volume for the boot AMI you launch.
  • If you work with multiple EBS volumes you will want to identify the purpose of each volume without attaching it to a running instance and looking at the label file you created. Unfortunately the EC2 API does not currently offer a way to tag EBS volumes. But the ElasticFox Firefox extension does – and I highly recommend it for this purpose. Note that the volume tags will only be visible in the browser that creates them, not on other machines. [See my article on Copying ElasticFox tags between browsers for a workaround.]

Cost Implications of Booting Instances from EBS

EBS costs money beyond what you pay for the disk drives that come as part of EC2 instances. There are four components of the EBS cost to consider:

  1. Allocated storage size ($0.10 per GB per month)
  2. I/O requests ($0.10 per million I/O requests)
  3. Snapshot storage (same as S3 storage costs; depends on the region)
  4. Snapshot transfer (same as S3 transfer costs; depends on the region)

The AWS Simple Monthly Calculator can help you estimate these costs. Here are some guidelines to help you figure out what numbers to put into these fields:

    • Component #1 is easy: just plug in the size of your bootable EBS volume.
    • Component #2 should be estimated based on the I/O usage of your existing application instance. You can use iostat to estimate this number as follows:
      iostat | awk -F" " 'BEGIN {x="0.0"} /^sd/ {x=x+$3} END {print x}'
      The result is the number of I/O transactions per second. Multiply this figure by 2592000 (60 * 60 * 24 * 30, the number of seconds in a month) to get the number of I/O transactions per month, then divide by 1 million. Or, better yet, use this instead:
      iostat | awk -F" " 'BEGIN {x="0.0"} /^sd/ {x=x+$3} END {printf "%12.2f\n", x*2.592}'
      For my test environment, this figure comes to 29 (million I/O requests per month).
    • Component #3 can be estimated with the help of the guidelines at the bottom of the EBS Product Info page:

Snapshot storage is based on the amount of space your data consumes in Amazon S3. Because data is compressed before being saved to Amazon S3, and Amazon EBS does not save empty blocks, it is likely that the size of a snapshot will be considerably less than the size of your volume. For the first snapshot of a volume, Amazon EBS will save a full copy of your data to Amazon S3. However for each incremental snapshot, only the part of your Amazon EBS volume that has been changed will be saved to Amazon S3.

As a conservative estimate of snapshot storage size, I use the same size as the actual data on the EBS volume. I figure this covers the initial compressed snapshot plus a month’s worth of delta snapshots.

    • Component #4 can also be estimated from the guidelines further down the same page:

Volume data is broken up into chunks before being transferred to Amazon S3. While the size of the chunks could change through future optimizations, the number of PUTs required to save a particular snapshot to Amazon S3 can be estimated by dividing the size of the data that has changed since the last snapshot by 4MB. Conversely, when loading a snapshot from Amazon S3 into and Amazon EBS volume, the number of GET requests needed to fully load the volume can be estimated by dividing the full size of the snapshot by 4MB. You will also be charged for GETs and PUTs at normal Amazon S3 rates.

My own monthly estimate includes taking one full EBS volume snapshot and creating one EBS volume from a snapshot. I keep around 20GB of data on the EBS volume. So I divide 20480 (20 GB expressed in MB) by 4 to get 5120. This is the estimated number of PUTs per month. It is also the estimated number of GETs per month.

For my usage in our test environment, the cost of running instances that boot from a 50GB EBS volume with 20GB of data on it in the US region comes to approximately $10.96 per month.

Is it financially worthwhile?

As long as your EBS cost estimate is less than the cost of running the instance for the hours it would sit unused, this solution saves you money. My test instance is an m1.large in the US region, which costs $0.40 per hour. If my instance would sit idle for at least (10.96/0.4=) 28 hours a month, I save money by using a bootable EBS volume and terminating the instance when it is unused. As it happens, my test instance is unused more than 360 hours a month, so for me it is a no-brainer: I use bootable EBS volumes.

EC2 instances that boot from an EBS volume offer the benefits of an always-on machine without the costs of paying for it when it is not in use. The implementation presented here was developed based on posts by the aforementioned forum users, and I thank them for their help.

Cloud Developer Tips

The “Elastic” in “Elastic Load Balancing”: ELB Elasticity and How to Test it

Update March 2012: Amazon published their own official guide to ELB’s architecture and building and testing applications that use it. The official material is very consistent with the presentation offered here, more than two and a half years prior.

Elastic Load Balancing is a long-anticipated AWS feature that aims to ease the deployment of highly-scalable web applications. Let’s take a look at how it achieves elasticity, based on experience and based on the information available in the AWS forums (mainly this thread). The goal is to understand how to design and test ELB deployments properly.

ELB: Two Levels of Elasticity

ELB is a distributed system. An ELB virtual appliance does not have a single public IP address. Instead, when you create an ELB appliance, you are given a DNS name such as Amazon encourages you to set up a DNS CNAME entry pointing (say) to the ELB-supplied DNS name.

Why does Amazon use a DNS name? Why not provide an IP address? Why encourage using a CNAME? Why can’t we use an Elastic IP for an ELB? In order to understand these issues, let’s take a look at what happens when clients interact with the ELB.

Here is the step-by-step flow of what happens when a client requests a URL served by your application:

  1. The client looks in DNS for the resolution of your web server’s name, Because you have set up your DNS to have a CNAME alias pointing to the ELB name, DNS responds with the name
  2. The client looks in DNS for the resolution of the name This DNS entry is controlled by Amazon since it is under the domain. Amazon’s DNS server returns an IP address, say
  3. The client opens a connection with the machine at the provided IP address The machine at this address is really an ELB virtual appliance.
  4. The ELB virtual appliance at address passes through the communications from the client to one of the EC2 instances in the load balancing pool. At this point the client is communicating with one of your EC2 application instances.

As you can see, there are two levels of elasticity in the above protocol. The first scalable point is in Step 2, when Amazon’s DNS resolves the ELB name to an actual IP address. In this step, Amazon can vary the actual IP addresses served to clients in order to distribute traffic among multiple ELB machines. The second point of scalability is in Step 4, where the ELB machine actually passes the client communications through to one of the EC2 instances in the ELB pool. By varying the size of this pool you can control the scalability of the application.

Both levels of scalability, Step 2 and Step 4, are necessary in order to load-balance very high traffic loads. The Step 4 scalability allows your application to exceed the maximum connections per second capacity of a single EC2 instance: connections are distributed among a pool of application instances, each instance handling only some of the connections. Step 2 scalability allows the application to exceed the maximum inbound network traffic capacity of a single network connection: an ELB machine’s network connection can only handle a certain rate of inbound network traffic. Step 2 allows the network traffic from all clients to be distributed among a pool of ELB machines, each appliance handling only some fraction of the network traffic.

If you only had Step 4 scalability (which is what you have if you run your own software load balancer on an EC2 instance) then the maximum capacity of your application is still limited by the inbound network traffic capacity of the front-end load balancer: no matter how many back-end application serving instances you add to the pool, the front-end load balancer will still present a network bottleneck. This bottleneck is eliminated by the addition of Step 2: the ability to use more than one load balancer’s inbound network connection.

[By the way, Step 2 can be replicated to a limited degree by using Round-Robin DNS to serve a pool of IP addresses, each of which is a load balancer. With such a setup you could have multiple load-balanced clusters of EC2 instances, each cluster sitting behind its own software load balancer on an EC2 instance. But Round Robin DNS has its own limitations (such as the inability to take into account the load on each load-balanced unit, and the difficulty of dynamically adjusting the pool of IP addresses to distribute), from which ELB does not suffer.]

Behind the scenes of Step 2, Amazon maps an ELB DNS name to a pool of IP addresses. Initially, this pool is small (see below for more details on the size of the ELB IP address pool). As the traffic to the application and to the ELB IP addresses in the pool increases, Amazon adds more IP addresses to the pool. Remember, these are IP addresses of ELB machines, not your application instances. This is why Amazon wants you to use a CNAME alias for the front-end of the ELB appliance: Amazon can vary the ELB IP address served in response to the DNS lookup of the ELB DNS name.

It is technically possible to implement an equivalent Step 2 scalabililty feature without relying on DNS CNAMEs to provide “delayed binding” to an ELB IP address. However, doing so requires implementing many features that DNS already provides, such as cached lookups and backup lookup servers. I expect that Amazon will implement something along these lines when it removes the limitation that ELBs must use CNAMEs – to allow, for example, an Elastic IP to be associated with an ELB. Now that would be cool.

How ELB Distributes Traffic

As explained above, ELB uses two pools: the pool of IP addresses of ELB virtual appliances (to which the ELB DNS name resolves), and the pool of your application instances (to which the ELBs pass through the client connection). How is traffic distributed among these pools?

ELB IP Address Distribution

The pool of ELB IP addresses initially contains one IP address. More precisely, this pool initially consists of one ELB machine per availability zone that your ELB is configured to serve. This can be inferred from this page in the ELB documentation, which states that “Incoming traffic is load balanced equally across all Availability Zones enabled for your load balancer”. I posit that this behavior is implemented by having each ELB machine serve the EC2 instances within a single availability zone. Then, multiple availability zones are supported by the ELB having in its pool at least one ELB machine per enabled availability zone. Update April 2010: An availability zone that has no healthy instances will not receive any traffic. Previously, the traffic would be evenly divided across all enables availability zones regardless of instance health, possibly resulting in 404 errors. Recent upgrades to ELB no longer exhibit this weakness.

Back to the scalability of the ELB machine pool. According to AWS folks in the forums, this pool is grown in response to increased traffic reaching the ELB IP addresses already in the pool. No precise numbers are provided, but a stream of gradually-increasing traffic over the course of a few hours should cause ELB to grow the pool of IP addresses behind the ELB DNS name. ELB grows the pool proactively in order to stay ahead of increasing traffic loads.

How does ELB decide which IP address to serve to a given client? ELB varies the chosen address “from time to time”. No more specifics are given. However, see below for more information on making sure you are actually using the full pool of available ELB IP addresses when you test ELB.

Back-End Instance Connection Distribution

Each ELB machine can pass through client connections to any of the EC2 instances in the ELB pool within a single availability zone. According to user reports in other forum posts, clients from a single IP address will tend to be connected to the same back-end instance.  According to AWS ELB does round-robin among the least-busy back-end instances, keeping track of approximately how many connections (or requests) are active at each instance but without monitoring CPU or anything else on the instances. It is likely that earlier versions of ELB exhibited some stickiness, but today the only way to ensure stickiness is to use the Sticky Sessions feature.

How much variety is necessary in order to cause the connections to be fairly distributed among your back-end instances? AWS says that “a dozen clients per configured availability zone should more than do the trick”. Additionally, in order for the full range of ELB machine IP addresses to be utilized, “make sure each [client] refreshes their DNS resolution results every few minutes.”

How to Test ELB

Let’s synthesize all the above into guidelines for testing ELB:

  1. Test clients should use the ELB DNS name, ideally via a CNAME alias in your domain. Make sure to perform a DNS lookup for the ELB DNS name every few minutes. If you are using Sun’s Java VM you will need to change the system property to be different than the default value of -1, which causes DNS lookups to be cached until the JVM exits. 120 (seconds) is a good value for this property for ELB test clients. If you’re using IE 7 clients, which cache DNS lookups for 30 minutes by default, see the Microsoft-provided workaround to set that cache timeout to a much lower value.
    Update 14 August 2008: If your test clients cache DNS lookups (or use a DNS provider that does this) beyond the defined TTL, traffic may be misdirected during ramp-up or ramp-down. See my article for a detailed explanation.
  2. One test client equals one public IP address. ELB machines seem to route all traffic from a single IP address to the same back-end instance, so if you run more than one test client process behind a single public IP address, ELB regards these as a single client.
  3. Use 12 test clients for every availability zone you have enabled on the ELB. These test clients do not need to be in different availability zones – they do not even need to be in EC2 (although it is quite attractive to use EC2 instances for test clients). If you have configured your ELB to balance among two availability zones then you should use 24 test clients.
  4. Each test client should gradually ramp up its load over the course of a few hours. Each client can begin at (for example) one connection per second, and increase its rate of connections-per-second every X minutes until the load reaches your desired target after a few hours.
  5. The mix of requests and connections that each test client performs should represent a real-world traffic profile for your application. Paul@AWS recommends the following:

    In general, I’d suggest that you collect historical data or estimate a traffic profile and how it progresses over a typical (or perhaps extreme, but still real-world) day for your scenario. Then, add enough headroom to the numbers to make you feel comfortable and confident that, if ELB can handle the test gracefully, it can handle your actual workload.

    The elements of your traffic profile that you may want to consider in your test strategy may include, for example:

    • connections / second
    • concurrent connections
    • requests / second
    • concurrent requests
    • mix of request sizes
    • mix of response sizes

Use the above guidelines as a starting point to setting up a testing environment that exercises you application behind an ELB. This testing environment should validate that the ELB, and your application instances, can handle the desired load.

A thorough understanding of how ELB works can go a long way to helping you make the best use of ELB in your architecture and toward properly testing your ELB deployment. I hope this article helps you design and test your ELB deployments.

Cloud Developer Tips

Sending Email from EC2

A common question about moving applications to EC2 is what to do about sending email. Web applications occasionally send email, to their administrators (“Help! I’m running out of disk space!”) or to their users (“Here is your forgotten password”), and this needs to work when the application is hosted within EC2 also. Unfortunately, it is not simple to send email reliably from within the cloud. Here are some tips about sending reliable email from EC2 instances.

What is the Problem?

One word: spam. Email sent from servers within EC2 is marked as spam by most ISPs and email providers. The cloud offers a very scalable way for anyone to launch email-sending machines, on-demand. Spammers know this. So do the anti-spammers. Luckily (for the anti-spammers), existing anti-spam safeguards are effective against attempts to send spam from within EC2. Unluckily (for you), these same anti-spam measures make your legitimate (I hope!) emails get flagged as spam.

The anti-spam safeguard that trips up your application’s emails is called reverse DNS lookup. This lookup checks that the name and IP address of the mailserver sending the message match the name and IP address specified in DNS records. Here’s how it works:

  1. A machine at receives a message from your mail server, whose IP address is (let’s say)
  2. looks in DNS for a PTR record for the IP address If none exists, the mail is not delivered. Otherwise, the PTR record specifies the FQDN of the machine that should have the address Let’s say this is specified as
  3. looks in DNS for the IP address of the host specified by the PTR lookup: If the IP address does not match the address from which the mail was received ( in our case), the message is not delivered.

The above is a bit of a simplification, but it is enough to explain the difficulty in sending email from within EC2. The problem lies in step 2, when the receiving mail server tries to find the PTR record for the IP address from which the mail was sent. The public IP addresses used by EC2 belong to Amazon, and the DNS lookups for these addresses will therefore look in Amazon’s DNS records. Amazon does not have PTR records pointing to your mailserver’s FQDN in their DNS, so the reverse DNS lookup fails at step 2, and your email message is not delivered.

Amazon, unlike traditional hosting companies, does not have a service that allows you to set a reverse DNS entry (a PTR record) within their DNS records. If they did offer such a service it would solve the reverse DNS lookup problem, and enable your EC2 application to send emails. Of course, such a service would also allow spammers to easily use EC2 for sending spam, so it is clear that Amazon cannot offer a service to allow custom PTR records without instituting some effective safeguards against abuse.

Update 30 October 2009: Amazon now has a new email policy in which outbound SMTP traffic is blocked (beyond miniscule usage). In order to be able to send email directly from EC2 you also need to provision an Elastic IP address for your instance and submit the following form. In return, Amazon will work to keep that Elastic IP of of the common anti-spam lists. More information here. Note that the comments in this article relating to reverse DNS are still applicable: some email providers do not check the PTR records, and Amazon’s new program will help email get through to those providers – but not all.

Update 25 January 2010: Amazon has announced a new private beta where they will set PTR records for your Elastic IP address. To participate in the private beta, join the EC2 developer forums and send a private message to ian@aws.

Update 23 March 2010: Reverse DNS for Elastic IPs is now officially supported!

Sending Email from EC2: What Works

In any case, let us explore how you can send email from within EC2. Here are a number of cases that work:

  • One case that works is sending email to email accounts that are hosted on the same EC2 instance, using sendmail as the mail transfer agent (MTA). This can be appropriate for a corporate setting, where your recipients can be told to configure their email readers accordingly. To do this, set up sendmail on your EC2 instance, then set up a POP3 or IMAP server on the same instance, and then set up your users’ email readers to fetch email from those accounts. Be sure to configure the sendmail and POP/IMAP services to accept connections only from machines you trust. (You’ll probably want to use an Elastic IP and an EBS drive also, to ensure that access to those email boxes can be recovered if the instance goes down.)This case does not have the reverse DNS lookup problem because sendmail will deliver the messages to the account on the local machine directly, bypassing the reverse DNS lookup.One limitation of this approach is that the users must learn to check this “special” email account. Some email readers make this easy by allowing more than one server-side account to be checked.

    A more significant limitation to this approach is that it will not work for sending mail to recipients who have accounts elsewhere. The reverse DNS lookup anti-spam measure described above will prevent those messages from getting through.

  • Sending email from EC2 to an address whose email delivery you control can work also. The application sends email from a known email account (say, “”) to your application administrators’ email accounts in your domain. In this approach you set up an SMTP server on your EC2 instance. You configure the SMTP installation on EC2 to relay mail directly to your domain’s mail servers. And you need to make sure that the anti-spam tools running on your domain’s mail servers (and in the user’s email reader) are configured to whitelist messages from the known email account.
  • You can use third-party SMTP services to send emails from your EC2 instance. There are many third-party email services available, including the free-but-limited Google Apps Email, and also including paid-but-less-limited solutions. A detailed discussion of how to set up your email to use a third-party SMTP server is beyond the scope of this article (but here’s a good one). However, a useful strategy is to run an SMTP server locally on your EC2 instance which is configured to relay the mail to the third-party server. This relaying setup allows your application to benefit from reduced connection time when sending messages, and relays the messages in a separate process.There are many third-party SMTP service providers available, each with their own pricing structure and Terms & Conditions.

    Google Apps Email’s limitations are an interesting subject, and I plan another blog article on strategies for dealing with these limitations.

The above three methods of sending email from your EC2 instances each have their pros and cons. If you’re only delivering messages to internal recipients, use the first or the second method. If you are also sending mails to arbitrary addresses, go for a third-party SMTP provider.

Cloud Developer Tips

EC2 Reserved Instance Billing Gotcha

So you purchased an EC2 Reserved Instance and you’re looking to see the reduced hourly rate take effect immediately.


Unlike all the other resources in EC2, which you can use immediately, reserved instances are not actually available until the credit card charge for your account is processed.

You’ll need to wait until the next billing period begins the charge clears your credit card before the reserved instance is actually available and the discounted hourly rate takes effect. Until your credit card charge goes through, your instances will continue to be billed at the regular hourly rates.

It’s a classic gotcha, but it’s in the EC2 FAQ.

Update: According to AWS folks, the charge for the reservation is posted to your credit card soon after you order the reserved instance, instead of waiting for the next billing period. Here is the AWS Forum thread where AWS says so.

Cloud Developer Tips

Tagging EC2 Instances Using Security Groups

Update September 2010: AWS has added support for tagging instances, volumes, snapshots, and many other EC2 resource types via the API and the AWS Management Console. The techniques described here are no longer necessary.

So you have a bunch of instances running in Amazon EC2 and you want to be able to tell them apart easily. The way to do this is to “tag” each instance with a descriptive tag, such as “database”, “web server”, “memcached”, etc. The easy, human-friendly way to tag your instances is to use ElasticFox, and the robust, programming-friendly way is to use Security Groups.

Tagging Instances in ElasticFox

The ElasticFox Firefox extension is a great UI for managing your EC2 resources. ElasticFox supports adding tags for instances (as well as tagging EBS volumes, AMIs, and Elastic IPs). These tags are stored in internal Firefox preferences.

Let’s tag an instance in ElasticFox:

1. Here is the ElasticFox extension. Note how it shows that I am running one EC2 instance.
ElasticFox showing one running EC2 instance. Note the Tag column, which is empty for the running instance. Notice the “Tag” column, which is empty.

2. Right-click on the entry for the instance we want to tag:
Right-click on the EC2 instance you want to tag and choose Add Tag. Now, add the tag in the dialog that pops up:
Add the tag in the ElasticFox dialog that pops up. And the result:
ElasticFox showing the tagged instance.
Tagging instances in ElasticFox is great if you are the only person managing EC2 instances, and if you only use ElasticFox from a single machine. Because the tags are stored in your local browser, they are only visible to you. But, if you need your tags to be visible to other people, or on multiple browsers/computers, tagging in ElasticFox will not help because the tags are not stored in the cloud, only in the local browser. Other browsers / users / computers will not see the tags that you supply using ElasticFox. And, if you need programmatic access (via the EC2 API or command-line tools) to the tags, ElasticFox tags will not help.

Update 15 July 2009: Check out my article about how to copy ElasticFox settings (including tags) between browsers.

Tagging Instances Using Security Groups

Amazon Security Groups can also be used as a tagging mechanism for EC2 instances. The basic idea is to create a custom security group named, for example, “#database” or “#web server”, and launch the instance(s) in that custom security group in addition to the other security groups you want. The custom security group is defined without any permissions so it will have no effect on any instances in the group. This works without affecting the “real” security because the effective security permissions for an instance are the union (the sum) of all the permissions on the security groups the instance belongs to, and the “tag” security group grants zero permissions: x + 0 = x.

Using security groups to tag instances is slightly more cumbersome because you must create the custom-named security group in a separate step before launching instances in it. However, tagging via security groups is a more robust solution for the following reasons:

  • Security groups are stored inside the cloud. When you add an instance to a security group, you do not need to “remember” anything on the client-side: you can change browsers or change computers and your assigned security groups will still be visible every time you look.
  • Security groups are visible via the API and the command-line tools. This mean you can write code, in bash/PHP/Java/python/etc., that reads and manipulates the instance “tags” (really security groups). In fact, if you choose a convention for naming your security groups that are really tags, you can programmatically distinguish between the “real” security group (such as “default”) and the “tag” security group. I use the prefix “# ” for naming the security groups that function as tags.

However, all this extra flexibility comes with a price: you cannot add an instance to a security group after it is running; you can only specify an instance’s security groups at launch time. If you use security groups to tag instances, your must specify all your security group tags when you launch your instance(s).

You can use ElasticFox to create tag security groups and to specify the security groups when you launch instances. There is one gotcha: when creating the tag security group, make sure you choose “I will authorize protocols for this group as needed” – this creates the group without adding any permissions.

Instead of walking through that process in ElasticFox, here is an example of using the command-line tools to manage tag security groups.

Managing Tag Security Groups from the Command-Line

To create a security group on the command-line, use the ec2-add-group command:

ec2-add-group "#web server" -d "tag web server"


GROUP #web server tag web server

This creates a security group called #web server with the description tag web server.

To launch an instance with this tag, use the ec2-run-instances command:

ec2-run-instances ami-12345678 --instance-type m1.small --key gsg-keypair --group default --group "#web server"


RESERVATION r-27e29b4e 614123456789 #web server,default
INSTANCE i-031b376a ami-12345678 pending gsg-keypair 0 m1.small 2009-06-30T11:36:30+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc monitoring-disabled

This launches a single instance of ami-12345678, in any availability zone in the US region (I got us-east-1c), using the gsg-keypair, and having the security groups default and #web server. In practice, you will substitute the AMI id you want to use and the name of your keypair. You may also want to add other arguments, so check out the docs or try ec2-run-instances --help for more command-line options. You can see in the output above that the security groups are default and #web server.

Once the instance starts running, you can use the ec2-describe-instances command to see all your running instances and the security groups they belong to:



RESERVATION r-27e29b4e 614123456789 #web server,default
INSTANCE i-031b376a ami-12345678 ip-10-250-18-244.ec2.internal running gsg-keypair 0 m1.small 2009-06-30T11:36:30+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc monitoring-disabled

Note the #web server security group in the output. This is the tag security group for this instance.

You can parse the output of the above command using shell utilities (awk, perl, cut, etc.) if you want to programmatically discover what instances belong to which groups.

Update 24 September 2009: AWS now supports the ability to add a long description to EBS snapshots.

Cloud Developer Tips

Welcome to Cloud Developer Tips

Here you will find practical tips for developers who are using cloud computing.

This is me on the Amazon Web Services EC2 support forum, where I’m a frequent poster answering questions and sharing practical advice based on my experience. My nickname there used to be “shl0m0”; you might see that reference in older posts.

My experience includes more than 15 years of software development and software architecture. I’ve been building cloud-computing-based applications for clients (including my own company MyDrifts, where I am CTO and co-founder) since 2007.