Tag: ami

Categories
Cloud Developer Tips

Creating Consistent Snapshots of a Live Instance with XFS on a Boot-from-EBS AMI

Eric Hammond has taught us how to create consistent snapshots of EBS volumes. Amazon has allowed us to use EBS snapshots as AMIs, providing a persistent root filesystem. Wouldn’t it be great if you could use both of these techniques together, to take a consistent snapshot of the root filesystem without stopping the instance? Read on for my instructions how to create an XFS-formatted boot-from-EBS AMI, allowing consistent live snapshots of the root filesystem to be created.

The technique presented below owes its success to the Canonical Ubuntu team, who created a kernel image that already contains XFS support. That’s why these instructions use the official Canonical Ubuntu 9.10 Karmic Koala AMI – because it has XFS support built in. There may be other AKIs out there with XFS support built in – if so, the technique should work with them, too.

How to Do It

The general steps are as follows:

  1. Run an instance and set it up the way you like.
  2. Create an XFS-formatted EBS volume.
  3. Copy the contents of the instance’s root filesystem to the EBS volume.
  4. Unmount the EBS volume, snapshot it, and register it as an AMI.
  5. Launch an instance of the new AMI.

More details on each of these steps follows.

1. Run an instance and set it up the way you like.

As mentioned above, I use the official Canonical Ubuntu 9.10 Karmic Koala AMI (currently ami-1515f67c for 32-bit architecture – see the table on Alestic.com for the most current Ubuntu AMI IDs).

ami=ami-1515f67c
security_groups=default
keypair=my-keypair
instance_type=m1.small
ec2-run-instances $ami -t $instance_type -g $security_groups -k $keypair

Wait until the ec2-describe-instances command shows the instance is running and then ssh into it:

ssh -i my-keypair ubuntu@ec2-1-2-3-4.amazonaws.com

Now that you’re in, set up the instance’s root filesystem the way you want. Don’t forget that you probably want to run

sudo apt-get update

to allow you to pull in the latest packages.

In our case we’ll want to install ec2-consistent-snapshot, as per Eric Hammond’s article:

codename=$(lsb_release -cs)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu $codename main" | sudo tee /etc/apt/sources.list.d/alestic-ppa.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
sudo apt-get update
sudo apt-get install -y ec2-consistent-snapshot
sudo PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2

2. Create an XFS-formatted EBS volume.

First, install the XFS tools:

sudo apt-get install -y xfsprogs

These utilities allow you to format filesystems using XFS and to freeze and unfreeze the XFS filesystem. They are not necessary in order to read from XFS filesystems, but we want these programs installed on the AMI we create because they are used in the process of creating a consistent snapshot.

Next, create an EBS volume in the availability zone your instance is running in. I use a 10GB volume, but you can use any size and grow it later using this technique. This command is run on your local machine:

ec2-create-volume --size 10 -z $zone

Wait until the ec2-describe-volumes command shows the volume is available and then attach it to the instance:

ec2-attach-volume $volume --instance $instance --device /dev/sdh

Back on the instance, format the volume with XFS:

sudo mkfs.xfs /dev/sdh
sudo mkdir -m 000 /vol
sudo mount -t xfs /dev/sdh /vol

Now you should have an XFS-formatted EBS volume, ready for you to copy the contents of the instance’s root filesystem.

3. Copy the contents of the instance’s root filesystem to the EBS volume.

Here’s the command to copy over the entire root filesystem, preserving soft-links, onto the mounted EBS volume – but ignoring the volume itself:

sudo rsync -avx --exclude /vol / /vol

My command reports that it copied about 444 MB to the EBS volume.

4. Unmount the EBS volume, snapshot it, and register it as an AMI.

You’re ready to create the AMI. On the instance do this:

sudo umount /vol

Now, back on your local machine, create the snapshot:

ec2-create-snapshot $volume

Once ec2-describe-snapshots shows the snapshot is 100% complete, you can register it as an AMI. The AKI and ARI values used here should match the AKI and ARI that the instance is running – in this case, they are the default Canonical AKI and ARI for this AMI. Note that I give a descriptive “name” and “description” for the new AMI – this will make your life easier as the number of AMIs you create grows. Another note: some AMIs (such as the Ubuntu 10.04 Lucid AMIs) do not have a ramdisk, so skip the --ramdisk $ramdisk arguments if you’ve used such an AMI.

kernel=aki-5f15f636
ramdisk=ari-0915f660
description="Ubuntu 9.10 Karmic formatted with XFS"
ami_name=ubuntu-9.10-32-bit-ami-1515f67c-xfs
ec2-register --snapshot $snapshot --kernel $kernel --ramdisk $ramdisk '--description=$description' --name=$ami_name --architecture i386 --root-device-name /dev/sda1 --block-device-mapping /dev/sda2=ephemeral0

This displays the newly registered AMI ID – let’s say it’s ami-00000000.

5. Launch an instance of the new AMI.

Here comes the moment of truth. Launch an instance of the newly registered AMI:

ami=ami-00000000
security_groups=default
keypair=my-keypair
instance_type=m1.small
ec2-run-instances $ami -t $instance_type -g $security_groups -k $keypair

Again, wait until ec2-describe-instances shows it is running and ssh into it:

ssh -i my-keypair ubuntu@ec2-5-6-7-8.amazonaws.com

Now, on the instance, you should be able to see that the root filesystem is XFS with the mount command. The output should contain:

/dev/sda1 on / type xfs (rw)
...

We did it! Let’s create a consistent snapshot of the root filesystem. Look back into the output of ec2-describe-instances to determine the volume ID of the root volume for the instance.

sudo ec2-consistent-snapshot --aws-access-key-id $aws_access_key_id --aws-secret-access-key $aws_secret_access_key --xfs-filesystem / $volumeID

The command should display the snapshot ID of the snapshot that was created.

Using ec2-consistent-snapshot and an XFS-formatted EBS AMI, you can create snapshots of the running instance without stopping it. Please comment below if you find this helpful, or with any other feedback.

Categories
Cloud Developer Tips

Copying ElasticFox Tags from One Browser to Another

The ElasticFox Firefox extension allows you to tag EC2 instances, EBS volumes, EBS snapshots, Elastic IPs, and AMIs. ElasticFox’s tags are stored locally within Firefox, so if you use ElasticFox from more than one browser your tags from one browser are not visible in any other browser. In this article I demonstrate how to copy tags from one ElasticFox installation to another.

Where Are ElasticFox Tags Stored?

My article about Tagging EC2 Instances Using Security Groups shows a brief walkthrough of using ElasticFox to tag instances. Under the hood, ElasticFox stores tags in the prefs.js file within your Firefox Profile directory. But instead of poking around in there directly, take a look at Firefox’s built-in interface to these preferences. It’s the about:config screen, and you get there by typing about:config into the address bar:

The Firefox about:config screen

Click “I’ll be careful, I promise!” Up comes the about:config screen, with all sorts of interesting-looking settings. We’re interested in the ElasticFox tags only, so type ec2ui.*tag in the Filter field, and the result looks like this:

Viola! The ElasticFox tags are stored here.

Viola! Those are the ElasticFox tags that I’ve set up.

Copying ElasticFox Tags to Another Browser Manually

You can copy ElasticFox tags from the about:config screen, via the clipboard, into a text file. First, open the about:config screen and use the Filter as described above to see the relevant settings. Create an empty text file and copy each setting (right-click on the entry and choose “Copy”) and paste it into the text file. Add each setting you want to transfer into the text file. You can email the text file to yourself (or put it on your USB key) and then use the text file to help you change the values of these settings in the target browser’s about:config screen.

Once copied setting looks like this:

ec2ui.eiptags.MyDrifts.us-east-1;({'75.101.15.8':'mydrifts.com'})

The setting’s name is ec2ui.eiptags.MyDrifts.us-east-1 and its value is ({'75.101.15.8':'mydrifts.com'}), separated from the name by a semicolon (;). For each setting you want to transfer to the target browser, select the value (after the semicolon) from the text file and copy it to the clipboard. Then, in the target browser’s about:config screen, right-click on the corresponding setting entry and choose “Modify”, and paste in the new value.

All that selecting and copying and pasting is annoying, and you might make a mistake when entering the values in the target browser. I feel your pain. There must be a better way.

Copying ElasticFox Tags to Another Browser with OPIE

OPIE is a Firefox extension that allows you to import and export preference settings. The good thing about it is that it can be used to export only some settings – and we only want to export the ElasticFox settings. Unfortunately, it does not support ElasticFox!

Please help lobby the OPIE developer to add ElasticFox support by posting in this forum thread – OPIE support would make copying ElasticFox tags (and indeed, entire ElasticFox setups) between browsers much easier.

Copying ElasticFox Tags to Another Browser with Shell Scripts

I’ve put together two shell scripts that can be used to directly export ElasticFox settings from the Firefox prefs.js file and to directly import them back in. These scripts really export and import all your ElasticFox settings, not just tags. Read on for directions on how to use these scripts, which work on Linux, Mac OS X, and Solaris, for Firefox 3.x and above.

A word of caution before you export and import via prefs.js directly: make sure Firefox is not running when you export or import. Firefox overwrites the prefs.js file upon exiting, and reads it upon starting up. In order to make sure all your ElasticFox tags (and all other settings, too) are in the export, Firefox must be closed before exporting. Likewise, in order to make sure Firefox picks up your imported settings you need to make sure it is closed before importing.

How to Export ElasticFox Settings

Here is a handy script to export ElasticFox preferences from Firefox. You can download it, or copy it from here and save it to your local machine:

#! /bin/bash

# the base name of the extension preferences
# that are to be imported
PREF_BASE=ec2ui
# ec2ui is the extension base name of ElasticFox

# backslash-escape special characters as you
# would in a regular expression
# the following command will only escape
# periods (".") but that should suffice
# for firefox extensions
ESCAPED_PREF_BASE=$(echo $PREF_BASE | sed -e "s/\./\\\\./g")

# export desired prefs
egrep "^user_pref\(\"$ESCAPED_PREF_BASE\." prefs.js

Once you save the script locally, make it executable:

chmod +x exportFirefoxPrefs.sh

The script expects to be run in the same directory as the prefs.js file, but it’s most convenient to keep this script in your home directory and create a soft-link there to the prefs.js file. On my machine, the command to make a soft-link to the prefs.js file is this:

ln -s /Users/shlomo/Library/Application\ Support/Firefox/Profiles/f0psntn6.default/prefs.js prefs.js

You’ll need to specify the location of your Firefox profile directory (instead of mine).

Remember to exit Firefox. Now you’re ready to run the export script.

./exportFirefoxPrefs.sh > savedPrefs

This will create a file called savedPrefs containing the ElasticFox settings. Move this file to the target browser’s machine, where we will import it.

How to Import ElasticFox Settings

Here is a script to import to import ElasticFox settings into Firefox. Download it, or copy it from here and save it to your local machine:

#! /bin/bash

WORKFILE=workfile.txt
OUTFILE=newPrefs.js
XFERFILE="$1"

# the base name of the extension preferences
# that are to be imported
PREF_BASE=ec2ui
# ec2ui is the extension base name of ElasticFox

if [ -z "$1" ]
then
  echo "Please provide the filename of the file containing the prefs to import."
else
# backslash-escape any  characters as you would
# in a regular expression
# this command only handles periods
# but that should suffice for firefox extensions
ESCAPED_PREF_BASE=$(echo $PREF_BASE | sed -e "s/\./\\\\./g")

# backup the existing prefs
cp prefs.js prefs.js.old

# get the file header
egrep -v "^user_pref\(" prefs.js > $OUTFILE

# add all other prefs
egrep "^user_pref\(" prefs.js | egrep -v "^user_pref\(\"$ESCAPED_PREF_BASE\." > $WORKFILE

# add desired prefs
cat $XFERFILE >> $WORKFILE

# sort all prefs into output
sort $WORKFILE >> $OUTFILE

# clean up
rm $WORKFILE

# uncomment this when you trust this script
# to replace your prefs automatically
#cp $OUTFILE prefs.js
#rm $OUTFILE
fi

After you save the script locally, make it executable:

chmod +x importFirefoxPrefs.sh

As with the export script above, the import script expects to be run in the same directory as the Firefox prefs.js file, so make a soft-link to the prefs.js file in the same directory as the script (see above).

Once you’ve made sure that Firefox is not running, you’re ready to run the import script:

./importFirefoxPrefs.sh savedPrefs

This will create a file called newPrefs.js in the same directory. newPrefs.js will contain the imported ElasticFox settings from the savedPrefs file, merged with the other already-existing Firefox settings from prefs.js. The script will also create a backup copy of prefs.js called prefs.js.old. All you need to do is to replace your Firefox’s prefs.js with newPrefs.js, as follows:

cp newPrefs.js prefs.js
rm newPrefs.js

Now, restart Firefox. It should contain the ElasticFox settings from the source browser. If anything doesn’t work, simply exit Firefox and restore the previous prefs.js:

cp prefs.js.old prefs.js

Once you are confident that the script works consistently, you can uncomment the last few lines of the script and let it automatically replace Firefox’s prefs.js file by itself – then you do not need to perform that step manually.

This method of copying ElasticFox settings using shell scripts is relatively easy, but not perfect. OPIE would be a friendlier way, and would support Windows as well (so please lobby the developer to support ElasticFox). If you’re looking for your ElasticFox tags to be available via the EC2 APIs, check out my article on using security groups as a way to tag instances (but unfortunately not Elastic IPs, EBS volumes and snapshots, or AMIs).