≡ Menu

Creating Consistent Snapshots of a Live Instance with XFS on a Boot-from-EBS AMI

Eric Hammond has taught us how to create consistent snapshots of EBS volumes. Amazon has allowed us to use EBS snapshots as AMIs, providing a persistent root filesystem. Wouldn’t it be great if you could use both of these techniques together, to take a consistent snapshot of the root filesystem without stopping the instance? Read on for my instructions how to create an XFS-formatted boot-from-EBS AMI, allowing consistent live snapshots of the root filesystem to be created.

The technique presented below owes its success to the Canonical Ubuntu team, who created a kernel image that already contains XFS support. That’s why these instructions use the official Canonical Ubuntu 9.10 Karmic Koala AMI – because it has XFS support built in. There may be other AKIs out there with XFS support built in – if so, the technique should work with them, too.

How to Do It

The general steps are as follows:

  1. Run an instance and set it up the way you like.
  2. Create an XFS-formatted EBS volume.
  3. Copy the contents of the instance’s root filesystem to the EBS volume.
  4. Unmount the EBS volume, snapshot it, and register it as an AMI.
  5. Launch an instance of the new AMI.

More details on each of these steps follows.

1. Run an instance and set it up the way you like.

As mentioned above, I use the official Canonical Ubuntu 9.10 Karmic Koala AMI (currently ami-1515f67c for 32-bit architecture – see the table on Alestic.com for the most current Ubuntu AMI IDs).

ami=ami-1515f67c
security_groups=default
keypair=my-keypair
instance_type=m1.small
ec2-run-instances $ami -t $instance_type -g $security_groups -k $keypair

Wait until the ec2-describe-instances command shows the instance is running and then ssh into it:

ssh -i my-keypair ubuntu@ec2-1-2-3-4.amazonaws.com

Now that you’re in, set up the instance’s root filesystem the way you want. Don’t forget that you probably want to run

sudo apt-get update

to allow you to pull in the latest packages.

In our case we’ll want to install ec2-consistent-snapshot, as per Eric Hammond’s article:

codename=$(lsb_release -cs)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu $codename main" | sudo tee /etc/apt/sources.list.d/alestic-ppa.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
sudo apt-get update
sudo apt-get install -y ec2-consistent-snapshot
sudo PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2

2. Create an XFS-formatted EBS volume.

First, install the XFS tools:

sudo apt-get install -y xfsprogs

These utilities allow you to format filesystems using XFS and to freeze and unfreeze the XFS filesystem. They are not necessary in order to read from XFS filesystems, but we want these programs installed on the AMI we create because they are used in the process of creating a consistent snapshot.

Next, create an EBS volume in the availability zone your instance is running in. I use a 10GB volume, but you can use any size and grow it later using this technique. This command is run on your local machine:

ec2-create-volume --size 10 -z $zone

Wait until the ec2-describe-volumes command shows the volume is available and then attach it to the instance:

ec2-attach-volume $volume --instance $instance --device /dev/sdh

Back on the instance, format the volume with XFS:

sudo mkfs.xfs /dev/sdh
sudo mkdir -m 000 /vol
sudo mount -t xfs /dev/sdh /vol

Now you should have an XFS-formatted EBS volume, ready for you to copy the contents of the instance’s root filesystem.

3. Copy the contents of the instance’s root filesystem to the EBS volume.

Here’s the command to copy over the entire root filesystem, preserving soft-links, onto the mounted EBS volume – but ignoring the volume itself:

sudo rsync -avx --exclude /vol / /vol

My command reports that it copied about 444 MB to the EBS volume.

4. Unmount the EBS volume, snapshot it, and register it as an AMI.

You’re ready to create the AMI. On the instance do this:

sudo umount /vol

Now, back on your local machine, create the snapshot:

ec2-create-snapshot $volume

Once ec2-describe-snapshots shows the snapshot is 100% complete, you can register it as an AMI. The AKI and ARI values used here should match the AKI and ARI that the instance is running – in this case, they are the default Canonical AKI and ARI for this AMI. Note that I give a descriptive “name” and “description” for the new AMI – this will make your life easier as the number of AMIs you create grows. Another note: some AMIs (such as the Ubuntu 10.04 Lucid AMIs) do not have a ramdisk, so skip the --ramdisk $ramdisk arguments if you’ve used such an AMI.

kernel=aki-5f15f636
ramdisk=ari-0915f660
description="Ubuntu 9.10 Karmic formatted with XFS"
ami_name=ubuntu-9.10-32-bit-ami-1515f67c-xfs
ec2-register --snapshot $snapshot --kernel $kernel --ramdisk $ramdisk '--description=$description' --name=$ami_name --architecture i386 --root-device-name /dev/sda1 --block-device-mapping /dev/sda2=ephemeral0

This displays the newly registered AMI ID – let’s say it’s ami-00000000.

5. Launch an instance of the new AMI.

Here comes the moment of truth. Launch an instance of the newly registered AMI:

ami=ami-00000000
security_groups=default
keypair=my-keypair
instance_type=m1.small
ec2-run-instances $ami -t $instance_type -g $security_groups -k $keypair

Again, wait until ec2-describe-instances shows it is running and ssh into it:

ssh -i my-keypair ubuntu@ec2-5-6-7-8.amazonaws.com

Now, on the instance, you should be able to see that the root filesystem is XFS with the mount command. The output should contain:

/dev/sda1 on / type xfs (rw)
...

We did it! Let’s create a consistent snapshot of the root filesystem. Look back into the output of ec2-describe-instances to determine the volume ID of the root volume for the instance.

sudo ec2-consistent-snapshot --aws-access-key-id $aws_access_key_id --aws-secret-access-key $aws_secret_access_key --xfs-filesystem / $volumeID

The command should display the snapshot ID of the snapshot that was created.

Using ec2-consistent-snapshot and an XFS-formatted EBS AMI, you can create snapshots of the running instance without stopping it. Please comment below if you find this helpful, or with any other feedback.

{ 52 comments… add one }

  • Anders Vinther February 11, 2010, 5:18 pm

    Great stuff!

    Sounds all too easy and too good to be true… can’t wait to give it a spin!

    Thanks!

    Anders

  • Anders Vinther February 19, 2010, 1:40 am

    You are missing a line break in step 2 where you format the XFS volume… :-)

    • shlomo February 19, 2010, 3:12 am

      @Anders Vinther,

      Thanks for pointing that out. I’ve fixed it.

      [An aside: When I moved from Blogger to self-hosted WordPress, all the code formatting got messed up. I fixed it up manually, but I missed some spots. It was completely worth it, though, because the WordPress authoring tools are much more flexible and capable.]

  • Anders Vinther February 19, 2010, 2:39 am

    Works a treat… thanks!

  • Anders Vinther February 22, 2010, 11:26 am

    Hmmm…. not quite sure what’s going on… but it seems as if an instance I am running with a scheduled create-consistent-snapshot died just after the cron job ran…

    Now I tried kicking off a manual snapshot and that seemed to work fine before I set up the cron job… However now I can’t seem to get on to the dead instance at all… It is quite possible this is not related to create-consistent-snapshot at all, and I don’t have a lot of info at this stage. I will look into it when I get some more time…

    Just wanted to know:
    - are you running this succesfully?
    - am I correct in assuming that an instance can take a snapshot of itself?

    Sorry for not being able to provide more info… am a bit strapped for time as we have a web site just gone live…

    Once again thanks a bunch for your very informative site!!

    Cheers,

    Anders

    • shlomo February 22, 2010, 1:07 pm

      @Anders Vinther,

      It’s possible the instance’s sudden shutdown was unrelated to the snapshot. In the manual snapshot you tried, did you use ec2-consistent-snapshot from on the instance, or did you use another method to initiate the snapshot?

      It’s also possible that your instance is still running – I assume you know to look in the AWS Management Console and, if it’s there, try to reboot it.

      There’s no issue in general with initiating a snapshot from within an instance. Many, including myself, have been using this technique, made easier ever since Eric Hammond distributed the ec2-consistent-snapshot utility.

      Hope this helps.

      • Anders Vinther March 2, 2010, 9:16 pm

        Hi Shlomo,

        Sorry to drop the ball on this one completely… I have simply been too busy with the site going live, so haven’t actually done anything further with this…

        The instance died completely. Tried rebooting and I still couldn’t get to it. Not sure what went on. In the end I reverted to the instance I created the XFS from, and am still running with it now…

        When I get some more time I’ll give it another shot…

        Thanks for letting me know that it should work…

        And, yes, you are right… the manual snapshot I did do from within the instance itself…

        Once again thanks for a great resource!

        Cheers,

        Anders

  • Francois March 29, 2010, 10:05 pm

    What would the restore procedure look like? Let’s say I want to recover a point in time consistent snapshot, and create an instance from it? Make Volume from snapshot, then register the ebs volume as a new AMI, then launch an instance of it?

    • shlomo March 30, 2010, 5:12 pm

      @Francois,

      You can simply register the snapshot as an AMI and launch it. For example:

      ec2-register --snapshot snap-12345678 --kernel aki-kkkkkkkk --ramdisk ari-rrrrrrrr --root-device-name /dev/sda1 --architecture x86_64 --name name_of_ami --description 'Long description of this AMI'

      substitute the values as follows:

      snap-12345678 : your snapshot ID
      aki-kkkkkkkk : the kernel ID from the original instance as it was launched. You can check this via ec2-describe-instances.
      ari-rrrrrrrr : the ramdisk ID from the original instance as it was launched. You can check this via ec2-describe-instances.
      /dev/sda1 : the root device name of the instance . You can check this via the output of ‘mount’ on the instance. Usually it will be /dev/sda1.
      x86_64 : ‘i386′ for 32-bit instance types (m1.small, c1.medium), ‘x86_64′ for 64-bit instances (all other instance types)

      The output of the command will display the new AMI ID. This can be launched directly, using the AWS Management Console or the ec2-run-instances command.

      • Daniel April 19, 2010, 10:04 pm

        Shlomo, you rock! This is exactly what I needed. Keep up the great work.

  • Remus May 5, 2010, 3:43 pm

    Shlomo I see the article and the instructions apply to instances running some version of Linux/unix.
    Do you know if for Windows Instances there is something similar?
    I mean right now I shut down the instance. Create a new AMI from it, and snapshots have the root + the EBS attached to it.
    My problem however is I have a SPOT instance which as you know can not be put in a stop state. So I did all this while the AMI was running but I am not sure if the new AMI will be ok or not… and I kind of a need to be sure of this if it holds important data.
    So are you aware of anything like this but for the Windows environment, running say server 2008 and Ms SQL Server (2005 or 2008)?

    thanks,
    Remus

    • shlomo May 5, 2010, 4:27 pm

      @Remus,

      I don’t know of a way to create a consistent snapshot of an EBS volume in Windows. Any readers know how?

      It looks like you’re trying to create an AMI from the snapshot you’ve taken. This can’t be done for Windows AMIs, you can only create them via the API.

      But you might not need an AMI if you have the volume you want to use as the root. You can use this technique described by Eric Hammond to swap out the root partition of another Windows instance and replace it with the desired Windows EBS root volume. I’m curious to know if this works when the desired swapped-in root volume was created from a snapshot – if you try it, please let me know.

      For spot EBS-backed instances you can create the spot request with the flag --block-device-mapping /dev/sda1=:false to prevent the volume from being deleted when the instance terminates. Then, after the spot instance terminates, you can use Eric Hammond’s technique to use that volume as the root volume for another instance.

  • Remus May 5, 2010, 4:52 pm

    Well for the record stated I am still a beginner with Amazon :) and I tend to use the Console as much as I can.
    I do not try to create an AMI from a snapshot. What I do I go in Console Management and under instances if you have a Windows one using EBS as root if you right click on it, it has this option “Create Image (EBS AMI).
    So the way I see it is: I have the initial instance I want to backup. I select it and run that. What it does it creates a new AMI for me, I see it listed after that. And under snapshots it creates the snapshots of the EBS used by that image.
    And I think when you want to launch that AMI it will create its own EBS from those snapshots.
    Now the big question is: When you do tha Create Image in Windows will it create a workable AMI? Meaning the SQL and WIndows on this one will they be ok?… I would hope so… otherwise it really sucks.

    Remus

    • shlomo May 7, 2010, 2:00 am

      @Remus,

      Yes, creating an image of a Windows instance via the AWS Management Console will create a bootable AMI.

  • jm May 7, 2010, 12:33 pm

    I followed all the steps and created the AMI from snaphot. When I launch the new instance, and try to connect via SSH, and returned “Connection refused”.

    I checked the firewall and is correct.

    In the log of the instance appears:
    [1.237071] sda1 XFS filesystem MOUNTING
    [2.640028] VFS: Mounted root (xfs filesystem) readonly device on 8:1.

    Is it normal to mount the filesystem in readonly mode?

    Any suggestions.

    • shlomo May 7, 2010, 2:37 pm

      @jm.

      In the course of booting the instance the root filesystem is first mounted read-only and then, later on, remounted read-write. There should be a console output line later on saying something like this:

      Remounting root filesystem in read-write mode: [ OK ]

      Does your console output have that line later? Does the console output show your opensshd server launching, and show the host key’s fingerprint?

      • jm May 7, 2010, 8:08 pm

        This is the next console output

        [ 1.535295] Filesystem “sda1″: Disabling barriers, trial barrier write failed
        [ 1.535640] XFS mounting filesystem sda1
        [ 1.612446] VFS: Mounted root (xfs filesystem) readonly on device 8:1.
        [ 1.613459] devtmpfs: mounted
        [ 1.613598] Freeing unused kernel memory: 228k freed
        [ 1.613735] Write protecting the kernel read-only data: 6416k
        init: console-setup main process (81) terminated with status 1
        %Ginit: plymouth main process (63) killed by SEGV signal
        init: plymouth-splash main process (228) terminated with status 2
        mountall: Disconnected from Plymouth

        In the console output not appears sshd init or host key’s fingerprint.

        Thanks for your help

        • shlomo May 8, 2010, 9:23 pm

          @jm,

          I’m trying to reproduce the issue you’re seeing.

          Using the Ubuntu 9.10 AMI, AKI, and ARI originally specified in this article (ami-1515f67c, aki-5f15f636, ari=0915f660) works without a problem.

          Using that same AMI/AKI/ARI and performing an apt-get upgrade before copying into the EBS volume also produces a bootable AMI.

          Using the latest 9.10 AMI/AKI/ARI (ami-bb709dd2, aki-5f15f636, ari-d5709dbc) also works without a problem, both when skipping the apt-get upgrade and when performing it.

          What Ubuntu AMI, AKI, ARI are you using?

        • shlomo May 8, 2010, 10:13 pm

          @jm,

          Now I understand – you’re using the Ubuntu 10.04 Lucid AMI as a starting point. See my response to @Michael Glass below – you need to register the AMI having an ephemeral drive device mapping. I’ve updated the article to include this in the ec2-register-image command.

          • jm May 10, 2010, 9:37 am

            Sorry, yes. I use an Ubuntu 10.04. With michael’s solution now works. Thanks

    • Mike May 8, 2010, 5:52 pm

      Hey. I believe that I have the same problem (can’t ssh, but pinging to public dns is ok). I did everything as stated above (I only skipped ari id at ec2-register, because I don’t know it – it is blank in aws mgmt console).
      http://pastebin.com/Nn8QwxCF – these are the last lines in console.

      Any idea? Anyway, thanks for tutorial.

      • Michael Glass May 8, 2010, 8:41 pm

        … sorry I didn’t read the comment history. I had similar errors before I specified /sda2 and /sda3 as specified below. Try it out when launching our ami or when registering your snapshot as an ami? (also, before you randomly assign /sda2 and /sda3 check out your /etc/fstab as to know what’s going where)

        good luck!

        • Mike May 10, 2010, 12:24 am

          Thank you mike.is and shlomo. It is working now.

  • Michael Glass May 7, 2010, 8:31 pm

    have you had any luck doing this with any of the 10.04 images? I try to stick to the LTS releases.

    Thanks
    -mike

  • Michael Glass May 8, 2010, 3:02 am

    I didn’t realize I had to also set up ephemeral storage from swap. others running into this, use a command similar to the following:

    ec2-register -n ami-name-here -d “description here” –block-device-mapping /dev/sda2=ephemeral0 –block-device-mapping ‘/dev/sda3=ephemeral1′ –snapshot snap-d03587b8 –architecture i386 –kernel aki-754aa41c

    for more info, go here:

    http://www.webadminblog.com/index.php/2010/03/23/amazon-ec2-ebs-instances-and-ephemeral-storage/

    (enjoy)

    • shlomo May 8, 2010, 10:10 pm

      @Michael Glass,

      Thanks for discovering the need for the 10.04 Lucid AMIs to have an ephemeral drive mount specified in the AMI registration. When you skip this step you get the strange behavior that @jm, @Mike, and you reported, where SSH doesn’t seem to start.

      I’ve edited the article to include the block-device-mapping specification for the ephemeral drives when registering the AMI.

      • Aruna Kulatunga June 23, 2010, 9:55 pm

        Hi Shlomo and Michael,

        Using the Lucid EBS boot ami-ab4d67df (the latest 64bit from Canonical) and copying Shlomo´s directions exactly, I still get the issue with ssh and connection is refused. As a second step, I added the extra block device sda3 as shown in Michael´s hint.

        This was my code to register the AMI


        ec2-register --snapshot snap-90b07bf9 --kernel aki-cb4d67bf --region eu-west-1 '--description=Ubuntu 10 Lucid formatted with XFS' --name=ubuntu-10.4-64-bit-ami-ab4d67df-xfs --architecture x86_64 --root-device-name /dev/sda1 --block-device-mapping /dev/sda2=ephemeral0 --block-device-mapping /dev/sda3=ephemeral1

        Note that I had to add the region as well as my instances are in eu-west. There was no issue with the AMI (ami-43103a37 – its private, not sure how to make it public!).

        I attached an Elastic IP to the running instance and attempted to connect.

        Here is the output from my terminal:


        ssh -i /Users/aruna/mykeys/mysecurity ubuntu@79.xxx.xxx.xxx
        ssh: connect to host 79.xxx.xxx.xxx port 22: Connection refused

        Here is the final lines from the console


        [ 1.523075] Filesystem "sda1": Disabling barriers, trial barrier write failed
        [ 1.569026] XFS mounting filesystem sda1
        [ 1.845787] VFS: Mounted root (xfs filesystem) readonly on device 8:1.
        [ 1.846784] devtmpfs: mounted
        [ 1.846899] Freeing unused kernel memory: 228k freed
        [ 1.847008] Write protecting the kernel read-only data: 6416k
        init: console-setup main process (81) terminated with status 1
        %Ginit: plymouth main process (63) killed by SEGV signal
        init: plymouth-splash main process (233) terminated with status 2
        Generating locales...
        en_GB.UTF-8... up-to-date
        Generation complete.
        mountall: Disconnected from Plymouth

        These two lines indicate that its mounted on an xfs volume (Hurrah!) but is yet read-only

        1.569026] XFS mounting filesystem sda1
        1.845787] VFS: Mounted root (xfs filesystem) readonly on device 8:1.

        I have refreshed the console a few times to see whether it would change status – no luck. Also rebooted the instance – same status.
        What am I doing wrong here…
        Grateful for any leads!

        • Aruna Kulatunga June 23, 2010, 10:21 pm

          Please disregard above. The issue was that I did not check the /etc/fstab to determine where the ephemeral storage should be mounted. It appears that on 64 bit ami´s its /dev/sdb mounted in /mnt, whereas I had specified /dev/sda2 and /dev/sda3. May I suggest that you also add some comment to to edit /etc/fstab and include the following two lines after editing out existing references to ephemeral storage.

          /dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0
          /dev/sda3 /mnt auto defaults,comment=cloudconfig 0 0

          With this, the following works very well for EBS booted 64 bit Lucid image ami-ab4d67df.
          Please substitute where I have XXXXX.


          ec2-register --snapshot snap-XXXXXXXX --kernel aki-cb4d67bf --region XXXXXX '--description=Ubuntu 10 Lucid formatted with XFS' --name=ubuntu-10.4-64-bit-ami-ab4d67df-xfs --architecture x86_64 --root-device-name /dev/sda1 --block-device-mapping /dev/sda2=ephemeral0 --block-device-mapping /dev/sda3=ephemeral1

          Many thanks and much appreciated.

          • Aruna Kulatunga June 24, 2010, 3:44 pm

            A restart of the instance (not terminate – but stop and start ) breaks the above. It appears that etc/fstab is reset to its original values, so my changes to /dev/sda2 and /dev/sda3 does not persist in /etc/fstab. I am investigating more. I think I might have to add the DeleteOnTermination flag to false for both the ephemeral storage devices. Will update once this is working

  • Ivan May 27, 2010, 10:08 am

    Hi,

    I followed the article but stumbled upon a problem on the boot of the newly created ami:
    Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(8,1)
    I tried to used this on a 10.04 Lucid (ami-a94d67dd from Alestic)
    I think maybe apt-get upgrade is breaking something but I can’t figure out what.
    I will try without doing the upgrade and post the results.

    Thanks

    Ivan

    • Ivan May 27, 2010, 1:56 pm

      Ok got it working with 10.04 when not doing apt-get upgrade.

      Will try again because I see no reason why it should not work.

      Thanks.

  • Miki June 15, 2010, 12:45 pm

    Everything was fine here with ami-a94d67dd (Ubuntu 10.4 32 bit). Thank you very much.

  • Hellmut June 20, 2010, 5:52 am

    I followed your article, its great!
    I built a new AMI to template my database servers (Postgres 8.4.4 + all GIS features) and this worked perfect. After I restarted the AMI the server offered me some updates, I ran the apg-get upgrade command to check and during the upgrade it told me it wanted to write a new GRUB … I figured what the heck lets test, it re-wrote the boot sector after the upgrade and … it actually works fine :) no problems on my end. It would be nice to know how I can spawn new servers resizing the volume… I know I can spawn a new volume from the AMI and give it a new size, but once the server comes up, whats the right process to allocate the space and not screw up my root file system?

    Thanks 1000000 again, and keep up the great blog :) super useful.

  • Hellmut June 20, 2010, 6:30 pm

    Thanks for pointing me to those articles Shlomo that resolves the question I had :)
    I have been reading your other posts, I was very interested in your post: How to Keep Your AWS Credentials on an EC2 Instance Securely back in August 2009 which was very comprehensive and educational for me, did you ever got to write a follow up article demonstrating the scripts to automate the procedures? I couldn’t find it in your previous posts, so I was curious. Thanks for all the great info!

    cheers,

    -H

    • shlomo June 20, 2010, 8:40 pm

      @Hellmut,

      Unfortunately I haven’t put together all the scripts to make those techniques easy. I will, but I can’t be sure when – unless I find a sponsor for the work.

  • RichardSanDiego June 21, 2010, 7:55 pm

    Thanks Shlomo for the great blog and MichaelGlass for figuring this out! i wanted to cross link to the amazon forum thread for those folks who might stumble across this blog entry and not realize the gem that’s contained within it. the following thread linked to this blog
    http://developer.amazonwebservices.com/connect/thread.jspa?messageID=183587

  • Aruna Kulatunga June 24, 2010, 5:40 pm

    Hi All,

    Great tutorial. I used the pesky Ubuntu Lucid EBS ami-ab4d67df as a base in eu-west-1 region and had a few issues. The first was the dreaded ssh connection refused when the time came to launch and log into the new instance, even after explicitly adding the ephemeral storage as per comment in No. 25. The second, after figuring out that I stumbled by just copying the directions instead of checking where the ephemeral storage is in m1.large instance (they go to /dev/sdb and /dev/sdc – instead of /dev/sda2 and /dev/sda3 ) was when I stopped the new instant and restarted again, to be refused entry, again!

    What finally worked for me is detailed below. As always YMMW.

    Before rsynching to /vol (or even after, you need to rsynch again) edit the etc/fstab so that the ephemeral drives are correctly mounted and root is specified as xfs.

    Example:

    # /etc/fstab: static file system information.
    #
    proc /proc proc nodev,noexec,nosuid 0 0
    /dev/sda1 / xfs defaults 0 0
    /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0
    /dev/sdc /log auto defaults,comment=cloudconfig 0 0

    Then, change the ec2-register instructions so that the root device, /dev/sda1, has its DeleteOnTermination flag set to false. This should be automatic given its an EBS volume and also we are not terminating, but simply stopping and restarting, but for some reason it appears that without this flag explicitly set, weird things happen to /etc/fstab and the ephemeral storage devices, which in turn affect services such as ssh.

    This is the command I used for the Lucid ami-ab4d67df . Replace the XXXXX with your unique information and change the size (30) immediately after the snap-xxxxxxx: again reflect the size of your own snapshot or a number bigger than your snapshot. If you specify a number smaller than the your snapshot, register AMI will fail.


    ec2-register --kernel aki-cb4d67bf --region xx-xxxx-x '--description=Ubuntu 10 Lucid formatted with XFS' --name=ubuntu-10.4-64-bit-ami-ab4d67df-xfs --architecture x86_64 --root-device-name /dev/sda1 -b /dev/sda1=snap-xxxxxxx:30:false --block-device-mapping /dev/sdb=ephemeral0:false --block-device-mapping /dev/sdc=ephemeral1:false

  • FreeMind June 30, 2010, 2:13 pm

    Hi,

    This was really useful.

    I am trying to use a bootable windows ebs volume with eucalyptus but I am not able to figure out how to make the boot ami point to the windows ebs volume.

    Any help is appreciated.

    Thanks

    • shlomo June 30, 2010, 4:28 pm

      @FreeMind,

      In EC2 the only way to create a bootable EBS volume with Windows on it is by starting with an EBS-based instance and using the API tools to create a new image from that instance – there’s no way to create a snapshot and then register it as a Windows AMI. I’m not familiar with how Eucalyptus does Windows, but I imagine it might have the same limitation.

  • Peter September 25, 2010, 10:14 pm

    Has anyone had any luck doing this with Lucid on a Micro instance? My new XFS AMI is always unbootable i.e. it has an empty System Log and is totally unreachable via ssh even though the instance shows as running in the AWS Console. My guess is that this is do to the lack of ephemeral storage on micro instances but without the System Log info I really don’t know where to start …

    I’ve updated the fstab to specify xfs as the filesystem on the root device but that didn’t seem to help at all.

    Any thoughts would be appreciated!

  • Peter September 26, 2010, 12:31 pm

    Unfortunately no. I was using the latest release last night for testing hoping that it would fix the issue but it didn’t. I’ll try again later this evening and document my steps and post here again so you can see what I did … and hopefully point out my mistake! :)

    • Peter September 27, 2010, 10:30 am

      After carefully walking through the steps in the posting and using the most recent Lucid AMI, I’ve had success! In my previous attempts, I was also installing some default packages before doing the rsync so maybe something we causing issues there. Now that I have my basic XFS AMI, I’m going to try installing the packages I need and creating a new AMI from that. If I run into any issues, I’ll report back. Otherwise, thanks for the great walkthrough and follow-up help!

  • Daniel November 3, 2010, 10:00 pm

    After failing for quite some time to get this working on a Micro instance using alestic.com’s Ubuntu 10.10 Maverick EBS boot (ami-548c783d) 64-bit, I finally discovered what the problem was.

    The above call to format the volume with XFS needs to use the -L flag. I discovered this by opening /etc/fstab and seeing “LABEL=uec-rootfs / ” instead of the expected “/dev/sda1 /”

    Therefore, use:

    sudo mkfs.xfs -L uec-rootfs /dev/sdh

  • Bart B. December 8, 2010, 3:56 pm

    I have a similar issue where my instance is inaccessible via ssh following a reboot. I’m using ami-8c0c5cc9 (Lucid 10.4.1 32bit) micro instance, with kernel aki-e40c5ca1. The root partition is on EBS volume. The only thing I updated yesterday was openssl then the instance was rebooted. It shows as running but I cannot ssh to it. Is there anything I can do to it now to resurrect it? I am sure it is the same fstab mapping issue as described above.

    Here are the last few lines of my System Log:
    [code]
    [ 0.566980] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.

    [ 0.592904] devtmpfs: mounted

    [ 0.592957] Freeing unused kernel memory: 216k freed

    [ 0.594161] Write protecting the kernel text: 4332k

    [ 0.594500] Write protecting the kernel read-only data: 1336k

    init: console-setup main process (63) terminated with status 1

    %Ginit: plymouth main process (45) killed by SEGV signal

    init: plymouth-splash main process (244) terminated with status 2

    cloud-init running: Wed, 08 Dec 2010 12:30:33 -0800. up 3.87 seconds
    mountall: Disconnected from Plymouth
    [/code]

  • Bart B. December 8, 2010, 7:45 pm

    I just wanted to say that I found a solution to this problem in this thread:
    https://forums.aws.amazon.com/thread.jspa?threadID=51267&start=25&tstart=0

    The last comment explains nicely how to recover the unbootable instance.

    • shlomo December 9, 2010, 2:39 am

      @Bart B.,

      Nice to know about that workaround for this issue. Thanks for sharing the solution.

  • ec2-do May 15, 2011, 4:03 pm

    Got this thing working on Ubuntu 10.10. 64-bit. Then tried using the script ec2-consistent-snapshot from cron, and the instance got totally stuck. Shortly after all IO got stuck. Incoming ssh connections also kept failing. Had to stop/start the instance from another instance using ec2 tools. And it’s consistently reproducible. Could it be that something fails in ec2-consistent-snapshot, and the script fails to release the xfs lock due to sloppy error handling? (I’m not using db locking)

    • shlomo May 16, 2011, 7:09 am

      @ec2-do,

      Sorry to hear that utility is giving you trouble. If it works from the command-line but not from cron then it’s very likely a PATH issue.
      Eric Hammond, the author of ec2-consistent-snapshot, has put great error-handling in that script. Try looking at (or adding to) the comments on Eric Hammond’s page for ec2-consistent-snapshot.
      BTW, it’s not necessary to fire up another EC2 instance just to use the command-line tools to start/stop instances. You can install those tools locally. Or you can use the AWS Management Console.

  • Anand Rao October 23, 2014, 9:56 am

    hi , I get a strange error when I try to create a volume …
    Any suggestions ?
    thanks
    Anand

    ubuntu@ubuntuamibuild:~$ aws ec2 create-volume –size 100 –availability-zone us-east-1a –dry-run
    Service ec2 not available in region “us-east-1″
    ubuntu@ubuntuamibuild:~$

    • Shlomo Swidler October 23, 2014, 10:08 am

      @Anand Rao,

      Strange indeed.

      Have you tried explicitly setting the region on the command line, as follows:
      aws --region us-east-1 ec2 create-volume –size 100 –availability-zone us-east-1a –dry-run
      If this works, then check your ~/.aws/config file to ensure the default region is set properly.

Leave a Comment