Comments on: Creating Consistent Snapshots of a Live Instance with XFS on a Boot-from-EBS AMI

By: What XFS configuration should I use for EC2 disks (mysql db-server) - Admins Goodies

Tue, 16 Aug 2011 10:08:09 +0000

[...] Somewhat related: http://shlomoswidler.com/2010/01/creating-consistent-snapshots-of-live.html [...]

By: shlomo

shlomo — Mon, 16 May 2011 12:09:57 +0000

@ec2-do,

Sorry to hear that utility is giving you trouble. If it works from the command-line but not from cron then it’s very likely a PATH issue.
Eric Hammond, the author of ec2-consistent-snapshot, has put great error-handling in that script. Try looking at (or adding to) the comments on Eric Hammond’s page for ec2-consistent-snapshot.
BTW, it’s not necessary to fire up another EC2 instance just to use the command-line tools to start/stop instances. You can install those tools locally. Or you can use the AWS Management Console.

By: ec2-do

ec2-do — Sun, 15 May 2011 21:03:48 +0000

Got this thing working on Ubuntu 10.10. 64-bit. Then tried using the script ec2-consistent-snapshot from cron, and the instance got totally stuck. Shortly after all IO got stuck. Incoming ssh connections also kept failing. Had to stop/start the instance from another instance using ec2 tools. And it’s consistently reproducible. Could it be that something fails in ec2-consistent-snapshot, and the script fails to release the xfs lock due to sloppy error handling? (I’m not using db locking)

By: shlomo

shlomo — Thu, 09 Dec 2010 07:39:12 +0000

@Bart B.,

Nice to know about that workaround for this issue. Thanks for sharing the solution.

By: Bart B.

Bart B. — Thu, 09 Dec 2010 00:45:07 +0000

I just wanted to say that I found a solution to this problem in this thread:
https://forums.aws.amazon.com/thread.jspa?threadID=51267&start=25&tstart=0

The last comment explains nicely how to recover the unbootable instance.

By: Bart B.

Bart B. — Wed, 08 Dec 2010 20:56:05 +0000

I have a similar issue where my instance is inaccessible via ssh following a reboot. I’m using ami-8c0c5cc9 (Lucid 10.4.1 32bit) micro instance, with kernel aki-e40c5ca1. The root partition is on EBS volume. The only thing I updated yesterday was openssl then the instance was rebooted. It shows as running but I cannot ssh to it. Is there anything I can do to it now to resurrect it? I am sure it is the same fstab mapping issue as described above.

Here are the last few lines of my System Log:
[code]
[ 0.566980] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.

[ 0.592904] devtmpfs: mounted

[ 0.592957] Freeing unused kernel memory: 216k freed

[ 0.594161] Write protecting the kernel text: 4332k

[ 0.594500] Write protecting the kernel read-only data: 1336k

init: console-setup main process (63) terminated with status 1

%Ginit: plymouth main process (45) killed by SEGV signal

init: plymouth-splash main process (244) terminated with status 2

cloud-init running: Wed, 08 Dec 2010 12:30:33 -0800. up 3.87 seconds
mountall: Disconnected from Plymouth
[/code]

By: Daniel

Daniel — Thu, 04 Nov 2010 03:00:34 +0000

After failing for quite some time to get this working on a Micro instance using alestic.com’s Ubuntu 10.10 Maverick EBS boot (ami-548c783d) 64-bit, I finally discovered what the problem was.

The above call to format the volume with XFS needs to use the -L flag. I discovered this by opening /etc/fstab and seeing “LABEL=uec-rootfs / ” instead of the expected “/dev/sda1 /”

Therefore, use:

sudo mkfs.xfs -L uec-rootfs /dev/sdh

By: Peter

Peter — Mon, 27 Sep 2010 15:30:31 +0000

After carefully walking through the steps in the posting and using the most recent Lucid AMI, I’ve had success! In my previous attempts, I was also installing some default packages before doing the rsync so maybe something we causing issues there. Now that I have my basic XFS AMI, I’m going to try installing the packages I need and creating a new AMI from that. If I run into any issues, I’ll report back. Otherwise, thanks for the great walkthrough and follow-up help!

By: Peter

Peter — Sun, 26 Sep 2010 17:31:16 +0000

Unfortunately no. I was using the latest release last night for testing hoping that it would fix the issue but it didn’t. I’ll try again later this evening and document my steps and post here again so you can see what I did … and hopefully point out my mistake!

By: shlomo

shlomo — Sun, 26 Sep 2010 08:12:39 +0000

@Peter,

This is likely related to an Ubuntu bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/634102 that was recently fixed. Do things work if you do
sudo apt-get update && sudo apt-get -y upgrade
?