Thanks to reader Ewout and his comment on my article How to Keep Your AWS Credentials on an EC2 Instance Securely for suggesting an additional method of transferring credentials: via a snapshot. It’s similar to burning credentials into an AMI, but easier to do and less prone to accidental inclusion in the application’s AMI.
Read on for a discussion of how to implement this technique.
How to Store AWS Credentials on an EBS Snapshot
This is how to store a secret on an EBS snapshot. You do this only once, or whenever you need to change the secret.
We’re going to automate as much as possible to make it easy to do. Here’s the command that launches an instance with a newly created 1GB EBS volume, formats it, mounts it, and sets up the root user to be accessible via ssh and scp. The new EBS volume created will not be deleted when the instance is terminated.
$ ec2-run-instances -b /dev/sdf=:1:false -t m1.small -k \ my-keypair -g default ami-6743ae0e -d '#! /bin/bash yes | mkfs.ext3 /dev/sdf mkdir -m 000 /secretVol mount -t ext3 -o noatime /dev/sdf /secretVol cp /home/ubuntu/.ssh/authorized_keys /root/.ssh/'
We have set up the root user to be accessible via ssh and scp so we can store the secrets on the EBS volume as the root user by directly copying them to the volume as root. Here’s how we do that:
$ ls -l total 24 -r--r--r-- 1 shlomo shlomo 916 Jun 20 2010 cert-NT63JNE4VSDEMH6VHLHBGHWV3DRFDECP.pem -r-------- 1 shlomo shlomo 90 Jun 1 2010 creds -r-------- 1 shlomo shlomo 926 Jun 20 2010 pk-NT63JNE4VSDEMH6VHLHBGHWV3DRFDECP.pem $ scp -i /path/to/id_rsa-my-keypair * email@example.com:/secretVol/
Our secret is now on the EBS volume, visible only to the root user.
We’re almost done. Of course you want to test that your application can access the secret appropriately. Once you’ve done that you can terminate the instance – don’t worry, the volume will not be deleted due to the “:false” specification in our launch command.
$ ec2-terminate-instance $instance $ ec2-describe-volumes VOLUME vol-7ce48a15 1 us-east-1b available 2010-07-18T17:34:01+0000 VOLUME vol-7ee48a17 15 snap-5e4bec36 us-east-1b deleting 2010-07-18T17:34:02+0000
Note that the root EBS volume is being deleted but the new 1GB volume we created and stored the secret on is intact.
Now we’re ready for the final two steps:
Snapshot the volume with the secret:
$ ec2-create-snapshot $secretVolume SNAPSHOT snap-2ec73045 vol-7ce48a15 pending 2010-07-18T18:05:39+0000 540528830757 1
And, once the snapshot completes, delete the volume:
$ ec2-describe-snapshots -o self SNAPSHOT snap-2ec73045 vol-7ce48a15 completed 2010-07-18T18:05:40+0000 100% 540528830757 1 $ ec2-delete-volume $secretVolume VOLUME vol-7ce48a15 # save the snapshot ID $ secretSnapshot=snap-2ec73045
Now you have a snapshot
$secretSnapshot with your credentials stored on it.
How to Use Credentials Stored on an EBS Snapshot
Of course you can create a new volume from the snapshot, attach the volume to your instance, mount the volume to the filesystem, and access the secrets via the root user. But here’s a way to do all that at instance launch time:
$ ec2-run-instances un-instances -b /dev/sdf=$secretSnapshot -t m1.small -k \ my-keypair -g default ami-6743ae0e -d '#! /bin/bash mkdir -m 000 /secretVol mount -t ext3 -o noatime /dev/sdf /secretVol # make sure it gets remounted if we reboot echo "/dev/sdf /secretVol ext3 noatime 0 0" > /etc/fstab'
This one-liner uses the
-b option of
ec2-run-instances to specify a new volume be created from
$secretSnapshot, attached to /dev/sdf, and this volume will be automatically deleted when the instance terminates. The user-data script sets up the filesystem mount point and mounts the volume there, also ensuring that the volume will be remounted if the instance reboots.
Check it out, a new volume was created for /dev/sdf:
$ ec2-describe-instances RESERVATION r-e4f2608f 540528830757 default INSTANCE i-155b857f ami-6743ae0e pending my-keypair 0 m1.small 2010-07-19T15:51:13+0000 us-east-1b aki-5f15f636 ari-d5709dbc monitoring-disabled ebs BLOCKDEVICE /dev/sda1 vol-8a721be3 2010-07-19T15:51:22.000Z BLOCKDEVICE /dev/sdf vol-88721be1 2010-07-19T15:51:22.000Z
Let’s make sure the files are there. SSHing into the instance (as the
ubuntu user) we then see:
$ ls -la /secretVol ls: cannot open directory /secretVol: Permission denied $ sudo ls -l /secretVol total 28 -r-------- 1 root root 916 2010-07-18 17:52 cert-NT63JNE4VSDEMH6VHLHBGHWV3DRFDECP.pem -r-------- 1 root root 90 2010-07-18 17:52 creds dr-------- 2 root root 16384 2010-07-18 17:42 lost+found -r-------- 1 root root 926 2010-07-18 17:52 pk-NT63JNE4VSDEMH6VHLHBGHWV3DRFDECP.pem
Your application running the instance (you’ll install it by adding to the user-data script, right?) will need root privileges to access those secrets.