Working with virtual infrastructure is not a whole lot different than using real hardware, but it’s easy to get confused when things are different. In this article I’ll take a look at the life cycle of Amazon EC2 instances and how it differs from the traditional life cycle of real computers.
Update January 2010: This article has been updated to incorporate new options introduced by the Boot-from-EBS feature.
The Life Cycle of “Real” Computers
By “life cycle” I mean “the different stages that a computer goes through from power-on to power-off”. This is best explained with a diagram:
Real computers begin their service lives “off”. Someone hits the power button and the computers start booting up, eventually settling into a “running” state. From there they can be rebooted, hibernated, suspended, or shut-down normally. (Of course, something abnormal can happen, such as a power failure, which would cause the machine to immediately return to the “off” state. This is not represented in the diagram above.)
The diagram highlights two important features of the life cycle of real computers:
- Real computers cost you money even when they are not in use. Of course they consume power when they are on, and you pay for that power. But even when they are off they cost money, if only for the space that they occupy. This is represented in the diagram by the green box: real computers cost you money no matter if they are in use or not.
- Even when they are “off”, real computers remain yours. The information stored on their hard drives stays there, waiting to be accessed the next time the machine is powered on. This is represented by the arrows leading into and exiting the “off” state: even if you turn a computer off, you can still turn it on again.
The Life Cycle of EC2 Instances
The above two features of real computers, costing money and being yours to control, are different for virtual computers in the cloud. Different clouds have different features, and in Amazon EC2 there are two different ways of booting up an instance. Depending on which boot method you use you get different life cycle models. You can boot from an S3-backed AMI (also called an instance store AMI), or from an EBS-backed AMI. (The pros and cons of each kind of AMI are beyond the scope of this article.)
When you boot from an S3-backed AMI the instance life cycle looks like this:
S3-backed AMI instances begin their lives when you launch them (such as with the ec2-run-instances
command). They spend some time in the “pending” state, waiting for the reservation to be fulfilled. Once the instances have begun to boot they are in the “running” state. Here they will stay until they are terminated (such as with the ec2-terminate-instances
command), which causes the instances to begin “shutting down”, ending with the instances being “terminated”.
EBS-backed AMIs have a life cycle more similar to that of real computers:
EBS-backed AMI instances begin their lives when launched, spending some time in the “pending” state until the order is fulfilled. When the instances start booting they are “running”. From there they can be rebooted or shut down, like S3-backed AMI instances. They can also be “stopped” (such as with the ec2-stop-instances
command), from which they can be started again (e.g. with the ec2-start-instances
command) or terminated.
The above diagrams illustrate how EC2 instances are different than real computers as far as costing money and being yours to control is concerned:
- EC2 instances only cost money when they are “running”. You do not pay for instance-hours for pending, stopped, shutting down, or terminated instances. This is represented in the diagrams by the green box, which only surrounds the “running” state. This is unlike real computers, which cost money even when they are hibernating or powered off. EC2 instances that are “stopped” are not charged for any instance-hours, but the attached EBS volumes still do cost money. This is represented by the purple box, which surrounds both the “running” and “stopped” states. Note that, by default, EBS-backed AMI instances delete the associated EBS boot volume when they are terminated, so the cost of the EBS boot volume is not incurred once the instance terminates. This behavior can be changed at launch time, leaving the associated EBS volume intact even after the instance terminates. This situation is not illustrated, but in this case the EBS volume costs money until it is deleted.
- You control instances that are “running” and “stopped” – you can attach and detach EBS volumes from them (but you can’t detach the boot volume from an EBS-backed AMI instance that is “running”). But, once an EC2 instance has begun “shutting down”, you no longer control it. In the diagrams above there is no way to get an instance from “shutting down” back to “running”. This means that you cannot get back an instance that was terminated – the data stored on that computer’s local disks (“ephemeral storage”) can not be retrieved. Real computers, on the other hand, are still controlled by you when they are not running – all you need to do to access their data is to power them on.
Follow-on Questions
If the data that you place on the “ephemeral storage” of an EC2 instance is no longer accessible once you terminate it, what happens to the data? Specifically:
- What processes does Amazon have in place to make sure that the ephemeral storage drives do not present a privacy risk, allowing other instances possibly belonging to other people to see the data left over on the ephemeral drives you have relinquished?
- What can you do to mitigate the privacy risk of your leftover data on ephemeral storage, without relying on Amazon to do it for you?
If you can’t restart a terminated instance and you can’t access the data on a terminated instance, how do you accomplish in the EC2 cloud all the “usual” things that you do with real computers? Specifically:
- How do you keep data and applications in EC2 without keeping an instance running? (Hint: use EBS-backed AMIs and “stop” & “start” the instances as needed.)
- How do you make sure that your data and applications can be easily restored if your instance terminates unexpectedly?
And, what does an AMI have to do with the life cycle model? What about the life cycle model changes if you bundle your instance into an AMI? (Spoiler: nothing).
These questions will be the subject of future articles, so stay tuned.
18 replies on “The EC2 Instance Life Cycle”
[…] EBS-backed AMI [2]. These two AMIs exhibit a number of differences, for example in their lifecycle [3] and data persistence characteristics […]
I found out the hard way that terminated instances with ebs boot partition cannot be restarted, haha. luckily it was just during a test, so nothing was lost.
Thanks for your article, good casual read.
@tim,
Check out Eric Hammond’s article providing a few ways to prevent accidental termination from having drastic effects: http://alestic.com/2010/01/ec2-instance-locking
I wonder how long these transitions take. The new micro option is starting to look attractive for my modest needs. If I set my SSH ProxyCommand to ec2-run-instances or ec2-startup-instances (plus socat magic), how soon will the machine be available?
I asked ServerFault and it seems to fall in the one-three minute range, but with some variability and no guarantee that there is a machine available at all.
@Tobu,
Yes, it can take a few minutes. That SSH ProxyCommand is a cool idea, but it’s not appropriate for seldom-but-immediate need.
Thanks for the great post (especially the diagram explaining EBS boot).
Is it possible to use auto-scaling to automatically revive an EBS instance that has been stopped?
@Nelson,
Autocscaling is only capable of launching new instances, not starting stopped instances. That’s a very interesting idea, though. It would require using your own autocscaling framework, not Amazon’s.
darn good analysis here.
what about detaching the EBS ? during a stop. is that possible.
how to make a S3 backed instance.
@Adrian,
EBS volumes can be attached and detached while the instance is “Running” or “Stopped”. Although you can detach the root EBS volume (on an EBS-backed instance) while the instance is Running, you probably don’t want to unless you really know what you’re doing (it has to do with the OS keeping open files on the root volume).
The decision about which type of instance – S3-backed (instance-store) or EBS-backed – is determined by the AMI. Launch an S3-backed AMI for an instance-store root volume instance, and launch an EBS-backed AMI for an EBS root volume.
[…] EC2 instance life cycle /* […]
Can a unit be stopped from the pending state?
Could this diagram be upgraded to show the states that get read back, like “stopped”, “pending”, “running”, “stopping” ?
@Dennis Gearon,
An instance can only be stopped from the “running” state:
shlomo$ ec2-run-instances -t m1.small ami-0baf7662
RESERVATION r-6da9ac08 540528830757 default
INSTANCE i-3c853c44 ami-0baf7662 pending m1.small 2012-07-10T05:07:14+0000 us-east-1c aki-427d952b monitoring-disabled ebs paravirtual
shlomo$ ec2-stop-instances i-3c853c44
Client.IncorrectInstanceState: Instance 'i-3c853c44' cannot be stopped as it has never reached the 'running' state.
[…] EC2 instance life cycle […]
Thanks for the nice explanation, Shlomo.
One thing I missed in the article, though, is that if you are using ElasticIP, the moment you stop your instance and do not dissociate and release the ElasticIP from your instance, you are charged a small hourly fee for the time it is off.
One other think that users should be very aware of is when they create an instance and let it on 24/7 for long time. They should really consider using reserved instances… for the sake of their pockets.
@Eduardo Kita,
Thanks for pointing out the Elastic IP charges and other ways to keep costs down. This post is meant to explain the instance lifecycle and when you are charged for instance usage – costs associated with other resources – such as networking, storage – are extra.
[…] texto é uma tradução do artigo ‘The EC2 Instance Life Cycle’, escrito por Shlomo Swidler. O texto original é em Inglês. Por favor, leia também a […]
[…] for successful management of EC2 instances is a good understanding of their life cycle. There are some resources on the topic, but they do not concern Spot Instances, are partly outdated and do not have […]