≡ Menu

Scalability and HA Limitations of AWS Marketplace AMIs

Reading AWS’s recent announcement of the AWS Marketplace you would think that it provides a catalog of click-to-deploy, highly-available, scalable applications running on EC2. You’d be partially right: the applications available in the AWS Marketplace are deployable in only a few clicks. But highly-available and scalable services will be difficult to build using Marketplace images. Here’s why.

Essential Ingredients of HA and Scalability on AWS

AWS makes it easy to run scalable, HA applications via several features. Not all applications use all of these features, but it would be very difficult to provide scalable and highly available service without using at least one of these:

  • Elastic Load Balancing
  • Auto Scaling
  • Elastic Block Storage volumes

ELB and AutoScaling both enable horizontal scalability: spreading load and controlling deployment size via first-class-citizen tools integrated into the AWS environment. They also enable availability by providing an automated way to recover from the failure of individual instances. [Scalability and availability often move in lock-step; improving one usually improves the other.] EBS volumes provide improved data availability: data can be retrieved off of dying instances – and often are used in RAID configurations to improve write performance.

AWS Marketplace Limitations

The AWS Marketplace has limitations that cripple two of the above features, making highly available and scalable services much more difficult to provide.

Marketplace AMI instances cannot be added to an ELB

Update 17 May 2012: The Product Manager for AWS Marketplace informed me that AWS Marketplace instances are now capable of being used with ELB. This limitation no longer exists.

Try it. You’ll get this error message:

 Error: InvalidInstance: ElasticLoadBalancing does not support the paid AMI or supported AMI of instance i-10abf677.

There is no mention of this limitation in the relevant ELB documentation.

This constraint severely limits horizontal scalability for Marketplace AMIs. Without ELB it’s difficult to share web traffic to multiple identically-configured instances of these AMIs. The AWS Marketplace offers several categories of AMIs, including Application Stacks (RoR, LAMP, etc.) and Application Servers (JBoss, WebSphere, etc.), that are typically deployed behind an ELB – but that won’t work with these Marketplace AMIs.

Root EBS volumes of Marketplace AMI instances cannot be mounted on non-root devices

Because all Marketplace AMIs are EBS-backed, you might think that there is a quick path to recover data if the instance dies unexpectedly: simply attach the root EBS volume to another device on another instance and get the data from there. But don’t rely on that – it won’t work. Here is what happens when you try to mount the root EBS volume from an instance of a Marketplace AMI on an another instance:

Failed to attach EBS volume 'New-Mongo-ROOT-VOLUME' (/dev/sdj) to 'New-Mongo' due to: OperationNotPermitted: 'vol-98c642f7' with Marketplace codes may not be attached as a secondary device.

This limitation is described here in AWS documentation:

If a volume has an AWS Marketplace product code:

  • The volume can only be attached to the root device of a stopped instance.
  • You must be subscribed to the AWS Marketplace code that is on the volume.
  • The configuration (instance type, operating system) of the instance must support that specific AWS Marketplace code. For example, you cannot take a volume from a Windows instance and attach it to a Linux instance.
  • AWS Marketplace product codes will be copied from the volume to the instance.

Closing a Licensing Loophole

Why did AWS place these constraints on using Marketplace-derived EBS volumes? To help Sellers keep control of the code they place into their AMI. Without the above limitations it’s simple for the purchaser of a Marketplace AMI to clone the root filesystem and create as many clones of that Marketplace-derived instance without necessarily being licensed to do so and without paying the premiums set by the Seller. It’s to close a licensing loophole.

AWS did a relatively thorough job of closing that hole. Here is a section of the current (25 April 2012) AWS overview of the EC2 EBS API and Command-Line Tools, with relevant Marketplace controls highlighted:

Command and API Action Description
ec2-create-volumeCreateVolume Creates a new Amazon EBS volume using the specified size or creates a new volume based on a previously created snapshot. Any AWS Marketplace product codes from the snapshot are propagated to the volume. For an overview of the AWS Marketplace, go to https://aws.amazon.com/marketplace/help/200900000. For details on how to use the AWS Marketplace, see AWS Marketplace.
ec2-attach-volumeAttachVolume Attaches the specified volume to a specified instance, exposing the volume using the specified device name. A volume can be attached to only a single instance at any time. The volume and instance must be in the same Availability Zone. The instance must be in the running or stoppedstate.

[Note] Note
If a volume has an AWS Marketplace product code:

  • The volume can only be attached to the root device of a stopped instance.
  • You must be subscribed to the AWS Marketplace code that is on the volume.
  • The configuration (instance type, operating system) of the instance must support that specific AWS Marketplace code. For example, you cannot take a volume from a Windows instance and attach it to a Linux instance.
  • AWS Marketplace product codes will be copied from the volume to the instance.

For an overview of the AWS Marketplace, go to https://aws.amazon.com/marketplace/help/200900000. For details on how to use the AWS Marketplace, see AWS Marketplace.

ec2-detach-volumeDetachVolume Detaches the specified volume from the instance it’s attached to. This action does not delete the volume. The volume can be attached to another instance and will have the same data as when it was detached. If the root volume is detached from an instance with an AWS Marketplace product code, then the AWS Marketplace product codes from that volume will no longer be associated with the instance.
ec2-create-snapshotCreateSnapshot Creates a snapshot of the volume you specify. After the snapshot is created, you can use it to create volumes that contain exactly the same data as the original volume. When a snapshot is created, any AWS Marketplace product codes from the volume will be propagated to the snapshot.
ec2-modify-snapshot-attributeModifySnapshotAttribute Modifies permissions for a snapshot (i.e., who can create volumes from the snapshot). You can specify one or more AWS accounts, or specify all to make the snapshot public.

[Note] Note
Snapshots with AWS Marketplace product codes cannot be made public.

The constraints above are meant to maintain the AWS Marketplace product code, the mechanism AWS uses to identify resources (AMIs, snapshots, volumes, and instances) that require Marketplace licensing integration. Note that not all AMIs in the AWS Marketplace have a product code – for example, the Amazon Linux AMI does not have one. AMIs that do not require licensing control (such as Amazon Linux, and Ubuntu without support) do not have AWS Marketplace product codes – but the rest do.

A Hole

There remains a hole in this lockdown scheme. Any instance whose kernel allows booting from a volume based on its volume label can be manipulated into booting from a secondary EBS volume. This requires root privileges on the instance. I have successfully booted an instance of the MongoDB AMI in the AWS Marketplace from a secondary EBS volume created from the Amazon Linux AMI. Anyone exploiting this hole can circumvent the product code lockdown.

Plugging the Hole

Sellers want these licensing controls and lockdowns. Here’s how:

  • Disable the root account.
  • Disable sudo.
  • Prevent user-data from being executed. On the Amazon Linux AMI and Ubuntu AMIs, user-data beginning with a hashbang is executed as root during the startup sequence.

Unfortunately these mitigations result in a crippled instance. Users won’t be able to mount EBS volumes – which requires root access – so data can’t be stored on EBS volumes for better recoverability.

Alternatively, you could develop your AWS Marketplace solutions as SaaS applications. For many potential Sellers this would be a long-term effort.

I’m still looking for good ways to enable scalability and HA of Marketplace AMIs. I welcome your suggestions.

Update 27 April 2012: Amazon Web Services PR has contacted me to say they are actively working on a fix for the ELB limitations, and are also working on removing the limitation related to mounting Marketplace-derived EBS volumes on secondary devices. I’ll update this article when this happens. In the meantime, AWS said that users who want to recover data from Marketplace-derived EBS volumes should reach out to AWS Support for help.

Update 17 May 2012: The Product Manager for AWS Marketplace informed me that AWS Marketplace instances are now capable of being used with ELB.

{ 12 comments… add one }
  • GPN April 25, 2012, 10:17 am

    Great post. Thanks.

    I do understand the value of the lockdown to AWS sellers.

    One question I have – why do you think the ELB limitation is needed as part of the lockdown? It sounds more like a bug than a feature. What am I missing?

    Thanks

    • shlomo April 26, 2012, 2:51 pm

      @GPN,

      I’m not sure why ELB doesn’t work with AWS Marketplace AMI instances. None of the features of ELB seem to me to be related to any Marketplace functionality.

      DevPay instances were also incompatible with ELB – that could be related to the networking bandwidth markup that DevPay providers can charge – though I’m not sure why that would matter either, but it is at least related. AWS Marketplace doesn’t offer markups for networking – only monthly charges and hourly over-and-above AWS’s rates.

  • GPN April 26, 2012, 3:25 pm

    Thank you

  • CJF April 30, 2012, 2:29 am

    If a buyer create a volume froma snaphot of a buyed Instance, “cloning” the machine on a different hardware.. will be billed for both instances? Only for the hourly use?

    • shlomo April 30, 2012, 6:45 am

      @CJF,

      Yes, you will be billed for the product use costs plus hourly costs for both instances. AWS Marketplace billing is based on product codes associated with the instance – you can see these product code associations in the AWS Management Console. As I showed above, the API manipulating EBS snapshots and volumes maintains the product codes and carries them over to instances booted from those volumes.

  • Uri May 10, 2012, 7:40 am

    Very helpful post!

    In AWS’s reply where they said they are going to lift the EBS limitation. Did they give any indication on how? I assume they will still need to let sellers protect their code against unauthorized copies.

    Thanks

  • Pablo Srabstein December 14, 2014, 11:15 am

    Why does ELB (or internet gateway) does not work also on the outgoing direction of the traffic as an abstraction layer? You mention NAT as a solution to this challenge , is there any other solution?

  • Pablo Srabstein December 14, 2014, 11:17 am

    Link to a similar question posted in Stack overflow sorry for multiple comments.

    http://stackoverflow.com/questions/27470819/multiple-ec2-instances-outgoing-outbound-traffic-presented-from-a-single-common

    • Shlomo Swidler December 22, 2014, 9:10 am

      @Pablo Srabstein,

      NAT is one solution. A proxy gateway will also do it. But the real question is, why do you want to have a single IP address for your source of traffic? I don’t know of any large-scale service that operates this way.

  • krunnal Gharre May 19, 2015, 1:48 pm

    Hi,

    Nice post, we were actually unable to find details about ami security online. I although have one more query regarding the above paid ami. We are developing a solution that requires users to modify our paid ami with necessary parameters, create a ami copy of the same and launch multiple instances of the modified ami. But at the same time, as you mentioned we are expecting that the user is charged for the extra modified instances they launch as its our proprietary solution they will be using. Does aws track this or we need to create some built in solution
    Thanks
    Krunnal .

    • Shlomo Swidler May 26, 2015, 10:58 am

      @krunnal Gharre,

      You can only register AMIs in the AWS Marketplace that you control. Once you give the customer control over the AMI, he will be able to re-register a new AMI, but it will be under his control.

      You need to store per-instance configuration details external to the AMI. User-data (which you cannot guarantee will be present or correct) and an API service under your control (with a customer-facing dashboard) are common techniques, often used together.

Leave a Comment