Elastic Grid Blog

Most of the applications/frameworks/application servers usually rely on multicast in order to the discovery of other service instances. Unfortunately multicast does not work on Amazon EC2.

For Elastic Grid, we decided to face this issue from the beginning because we believe discovery of services is a key point in order to ease usage of Amazon EC2. In Elastic Grid, we use an Application Monitor who is in charge of provisioning the applications/services. The monitor provision services on Application Agents who are the receptacle for services.

The agents need to connect with the monitors so that the monitor knows which agents are running and their capabilities. We saw some suggestions on the EC2 forums which we will review shortly and explain what we ended up with for Elastic Grid.

First solution is to use Amazon SDB: each agent inserts in a SDB Domain the IP address it is running from. Then the monitors polls the SQS Domain regularly in order to find out if there is some new agents or if some of them are gone. The first problem with this solution is that your reaction can’t be faster than the polling interval. The second problem is that SDB now becomes another requirement for running Elastic Grid. Finally the worst is what happens if an agent dies? Its “record” in SDB would still be there, so the monitors would need to purge the SDB Domain from “dead records”.

Second solution is to use EC2 launch meta-data: when you ask to start an EC2 instance you can give some launch meta-data that this instance will be able to retrieve at boot time (the process of retrieval of the user meta-data is explained in the Developer Guide). The idea is that you start first the monitor, then retrieve the IP address of that monitor instance. Next, you start some agents using a launch meta-data whose value is the IP address (usually the private IP) of the previously started monitor. That is the strategy most of the solutions available on EC2 use. The problem with this solution is that if the monitor dies (and you restart it somewhere else), how do you make sure the currently running agents will be updated?

Now, back to how Elastic Grid tackles this problem. What we need is a way, at boot-time for an EC2 instance to find out where the other EC2 instances are, and more importantly what kind of “profile” they are (monitor or agent). First of all, have a look at the output we have when we do a DescribeInstances query using the EC2 command-line tools:

RESERVATION	r-05e5286c	154066937112	default

INSTANCE	i-2271a74b	ami-c140a5a8	ec2-75-101-202-32.compute-1.amazonaws.com
    ip-10-251-199-99.ec2.internal	running	eg-gsg-keypair	0		m1.small
    2008-06-30T09:41:43+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc

The default which appears in bold above is the name of the security group we used when we launched that instance. What this means is that at any time you call get the list of running instances and know in which security groups they are into.

The solution we came up with is to create some empty security groups (meaning security groups with no rules) used as tags. For Elastic Grid, we use two groups: one called eg-monitor and another one called eg-cybernode (the technical name for an agent).

When we start a monitor, we simply make sure it is started using that security group. Here is for example the shell script for starting an Elastic Grid monitor:

ec2run ami-c140a5a8 -g eg-monitor -g elastic-grid -g default -k eg-gsg-keypair -f ec2params.config

Starting an agent is pretty much the same, except we use the eg-cybernode group instead:

ec2run ami-c140a5a8 -g eg-cybernode -g elastic-grid -g default -k eg-gsg-keypair -f ec2params.config

When the Elastic Grid instance is started, it runs a DescribeInstances command, and scan each instance for its security groups. It the instance running is a monitor, it will only register with the other monitors (they peer themselves for failover). It the instance is an agent, then it will register with all the monitors it find.

Here is the output from DescribeInstances when there is a monitor running and an agent.

RESERVATION	r-05e5286c	154066937112	default,eg-monitor,elastic-grid INSTANCE	i-2271a74b
    ami-c140a5a8	ec2-75-101-202-32.compute-1.amazonaws.com	ip-10-251-199-99.ec2.internal
    running	eg-gsg-keypair	0		m1.small	2008-06-30T09:41:43+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc
RESERVATION	r-bae528d3	154066937112	default,elastic-grid,eg-cybernode INSTANCE	i-df71a7b6
    ami-c140a5a8	ec2-75-101-238-91.compute-1.amazonaws.com	ip-10-251-199-131.ec2.internal
    running	eg-gsg-keypair	0		m1.small	2008-06-30T10:00:41+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc

In fact this works so well that we use also this solution as a way to create logical cluster. For each cluster, we create a specific security group and only allow traffic to happen from EC2 instances within the same cluster group.

I hope this small how-to will help you, and feel free to post comments about alternatives you may have identified and or shortcomings we may have missed with our solution.

O’Reilly should release a new book called Programming Amazon Web Services: S3, EC2, SQS, FPS, and SimpleDB (Programming) in March 2008.

This book looks really promising. I’ll buy the book as soon as possible and will publish a review of it when done reading it.

Here is the book description:

Building on the success of its storefront and fulfillment services, Amazon now allows businesses to “rent” computing power, data storage and bandwidth on its vast network platform. This book demonstrates how developers working with small- to mid-sized companies can take advantage of Amazon Web Services (AWS) such as the Simple Storage Service (S3), Elastic Compute Cloud (EC2), Simple Queue Service (SQS), Flexible Payments Service (FPS), and SimpleDB to build web-scale business applications.

With AWS, Amazon offers a new paradigm for IT infrastructure: use what you need, as you need it, and pay as you go. Programming Web Services explains how you can access Amazon’s open APIs to store and run applications, rather than spend precious time and resources building your own. With this book, you’ll learn all the technical details you need to:

• Store and retrieve any amount of data using application servers, unlimited data storage, and bandwidth with the Amazon S3 service
• Buy computing time using Amazon EC2’s interface to requisition machines, load them with an application environment, manage access permissions, and run your image using as many or few systems as needed
• Use Amazon’s web-scale messaging infrastructure to store messages as they travel between computers with Amazon SQS
• Leverage the Amazon FPS service to structure payment instructions and allow the movement of money between any two entities, humans or computers
• Create and store multiple data sets, query your data easily, and return the results using Amazon SimpleDB.
• Scale up or down at a moment’s notice, using these services to employ as much time and space as you need

Whether you’re starting a new online business, need to ramp up existing services, or require an offsite backup for your home, Programming Web Services gives you the background and the practical knowledge you need to start using AWS. Other books explain how to build web services. This book teaches businesses how to take make use of existing services from an established technology leader.