Elastic Grid Blog

Most of the applications/frameworks/application servers usually rely on multicast in order to the discovery of other service instances. Unfortunately multicast does not work on Amazon EC2.

For Elastic Grid, we decided to face this issue from the beginning because we believe discovery of services is a key point in order to ease usage of Amazon EC2. In Elastic Grid, we use an Application Monitor who is in charge of provisioning the applications/services. The monitor provision services on Application Agents who are the receptacle for services.

The agents need to connect with the monitors so that the monitor knows which agents are running and their capabilities. We saw some suggestions on the EC2 forums which we will review shortly and explain what we ended up with for Elastic Grid.

First solution is to use Amazon SDB: each agent inserts in a SDB Domain the IP address it is running from. Then the monitors polls the SQS Domain regularly in order to find out if there is some new agents or if some of them are gone. The first problem with this solution is that your reaction can’t be faster than the polling interval. The second problem is that SDB now becomes another requirement for running Elastic Grid. Finally the worst is what happens if an agent dies? Its “record” in SDB would still be there, so the monitors would need to purge the SDB Domain from “dead records”.

Second solution is to use EC2 launch meta-data: when you ask to start an EC2 instance you can give some launch meta-data that this instance will be able to retrieve at boot time (the process of retrieval of the user meta-data is explained in the Developer Guide). The idea is that you start first the monitor, then retrieve the IP address of that monitor instance. Next, you start some agents using a launch meta-data whose value is the IP address (usually the private IP) of the previously started monitor. That is the strategy most of the solutions available on EC2 use. The problem with this solution is that if the monitor dies (and you restart it somewhere else), how do you make sure the currently running agents will be updated?

Now, back to how Elastic Grid tackles this problem. What we need is a way, at boot-time for an EC2 instance to find out where the other EC2 instances are, and more importantly what kind of “profile” they are (monitor or agent). First of all, have a look at the output we have when we do a DescribeInstances query using the EC2 command-line tools:

RESERVATION	r-05e5286c	154066937112	default

INSTANCE	i-2271a74b	ami-c140a5a8	ec2-75-101-202-32.compute-1.amazonaws.com
    ip-10-251-199-99.ec2.internal	running	eg-gsg-keypair	0		m1.small
    2008-06-30T09:41:43+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc

The default which appears in bold above is the name of the security group we used when we launched that instance. What this means is that at any time you call get the list of running instances and know in which security groups they are into.

The solution we came up with is to create some empty security groups (meaning security groups with no rules) used as tags. For Elastic Grid, we use two groups: one called eg-monitor and another one called eg-cybernode (the technical name for an agent).

When we start a monitor, we simply make sure it is started using that security group. Here is for example the shell script for starting an Elastic Grid monitor:

ec2run ami-c140a5a8 -g eg-monitor -g elastic-grid -g default -k eg-gsg-keypair -f ec2params.config

Starting an agent is pretty much the same, except we use the eg-cybernode group instead:

ec2run ami-c140a5a8 -g eg-cybernode -g elastic-grid -g default -k eg-gsg-keypair -f ec2params.config

When the Elastic Grid instance is started, it runs a DescribeInstances command, and scan each instance for its security groups. It the instance running is a monitor, it will only register with the other monitors (they peer themselves for failover). It the instance is an agent, then it will register with all the monitors it find.

Here is the output from DescribeInstances when there is a monitor running and an agent.

RESERVATION	r-05e5286c	154066937112	default,eg-monitor,elastic-grid INSTANCE	i-2271a74b
    ami-c140a5a8	ec2-75-101-202-32.compute-1.amazonaws.com	ip-10-251-199-99.ec2.internal
    running	eg-gsg-keypair	0		m1.small	2008-06-30T09:41:43+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc
RESERVATION	r-bae528d3	154066937112	default,elastic-grid,eg-cybernode INSTANCE	i-df71a7b6
    ami-c140a5a8	ec2-75-101-238-91.compute-1.amazonaws.com	ip-10-251-199-131.ec2.internal
    running	eg-gsg-keypair	0		m1.small	2008-06-30T10:00:41+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc

In fact this works so well that we use also this solution as a way to create logical cluster. For each cluster, we create a specific security group and only allow traffic to happen from EC2 instances within the same cluster group.

I hope this small how-to will help you, and feel free to post comments about alternatives you may have identified and or shortcomings we may have missed with our solution.

Last friday I presented Elastic Grid to the local Java Open Source developers group called OSSGTP. The audience of this group usually is made of skilled Java developers working on many famous Java projects, such as Spring, Hibernate, XWiki, eXo Portal, Restlet, jGuard, JCapthcha, etc (sorry, I probably forgot to cite many of them…).

This talk lasted 2 hours and half and was really interesting because of its format: this session was highly interactive and I was asked many questions about AWS and Elastic Grid.

The content of the talk was pretty much the same as the one Dennis and I did at JavaOne, except that I added two other demonstrations illustrating what EG (Elastic Grid) brings to the developers for ease of deployment of JEE applications. The first demonstration illustrated a deployment of XWiki whereas the second one was focused on eXo Portal.

For both demonstrations, I showed how EG is actively monitoring the processes it started when deploying the applications/servers. I simulated some crashes by killing some applications servers and could explain how the application servers were handled.

The feedback I received the day after, such as the blog post from Nicolas Martignole (in French only, sorry…), was really great and I truly thank all the participants.

Many news related to the Elastic Grid project should come soon, so stay tuned :=)

The JiBX plugin for IntelliJ IDEA has been updated for both IDEA 7.x and the latest EAP release (future 8.x release).

This update upgrades the embedded JiBX distribution to the latest release, that is version 1.1.6a.

The plugin for Amazon EC2 has been updated in order to:

• add support for the new instance types (High-CPU Medium and High-CPU Extra Large),
• update the Typica library.

During JavaOne 08, it’s been hard to update the blog. Here are below all the interesting things you may have missed.

Entries about Amazon Web Services:

• JBoss AMI - Ubuntu, Apache and JBoss. When Elastic Grid will be released, you won’t need to use a AMI per kind of application/environment you are trying to set up. More on that explained on the BoF slides and the screencast to come.
• Browser Uploads to S3 using HTML POST Forms - Interesting article from James Murty, the author of JetS3t.
• Processing Pipeline with Amazon S3, SQS, EC2, Ruby, Rails and ActiveMessaging
• Building for Performance and Reliability with Amazon SimpleDB
• SQS-EC2 Job Processor Sample - Quite similar to the demo we made during the BoF on EC2 and Elastic Grid at JavaOne 08.
• Sample application to get started with Amazon SQS and Amazon EC2
• dotCMS : Open Source Web Content Management System - Although the idea of pulling the latest version of the app from Subversion may sound interesting, I see more problems than solutions with that approach; what would happen if someone commits something broken into Subversion?
• Beta release of S3 copy feature
• Tips for Securing Your EC2 Instance
• Redundant Disk Storage Across Multiple EC2 nodes - Useful until most of us can use the Persistent Storage feature and probably still useful after that depending on your needs

The slides of the BoF on Elastic Grid and EC2 are finally available!

Thanks for all of you who could come. We had some interesting discussions and feedback after the talk, and even though Dennis has to come home tonight, I will stay in San Francisco up to the 13th, so feel free to get in touch with me if you’d like to discuss or have some other demo.

For those of you who could not make it for our BoF session, as I said during the talk, the plan is to make a screencast available shortly after I come back home, in Paris, France.

P.S.: of course this presentation is made available from Amazon S3…

This is huge!

Sun is now proposing a whole stack of products/solutions for Amazon EC2.

They are proposing OpenSolaris on Amazon EC2 AMI, but also some MySQL on EC2 solution and support!

This is supposed to be announced at the beginning of JavaOne in one of the keynotes.
Just some good PR before our BoF session on Amazon EC2.

A new HOW-TO called How To Start Rio on Amazon EC2 has been published.

As said previously, I’m going to JavaOne this year for a BoF on Elastic Grid.

I will arrive at San Francisco on April, 30th and leave by May, 13th.

If you’d like to talk to me when I’m there, feel free to drop me an email at jerome DOT bernard AT elastic-grid DOT com.

I would be pleased talking with you about Amazon services, Rio and Jini, or whatever else…

Entries about Amazon Web Services:

• Amazon S3 Copy Proposal: at last… This will allow to copy, rename, move, and relocate (from US to EU or the contrary) buckets without having to download/upload.
• Amazon FPS: How to create a Marketplace Application
• Amazon FPS: Pay Now Widget Module for OsCommerce
• Monetize your Amazon S3 objects using Amazon FPS
• Hadoop AMI and its associated tutorial Running Hadoop MapReduce on Amazon EC2 and Amazon S3
• Taking Massive Distributed Computing to the Common Man - Hadoop on Amazon EC2/S3
• Hadoop on Demand: provisioning of Hadoop on a virtual cluster, but not EC2…