Update: the slides have been published!

We are thrilled to be present this year in Antwerp, Belgium for Devoxx. Our talk, Dynamic deployment and scalability for the cloud will be on Monday, 8th at 9:00pm.

We really hope to see you there for one of the biggest Java developers event in Europe.

Here is the abstract of our session:

Cloud Computing providers allow a fantastic way to deploy scalable machine images easily and on demand. However, there is a finer grain of scalability that must be provided, allowing individual application assets to easily scale to meet the demands of a running system. Our session discusses the Elastic Grid, an approach that provides dynamic allocation, management and scalability of applications through the cloud.

Intrinsic to the Elastic Grid are a set of dynamic capabilities and reliance on Policy-based and Quality of Service mechanisms that extend capabilities currently found in available cloud computing technologies

Being able to inject rules & policies into cloud focused infrastructure allows greater automation, scalability and controlled behavior. Ultimately, cloud based deployments can provide advanced capabilities surrounding self-healing, self optimization & self configuration. The Elastic Grid provides an approach using a cloud focused Domain Specific Language to declaratively include behavior as SLA policy declarations.

Discover how to easily deploy your Java applications (but not only!) on the Cloud (especially Amazon EC2) during this BoF with both an introduction to how Elastic Grid ease deployments, a real demo done live and some feedback from real uses cases.

If you’re in the SaaS (Software as a Service) business, and are following the Cloud Computing trends, you really need to understand the difference between the offers. Of course, I’m not talking about Cloud, like Apple’s Mobile , but only Cloud Computing.

As of today, we have at least 3 big players (Amazon, Google and Microsoft) and some other ones from smaller players (like GoGrid, Flexiscale and Mosso to name a few of them). What is really important when choosing a Cloud Computing Platform is to understand what they offer and how they are different from each others. Of course the pricing models can vary quite a lot, with some providers offering SLAs, some offering virtualized hardware whereas some others one mean real hardware, etc.

But more than that, what really matters is do you need an infrastructure or a plaform, or if you’re fond of acronyms, are you the IaaS guy or the PaaS one?

That question is really important as it’s not easy to switch from one to the other.

Let’s explain the difference by analyzing the Cloud Computing offers from the 3 big players named above.

Google, through it’s App Engine offer, enable you to host your websites and your data on the Cloud and enable you to use some of their successful services like BigTable. What is important to understand is their pricing model and their APIs/Services. Two things are important to note :

• You can only host Python applications (erh… web applications) as of today and you pay per bandwidth, storage and CPU used for hosting this web application,
• You can easily integrate with other Google Services and Google Accounts.

Microsoft, offers a way to host your .Net applications on the Cloud with a pricing model yet to be officially announced, and offers integration with some Microsoft services/applications.

Amazon on the other end, does not offer a way to host your web applications out-of-the-box on the Cloud, but simply provide virtualized hardware on which you can do whatever you’d like to (well, as usual it’s it a bit more complex than that, but that’s pretty much it).

So basically, Google and Microsoft offers are PaaS solutions: they offer a Platform on which you can deploy your applications. On the other end, Amazon offers an IaaS solution: an Infrastructure which you can use.

So your choice has to be made carefully and need to analyze:

• Vendor Lock-in: if you deploy on Google or Microsoft Clouds, you make a choice on both technologies and with which services you’d like to integrate,
• Ease of Use: deploying a web application is easier to do than deploying and managing a complete infrastructure,
• PaaS or IaaS: do you want the Cloud Provider to offer you a way to host your applications (if you can accomadate with their technical restrictions) or an infrastructure allowing you to host your applications (without restrictions) the way you want?

I guess, it must be quite clear from this blog post that I’m not really fond of PaaS. Surely IaaS means more work for the customer, but you’re the one in charge of that virtualized infrastructure, allowing you to deploy complex applications on the Cloud, something I guess most of the time you won’t be able to do with the PaaS approach.

On the other end, if your goals are integration with Google and Microsoft services and you want to make sure you Gadget (don’t know the word for Microsoft sites) can scale, then it makes sense to deploy on those platforms.

Elastic Grid team will be present in early afternoon on December, 2nd in Paris at the Open World Forum for a panel titled Technological and economic breakthroughs: Challenge or opportunity for FLOSS?

Abstract:

What impacts will SAAS, Cloud Computing, Ambient Computing… have on FLOSS, and what challenges will they bring? Do the new technological and business model breakthroughs represent a threat or an opportunity for FLOSS?

We hope to see you there. Feel free to get in touch with us during/after the conferences.

The OW2 Quarterly Meeting happened yesterday morning. We did our official introduction of Elastic Grid to the Technical Council and we can now announce Elastic Grid being part of the OW2 Community.

We had been already working with some of its members and some of the OW2 projects for a while, like XWiki and eXo Portal, so this move will ease partnerships with some of the other projects.

So now our project page on OW2 is available at http://forge.objectweb.org/projects/elastic-grid/. Our Subversion repository is also available until we complete our roadmap for the first release. Some mailing-lists have also been set up and you can subscribe to them.

We are thrilled by the opportunity to be part of the OW2 Community!

The slides of the BoF on Elastic Grid and EC2 are finally available!

Thanks for all of you who could come. We had some interesting discussions and feedback after the talk.
This is the first time I had a chance to go to JavaZone and I must say it’s been quite interesting. A few friends and I had the opportunity to discuss with Tom J. Bang who gave us the opportunity to taste local stuff.

I also had the opportunity to discuss with a lot of people there: my friends from our local user group (Guillaume Laforge, Julien Viet, Alexis Moussine-Pouchkine, Jerome Lacoste) but also Scott Davis and Eitan Suez. (Sorry for all the other ones I forgot to cite.)

For those of you who could not make it for our BoF session, the organizers of the JavaZone event will provide a video. I’ll update the blog when the video is available.

Update: the slides have been published! I finally did not demonstrate the FPS webapp and added more fresh content on the presentation. The demo part of the talk has also been improved a lot as it is now even easier to use Elastic Grid. The webcast is supposed to be provided at some point. I’ll update this blog post when the video is available.

I will be doing a talk like the one we did for JavaOne 08, except this time the demo will a bit shorten compared to the one we did at JavaOne. I will also demonstrate the Amazon FPS webapp we’ve been working on (more about this to come soon on this blog).

Our talk How Can Amazon EC2 Benefit from the Elastic Grid Solution? is scheduled on September 18th (2nd day of the JavaZone conference) at 3:45pm in Lab 6.

If you are in Oslo for JavaZone this year, let me know. I would be pleased to see you and discuss anything like AWS, Elastic Grid, or even for drinking a beer :-p

Most of the applications/frameworks/application servers usually rely on multicast in order to the discovery of other service instances. Unfortunately multicast does not work on Amazon EC2.

For Elastic Grid, we decided to face this issue from the beginning because we believe discovery of services is a key point in order to ease usage of Amazon EC2. In Elastic Grid, we use an Application Monitor who is in charge of provisioning the applications/services. The monitor provision services on Application Agents who are the receptacle for services.

The agents need to connect with the monitors so that the monitor knows which agents are running and their capabilities. We saw some suggestions on the EC2 forums which we will review shortly and explain what we ended up with for Elastic Grid.

First solution is to use Amazon SDB: each agent inserts in a SDB Domain the IP address it is running from. Then the monitors polls the SQS Domain regularly in order to find out if there is some new agents or if some of them are gone. The first problem with this solution is that your reaction can’t be faster than the polling interval. The second problem is that SDB now becomes another requirement for running Elastic Grid. Finally the worst is what happens if an agent dies? Its “record” in SDB would still be there, so the monitors would need to purge the SDB Domain from “dead records”.

Second solution is to use EC2 launch meta-data: when you ask to start an EC2 instance you can give some launch meta-data that this instance will be able to retrieve at boot time (the process of retrieval of the user meta-data is explained in the Developer Guide). The idea is that you start first the monitor, then retrieve the IP address of that monitor instance. Next, you start some agents using a launch meta-data whose value is the IP address (usually the private IP) of the previously started monitor. That is the strategy most of the solutions available on EC2 use. The problem with this solution is that if the monitor dies (and you restart it somewhere else), how do you make sure the currently running agents will be updated?

Now, back to how Elastic Grid tackles this problem. What we need is a way, at boot-time for an EC2 instance to find out where the other EC2 instances are, and more importantly what kind of “profile” they are (monitor or agent). First of all, have a look at the output we have when we do a DescribeInstances query using the EC2 command-line tools:

RESERVATION	r-05e5286c	154066937112	default

INSTANCE	i-2271a74b	ami-c140a5a8	ec2-75-101-202-32.compute-1.amazonaws.com
    ip-10-251-199-99.ec2.internal	running	eg-gsg-keypair	0		m1.small
    2008-06-30T09:41:43+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc

The default which appears in bold above is the name of the security group we used when we launched that instance. What this means is that at any time you call get the list of running instances and know in which security groups they are into.

The solution we came up with is to create some empty security groups (meaning security groups with no rules) used as tags. For Elastic Grid, we use two groups: one called eg-monitor and another one called eg-cybernode (the technical name for an agent).

When we start a monitor, we simply make sure it is started using that security group. Here is for example the shell script for starting an Elastic Grid monitor:

ec2run ami-c140a5a8 -g eg-monitor -g elastic-grid -g default -k eg-gsg-keypair -f ec2params.config

Starting an agent is pretty much the same, except we use the eg-cybernode group instead:

ec2run ami-c140a5a8 -g eg-cybernode -g elastic-grid -g default -k eg-gsg-keypair -f ec2params.config

When the Elastic Grid instance is started, it runs a DescribeInstances command, and scan each instance for its security groups. It the instance running is a monitor, it will only register with the other monitors (they peer themselves for failover). It the instance is an agent, then it will register with all the monitors it find.

Here is the output from DescribeInstances when there is a monitor running and an agent.

RESERVATION	r-05e5286c	154066937112	default,eg-monitor,elastic-grid INSTANCE	i-2271a74b
    ami-c140a5a8	ec2-75-101-202-32.compute-1.amazonaws.com	ip-10-251-199-99.ec2.internal
    running	eg-gsg-keypair	0		m1.small	2008-06-30T09:41:43+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc
RESERVATION	r-bae528d3	154066937112	default,elastic-grid,eg-cybernode INSTANCE	i-df71a7b6
    ami-c140a5a8	ec2-75-101-238-91.compute-1.amazonaws.com	ip-10-251-199-131.ec2.internal
    running	eg-gsg-keypair	0		m1.small	2008-06-30T10:00:41+0000	us-east-1c	aki-a71cf9ce	ari-a51cf9cc

In fact this works so well that we use also this solution as a way to create logical cluster. For each cluster, we create a specific security group and only allow traffic to happen from EC2 instances within the same cluster group.

I hope this small how-to will help you, and feel free to post comments about alternatives you may have identified and or shortcomings we may have missed with our solution.

Last friday I presented Elastic Grid to the local Java Open Source developers group called OSSGTP. The audience of this group usually is made of skilled Java developers working on many famous Java projects, such as Spring, Hibernate, XWiki, eXo Portal, Restlet, jGuard, JCapthcha, etc (sorry, I probably forgot to cite many of them…).

This talk lasted 2 hours and half and was really interesting because of its format: this session was highly interactive and I was asked many questions about AWS and Elastic Grid.

The content of the talk was pretty much the same as the one Dennis and I did at JavaOne, except that I added two other demonstrations illustrating what EG (Elastic Grid) brings to the developers for ease of deployment of JEE applications. The first demonstration illustrated a deployment of XWiki whereas the second one was focused on eXo Portal.

For both demonstrations, I showed how EG is actively monitoring the processes it started when deploying the applications/servers. I simulated some crashes by killing some applications servers and could explain how the application servers were handled.

The feedback I received the day after, such as the blog post from Nicolas Martignole (in French only, sorry…), was really great and I truly thank all the participants.

Many news related to the Elastic Grid project should come soon, so stay tuned :=)

The JiBX plugin for IntelliJ IDEA has been updated for both IDEA 7.x and the latest EAP release (future 8.x release).

This update upgrades the embedded JiBX distribution to the latest release, that is version 1.1.6a.

The plugin for Amazon EC2 has been updated in order to:

• add support for the new instance types (High-CPU Medium and High-CPU Extra Large),
• update the Typica library.