The slides of the BoF on Elastic Grid and EC2 are finally available!

Thanks for all of you who could come. We had some interesting discussions and feedback after the talk.
This is the first time I had a chance to go to JavaZone and I must say it’s been quite interesting. A few friends and I had the opportunity to discuss with Tom J. Bang who gave us the opportunity to taste local stuff.

I also had the opportunity to discuss with a lot of people there: my friends from our local user group (Guillaume Laforge, Julien Viet, Alexis Moussine-Pouchkine, Jerome Lacoste) but also Scott Davis and Eitan Suez. (Sorry for all the other ones I forgot to cite.)

For those of you who could not make it for our BoF session, the organizers of the JavaZone event will provide a video. I’ll update the blog when the video is available.

This milestone release is significant, and provides important enhancements and bug fixes. An overview of some of the enhancements follow:

External Service execution
The Rio execution framework provides the ability to encapsulate the control and monitoring of external services. Service control adapters represent applications/services, adding network wide visibility and control. Using this approach, we can attach monitoring, metering and SLA control to existing applications. You can have a look at the tutorial here:
http://www.rio-project.org/confluence/display/rug/Service+Exec+Introduction

Data staging
Provides the ability to download artifacts that a service needs to process. More information can be obtained here:
http://www.rio-project.org/confluence/display/rug/Data+Staging+and+Persistent+Provisioning

ServiceBean fork & exec
The ability to instantiate a service in it’s own JVM. This option provides the flexibility to create a new JVM for a service that is best executed in it’s own JVM, instead of contained withint the CYbernode in a multi-service JVM

Groovy integration
The parsing of the OperationalString is quite complex. Adding groovy to make this part of Rio more maintainable is a key contribution. Based on this ability we will also be creating a Groovy Domain Specific Language (DSL). You can view examples both in the distribution and online here:
http://www.rio-project.org/confluence/display/rug/Instantiation+Properties

Package name change
We are moving from the org.jini.rio namespace to the org.rioproject namespace. We realize this is a big change, but it is a needed change. The current package namespace has Rio under jini.org. This namespace is no longer relevant

License change
We have moved to LGPLv3

UI enhancements
A number of enhancements have been added to the new Rio UI, including better visualization of contained services, Watch Viewer value displays and deployment graph orientation

Service selection strategies
All association proxies have a service selection strategy. The service selection strategy provides a way to determine how services in the collection of discovered services are invoked. The current service selection strategies are fail-over, round-robin and utilization (note all service selection strategies will also prefer local services over remote services).

The release notes are available on Jira:
http://www.rio-project.org/jira/secure/ReleaseNote.jspa?projectId=10000&styleName=Html&version=10010

Please note also that some work has been started in the Wiki:
http://www.rio-project.org/confluence

We welcome you to download and try the 4.0 M1 release. All feedback is  welcome.

That XML format, although quite extensible is also quite verbose, so a Domain Specific Language (DSL for short), means a lot of sense in that context in order to reduce verbosity.

In order to ease the code for such a DSL, Rio deployment descriptors (called OpStrings) parsing was rewritten using Groovy. This is something which turned out to be quite easy actually and is mostly done. I spent about 4 or 5 days on this task, including writing some Groovy Tests in order to reduce the risk of regression. This work is interesting because I was able to at least cut the code to half its initial size and it is now more easy to read, hence maintain.

Armed with this test suite, I started to work on the DSL. The option I have taken is to transform the DSL syntax into the current XML file and then use the usual XML parser. The trade off is that it minimize the work needed to be done but of course it would be better to use the DSL in order to create the Java model already using when parsing the XML format (this is still being discussed).

So here is how it looks like. Below is the XML format taken from one of Rio examples:
[sourcecode language='xml']





rio

calculator/lib/calculator.jar

calculator/lib/calculator-dl.jar

calculator.Calculator

1

calculator.Add





1

calculator.Subtract





1

calculator.Multiply





1

calculator.Divide





1

[/sourcecode]

Here is now the DSL in action for the same example:
[sourcecode language='java']
opstring(name:’Calculator’) {
groups(‘rio’)

resources(id: ‘impl.jars’, ‘calculator/lib/calculator.jar’)
resources(id: ‘client.jars’, ‘calculator/lib/calculator-dl.jar’)

service(name: ‘Calculator’) {
interfaces {
classes(‘calculator.Calculator’)
resources(ref: ‘client.jars’)
}
implementation(class: ‘calculator.service.CalculatorImpl’) {
resources(ref: ‘impl.jars’)
}
associations {
association(name: ‘Add’, type: ‘requires’, property: ‘add’)
association(name: ‘Subtract’, type: ‘requires’, property: ’subtract’)
association(name: ‘Multiply’, type: ‘requires’, property: ‘multiply’)
association(name: ‘Divide’, type: ‘requires’, property: ‘divide’)
}
maintain 1
}

service(name: ‘Add’) {
interfaces {
classes(‘calculator.Add’)
resources(ref: ‘client.jars’)
}
implementation(class: ‘calculator.service.AddImpl’) {
resources(ref: ‘impl.jars’)
}
maintain 1
}

service(name: ‘Subtract’) {
interfaces {
classes(‘calculator.Subtract’)
resources(ref: ‘client.jars’)
}
implementation(class: ‘calculator.service.SubtractImpl’) {
resources(ref: ‘impl.jars’)
}
maintain 1
}

service(name: ‘Multiply’) {
interfaces {
classes(‘calculator.Multiply’)
resources(ref: ‘client.jars’)
}
implementation(class: ‘calculator.service.MultiplyImpl’) {
resources(ref: ‘impl.jars’)
}
maintain 1
}

service(name: ‘Divide’) {
interfaces {
classes(‘calculator.Divide’)
resources(ref: ‘client.jars’)
}
implementation(class: ‘calculator.service.DivideImpl’) {
resources(ref: ‘impl.jars’)
}
maintain 1
}
}
[/sourcecode]

What’s interesting is that if you have a close look at the above syntax, you will notice that the Add, Sutract, Divide and Multiply services are configured quite the same. Wouldn’t it be nicer to define the configuration once for all those services?

This is something which is accomplished for free because the DSL is a Groovy script, so any Groovy syntax is valid in it:
[sourcecode language='java']
opstring(name:’Calculator’) {
groups ‘rio’

resources id:’impl.jars’, ‘calculator/lib/calculator.jar’
resources id:’client.jars’, ‘calculator/lib/calculator-dl.jar’

service(name: ‘Calculator’) {
interfaces {
classes ‘calculator.Calculator’
resources ref:’client.jars’
}
implementation(class:’calculator.service.CalculatorImpl’) {
resources ref:’impl.jars’
}
associations {
association name:’Add’, type:’requires’, property:’add’
association name:’Subtract’, type:’requires’, property:’subtract’
association name:’Multiply’, type:’requires’, property:’multiply’
association name:’Divide’, type:’requires’, property:’divide’
}
maintain 1
}

['Add', 'Subtract', 'Multiply', 'Divide'].each { s ->
println “Service is $s”
service(name: s) {
interfaces {
classes “calculator.$s”
resources ref:’client.jars’
}
implementation(class: “calculator.service.${s}Impl”) {
resources ref:’impl.jars’
}
maintain 1
}
}
}
[/sourcecode]

Our OpString file has now been reduced from 71 lines to 34 ones!

The only drawback in that case is that using the DSL you loose the autocompletion provided by any good XML editor.

So how does the code look like for that DSL:
[sourcecode language='java']
class GroovyDSLOpStringParser implements OpStringParser {
def OpStringParser xmlParser = new XmlOpStringParser()
def static final Logger logger = Logger.getLogger(GroovyDSLOpStringParser.class.name);

public List parse(Object source, ClassLoader loader, boolean verify, String[] defaultExportJars,
String codebaseOverride, String[] defaultGroups, boolean processingOverrides,
Object loadPath) {
logger.info “Parsing source $source”
ExpandoMetaClass.enableGlobally()

def tempFile = File.createTempFile(‘rio-dsl’, ‘xml’)
def writer = new FileWriter(tempFile)
def builder = new groovy.xml.MarkupBuilder(writer)
Script dslScript = new GroovyShell().parse(source)

dslScript.metaClass = createEMC(dslScript.class, {
ExpandoMetaClass emc ->
emc.opstring = { Map attributes, Closure cl ->
builder.printer.println(‘‘)
builder.opstring {
OperationalString(Name: attributes.name) { cl() }
}
writer.close()
xmlParser.parse(tempFile, loader, verify, defaultExportJars, codebaseOverride, defaultGroups,
processingOverrides, loadPath)
}
emc.groups = { String… groups ->
builder.Groups {
groups.each { Group(it) }
}
}
emc.cluster = { String… machines ->
builder.Cluster {
machines.each { Machine(it) }
}
}
emc.service = { Map attributes, Closure cl ->
builder.ServiceBean(Name: attributes.name) { cl() }
}
emc.serviceExec = { Map attributes, Closure cl ->
builder.ServiceExec(Name: attributes.name) { cl() }
}
emc.spring = { Map attributes, Closure cl ->
builder.SpringBean(Name: attributes.name, config: attributes.config) { cl() }
}
emc.interfaces = { Closure cl ->
builder.Interfaces { cl() }
}
emc.implementation = { Map attributes, Closure cl ->
builder.ImplementationClass(Name: attributes.class) { cl() }
}
emc.execute = { Map attributes ->
builder.Exec(nohup: attributes.nohup ? ‘yes’ : ‘no’) {
if (attributes.inDirectory)
WorkingDirectory(attributes.inDirectory)
def String[] cmd = attributes.command.split()
CommandLine(cmd[0])
if (cmd.size() – 1 > 0)
cmd[1..cmd.size() - 1].each { InputArg(it) }
}
}
emc.classes = { String… interfaceClasses ->
interfaceClasses.each { builder.Interface(it) }
}
emc.resources = { String… resources ->
builder.Resources {
resources.each { JAR(it) }
}
}
emc.resources = { Map attributes ->
builder.Resources(attributes)
}
emc.resources = { Map attributes, String… resources ->
builder.Resources(id: attributes.id) {
resources.each { JAR(it) }
}
}
emc.configuration = { String configuration ->
builder.Configuration(configuration)
}
emc.associations = { Closure cl ->
builder.Associations { cl() }
}
emc.association = { Map attributes ->
builder.Association(Name: attributes.name, Type: attributes.type,
Property: attributes.property, MatchOnName: attributes.matchOnName ? ‘yes’ : ‘no’)
}
emc.maintain = { Integer maintain ->
builder.Maintain(maintain)
}
emc.maxPerMachine = { Integer max ->
builder.MaxPerMachine(max)
}
emc.maxPerMachine = { Map attributes, Integer max ->
if (attributes.type)
builder.MaxPerMachine(max, type: attributes.type)
else
builder.MaxPerMachine(max)
}
emc.software = { Map attributes ->
builder.SystemRequirements {
SystemComponent(Name: ‘SoftwareSupport’) {
Attribute(Name: ‘Name’, Value: attributes.name)
Attribute(Name: ‘Version’, Value: attributes.version)
}
}
}
emc.software = { Map attributes, Closure cl ->
builder.SystemRequirements {
SystemComponent(Name: ‘SoftwareSupport’) {
Attribute(Name: ‘Name’, Value: attributes.name)
Attribute(Name: ‘Version’, Value: attributes.version)
builder.SoftwareLoad(removeOnDestroy: attributes.removeOnDestroy ? ‘yes’ : ‘no’) { cl() }
}
}
}
emc.download = { Map attributes ->
builder.Download(InstallRoot: attributes.installRoot,
Unarchive: attributes.unarchive ? ‘yes’ : ‘no’, Source: ”) {
Location(attributes.source)
}
}
emc.postInstall = { Map attributes, Closure cl ->
builder.PostInstall(RemoveOnCompletion: attributes.removeOnCompletion ? ‘yes’: ‘no’) { cl() }
}
emc.serviceLevelAgreements = { Closure cl ->
builder.ServiceLevelAgreements { cl() }
}
emc.sla = { Map attributes, Closure cl ->
builder.SLA(ID: attributes.id, Low: attributes.low, High: attributes.high) { cl() }
}
emc.policy = { Map attributes ->
builder.PolicyHandler(type: attributes.type, max: attributes.max,
lowerDampener: attributes.lowerDampener, upperDampener: attributes.upperDampener)
}
emc.logging = { Closure cl ->
builder.Logging { cl() }
}
emc.logger = { String name, Level level = Level.INFO, Closure cl ->
builder.Logger(Name: name, Level:level.toString()) { cl() }
}
emc.handler = { String name, Level level = Level.INFO ->
builder.Handler(ClassName: name, Level:level.toString())
}
})
List opstrings = dslScript.run()
return opstrings
}

public parseElement(Object element, GlobalAttrs global, ParsedService sDescriptor, OpString opString) {
throw new UnsupportedOperationException()
}

static ExpandoMetaClass createEMC(Class clazz, Closure cl) {
ExpandoMetaClass emc = new ExpandoMetaClass(clazz, false)
cl(emc)
emc.initialize()
return emc
}

}
[/sourcecode]

As you can see, the above code uses the XML MarkupBuilder which makes it very easy to generate the XML format. After that, it’s just a matter of delegating the real work to the XML parser!

I will arrive at San Francisco on April, 30th and leave by May, 13th.

If you’d like to talk to me when I’m there, feel free to drop me an email at jerome DOT bernard AT elastic-grid DOT com.

I would be pleased talking with you about Amazon services, Rio and Jini, or whatever else…

Here is on the left a screenshot of it in action…

Dennis Reedy and I have been selected as speakers for JavaOne 08.

Here is the abstract of our session called How can Amazon EC2 benefic from the Elastic Grid solution?

Amazon Elastic Compute Cloud (EC2) provides a fantastic way to deploy scalable machine images, but what to do when you want an application to scale across the machine images you have provisioned? This session discusses the Elastic Grid, an approach that provides dynamic allocation, management, and scalability of applications, using Amazon EC2 as the backbone. It also introduces the open-source technologies Elastic Grid is based on: Rio and Apache River (Jini™ network technology).The Elastic Grid provides an architecture for developing, deploying, and managing distributed applications composed of services. Key to the architecture are a set of dynamic capabilities and reliance on policy-based and quality-of-service mechanisms. The Elastic Grid reduces the complexity surrounding the development of dynamic services by introducing Jini network technology remoting for POJOs as well as by providing a simple component model.

The Elastic Grid extends Amazon EC2’s virtual grid environment, enabling users to manage and dynamically scale Amazon Machine Images (AMIs) based on declarable SLAs, as well as deal with partial failure of AMI instances. Rio reduces the complexity surrounding the development of dynamic services by introducing dynamic Jini network technology remoting for POJOs, as well as providing a simple component model.

The presentation demonstrates how IntelliJ plug-ins for Amazon EC2 and Rio ease building and deploying a sample application distributed over the Amazon EC2 grid. With the Elastic Grid solution, the application will scale on the Amazon EC2 grid by starting and stopping Amazon EC2 instances accordingly to declared SLAs (service-level agreements).

We hope to see you there!

Basically this plugin allows to deploy new services, undeploy them and visually watch the state of the deployments. You can also launch administration UI of the services, which also mean increasing/trimming the number of instances of your services.

There is a screenshot of this plugin in the dedicated page.

The test we did was quite simple: one instance of our Amazon image (let’s call it server1) ran Rio (rio start all), whereas another instance (let’s call it server2) ran CLI (Rio command-line interface). Multicast being disabled between Amazon EC2 instances, we had to use a Unicast lookup. This can be done in CLI with the following command:
set locators jini://PRIVATE_DNS_NAME_OF_RIO_SERVER_1

We were able to list from server2 the services running on server1. CLI uses the default ServiceDiscoveryManager settings, so when we stop Rio on server1, we had to wait 10 minutes before the services disappeared.

So, do not forget to customize your Rio/Jini environment when using Amazon EC2.

We will soon provide a public AMI, with a customized Rio distribution tuned for Amazon EC2.

Update: there is now a tutorial for running Rio on Amazon EC2 and Rio AMI!