Executable Wars with Gradle and Jetty

One of the things I recently wanted to do was create a set of Java based utility components that could be easily packaged (aka one delivery file), run together, and that all leveraged assets created inside a webapp.  Normally this would involve creating a 'fat jar', which takes all exisitng library classes and flattening them into the fat jar.

The upside of this is that you don't need any special class loaders, all the classes required by the application are now packaged directly in the jar.  The downside is that anything that used to live in META-INF folders in the third party libraries now get clobbered together in a single META-INF.  Of course, anything that used to rub up against servlet APIs, web aplication contexts and the what not will also seemingly break.

After poking around for a bit, the executable war file seemed like the way to go since it avoided some of these pitfalls and has the following benefits:

• All third party jars can be packaged in a WEB-INF/lib
• Solid and true jetty-6 provides a stable foundation for running a quick embedded container to run the war (i.e. itself)
• All code written for a webapp can be immediately consumed
• Remotability for the tool is immediately available
• Once you open this pandora's box, wild-eyed ideas sprout up like the first executable war could take a list of wars as a command line arguments and deploy them all in itself.. "it's war files all the way down!"

Riding the Gradle band wagon, I wanted to try doing this in straight up Gradle script without re-using existing Ant tasks.  The following was done with Gradle 1.0-rc5


Some notes on this evening's experiment:

• Jetty 8 has an 'orbit' file that Gradle doesn't yet handle gracefully.  There were some workarounds online, but I wanted this at a one hour research task so jetty 6 it was..
• Tomcat 7 has a simple API for instantiating and running an embedded Tomcat, I just haven't gotten around to trying that out yet
• "Gradle as Jetty Runner", "Gradle as Tomcat Runner", or plain old Groovy command lines are all valid options for doing the same thing, but in this case I wanted 1 file, 1 command
• After so many years of Maven relaxation, it was kinda fun having direct control over build configuration primitives in the build file again and be able to use them in simple one liners

This already has me thinking of another little experiment -- self deployable agent wars using the above + MBeans, but that'll be for another night!

Configuration Automation with Gradle

For a while I've been following the Gradle ecosystem, seeing it grow by leaps and bounds.  As a build system and automation platform, Gradle provides a strong value proposition around a pluggable lifecycle model combined with easy task definition and the Groovy programming language.  One of the areas where Java still has some pain points is in configuration management.  You can bootstrap your development environment easily enough with Eclipse template projects, Grails, Spring Roo, Maven Archetypes and the like.  However, what about your deployment environment?  With the large number of Java app servers, message brokers, cache servers, and other interesting things being developed - an automation system around continuous deployment looks like the next logical step.  Gradle is positioned to take that step in my opinion.  What follows is a collision of ideas - Gradle, continuous integration, continuous deployment, cloud computing, and where we can find the next evolution in Java automation.

Today this space is largely filled by solutions like Puppet, Chef, and a handful of other tools that tackle server administration automation generically - usually following a concept of cookbooks and repeatable dependency management for the platform.  While they do support the various Java environments (Tomcat, ActiveMQ, etc), the lack of pure Java integration in the automation stack means you cannot exploit Java's capabilities directly without jumping through an interop layer.  Here are some example ideas on what it would be nice to do:

•  Have a configuration management automation system that integrated with JMX, feeding information back to a management console
•  Share or re-use Java assets in the automation workflow - e.g. using your Spring Batch beans as part of automating the setup of your database
•  Leverage Java APIs in the automation system to distribute capabilities - e.g. start tomcat, start activeMQ, generate test JMS messages to validate connectivity, or perhaps use JMX to interrogate server status to validate sanity
•  Build configuration artifacts shared directly into integration and production server environments (e.g. properties files, Spring bean files, etc)
•  Provide a platform for next-generation Java platforms (OSGi, Cloud, etc)

The more I think about this, the more it makes sense to introduce some Gradle plugins that expand current task models into the continuous deployment and configuration management space.  Here is a sample list of tasks that could be executed in a Gradle build 

1.  Compile, test, package
2.  Jetty/tomcat integration tests, bootstrapping their configuration
3.  Deploy to Tomcat cluster, update local config - updating the software artifacts with local software config (e.g. hostname) as necessary
4.  Validate tomcat cluster sanity
5.  Initialize database - not with bash scripts running SQL commands, but Groovy SQL, your data access layer jar being invoked, etc

Steps 1 and 2 are what you do today with Gradle in your own dev environment.  When you think about steps 3-5 in the various environments out there - from your home grown environments, to larger app servers, to virtual machines, a task framework around server management within Gradle looks more and more attractive. 

Stardog and Spring Framework

Last week, Clark&Parsia released an initial integration between Stardog and Spring. To quote the Stardog site, Stardog is a commercial RDF database: insanely fast SPARQL query, transactions, and world-class OWL reasoning support. Of course, Spring provides a leading technology stack for rapid development of Java applications. Almost all projects support Spring integration in one form or another - with the exception of the Semantic Web technology stacks. So, working with C&P, we came up with an initial integration of Stardog and Spring.

Stardog-Spring 0.0.1 provides the initial groundwork for Spring developers to get started with Stardog, and in general, Semantic Web technology. Over time, the Stardog Spring integration will be expanded to support some of the larger enterprise capabilities from Spring, such as Spring Batch. Stardog-Spring is open source, available on Github, and licensed under the Apache 2.0 license.

For 0.0.1, there are three fundamental capabilities:

1. DataSouce and DataSourceFactoryBean for managing Stardog connections
2. SnarlTemplate for transaction- and connection-pool safe Stardog programming
3. DataImporter for easy bootstrapping of input data into Stardog

The implementations follow the standard design patterns used across the Spring Framework, so if you are familiar with JdbcTemplate, JmsTemplate, etc you will be right at home with the SnarlTemplate.  The SnarlTemplate provides interface callbacks for querying, adding, and removing data - abstracting away the boilerplate connection handling and transaction handling for you.  Likewise, the DataSource and the FactoryBean look and feel very much like SQL dataSource's and factory beans within Spring.  

You can read the documention here and get the source here.  There is also a downloadable jar from Github as well.

This implementation was built with Gradle, and you need edit the build.gradle file to point at your Stardog release for it to build.  Of course Stardog-Spring works well with Spring Jena and Groovy SPARQL.

Last but not least, you will have to sign up with the Stardog testers to get the current version.  Eventually there will be a community-style edition and enterprise style edition of Stardog.

Linked Data Microframework: Linked Ratpack

The other day I ran across some of the Sinatra inspired web microframeworks available in various languages, including Ratpack for Groovy.  Given RDF builder DSL in Groovy Sparql, I thought it would be a nice thought expertiment to create a microframework for linked data and RDF.  After an afternoon of coding and testing, the results look quite promising.  So here it is - Linked Ratpack, a microframework for Linked Data.

Linked RP works the same way Ratpack does - you provide a single Domain Specific Language (DSL) script where you write your methods to perform some function on a URL, and it weaves those in to a Jetty container.  In this case, I've added some capabilities to Ratpack to work with linked data:

• RDFBuilder from Groovy SPARQL is automatically available to the DSL script under the 'rdf' variable
• link(String endpoint) is available as a function to get an instance of the Groovy SPARQL Sparql class for performing Sparql queries.
• resolve(String uri) is a new piece of functionality that uses Groovy's HTTPBuilder DSL and Jena to retrieve a URL and read it into RDF.  It should work across various RDF serialization types, and likely bomb out on HTML or anything else if you feed it an incorrect URI

The following Gist illustrates everything fairly nicely:



You can now browse to the following URLs:

• localhost:4999/
• localhost:4999/tim
• localhost:4999/groovy

Note: since Jena models being returned by those functions get automatically serialized back out - if you want to do serialization inline - return null

To get started with Linked Ratpack, you must do the following:

1. Get Groovy SPARQL from Github, and build/install it with Gradle
2. Get Linked Ratpack from Github, and build it
3. Create simple groovy scripts, like the above gist, and run "ratpack path/to/whatever.groovy"

This will start an HTTP server on whatever port you define in the DSL.  After that, you can start browsing to your URLs, hooking up SPARQL endpoint and generating RDF.

For me, this is one of the missing pieces in building linked data applications - an easy way to stand up little RDF servers to test walking RDF graphs hop-by-hop and perform URI de-referencing, and experimenting with generating derivative RDF sites from other RDF data sources (e.g. SPARQL Construct).

Many thanks to Justin Voss ( @ github ) for creating Ratpack in the first place, it was a solid foundation to build off of.

Enjoy!

Groovy SPARQL 0.2 Available

Version 0.2 of Groovy SPARQL is now available.  This minor release includes a Groovy DSL for RDF, now you can build RDF and then query it.  The Groovy DSL is fairly flexible and takes advantage of a number of Groovy features including:

1. Optional syntax in Groovy 1.8 for more fluid DSLs
2. GPars, aka the Groovy Parallelizer for asynchronous output hooks
3. Usage of the BuilderSupport class

 Per previous posts on the blog, if you want to use it in Grails / GroovyConsole / other apps, I recommend downloading, doing the Gradle build, install into your local Maven repo and then you can include it easily enough in whatever build environment you are using.

Here is a GIST showing the RDFBuilder DSL in action, with comments noting all of the 'features' available so far.  This is still a work in progress, and the more I attempt to use it to build FOAF and other vocabularies, I'm sure I'll be shaking some bugs out (not the least of which is the wonderful world of URI fragments).



Enjoy!

Gradle, Maven, and Grapes Working Together

For Groovy SPARQL and Spring Jena, I wanted to start leveraging these in little test Groovy scripts running in the console or command line.  At first, I assumed the maven install that happens in their Gradle builds would immediately be picked up by Grape.  However, Grape uses Ant+Ivy, and Ivy does not look in your maven repo by default (doesn't it seem like it should?).

So here are the missing pieces of the puzzle:

1. Setup your Gradle build to create a POM and install your jars into your local maven repo
2. Add  Maven repo support to your Grape configuration.  Grape configuration is in ~/.groovy/grapeConfig.xml - and it's an Ivy file in disguise.  See [*] below for an example which is down near the bottom of the Grape documentation.
3. Install your POM artifacts into Grape.  For Groovy SPARQL, the command is:    grape install org.codehaus.groovy.sparql groovy-sparql 0.1
4. Now you can use Grape, e.g. @Grab('org.codehaus.groovy.sparql:groovy-sparql:0.1')

You can also use grape list to see what jars are now available.

All in all, this makes tools like the Groovy Console an excellent REPL for both Java, Groovy, and presumably other Java polyglot programming.

 * Here is the sample grapeConfig.xml file from the Groovy documentation.

Announcing Spring Jena

On the heels of Groovy SPARQL, here is an initial code base for standard Java and Spring applications -- Spring Jena!

The Spring folks have been putting together an impressive portfolio of data oriented capabilities for NoSQL data stores.  To compliment the those capabilities, here is Spring Jena - a project I hope to propose back to the Spring community to provide direct Jena API support and direct SPARQL.

Much like Groovy SPARQL, this is a relatively simple code base that applies the template design pattern to Jena and ARQ to simplify every day needs for creating, modifying, and querying RDF data.  There is a lot more work to do here, most noteably the parameterized queries.

Get Spring Jena @ Github here.

The roadmap includes:

• Spring datastore/mapping support for object relational mapping, once those projects reach 1.0
• Spring Transaction support - wrap Jena native transactions or provide app-level transaction management via Spring
• Abstraction for triple stores - likely aligned against the Datastore interface in Spring Data
• QuerySolutionMap overloading to the methods in the SparqlTemplate
• Web / MVC capabilities, such as a taglib

Here is a GIST to get you going:



Enjoy!