The lite at the end of webservice tunnels

In the last post, I alluded to how shifting complexity from one area to another can actually improve perceived simplicity. Here is a concrete example of how we got rid of scores of wsdl-client generated jaxb classes and brought in a fresh wave of simplicity to our Grails applications.

Problem

One or more of our Grails application uses several our own plugins and each of them talk to several web services that share a lot of xsd schemas, the java client generation of which end up proliferating duplicated jaxb classes in each of our plugins and also in the applications which use them, thus resulting in multiple classpath conflicts.

Also we dont like the date conversions from Date to Calendar to GregorianCalendar to XmlGregorianCalendar or other conversions from Byte to Int to Shorts. Also jaxb does some non-intuitive magic with some of the xsd:any types in the schemas. If xsd schemas want strict type checking, fine. We from the “def” world, just look everything as plain data. And finally the jaxb classes look ugly.

Solution

With a combination of groovy-wslite, grails wslite plugin and MarkupBuilder and Config, the code become pretty straight-forward, almost.

From the grails command prompt, run install-plugin wslite

def grailsApplication

public def amazonBookListMania() {
  WsliteConnector.instance.withSoap(serviceURL: grailsApplication.config.amazonBookListManiaServiceUrl) {
    def soapResponse = send(SOAPAction: grailsApplication.config.amazonBookListManiaSoapAction) {
      //Get the namespaces, which is a simple Groovy map
      //['xmlns:amz':'http://www.amazon.com/ListMania']
      envelopeAttributes grailsApplication.config.amazonBookListManiaNamespaces
      body {
        //Get the raw xml and slap it into soap body
        mkp.yieldUnescaped(createXmlRequest())
      }
    }
    //Wslite automatically xmlslurps the result and gives you the closure
    def response = soapResponse.body.bookListManiaResponseRoot
    println response.Items.count()
  }
}

The “withSoap” is a DSL, that is supposed to be injected automatically into controllers, but on other classes it didnt seem to work. I tried using WsliteEnhancer, but that didnt help either. So the direct call to withSoap using WsliteConnector singleton.

The example shows all params injected from grailsApplication config, to allow webservice urls for different environments.

The interesting aspect is the envelopeAttributes, which is a map of namespaces. Since we are not using generated client classes, whcih does the work for us, the namespace prefixes must be specified somehow in the xml itself as part of soap request. Adding namespaces to envelopeAttributes will append the namespaces to the soap-env tag. No need to include the namespace for soap-env itself. The wslite plugin does that.

public String createXmlRequest() {
    def writer = new StringWriter()
    def builder = new MarkupBuilder(writer)
    builder.'amz:ListMania' {
      //tag attributes
      'amz:credentials'(username: 'vasya10', password: 'yeahright')
      //tag value
      'amz:dateRequested'(new SimpleDateFormat('yyyy/MM/dd').format(new Date()))
    }
    //Returns <amz:ListMania><amz:credentials username='vasya10' password='yeahright'></amz:credentials><amz:dateRequested>2013/01/01</amz:dateRequested></amz:ListMania>
    return writer.toString()
}

With the complexity of creating the xml shifted into MarkupBuilder, creating soap requests become trivial, even more so with the withSoap DSL.

Wsdl import generated classes
Pros
No need to worry about namespaces
Set and Get of values use Java-like syntax
Setting values does not matter as xml marshalling happens at a later time than setting the values
Setting tag attribute or tag value is no different (ie code doesn’t care)

Cons
Every minor change in wsdl forces wsdl client generation
Duplicated client generated classes could cause classpath conflicts
The hierarchy of the xml elements is not apparent in the code, the only way to know the hierarchy is to look at the xml samples

WSLite + MarkupBuilder
Pros
No confusion about which client to use: there are scores of them and each have some unique issue (jax-ws, jax-b, cxf and so on)
No wsdl import client generated sources and jars
No crazy xerces, xalan and other marshalling runtime errors, especially if deploying to different app servers (tomcat, weblogic etc)
If there are trivial changes to elements (saying adding a new optional element), will still work without changing client code (if that element is not used)
Creation of xml is very straight-forward
Code documents the hierarchy of the xml!
Response returns XmlSlurped data, so dot-notation can directly be applied on the objects

Cons
Code must be namespace-prefix aware
Must know tag attributes and tag values to create the xml correctly (this must be a Pro)
Code must generate xml in the same sequence of elements; blocks of elements can still be called out-of-order, but the final xml body must be created in a sequence
Unvalidated namespace prefixes increases testing time
Extra testing time for un-type-checked variables

Compared to our previous jaxb code (with ObjectFactory, creation of new Objects etc.), the code is now 40% smaller, and additionally got rid of the client sources too.

The motorcycle diaries of webservices

Software is a big grandpa pendulum clock. It oscillates between simplicity and complexity perpetually. Ideas and concepts that start off simple, rapidly become complex and eventually complex ideas are broken down into simple components and again these simple components evolve into complex creatures of their own. After all we evolved from a single cell into complex cell living beings, why blame the software for that?

Webservices necessitated from complex interaction between disparate systems (remember CORBA, JNI, DCOM, RPC, mainframe apis?). All those were dissolved into ‘simple’ wsdl based services, as long as client and server maintain the contract, everything was promised to be a simple hookup.

But it didnt end there. Generating wsdl clients either in Java or .NET were complex in its own terms. Java discovered its way, as usual, with hundreds of wsdl clients and xml jars (anyone still using xerces, xalan, sax4j directly?) – jaxb, jax-rs, jws, cxf and several more proprietary ones). .Net, was not a A-shot either: first asmx, then wcf and then several flavors of .net versions on top of that. Most of the time, its not a one-step process to call a WCF wsdl from a Java based client, there are always some surprises around the corner in terms of configurations. Its heartening to see nobody got it right the first time.

If you think about it, what was the necessity to generate classes from wsdl in order to send plain xml text data? The early xml apis described the xml in terms of hardcore hierarchy: root, nodes, parent, children, siblings, single. It could have been easily extended to represent a oh-so loveable joint-family: cousins, uncles, aunts, nephews and nieces:

public getAuntyNode() {
return currentNode.getParent().getFather().getSister().toNode();
}

public getDistantUncleNode() {
return currentNode.getParentNode().getMother().getSisterInLaw().getHusband().getSibling().toNode()
}

public getImMyOwnGrandpaNode() {
//Outsourced
}

Such family friendly APIs would have thrilled developers who often work only after the family goes to sleep.

I am also starting to see that xml and xsd schemas are not an ideal way to describe data models. Describing a model in xsd using the semantics of xml itself is over complicated. On top of that extensions, simpleTypes, complexType – are way too tedious and garrulous. If you are exposed to C++/Ruby/Groovy world, you can visualize the models pretty quickly in terms of encapsulations, compositions and interfaces. In the xml world, I only see tags and attribute names. Lots and lots of them. The content is lost in the tag-tsunami.

While reviewing a project recently, where all our models are xsd based, data models extending from base xsds, and several simple, complex and xsd:any types, pretty soon I found them going above my head. Im not even thinking how xsds maintain versions of models and references.

Coming to actually using them in our Grails project, a colleague of mine tried to generate the client classes using jaxb (before trying a few other clients) and pretty soon found himself in a maze of conflicting classpaths from different generated wsdls. Jaxb generates classes for all the dependent xsds (obviously) and generating them for different wsdls (in the same project) ended up with several classpath issues.

Taking a break, we decided to ask a fundamental question: “Why do we have to worry about all the xml types in the Grails code, if all we need is to send a plain textual xml data?”.

Who cares what the model hierarchy of xsd is? Most of the time that model is useless to the Grails code, where typically the representation of model is different from the xsd models. All those xsd generated classes, objects, factories, abstract factories were required because the complex familial Java APIs were the only way to process xml (unless using StringBuffer and appending data manually).

Enter Groovy’s MarkupBuilder, it is much easier to build an xml than generating the client. So if we all care is to send data and generating that data itself is easier than generating clients, why bother at all with the later. And we switched to Grails wslite plugin, with which we are directly creating soap requests and sending it to server.

So thats how the pendulum switched back to its other position, back to creating hand made soaps. Eventually this may get complex too. But again, the important thing to remember is that the original complexity is not the same level of complexity of the broken down components. The “absolute complexity” remains unaltered and the “perceived complexity” depends on how lazy or smart we are.

Extensions via metaclass in Grails

Even after several releases and wrapper utilities, representation of time in Java seems to be an unconquerable mess that it originally was. At the same time, surprisingly, web services have been there for a long time now, still there is no magic wand to create webservices and clients.

Webservices is something Groovy/Grails should have made it ridiculously easy to work with, by default. When xml, json, http and so on have been outrageously simplified, the natural expectation is that the webservice call would be like:

WebServiceObject ws = new WebServiceSlurper().parse('http://someweathersite.com/weather.wsdl')
WeatherObject w = ws.getWeatherObject()
Temperature[] temperatures = w.findTemperature(zipcode: '11505', days: 5, unit: 'farenheit')

There are so many wrapper api-s, every webservice client generation is lacking in someway, making a new comer to Grails to try and find what works. axis, axis2, groovyws, xfire, cxf, jws, jaxb – there are so many acronyms and some of them are just wrappers over others, the choice may be overwhelming. Forums do provide lot of answers, but most times what works for one goose, does not work for even for its goose twin.

So after my friend pointed out the pot-holes of webservices in Grails, I started using the jws. All good until I hit the xs:date object in our wsdl, which translates to XMLGregorianCalendar. Like we all wanted a new class to represent date and time. I seriously hope Java/Groovy will deprecate and delete all its Date and Calendar classes and introduce just one class with a philosophical outlook:

/**
* Dead yesterday, unborn tomorrow, why fret about them, if today be sweet?
*/
class StarDate{
public double day;
public double increment(double unit) { day += unit }
public String toString() { return "Captain's log, stardate $day"  }
}

Conversion between a date in String to XMLGregorianCalendar goes like this:

SimpleDateFormat sdf = new SimpleDateFormat("MM/dd/yyyy")
Date d = sdf.parse(dateString)
GregorianCalendar g = new GregorianCalendar()
g.setTime(d)
def date = DatatypeFactory.newInstance().newXMLGregorianCalendar(g)

So much for a date? I tried to generate the client classes with a custom binding to Date instead of XMLGregorianCalendar. No matter what I tried, the value inside the webservice always ended up as long format “Sun Dec 4 …”. Fed up, I changed it back to XMLGregorianCalendar and now I saw the date as 1995-03-28T06:00:00. So where did the timezone come from and how do I avoid it? Stackoverflow had an answer:

dob.setTimezone(DatatypeConstants.FIELD_UNDEFINED)

Seriously, DatatypeConstants? Minimum integer value to specify undefined? Anyway, with that undefining of the timezone, the webservice was correctly seeing the date as just 1995-03-28. But I still dont like the fact that I had to convert this in the service class or that I had to put this in an utility class.

And all this while I was developing a Grails plugin. I wanted to make the conversion as an extension to String, so that my call is like:

dateString.toXmlGregorianCalendar()

Recall that Grails plugin will load the *BootStrap.groovy when it starts up. We need to inject a metaclass method into java.lang.String so Grails runtime knows about it. (Only when run standalone, not after installing within an application).

public class MyPluginBootStrap {
def init = { servletContext ->
println "booting the plugin"
new GroovyShell(this.getClass().getClassLoader()).evaluate(new File("grails-app/conf/MetaFunctions.groovy"))
}
}

And define the MetaFunctions.groovy:

String.metaClass.toXmlGregorianCalendar = {
SimpleDateFormat sdf = new SimpleDateFormat("MM/dd/yyyy")
Date d = sdf.parse(delegate)
GregorianCalendar gcal = new GregorianCalendar()
gcal.setTime(d)
def dob = DatatypeFactory.newInstance().newXMLGregorianCalendar(gcal)
dob.setTimezone(DatatypeConstants.FIELD_UNDEFINED)
return dob
}

In .Net doing this may be simpler – just define StringExtensions. But they are static methods and cannot override existing methods. But in Groovy that is entirely possible, which makes a powerful feature, because if you don’t like the behaviour of a plugin class, just override it via metaclass using the above technique.