Thoughts on SpringBoot

In its new avatar, Spring enters the opinionated frameworks marketspace, and finally takes the Sinatra/Scalatra/Ratpack/Dropwizard/Spark route, where embedded containers (ie no external server containers) will start dictating the future. Whats an opinionated framework anyway? Aint all frameworks opinionated? To me every piece of code is just as opinionated as the developer who wrote it. Here are my observations about SpringBoot, after having worked with it for about 6 weeks.

1. Build support for gradle (+)
SpringBoot apps can be built using both maven and gradle. But I believe Gradle will eventually dominate the world of buildscripts. So the sooner your team moves to gradle, better will be your quality of this life and perhaps, next life too. Are you listening, .Net?

2. Starter poms (+)
Remember your 500 line pom.xml, where the tag <dependency> occurs 25000 times? Say good bye and start using starter-poms. Combined with gradle, a typical build file is now about 30 lines.

3. Configurations (+/-)
Spring is among the last standing action heros of xml-based application configuration in jvm world. Super hit when introduced, but soon became a liability. Thankfully, many developers challenged the xml tag soup and created innovative web frameworks (Wicket, Play etc.). Those who work with the Spring-based framework Grails, may have not touched an xml for a good period of time. If Spring MVC developers are still stuck with Spring without adapting Grails, I fail to understand why (well except for business impositions).

Ironically, SpringBoot seems to have many opinions when it comes to configuration – it supports 3 flavors of bean configurations: xml, annotation and groovy dsl. Having worked with Grails and other non-xml based applications for the last few years, picking up SpringBoot was bit of a chore. Grails autowires beans by name, and when custom beans are required, there is this wonderful bean builder. For me, java annotations are no better than xml configuration. Xml configuration proliferates verbosity but annotations defeat the purpose of wiring objects separate from code. While it brings type-safety, it also lets you mix code, configuration and logic and I think its a total mess. Annotations are like antibiotics – good in small doses (@ToString, @RestController, @AutoClone etc.), too many side-effects with overdose – building whole applications using annotations is @NotMyStyleOfTea.

A typical SpringBoot application has many many annotations – @EnableAutoConfiguration, @Configuration, @ComponentScan, @Conditional etc. Sometimes you end up with more annotations than the code itself and its not intuitive which is doing what when how. SpringBoot is certainly simpler for developers from xml background, but to me as a Grails developer, annotations have been a bit intimidating and sprawling. I don’t see CoC (Convention-over-Configuration), instead I see CoC (Configuration-over-Code).

Thankfully, SpringBoot has great (although not 1:1 xml equivalent) support for groovy beans dsl (via the GroovyBeanDefinitionReader). Groovy dsl is elegant, concise, intuitve and very readable. For some loss of type-safety (which could have been compensated by good tooling support), it comes with a great punch – wiring beans, environmental configurations, programmatic startup logic (as opposed to declarative) etc. I feel Spring can standardize groovy bean as the only configuration mechanism and shed all the fat of xmls and annotations. It would make the framework pretty lean and competitive in the already crowding lean frameworks market. May be thats what grails-boot is?

4. Properties (+/-)
Just like configurations, there are a few ways of defining and injecting properties. Coming from Grails background, the many ways of injecting properties was a bit confusing. SpringBoot supports both .properties and .yaml files. AutoConfiguration uses many default values for properties, but there is no comprehensive documentation on these properties on whats default. Again there are many annotations related to properties @ConfigurationProperties, @PropertySource, @Value, @EnableConfigurationProperties etc. Grails has this amazing single-point grailsApplication bean based on ConfigObject (a glorified Map) and allows nested and runtime evaluate-able configurations – helpful in dynamic scenarios. Again I wish Spring had defacto support for this (injecting properties from a config.groovy).

5. Rest Support (+)
SpringBoot makes it very easy to create rest controllers via @RestController. Instead of creating full-blown web applications, Spring’s eco can be used to create well-rounded rest services backed by Batch/Data/Integration/Security and use js frameworks like Angular/Knockout/Backbone etc. for front end. If using rest over json, Groovy 2.3 is promising to come up with fast json marshaller. While I like the cleanliness of Thymeleaf, the modern js frameworks have a clear advantage over server-side html generators.

6. Logging (+/-)
Yet again, too many logging frameworks in the bag: log4j, log4j2, slf4j, slf4j2, commons.log, java.util.log, logback. I spent some time resolving dependency conflicts, until I finally gave up and switched to logback. Spring team strongly recommends logback – Just Go with it – there is probably a good reason.

7. Data Access (+)
No question about Spring’s versatility here. Name any db and you have SpringBoot autoconfiguration support. Plus the Grails team has done a great job of spinning-off Gorm to standalone component. TexMex, I would say.

8. Testing (+/-)
Many examples still show use of JUnit, but here is a good start on how to use the incredible Spock framework in SpringBoot. Spock is like mom’s recipe – once you taste it, others aint the best.

9. Documentation (+)
It has improved a lot with newer minor releases. It takes time to sift through some old examples in the community, but lets just blame Google for not knowing what you exactly want, though it seems to know all about what food you want to eat when you are near a restaurant.

10. Finally
I think Grails3 (grails-boot?) is going down the trending route of embedded container deployments. I think that’s the only thing against Grails in the current trend. SpringBoot has got there first but I still feel it lacks the simplicity of Grails or the leanness of Play. It has certainly been simplified, but not simple.

If your existing eco system depends a lot on Spring based frameworks, it is worthwhile to adapt SpringBoot. Honestly, Im hoping Grails3 isn’t far off!

The GroovySpringBootBatchGormGroovyDslBeanFactory

See spring-boot-batch-sample at github for the updated source code.

I was recently working on setting up a not-so-trivial Spring Batch application and wanted to use SpringBoot, because of its embeded-web container capabilities. I could have used Grails and Spring Batch plugin, but wanted to give an opportunity for our Ops/DevOps team a peek at running and maintaining apps via plain “java -jar”. I hadn’t used Spring since 2009 (switching between SharePoint and Grails). Xml is no more a necessity for Spring and it has good support for java annotation configuration. But I am not very comfortable with annotation oriented programming. Xml provided the separation of wiring dependencies from code, but its verbose. Annotations are less verbose, but allows to mix configurations, logic and code, which I feel can spiral out of control pretty soon. I still feel wiring dependencies independently of “code” is a very desirable feature for large applications. Both xml and annotations seem to be at opposite ends of the spectrum.

Fortunately there is a middle ground. SpringBoot provides support for Groovy beans dsl, via the GroovyBeanDefinitionReader. Groovy bean dsls solve the problems of xml and annotations: DI wiring + concise readable syntax + some logic (eg environment-based). But this comes at a price of loosing type-safety. I am surprised that groovy dsl has not yet gone mainstream with Spring apps. If Spring comes with a Groovy DSLD schema (may be there is one already?), it could be a killer feature. For example, Spring Integration is already offering a dsl based “workflow”, which is pretty elegant to read, write and maintain.

I was also new to SpringBatch so it took a while to get them all wired up. So here is a starting point, if you want to use Groovy lang + SpringBoot + SpringBatch + Groovy DSL + Gorm. I haven’t figured out dsl equivalents of @EnableScheduling and @Scheduled yet.

Part 1: appcontext.groovy

//Note that config is not defined as a bean, but directly evaluated and then injected into other beans
//the 'grailsApplication' equivalent
ConfigObject configObject = new ConfigSlurper().parse(Config)

//Note the syntax beans {} not beans = {} like in Grails resources.groovy
beans {

	xmlns([ctx: 'http://www.springframework.org/schema/context', batch: 'http://www.springframework.org/schema/batch'])
	ctx.'component-scan'('base-package': 'org.mypackage')
	ctx.'annotation-config'()

	myController(MyController) {
		config = configObject 
	}

	myService(MyService) {
		config = configObject 
  }

  //MyItemReader implements ItemReader
	myItemReader(MyItemReader) { bean ->
		bean.initMethod = 'init' //required if initializing some data from external dao
		bean.scope = 'step' //for job restartability
	}

	myItemProcessor(MyItemProcessor) {
		myService = ref('myService')
	}

	myItemWriter(FlatFileItemWriter) {
		lineAggregator = new DelimitedLineAggregator(delimiter: ',', fieldExtractor: new BeanWrapperFieldExtractor(names: ["id", "title"]))
		resource = '/apps/springboot/myproject/output'
	}

  //create a job
	batch.job(id: 'job1') {
		batch.step(id: 'step1') {
			batch.tasklet {
				batch.chunk(
					reader: 'myItemReader',
					writer: 'myItemWriter',
					processor: 'myItemProcessor',
					'commit-interval': 10
				)
			}
		}
	}

  //the following beans are minimum mandate because there is no equivalent of xml's <batch:job-repository /> in groovy dsl
  //http://stackoverflow.com/questions/23436477/groovy-bean-syntax-for-spring-batch-job-repository
  //thanks to https://github.com/johnrengelman/grails-spring-batch/blob/master/SpringBatchGrailsPlugin.groovy for the bean definitions
	jobRepository(MapJobRepositoryFactoryBean) {
		transactionManager = ref('transactionManager')
	}

	jobRegistry(MapJobRegistry) { }

	jobLauncher(SimpleJobLauncher) {
		jobRepository = ref('jobRepository')
		taskExecutor = { SyncTaskExecutor executor -> }
	}

	jobExplorer(JobExplorerFactoryBean) {
    //dataSource is auto-configured
		dataSource = ref('dataSource')
	}

	jobOperator(SimpleJobOperator) {
		jobLauncher = ref('jobLauncher')
		jobRepository = ref('jobRepository')
		jobRegistry = ref('jobRegistry')
		jobExplorer = ref('jobExplorer')
	}

}

Part 2: Application and Scheduler

@Configuration
@ComponentScan
@EnableAutoConfiguration
@EnableScheduling
class MyJobApplication {

	private static final Logger logger = LoggerFactory.getLogger(MyJobApplication.class.getName())

	@Autowired
	JobLauncher jobLauncher

	@Autowired
	Job myJob

//  You can also create configObject bean like this and refer back in beans.groovy using ref('config')
//	@Bean(name="config")
//	ConfigObject configObject() {
//		return new ConfigSlurper().parse(Config)
//	}

	
  @Scheduled(fixedDelayString = '${myJobFixedDelay}', initialDelayString = '${myJobInitialDelay}')
	public void startMyJob() {
		logger.info "startMyJob()"
    //Add time if your job runs repeatedly on different parameters - this will make it an unique entry in the batch-job tables
		JobParameters jobParameters = new JobParametersBuilder().addLong("time",System.currentTimeMillis()).toJobParameters()
		jobLauncher.run(myJob, jobParameters)
	}

	public static void main(String[] args) {
		logger.info "Starting MyJobApplication..."
		Object[] sources = [MyJobApplication.class, new ClassPathResource("appcontext.groovy")]
		SpringApplication.run(sources, args);
	}
}

Part 3: Datasource

Datasource bean is autoconfigured via @EnableAutoConfiguration and if you define the application.properties (or application.yaml). Just add the right driver in your build.gradle. If you dont specify any, hsql db is used.

Part 4: build.gradle

buildscript {
	ext {
		springBootVersion = '1.0.2.RELEASE'
		spockVersion = '0.7-groovy-2.0'
	}
	repositories {
		mavenLocal()
		mavenCentral()
		maven { url "http://repo.spring.io/libs-snapshot" }
	}
	dependencies {
		classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
	}
}

apply plugin: 'groovy'
apply plugin: 'idea'
apply plugin: 'spring-boot'

jar {
	baseName = 'myapp'
	version = '0.1'
}

repositories {
	mavenCentral()
	mavenLocal()
	maven { url "http://repo.spring.io/libs-snapshot" }
}

dependencies {
	compile("org.springframework.boot:spring-boot-starter-web")
	//if you want to use Jetty, instead of Tomcat, replace the above line with the next two lines
	//compile("org.springframework.boot:spring-boot-starter-web:${springBootVersion}") { exclude module: "spring-boot-starter-tomcat" }
	//compile("org.springframework.boot:spring-boot-starter-jetty:0.5.0.M2")
	compile("org.springframework.boot:spring-boot-starter-actuator")
	compile("org.springframework.boot:spring-boot-starter-batch")
	compile("org.springframework.boot:spring-boot-starter-logging")
	compile("org.springframework.boot:spring-boot-starter-jdbc")

  compile("org.codehaus.groovy:groovy-all:2.2.2")
	
  compile("org.springframework:spring-orm:4.0.3.RELEASE")
  //For those Grails guys, just throw in the new and shiny standalone gorm dependency
	compile("org.grails:gorm-hibernate4-spring-boot:1.0.0.RC3") {
    //currently brings in spring-orm:3.2.8, exclude it and explicitly include new one above
		exclude module: 'spring-orm'
	}

	testCompile "org.springframework.boot:spring-boot-starter-test"
	testCompile "org.spockframework:spock-core:${spockVersion}"
	testCompile "org.spockframework:spock-spring:${spockVersion}"
}

task wrapper(type: Wrapper) {
	gradleVersion = '1.11'
}

A few more features for the future:

1. Defining multiple datasources in groovy dsl
2. Using spring-loaded for hot-swapping runtime (as of now I can’t get this to work with Intellij Idea)
3. Using Spring-batch-admin to control jobs via UI

Spring Boot with Groovy config

While working on converting a Grails application to a SpringBoot based application, I was trying to understand how the auto configurations and custom configurations work under the hood. Just like CoC of Groovy/Grails, there seems to be a lot of hide-and-seek going on under the Spring boot hood. Admittedly its been a while for me with good ol’ raw Spring, so the many different ways of skinning the configuration felt a bit overwhelming. I counted several concepts around configurations – application.properties, application.yaml, PropertySource, PropertyPlaceHolder, @Value, @EnableConfiguration, @EnableConfigurationProperties, @EnableAutoConfiguration, @Configuration, @ConfigurationProperties, pre-defined property name prefixes, creating classes specifically for holding configuration values – so much information just for accessing key-value pairs? Compare to Grails – one Config.groovy (or multiple externalized) and a simple grailsApplication injection takes care of making configuration available anywhere. When I read Spring Boot support for Groovy, I somehow assumed that support for a groovy config is automatically enabled.

My Grails application already had a groovy config, and I was not in a mood to convert them to .properties or .yaml. Groovy configs are DSLs in essence – it provides many advantages over the traditional properties file: key=value (like .properties), a structured format like .yaml – sans the indent-erasure-phobia, type-enriched config values like list/map etc, functions via closures and more.

So, here is a very simple way to expose the groovy config to your Spring Boot apps:

Create a Config.groovy in the same folder as the ConfigProperties class.

/** Config.groovy **/

//simple key-value
spring.boot.guide = 'https://github.com/spring-guides?page=1'
//formatted structure
google {
searchurl = 'http://www.google.com'
apiKey = '1234'
}

/** ConfigProperties.groovy **/

import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration

@Configuration
class ConfigProperties {

@Bean(name="configObject")
ConfigObject configObject() {
return new ConfigSlurper().parse(Config)
}

}
/** HelloController.groovy **/

import ConfigProperties
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.web.bind.annotation.RequestMapping
import org.springframework.web.bind.annotation.RestController

@RestController
class HelloController {
@Autowired
ConfigObject configObject

@RequestMapping("/")
String home() {
return "Hello World! " + configObject.spring.boot.guide
}

}

/** Application.groovy **/

@EnableAutoConfiguration
@ComponentScan
class Application {
static void main(String[] args) {
Object[] sources = [Application.class, ConfigProperties.class]
SpringApplication.run(sources, args)
}
}

Note: @Bean returns must be typed. Habitually, I wrote def configObject() { return new ConfigSlurper().parse(Config) } and the @Autowired did not wire it up.

The lite at the end of webservice tunnels

In the last post, I alluded to how shifting complexity from one area to another can actually improve perceived simplicity. Here is a concrete example of how we got rid of scores of wsdl-client generated jaxb classes and brought in a fresh wave of simplicity to our Grails applications.

Problem

One or more of our Grails application uses several our own plugins and each of them talk to several web services that share a lot of xsd schemas, the java client generation of which end up proliferating duplicated jaxb classes in each of our plugins and also in the applications which use them, thus resulting in multiple classpath conflicts.

Also we dont like the date conversions from Date to Calendar to GregorianCalendar to XmlGregorianCalendar or other conversions from Byte to Int to Shorts. Also jaxb does some non-intuitive magic with some of the xsd:any types in the schemas. If xsd schemas want strict type checking, fine. We from the “def” world, just look everything as plain data. And finally the jaxb classes look ugly.

Solution

With a combination of groovy-wslite, grails wslite plugin and MarkupBuilder and Config, the code become pretty straight-forward, almost.

From the grails command prompt, run install-plugin wslite

def grailsApplication

public def amazonBookListMania() {
  WsliteConnector.instance.withSoap(serviceURL: grailsApplication.config.amazonBookListManiaServiceUrl) {
    def soapResponse = send(SOAPAction: grailsApplication.config.amazonBookListManiaSoapAction) {
      //Get the namespaces, which is a simple Groovy map
      //['xmlns:amz':'http://www.amazon.com/ListMania']
      envelopeAttributes grailsApplication.config.amazonBookListManiaNamespaces
      body {
        //Get the raw xml and slap it into soap body
        mkp.yieldUnescaped(createXmlRequest())
      }
    }
    //Wslite automatically xmlslurps the result and gives you the closure
    def response = soapResponse.body.bookListManiaResponseRoot
    println response.Items.count()
  }
}

The “withSoap” is a DSL, that is supposed to be injected automatically into controllers, but on other classes it didnt seem to work. I tried using WsliteEnhancer, but that didnt help either. So the direct call to withSoap using WsliteConnector singleton.

The example shows all params injected from grailsApplication config, to allow webservice urls for different environments.

The interesting aspect is the envelopeAttributes, which is a map of namespaces. Since we are not using generated client classes, whcih does the work for us, the namespace prefixes must be specified somehow in the xml itself as part of soap request. Adding namespaces to envelopeAttributes will append the namespaces to the soap-env tag. No need to include the namespace for soap-env itself. The wslite plugin does that.

public String createXmlRequest() {
    def writer = new StringWriter()
    def builder = new MarkupBuilder(writer)
    builder.'amz:ListMania' {
      //tag attributes
      'amz:credentials'(username: 'vasya10', password: 'yeahright')
      //tag value
      'amz:dateRequested'(new SimpleDateFormat('yyyy/MM/dd').format(new Date()))
    }
    //Returns <amz:ListMania><amz:credentials username='vasya10' password='yeahright'></amz:credentials><amz:dateRequested>2013/01/01</amz:dateRequested></amz:ListMania>
    return writer.toString()
}

With the complexity of creating the xml shifted into MarkupBuilder, creating soap requests become trivial, even more so with the withSoap DSL.

Wsdl import generated classes
Pros
No need to worry about namespaces
Set and Get of values use Java-like syntax
Setting values does not matter as xml marshalling happens at a later time than setting the values
Setting tag attribute or tag value is no different (ie code doesn’t care)

Cons
Every minor change in wsdl forces wsdl client generation
Duplicated client generated classes could cause classpath conflicts
The hierarchy of the xml elements is not apparent in the code, the only way to know the hierarchy is to look at the xml samples

WSLite + MarkupBuilder
Pros
No confusion about which client to use: there are scores of them and each have some unique issue (jax-ws, jax-b, cxf and so on)
No wsdl import client generated sources and jars
No crazy xerces, xalan and other marshalling runtime errors, especially if deploying to different app servers (tomcat, weblogic etc)
If there are trivial changes to elements (saying adding a new optional element), will still work without changing client code (if that element is not used)
Creation of xml is very straight-forward
Code documents the hierarchy of the xml!
Response returns XmlSlurped data, so dot-notation can directly be applied on the objects

Cons
Code must be namespace-prefix aware
Must know tag attributes and tag values to create the xml correctly (this must be a Pro)
Code must generate xml in the same sequence of elements; blocks of elements can still be called out-of-order, but the final xml body must be created in a sequence
Unvalidated namespace prefixes increases testing time
Extra testing time for un-type-checked variables

Compared to our previous jaxb code (with ObjectFactory, creation of new Objects etc.), the code is now 40% smaller, and additionally got rid of the client sources too.

The motorcycle diaries of webservices

Software is a big grandpa pendulum clock. It oscillates between simplicity and complexity perpetually. Ideas and concepts that start off simple, rapidly become complex and eventually complex ideas are broken down into simple components and again these simple components evolve into complex creatures of their own. After all we evolved from a single cell into complex cell living beings, why blame the software for that?

Webservices necessitated from complex interaction between disparate systems (remember CORBA, JNI, DCOM, RPC, mainframe apis?). All those were dissolved into ‘simple’ wsdl based services, as long as client and server maintain the contract, everything was promised to be a simple hookup.

But it didnt end there. Generating wsdl clients either in Java or .NET were complex in its own terms. Java discovered its way, as usual, with hundreds of wsdl clients and xml jars (anyone still using xerces, xalan, sax4j directly?) – jaxb, jax-rs, jws, cxf and several more proprietary ones). .Net, was not a A-shot either: first asmx, then wcf and then several flavors of .net versions on top of that. Most of the time, its not a one-step process to call a WCF wsdl from a Java based client, there are always some surprises around the corner in terms of configurations. Its heartening to see nobody got it right the first time.

If you think about it, what was the necessity to generate classes from wsdl in order to send plain xml text data? The early xml apis described the xml in terms of hardcore hierarchy: root, nodes, parent, children, siblings, single. It could have been easily extended to represent a oh-so loveable joint-family: cousins, uncles, aunts, nephews and nieces:

public getAuntyNode() {
return currentNode.getParent().getFather().getSister().toNode();
}

public getDistantUncleNode() {
return currentNode.getParentNode().getMother().getSisterInLaw().getHusband().getSibling().toNode()
}

public getImMyOwnGrandpaNode() {
//Outsourced
}

Such family friendly APIs would have thrilled developers who often work only after the family goes to sleep.

I am also starting to see that xml and xsd schemas are not an ideal way to describe data models. Describing a model in xsd using the semantics of xml itself is over complicated. On top of that extensions, simpleTypes, complexType – are way too tedious and garrulous. If you are exposed to C++/Ruby/Groovy world, you can visualize the models pretty quickly in terms of encapsulations, compositions and interfaces. In the xml world, I only see tags and attribute names. Lots and lots of them. The content is lost in the tag-tsunami.

While reviewing a project recently, where all our models are xsd based, data models extending from base xsds, and several simple, complex and xsd:any types, pretty soon I found them going above my head. Im not even thinking how xsds maintain versions of models and references.

Coming to actually using them in our Grails project, a colleague of mine tried to generate the client classes using jaxb (before trying a few other clients) and pretty soon found himself in a maze of conflicting classpaths from different generated wsdls. Jaxb generates classes for all the dependent xsds (obviously) and generating them for different wsdls (in the same project) ended up with several classpath issues.

Taking a break, we decided to ask a fundamental question: “Why do we have to worry about all the xml types in the Grails code, if all we need is to send a plain textual xml data?”.

Who cares what the model hierarchy of xsd is? Most of the time that model is useless to the Grails code, where typically the representation of model is different from the xsd models. All those xsd generated classes, objects, factories, abstract factories were required because the complex familial Java APIs were the only way to process xml (unless using StringBuffer and appending data manually).

Enter Groovy’s MarkupBuilder, it is much easier to build an xml than generating the client. So if we all care is to send data and generating that data itself is easier than generating clients, why bother at all with the later. And we switched to Grails wslite plugin, with which we are directly creating soap requests and sending it to server.

So thats how the pendulum switched back to its other position, back to creating hand made soaps. Eventually this may get complex too. But again, the important thing to remember is that the original complexity is not the same level of complexity of the broken down components. The “absolute complexity” remains unaltered and the “perceived complexity” depends on how lazy or smart we are.

Xml Subtractor

A few weeks ago, had to work with two xmls where the first xml was a superset of the second xml and had to figure out only the xml nodes in first xml that were not in the second xml.

Mathematically expressed,

Given, xml2 Ε xml1
Find: xml1 – xml2

I had written a small program to do that and forgotten about it. Well today faced the same issue but with two substantially large xmls. Glad had the program earlier and it helped save a ton of time.

Groovy makes xml processing very simple and perhaps create an “xml arithmetic library”, using Groovy Categories et al, for what its worth.

void xmlMinus() {
String xmlText1 = new File("filename1.xml").text
String xmlText2 = new File("filename2.xml").text

def xml1 = new XmlParser().parseText(xml1)
def xml2 = new XmlParser().parseText(xml2)

xml2.each {
def uniqueId = it.'@uniqueId'
println "finding $uniqueId
def node = xml1.find { it.'@uniqueId' == uniqueId }
xml1.children().remove(node)
}

java.io.PrintWriter fw = new java.io.PrintWriter(new FileWriter("xml1-minus-xml2.xml"))
XmlNodePrinter xnp = new XmlNodePrinter(fw)
xnp.preserveWhitespace = true
xnp.print(xml1)
fw.close()
}

Example:

File List: filename1.xml

<Item uniqueId="1">
  <Details />
</Item>
<Item uniqueId="2">
  <Details />
</Item>
<Item uniqueId="3">
  <Details />
</Item>
<Item uniqueId="4">
  <Details />
</Item>

File Listing: filename2.xml

<Item uniqueId="3">
  <Details />
</Item>
<Item uniqueId="1">
  <Details />
</Item>

File Listing: xml1-minus-xml2.xml

<Item uniqueId="2">
  <Details />
</Item>
<Item uniqueId="4">
  <Details />
</Item>

The peekaboo column type

Alright, I will have to admit that the misuse of xml is irritating me quite a bit. Xml is reaching a critical desalination point (sorry Quaid) and is set to freeze the world over. I sincerely hope that xml abuse will be the cause of next Internet meltdown. Disproportionate amount of control information thrown along with a tiny amount of data is going to upset someone over the wire, causing a data-strife and leading to abrupt strike by tcp services.

That was pretty much my dream when I struggled over what I thought was a simple solution to a really simple problem. I was working on a data-migration project, you know, the routine thingy, read data from a legacy database, convert to xml and stuff down the throat of another database. I picked up Grails to get it done, uh well, with a premise that I do not have to handwrite a single query and can simply map tables to domains and magically invoke “save()”.

I was short-circuiting Grails by using it only for GORM features, but running as a grails script, and not for any ui, scaffolding and many of it’s “traditional” strenghts.

1. GORM hates tables with no keys

The legacy tables I dealt with did not have any PKs at all. And GORM does not like them because of required implicit id. Yes composite keys work fine, but the legacy tables did not have any unique rows either. So the solution the was to simply create a view on the table:

select rownum as uniquekey, t.* from table t

I could care a damn for what the rownum was, as long as it was unique. I mapped the GORM id to the uniquekey and all was okay.

2. Grails Spring Context in a script environment

Grails does an amazing job of injecting Spring beans whereever you expect to. All’s well when running as a web app, but I was running this as a grails script, so the Spring services were not getting injected.

MyScript.groovy

void eatPrayAndStuff() {
MyService service = new MyService()
service.eatFromLegacyDb()
service.prayConvertToXml()
service.stuffXmlDownTheThroatOfNewDatabase()
}

class MyService {

void eatFromLegacyDb()
def query = sessionFactory.currentSession().createSQLQuery(sql)
}
}

The script is run as following:

# grails –stacktrace dev run-script MyScript.groovy

Obviously, the service object was not null, but I was promptly getting null on currentSession. I understood Spring cannot inject its singleton service into the script, but I just didn’t know how.

Thanks to a quick response in stackoverflow:

MyService service = ctx.MyService

instead of new MyService(). That’s one of the great strenghts of grails, the spring context is available throughout the script environment (not just Web and Unit Test) and its easy to experiment with. It just takes a bit of hunting, and asking for help.

3. XMLType – The Clown Prince of Column types

I ran the script and it ran well about a 1000 records and I almost declared success. Almost. And then came the bouncer.

Colleges and Universities teach a lot about how to do programming. Sadly there are no professors who take a class on “How to write BAD code”. And set a test on some really professional-industry-level bad code. The experience gained from doing wrong things is hardly equivalent to making the right choices. Yeah, you know what Edison said.

Oracle’s SQL Errors (and JDBC Sql Exceptions) will always be dear to my heart in realizing how to write bad code. So I got the error:

ORA-01461: can bind a LONG value only for insert into a LONG column

I had a few long values in my domain, so I scratched off one by one, like a senile lottery ticket buyer, and after some debugging found that all numbers are good (Damn lottery pun!). By process of elimination I realized it was the XMLType column that wasn’t behaving. The reason was the size of the xml data was 4100 bytes in length. And Oracle’s XMLType is an “XMLType” if its data size < 4kb, but if its above 4kb its a CLOB. Good deal! A dynamically self-morphing peekaboo data column type, FWIW.

Someone suggested to use a stored procedure instead of domain mapping, so I cajoled my co-worker to quickly write a wrapper storedproc for me. Try what ever I could not get past the next error:

ORA-06553: PLS-306: wrong number or types of arguments in call.

When things go wrong, switch to basics, so they say. So I went back to basics of counting numbers by my fingers, nails and toes: all arguments are correct, all types are correct. Yet Oracle thinks I pass wrong arguments.

I abandoned the stored procedure route and turned to using Oracle XDB for the XMLType. Oracle does not respect Maven community and somehow I located an xdb.jar and dropped into the grails lib directory. But STS Eclipse has issues with properly refreshing libraries. So I had to clean, close, sweep, brush the project from both STS and command line to get it included in the classpath. But surprise, xdb.jar has its own SAXParser classes which interfere with the Grails scripts.

Googling around, there was a suggestion to use xmlparserv2.jar in grails/lib. I again I hunted the jar down, but If I put it in the lib directory, grails wouldn’t even compile Then I put it in the classpath at runtime, but I simply got ClassNotFoundException.

# grails –stacktrace -cp lib/xmlparserv2.jar dev run-script MyScript.groovy

I guess the -cp option works fine if running as a run-app, but while invoke the gant scripts, the -cp is not honored (Grails 2.1.0). I realized I may have to tweak the Run-Script, but I didnt want to go that far, because the script will be run in environments I dont even know and didnt want to assume a lot.

I abandoned the XMLType route. Oracle’s half-cooked XMLType implementation and its support api isnt for someone who likes to be productive. I took a break, wondering whats the need for storing a 200 byte data – yes, thats what the actual datasize was, but ended up  as a 4100 byte xml with all kinds of tags. Thats the anguish about xml abuse and the internet meltdown that I fervently hope.

4. When in Rome, treat everybody like British

Finally I found a piece of code, which converted the XMLType into a Clob in a round-about way. Not sure why converting one data type to another needs a javax.sql.Connection parameter. This is like, if you want to pour water from one container to another container, you need to send it to a hydro-electric plant.

private CLOB getCLOB(String xmlData, OracleConnection oracleConnection) throws SQLException{
CLOB clob = null
try {
clob = CLOB.createTemporary(oracleConnection, true, CLOB.DURATION_SESSION)
clob.open(CLOB.MODE_READWRITE)
Writer clobWriter = clob.getCharacterOutputStream()
clobWriter.write(xmlData)
clobWriter.flush()
clobWriter.close()
clob.close()
} catch(SQLException sqlx){
sqlx.printStackTrace(System.out)
clob?.freeTemporary()
} catch(Exception x){
x.printStackTrace(System.out)
clob?.freeTemporary()
}
return clob
}

Only to discover that the second parameter OracleConnection is not a javax.sql.Connectin per se, and certainly not what I have in the Hibernate’s sessionFactory.currentSession().connection() which is Proxy19 – a connection proxy. So how do I get an OracleConnection from a proxy connection?

5.Droid to the Rescue

Connection connection = sessionFactory.currentSession.connection()
PreparedStatement ps = connection.prepareStatement(SQL)
C3P0NativeJdbcExtractor cp30NativeJdbcExtractor = new C3P0NativeJdbcExtractor();
OracleConnection oc = (OracleConnection) cp30NativeJdbcExtractor.getNativeConnection(ps.getConnection());
CLOB c = getCLOB(xml, oc)
ps.setObject(3, c)

And so I finally after a long self-guided tour of google, oracle, stackoverflow and several more sites that even my browser history could not remember, got the xmltype working, but not before I scratched around to fix the sql statement itself:

String SQL = /insert into table1(?,?,?) values (?,?,xmltype(?))/

And so the conspiracy was settled – Grails, Hibernate, Script, Spring, Oracle, XMLType, CLOB, Connection Proxy – all concocted together to make a nightmare of a simple insert statement.

Follow

Get every new post delivered to your Inbox.

Join 191 other followers