The Innovation recycle bin

Time is cyclic, they say. The Sun, moon, seasons, tides, rains, storms, tornadoes and likely the universe itself – they all go and come back again with indifferent precision. There is something beyond the universe which definitely looks like huge unimaginable recycle bin. Similarly, innovation is also cyclic. Old ideas are thrown into a recycle bin and they come out as what we feel as a new idea. What we worked before, we work on again. Its not just déjà vu, but also déjà fait.

I was working on a project in Oracle Fusion middleware. Sitting in the bus back to home, set a train of thought.

<assign name=”assignVar”>

That’s an interesting syntax for setting a variable, isn’t it? For 400 years since Galileo, we are used to mathematical notations like:

outputVariable.response = 404

Any one remembers writing in 8085 instruction set? Here is refresher:

MVI A, 194H
STA 2000H

If compilers can understand ‘a = 404’ and if humans can understand that the same ‘a = 404’ really means ‘a = 404’, who are we pacifying with the above xml construct? The only participant that is most satisifed in this complicated piece of assignment is the schema validator. Even the assembler instructions look simpler compared to the xml-instruction set. So we progressed from complicated assembler copy instruction to simpler a = b to a complicated xml <copy> instruction. Innovation is cyclic.

When xmls became a rage a few years ago, every bit of configuration imaginable were ported to xml. We all enjoyed learning Spring xml configuration where we could create classes and reference objects using declarative xml. Suddenly everything was hierarchical. Eventually the limitations of xml was hard felt and out came the non-xml frameworks (Wicket, Rails for eg). Recent versions of Spring, Grails also do not rely on xml anymore as the backbone of configuration. Json and Dsl have made us realize the simplicity of representing data without a purposefully complicated hierarchical structure is still possible. We progressed from non-standard-simple-configurations to standard-hierarchical- configurations-in-xml to standard-non-xml-configurations. Innovation is cyclic.

But again, Json and Dsl go back to a similar format of key=value, just like the properties files. So we have moved from flat-file-properties to hierarchical-xml-configuration to flat-file-look-alike-optionally-hierarchical-dsl config files. Innovation is cyclic.

From Javascript validations to everything-must-be-on-server-validations back to ajax-type-javascript-validations. Innovation is cyclic.

Every data transmitted, whether it is across network, or between circuits, there is always a small amount of control data that is sent along with it. Like source, destination, checksum etc. Yet somehow we bought into the idea of xml – where the size of control data is about like 100 times the size of actual data itself. We have been wasting a huge amount of network bandwidth, sending the same tags for a million records. We wrap 10 bytes of data with about 1kb of tags, so that control data could be ripped off and the actual data be used. Much like an silk-embroidered-golden gift wrap for a lego toy.

Xml is not wrong. It provides a great way to exchange data between two systems when they don’t agree to agree with each other on anything, like a divorced couple. Or a teen and his/her dad. But using xml hierarchy to write a mathematically or logically expressible code like the one above or a similar cousin – the inexplicable-complexity-on-steroids Caml query, superimposing an overtly artificial hierarchy strucuture, where there is only a conceptual linearity, is like asking a husband to do laundry on a superbowl night.


Extensions via metaclass in Grails

Even after several releases and wrapper utilities, representation of time in Java seems to be an unconquerable mess that it originally was. At the same time, surprisingly, web services have been there for a long time now, still there is no magic wand to create webservices and clients.

Webservices is something Groovy/Grails should have made it ridiculously easy to work with, by default. When xml, json, http and so on have been outrageously simplified, the natural expectation is that the webservice call would be like:

WebServiceObject ws = new WebServiceSlurper().parse('')
WeatherObject w = ws.getWeatherObject()
Temperature[] temperatures = w.findTemperature(zipcode: '11505', days: 5, unit: 'farenheit')

There are so many wrapper api-s, every webservice client generation is lacking in someway, making a new comer to Grails to try and find what works. axis, axis2, groovyws, xfire, cxf, jws, jaxb – there are so many acronyms and some of them are just wrappers over others, the choice may be overwhelming. Forums do provide lot of answers, but most times what works for one goose, does not work for even for its goose twin.

So after my friend pointed out the pot-holes of webservices in Grails, I started using the jws. All good until I hit the xs:date object in our wsdl, which translates to XMLGregorianCalendar. Like we all wanted a new class to represent date and time. I seriously hope Java/Groovy will deprecate and delete all its Date and Calendar classes and introduce just one class with a philosophical outlook:

* Dead yesterday, unborn tomorrow, why fret about them, if today be sweet?
class StarDate{
public double day;
public double increment(double unit) { day += unit }
public String toString() { return "Captain's log, stardate $day"  }

Conversion between a date in String to XMLGregorianCalendar goes like this:

SimpleDateFormat sdf = new SimpleDateFormat("MM/dd/yyyy")
Date d = sdf.parse(dateString)
GregorianCalendar g = new GregorianCalendar()
def date = DatatypeFactory.newInstance().newXMLGregorianCalendar(g)

So much for a date? I tried to generate the client classes with a custom binding to Date instead of XMLGregorianCalendar. No matter what I tried, the value inside the webservice always ended up as long format “Sun Dec 4 …”. Fed up, I changed it back to XMLGregorianCalendar and now I saw the date as 1995-03-28T06:00:00. So where did the timezone come from and how do I avoid it? Stackoverflow had an answer:


Seriously, DatatypeConstants? Minimum integer value to specify undefined? Anyway, with that undefining of the timezone, the webservice was correctly seeing the date as just 1995-03-28. But I still dont like the fact that I had to convert this in the service class or that I had to put this in an utility class.

And all this while I was developing a Grails plugin. I wanted to make the conversion as an extension to String, so that my call is like:


Recall that Grails plugin will load the *BootStrap.groovy when it starts up. We need to inject a metaclass method into java.lang.String so Grails runtime knows about it. (Only when run standalone, not after installing within an application).

public class MyPluginBootStrap {
def init = { servletContext ->
println "booting the plugin"
new GroovyShell(this.getClass().getClassLoader()).evaluate(new File("grails-app/conf/MetaFunctions.groovy"))

And define the MetaFunctions.groovy:

String.metaClass.toXmlGregorianCalendar = {
SimpleDateFormat sdf = new SimpleDateFormat("MM/dd/yyyy")
Date d = sdf.parse(delegate)
GregorianCalendar gcal = new GregorianCalendar()
def dob = DatatypeFactory.newInstance().newXMLGregorianCalendar(gcal)
return dob

In .Net doing this may be simpler – just define StringExtensions. But they are static methods and cannot override existing methods. But in Groovy that is entirely possible, which makes a powerful feature, because if you don’t like the behaviour of a plugin class, just override it via metaclass using the above technique.

Recover site collection from a backup

I had accidentally restored a site collection (using Restore-SPSite) from another web application over our ongoing development site collection. Which meant the front-end (SharePoint Designer) work done on the development was lost.

Luckily I had a SQL database backup of the content database. I got the database restored into another similar existing web application (instead of creating a new web application again).

When I hit the site, I got the error

400 BAD Request

Now in the Central Admin, Manage Content Databases, I see that the content database indeed had one site collection. Ok, site collection is there, but is there a web?

$s = Get-SPSite http://mysite


So, the rootweb is missing or not recognized. When I googled various forums, surprisingly there is not a solution to be found for this. I tried recycling the apppool, iisreset, mounting the content database with a new assignment id. Nothing worked.

But the solution itself is simple. Just detach and re-attach the content database to the web application.

Dismount-ContentDatabase $contentDb
Mount-ContentDatabase -ContentDatabase $contentD -DatabaseServer $dbServer -WebApplication http://mysite

Inconspicuous SharePoint Quirks #6 – Automorphing internal column names

After a brief flirting with Groovy and Grails, its time for a SharePoint leather hunt. I had to add an existing site column “Order” (in the sense of sequencing) to an existing content type “CoolLinks”. Column added fine, and I added some data into the list using that content type.

I looked up the StaticName for that field, because thats what the code uses to query the data.

$s = Get-SPSite http://mysite
$w = $s.openWeb()
$f = $w.Fields["Order"]
$f.Title, $f.StaticName, $f.InternalName

Order, Order1, Order1

The webpart that queried that list, has the following code, using StaticName to query the data:

DataTable dt = web.List["CoolLinks"].Items.GetDataTable();
List<CoolLink> items = new List<CoolLink>();
items = (from item in dt.AsEnumerable()
select new CoolLink() {
LinkUrl = new PublishingLink(item.Lookup<string>("URL")),
Order = Convert.ToInt32(item.Lookup<double>("Order1")),

Yet my webpart is not retrieving the list data. I checked the ULS Logs and it has an error –

"Cannot find column 'Order1'"

I also had another List called Alerts, with another content type, which also used the same site column “Order”. The query to retrieve that data was exactly same as that of CoolLinks. Yet Alerts worked fine, but not CoolLinks.

So I startup Powershell, hoping to get a peek into the internal structure of the list, since the content types looked okay.

$s = Get-SPSite http://mysite
$w = $s.openWeb()
$list = $w.Lists["CoolLinks"]

This displays the data as it is queried. And what do I find —

URL              : http://somesite
Order0           : 1
ID               : 43
ContentType      : CoolLinks

Order0? Where did the 0 come from? The SharePoint API quietly changed the name of the column! I added the Order site column to another content type, it is still Order0. So now I have some Lists in my site collection which use Order1 and some which use Order0, though they are from the same site column.

I ran the .Net Reflector on Microsoft.SharePoint assembly, SPFieldCollection class. The code does check an internal name dictionary in the GetFieldByInternalName() method, which probably keeps a list of “keyword-type” column names and apparently appends arbitrary numbers to it.

Well, that’s the story. So if your Visual Studio goes on the fritz, or your code blows up, or your site collection conks out; before you call the Microsoft Support, turn on all the lights, check all the closets and cupboards, look under all the beds, ’cause you never can tell there just might be a different internal field name in your Content Type.

(Okay, you know where that classic line is from).

Groovy #7 – if there is no if

The most minimal high level programming language just needs three statements:

  1. assignment
  2. if
  3. goto

All other syntaxes can be replaced with a combination of these. As programming languages evolved, goto bore the ignominy of being the first to lose favor with the programmer-theorists. It was replaced with various sophisticated constructs like for, while, do, switch, subroutines and functions.

The next one is the ‘if’ condition. With the applications growing in complexity, the if-s also drive the cyclomatic complexity. But to some extent, OOP languages eliminate if conditions at a higher-level.

Consider a simple example:

abstract class Animal {
abstract string voice();

class Dog extends Animal {
string voice() { return "Bark" }

class Cat extends Animal {
string voice() { return "meow" }

Animal a = new Dog()
assert a.voice() == 'Bark'

Without the inheritance feature, a procedural language programmer would have to rely on if condition to make the above work. Obviously the if condition is still there somewhere down the line (compiler will have to determine the type of the object to know which method to invoke), but its not in the reviewed code. Yet, the above looks like an elaborate code-trap to just avoid one if condition.

By induction, can ‘if’ conditions be fully replaced by abstractions? Lets look a simple Groovy example, which uses the metaprogramming technique to do something like that.

Boolean.metaclass.methodMissing = { name, args -> println args[0].call() }

def find_nativity_by_greetings(def greetings) {
greetings in ['howdy', 'hiya'] { 'Texan!' }
greetings in ['hola', 'buenos dias'] { 'Mi amigo!' }
greetings == null { 'Yankee' }


Since if conditions evaluate to true/false and are automatically represented as Boolean in Groovy, by providing a missingMethod override implementation for the anonymous closure and calling that closure, the “if” keyword vanishes from code. The above construct is now equivalent to a if..elseif..elseif..elseif.. construct.

How is this helpful? The above constructs can actually be written in a configuration file by a business owner or it could even be in a excel spreadsheet. The core Groovy code just has to slurp the config file, evaluate and execute the statements as if there are no if-s.

Groovy OLC #3 – Perfect numbers

At a time when new JVM based languages are mushrooming as fast as frameworks were being written a few years ago, it is interesting to reflect upon a few trends. About 6th century CE, there were already several languages in India and several grammar works were being written. But there is one work outstanding in its contribution to the world of linguistics. A linguist named Bhartruhari wrote vAkyapadIya (Book of sentences and words). Instead of grammar, he pondered on the Philosophy of Languages, where he traces formation of a sentence all the way back to the abstractness of sound. This sixth century work contains some very fine abstraction thinking, which has formed the basis of much of 20th century linguistics.

An analogy can be seen with programming languages too. The underlying “language” of ’em all is 1s and 0s, but the syntaxes and semantics vary vastly. With so many languages being created, one question to ask is the Philosophy of the Programming Language. Every language, every framework must have a philosophy, or in loose terms, the “why” behind it. Why did you select your syntax the way it is? Why did you name something this and not that? What can you solve with your language that the others cant?

The philosophy of Java is “Write once run anywhere. And screw the pointers”. The philosophy of C# is like “Something like Java, but sorry, no Js. And screw the conventions”. The philosophy of Groovy seems to be “You are a programmer, you know better. I will standby and hint functions, you do whatever you want with it. And screw the configurations”. The philosophy of TFS, the rich man’s poor version control, is like “I know everything. You programmers don’t know sh*t. Beg me to keep your source safe. And screw You”. In essence it is like “I dont like this feature in this language. So I am creating a new language without that feature”.

So on the lines of the philosophy of Groovy, in which simplicity and intuitiveness seem to be the gravest concern, there are three built-in closures, that I find extremely powerful – the find, collect and inject. If used correctly, they reduce some complex code to really trivial and make a mockery of standard loop syntaxes.

Perfect numbers are indeed a very fascinating math phenomenon. They are few and far between and whether there are infinite perfect numbers is a not yet solved mystery. I will spare writing about perfect numbers as there is an abundance of online literatue on them. Here’s a very simple Groovy OLC that finds perfect numbers upto a given number.

def find_perfect_numbers(def number) {
(2..number).findAll { x-> (1..(x/2)).findAll{ x % it == 0 }.sum() == x }

assertEquals(find_perfect_numbers(10000), [1,6,28,8128]

The internal closure finds all the factors of a give number, while the outside closure finds all those whose sum is equal to that number.