The Mutable State-of-Art

Once very often, social media lights up with opinions about the Javascript language, frameworks and its ecosystem being terrible. Some time ago, there was one around React framework, and the question was whether putting all HTML/CSS/JavaScript in one file is a really a good thing, which is against the age old practice of separation of these technologies.

Whenever people question the new ways of doing things, I am reminded of one of the greatest quotes by the Sanskrit poet Kalidasa (4th century CE).

“Just because its old, something cannot be assumed right. Just because its new, something can’t be brushed away. The smart ones analyze and adapt, the fools are swayed by others’ opinions”.

Many retweets, faves, dislikes, epics, savages later, the question still remains: What is the right language or framework?

The separation of technologies then, say 15 years ago, was not intentional. It was a necessity of times. CSS, HTML, JS were advancing independently, performance in browsers were so poor and finding an engineer who could do all was almost impossible. But now, we still have specialists designers and developers, but in general they have a good idea about the other technologies. I still remember back in my IBM days, pointing out to JSP structures to our designer, on where to make style changes. But now these frameworks and their idiosyncrasies are not strange to designers.

Here are my thoughts on why it is pointless criticizing the Javascript world for its organic growth, and equally pointless to criticize the criticism.

Whether “language purists” like it or not, Javascript is ubiquitous. Yes, that’s very hard to digest. Despite its “severe short comings”, it has conquered the net. Why? As IT engineers, we keep holding ourselves to this arbitrary sense of “perfection” and trying to judge everything against it. Typed languages, OO Patterns, Functional paradigms, Modularity, Testability – we seem to know what should be “right” and we are inching towards it – but we just aren’t there yet, we say. But we will know when we reach it, we say. Really? Every language invented since Javascript, seems to be solving the same problem of types, patterns, modularity, dependency etc, only in different syntax combinations.

Here is a fundamental question: Have we got anything right at all in the programming industry at the first stab? Or is 50 years too short to judge? The real limit of the problem is that physiologically, human brains can only process a limited amount of information, so it has to break complex into simple pieces. Eventually the simple pieces evolve to become complex enough, or perhaps our brains shrink so it perceives simple things as complex. So they have to be broken again. But the broken simple pieces have something in common with previously broken simpler pieces that they have to re-integrated. Thus we swing into an eternal pendulum of centralization and distribution.

Here is an illustration with evolution of Java, but you can substitute it with any starter technology:

  1. Java and JVM were originally built for small devices
  2. But once it entered application server space, EJBs were added to tackle distributed environments.
  3. EJBs became so complex, it was abandoned and broken into smaller pieces
  4. Smaller pieces were dependency-injected to form a bigger solution, but eventually they grouped into micro services
  5. Micro-services combined with distributed systems to facilitate container architectures, like Docker
  6. Container architectures are becoming complex and are yielding to orchestration solutions like Swarm and Kubernetes
  7. Predictably, By induction, orchestration solutions will become complex and will yield to Local Clouds
  8. Predictably, commodity local clouds will be broken into Fogs, Cirrus, Altostratus, Cumulus … and delivered by drones.
  9. Predictably, Drones functionality will super-evolve, so they will eventually need a new programming language specifically written for small devices… well…

Much like a fractal, as we dissolve complex problems into simple solutions, they just take a new shape of complexity, due to external influences (mobile, bandwidth for eg). The point is – This cycle will never end.

Working as a Web + Network application engineer (on the SDN controller OpenDaylight) I am seeing two ends (philosophically speaking, they are the ends and the means respectively) of a spectrum. Both UI and Network are always evolving in interesting ways, yet we can analyze these two into a fundamental problem statement:

Network: Deliver content (a packet) from one place to another place, efficiently.
UI: Deliver content (html) from one place to another place, for a user friendly read.

(Of course I am intentionally oversimplifying, but as an end user, when your bandwidth drops below 1Mbps, you call up the customer service to negotiate a discount, not enquire the complexity of the network protocols).

But just to solve the packet delivery problem, think about myriads of protocols and schemes created – TCP/IP, VLAN, MPLS, BGP, VXLAN, OpenFlow, NETCONF and many combinations of these. Similarly in the UI world, all we have to do is deliver readable content, but there are so many aspects and choose-your-noun libraries CSS, SASS, WebPack, D3, Material, visualization, accessibility, modularity, layouts and the most subjective of all – Colors!

So why do front-end developers keep seeking and preferring newer frameworks? There are many reasons and not all are technical. The fundamental reason is: Boredom. Human mind always seeks something new. Not because its useful, but it has to be something just different. And perhaps have a personality.

For any framework in general, to be popular and become sustaining, it should satisfy ALL these conditions:

  1. A very nice familiar sounding easy-to-pronounce name
  2. Should NOT be one person driven
  3. Code should be easy to change and testable
  4. Good documentation
  5. Not do too much magic
  6. Ability to teach it easily to others

The golden rule of framework success is this: It takes a few smart developers to create an idea. It takes a few hundred average developers to sustain it, because the smart developers have moved on to a new idea.

Advertisements

His Last Framework

Its been a while since I visited my pyschiatrist and now, Chiropractor Dr. Pear – Perspective Readjustment Specialist. Last time, he helped me cope up an aggressive experience with pair programming. The front-desk handed me a big chunk of forms to fill up my software history.

Years of experiencing web frameworks: About 18.
First framework of pain: JSP Model 2, perhaps Applet, can’t remember.
Latest framework of pain: AngularJS
History of major pain points: EJB, WebSphere, WebLogic, Spring, Fusion Middleware, Wicket, ASP.Net Web/MVC, SharePoint, Grails, SpringBoot, Backbone, Ember, AngularJS and innumerable minor ones.
What type of pain is this? (Select one: Sharp, Shooting, Numb): All of them
Approximate location of pain: Initially Back-end. Then moved to Front-end.
Exact location of pain: Depends on Mood.
Frequency of occurrence of major pain: About every 18 months.

I physically submitted the forms and started checking my tweet feed. New tweets from dzone on Top 20 Javascript frameworks; AngularJS Jobs; Best 35 CSS Frameworks; Top 15 build tools for Javascript; AngularJS vs.., no wonder I needed a therapy.

The nurse called me in and after the initial handshake of pulses and vital stats, she said 200. I was idly looking at the magazines – PlayWeb, Maximum Frameworks… Dr. Pear entered after a knock.

Dr. Pear (browsing thru my history rather than looking at me): Oh, Hello! Its been a while from your last visit.
Me: Yes, Doctor.
Dr. Pear: And what brings you here?
Me: Well, Ive been spinning my head with the next web application framework and I haven’t been able to decide quite…
Dr. Pear: Well, the short answer is “As a consultant you should learn all of them !”, ha ha ha… (after a pause) Yeah, yeah I know. Its the viral season of frameworks. The Javascriptarix flu shot we gave last month was really a placebo. So what kind of framework you are really looking for?
Me: (a bit hesitantly) Not sure, Ive been taking the Javascript MVC Kool-aid for the last couple of years, but it looks like the pain isn’t gone…
Dr. Pear looked at me a bit condescendingly and said “MVC eh? in Javascript? Didn’t you get high enough on it in the Java/.Net world?”.
Me: Well, the Java web frameworks are kinda relegated to providing data services these days…

I had not finished. It must have triggered a raw nerve in him and he went on to a mini lecture of sorts.

“You see, web frameworks come in two general flavors – MVC and Component. Java failingly flirted with the Applet component model initially, and then took a stab at MVC via JSP Model 2, while ASP.Net took the component route after its success with Visual Basic. Interestingly, Java increasingly adapted the component framework model, but Microsoft abandoned ASP.Net Web in favor of ASP.Net MVC. Java/JVM probably tasted all kinds of frameworks in many levels – MVC (Struts, Stripes, Spring, Grails, Play) and Component oriented (Seam, Tapestry, JSF & Co, Wicket, GWT). The fundamental problem with MVC is – it sprays your logic and model everywhere. Many developers struggle to organize it between “what-they-are” vs “what-they-do”. As the complexity increases, the code gets re-organized by features, each of which in turn have the m-v-and-c. Add on top of that, the domain models and view-models are usually different, the whole thing about models start looking like a beauty pageant.”

He paused a second, probably aiming for me to get the pun.

“Component frameworks couple model and view tightly, because a view makes no sense without a model. But the biggest problem for them came from somewhere else: Javascript. Many pure java web frameworks are relatively stable, because of a stable and backward-compatible Servlet API. But component frameworks relied too much on encapsulating Javascript and HTML and in JS world, Friday’s release is outdated by next day Matinee show. Ajax, jQuery and related libraries innovated way too fast and library encapsulation really made no sense. When the JS libraries went unsupported, it dragged the Java frameworks along. With NodeJs, NPM-modules and unit-testing frameworks offering a vastly superior build ecosystem, life became miserable for frontend developers, bogged down by verbosity and slowness of JVM build tools.”

Me: Well, that’s exactly why I am here for. What is the JS framework you would really recommend?
Dr. Pear: Of course Backbone. I love backbones, especially if they aren’t straight. Get it? Ha ha !

He realized I didn’t look at him very admiringly. He continued.

Dr. Pear: Ok. Well, I hate to say it depends, because you didn’t pay me to hear that. So let me tell you something else. With JS frameworks like Backbone, Knockout and Ember becoming first class citizens, server-side slowly mellowed down to doing CRUDGE-work (Create, Read, Update, Delete, Gather and throw Exceptions), now gloriously termed as Microservices Architecture. With AngularJS offering a complete MVC stack, it doesn’t make sense to write Views in server-side anymore.”

Me: So should I still stick with AngularJS MVW ? Besides its barrage of new vocabulary redefinitions like service, factory, provider, scope, directive, the next version 2.0 promises to be completely different and backward incompatible…
Dr. Pear: Yeah, I heard that. It appears that you really need to work on your persuasion skills on convincing your team about writing in new framework every year.
Me: (I looked at him awkwardly, wondering what to say)
Dr. Pear: (Ignoring my awkward pause) Well, the web has to evolve. Why do you think Javascript is going bonkers with frameworks – they have to compete with rapidly advancing browser features, handheld and wearable devices. Who knows, Javascript could be running inside your brain tomorrow. Ok, so you are now tired of MVC in Javascript world also. What about Facebook ReactJS?
Me: ReactJS. Yes I’ve heard of it. Isn’t that a component oriented JS framework?
Dr. Pear: It sure is. You write HTML directly within Javascript or CoffeeScript or Typescript in an XML like syntax and compile them to Javascript.
Me: Im not sure I like the sound of that. Thats not even the best practice.
Dr. Pear: Well, its time to Refactor the Best Practices too. IMO, ReactJS may indeed work out well, because unlike the Java component frameworks, its closer to Javascript, it uses Virtual DOM providing fantastic rendering speeds and helps write isomorphic apps. If you get used to JSX, you will love it.
Me: What about Services?
Dr. Pear: Use one of the many flavors of Flux, Reflux, Fluxxor, Alt – the Action oriented one-way-data-flow-store.
Me: Hmm, that eerily reminds me of Struts. So, there is no really a best framework?
Dr. Pear: There is one – come up with your own. You will surely get a few fans.
Me: What if I get bored and jump ship to another framework? What about the developers who depend on my framework?

“Send them to me!” Dr. Pear said calmly, washing his hands for no reason. “If anyone thinks JS frameworks have more than 18 months of fame, they need a Perspective Readjustment”.

As I was leaving, he gave a parting shot, “Just remember one thing, if it makes you feel any better: no matter which framework, you are solving only a single problem: Delivering html to a browser. And by the way, if you haven’t selected your styling framework, I am open on Saturday afternoon”.

“Er.. Thanks, I will think about it”, I said as I flipped my phone to follow @ReactJS feed on my twitter.

As I was walking out I could see him frantically typing on the screen “Successfully readjusted for component framework. Will likely come here in the next 12 months looking for Web Component and Polymer adjustment.”

Note: Title is a tribute to AC Doyle’s His Last Bow.

Thoughts on SpringBoot

In its new avatar, Spring enters the opinionated frameworks marketspace, and finally takes the Sinatra/Scalatra/Ratpack/Dropwizard/Spark route, where embedded containers (ie no external server containers) will start dictating the future. Whats an opinionated framework anyway? Aint all frameworks opinionated? To me every piece of code is just as opinionated as the developer who wrote it. Here are my observations about SpringBoot, after having worked with it for about 6 weeks.

1. Build support for gradle (+)
SpringBoot apps can be built using both maven and gradle. But I believe Gradle will eventually dominate the world of buildscripts. So the sooner your team moves to gradle, better will be your quality of this life and perhaps, next life too. Are you listening, .Net?

2. Starter poms (+)
Remember your 500 line pom.xml, where the tag <dependency> occurs 25000 times? Say good bye and start using starter-poms. Combined with gradle, a typical build file is now about 30 lines.

3. Configurations (+/-)
Spring is among the last standing action heros of xml-based application configuration in jvm world. Super hit when introduced, but soon became a liability. Thankfully, many developers challenged the xml tag soup and created innovative web frameworks (Wicket, Play etc.). Those who work with the Spring-based framework Grails, may have not touched an xml for a good period of time. If Spring MVC developers are still stuck with Spring without adapting Grails, I fail to understand why (well except for business impositions).

Ironically, SpringBoot seems to have many opinions when it comes to configuration – it supports 3 flavors of bean configurations: xml, annotation and groovy dsl. Having worked with Grails and other non-xml based applications for the last few years, picking up SpringBoot was bit of a chore. Grails autowires beans by name, and when custom beans are required, there is this wonderful bean builder. For me, java annotations are no better than xml configuration. Xml configuration proliferates verbosity but annotations defeat the purpose of wiring objects separate from code. While it brings type-safety, it also lets you mix code, configuration and logic and I think its a total mess. Annotations are like antibiotics – good in small doses (@ToString, @RestController, @AutoClone etc.), too many side-effects with overdose – building whole applications using annotations is @NotMyStyleOfTea.

A typical SpringBoot application has many many annotations – @EnableAutoConfiguration, @Configuration, @ComponentScan, @Conditional etc. Sometimes you end up with more annotations than the code itself and its not intuitive which is doing what when how. SpringBoot is certainly simpler for developers from xml background, but to me as a Grails developer, annotations have been a bit intimidating and sprawling. I don’t see CoC (Convention-over-Configuration), instead I see CoC (Configuration-over-Code).

Thankfully, SpringBoot has great (although not 1:1 xml equivalent) support for groovy beans dsl (via the GroovyBeanDefinitionReader). Groovy dsl is elegant, concise, intuitve and very readable. For some loss of type-safety (which could have been compensated by good tooling support), it comes with a great punch – wiring beans, environmental configurations, programmatic startup logic (as opposed to declarative) etc. I feel Spring can standardize groovy bean as the only configuration mechanism and shed all the fat of xmls and annotations. It would make the framework pretty lean and competitive in the already crowding lean frameworks market. May be thats what grails-boot is?

4. Properties (+/-)
Just like configurations, there are a few ways of defining and injecting properties. Coming from Grails background, the many ways of injecting properties was a bit confusing. SpringBoot supports both .properties and .yaml files. AutoConfiguration uses many default values for properties, but there is no comprehensive documentation on these properties on whats default. Again there are many annotations related to properties @ConfigurationProperties, @PropertySource, @Value, @EnableConfigurationProperties etc. Grails has this amazing single-point grailsApplication bean based on ConfigObject (a glorified Map) and allows nested and runtime evaluate-able configurations – helpful in dynamic scenarios. Again I wish Spring had defacto support for this (injecting properties from a config.groovy).

5. Rest Support (+)
SpringBoot makes it very easy to create rest controllers via @RestController. Instead of creating full-blown web applications, Spring’s eco can be used to create well-rounded rest services backed by Batch/Data/Integration/Security and use js frameworks like Angular/Knockout/Backbone etc. for front end. If using rest over json, Groovy 2.3 is promising to come up with fast json marshaller. While I like the cleanliness of Thymeleaf, the modern js frameworks have a clear advantage over server-side html generators.

6. Logging (+/-)
Yet again, too many logging frameworks in the bag: log4j, log4j2, slf4j, slf4j2, commons.log, java.util.log, logback. I spent some time resolving dependency conflicts, until I finally gave up and switched to logback. Spring team strongly recommends logback – Just Go with it – there is probably a good reason.

7. Data Access (+)
No question about Spring’s versatility here. Name any db and you have SpringBoot autoconfiguration support. Plus the Grails team has done a great job of spinning-off Gorm to standalone component. TexMex, I would say.

8. Testing (+/-)
Many examples still show use of JUnit, but here is a good start on how to use the incredible Spock framework in SpringBoot. Spock is like mom’s recipe – once you taste it, others aint the best.

9. Documentation (+)
It has improved a lot with newer minor releases. It takes time to sift through some old examples in the community, but lets just blame Google for not knowing what you exactly want, though it seems to know all about what food you want to eat when you are near a restaurant.

10. Finally
I think Grails3 (grails-boot?) is going down the trending route of embedded container deployments. I think that’s the only thing against Grails in the current trend. SpringBoot has got there first but I still feel it lacks the simplicity of Grails or the leanness of Play. It has certainly been simplified, but not simple.

If your existing eco system depends a lot on Spring based frameworks, it is worthwhile to adapt SpringBoot. Honestly, Im hoping Grails3 isn’t far off!

Taming of the Pair Programmer

In a recent project at work, I went through an intense experience of pair programming with some of the thought leaders in pair programming work model. While I enjoyed the experience, I do feel that pair programming psychology has been given a serious miss.

So I visited the world famous psychiatrist Dr. Pear for consultations but unfortunately, the notes were leaked out due to a security breach from the cloud platform where the medical records were stored. Here is the transcription.

Dr.Pear: Hello! (looking at my records) May I know the reason for your visit today?
Me: Sure Dr. Pear. I have been having some delusions about pair programming and I want to understand how to overcome pair programming anxiety.
Dr.Pear: (notes down): Oh, a case of PTPPSD (Post Traumatic Pair Programming Stress Disorder). Have you tried the OTC medicine – Pairitin ?
Me: Yes, doctor. But it hasn’t helped, obviously. Placebo.
Dr.Pear: Oh, well … Ok, I have a few questions. Can you describe your regular office day?
Me: I check my mails while driving to office, reply to them on the elevator, get a cup of coffee, then find a pair and start doing things. We switch roles of driver-navigator till the end of the day. And the next day we switch pairs and do the same stuff.
Dr.Pear: Sounds normal. What about breaks?
Me: Yeah, we do take breaks. And I always explicitly communicate what I am about to do – whether its bathroom breaks, get coffee, drink water, check my personal emails, answer my phone, stand up, sit down or shake my head. One time I experienced a TalkOverflowError, by recursively saying “I am about to talk” before actually talking. I have come to believe that Pair programming is just a catchy name for “Thinking through mouth”…
Dr.Pear: How do you feel about the quality of work?
Me: Its definitely improved. It has helped me identify minor bugs pretty early, that could have been a pain later. But there are times where I have completely rewritten my code at night, after an all day unsuccessful pairing.
Dr.Pear: Yeah, I have heard that happen. In general, do you feel you are productive?
Me: Sometimes yes, sometimes not. It does take longer time to achieve things initially.
Dr.Pear: Thats natural. Pairing is a trade-off between delivery-time and knowledge-spread. Its an insurance against concentrated knowledge. The preimum for that insurance is the early time investment. It also psychologically eliminates situations to unfairly blame others like ‘I cant touch that person’s code, I dont know what’s in there’. Eventually, its all about hitting the sweetspot of productivity. It does take some trial and error.
Me: Ok, I get that… But I have also developed some phobias recently, like mysophobia
Dr.Pear: These are side effects of this disorder. When you are pairing, you have to ensure that your personal hygiene is never compromised, especially if you are sharing the keyboard and mouse. Always clean your keyboards, do not eat while typing and wash your hands often. And don’t ever fall sick.
Me: Should I shower everyday?
Dr.Pear: I’m going to pretend I did not hear that…. now, this may sound unrelated, but nevertheless, how much time do you spend with your wife everyday?
Me: 1-2 hours?
Dr.Pear: …not including like helping with dishes, watching TV, you know.. just sit around..
Me: Just sit with her?
Dr.Pear: Yes, just quality talk time…face to face…husband to wife…
Me: 15 mts..?
Dr.Pear: So you spend 15 mts with your life-time companion everyday, but sit with your coworker for 8 hours straight talking face to face about every action you do?
Me: Apparently, Yes.
Dr.Pear: Hmm, recent scientific studies have proven that any amount of time spent with a coworker as a pair more than that you could with your own spouse is detrimental.
Me: And which study is this?
Dr.Pear: Its called the Pear’s Pair Limit. If (time-spent-with-pair > 1.5*time-spent-with-spouse), you end up in whats called Productivity Blackhole.
Me: I see… Does that mean I can pair only for 15 minutes a day?
Dr.Pear: Or you could spend 8 hours of quality talk time per day with your spouse…
Me: (blinking and awkward silence)
Dr.Pear: Got yaa! Ha ha ha! You developers always fall for that… Ok.. how about remote pairing? Like with people you have never met before face to face.
Me: Happens all the time… This one time, I ended up talking on headphone for over 10 hours. I ended up with a soar throat and my younger kid wondered why I was talking to the computer all day long.
Dr.Pear: Ha ha ha! Your kid is very observant, indeed.
Me: ???
Dr.Pear: Was just kidding. (murmurs) Get some humour, you techies… Anyways, so who else practices pairing in your office?
Me: All programmers, of course.
Dr.Pear: No, I mean pairing, not just pair programming…
Me: I thought pair programming is meant for programmers, Duh!
Dr.Pear: Er.. not exactly. Again scientific studies have shown that pairing is beneficial only if applied at all levels of an organizational structure. You know, like you can’t have one agile team and the rest of the organization is waterfall. At the programmers level, a risk of a bug usually costs much lesser than an issue that is not addressed at the architecture or management level.
Me: So like a Pair Architecting, Pair Managing?
Dr.Pear: More than that. “Pair Resourcing”. A programmer’s bug is lot more tangible, than a decision made at an architecture or management level. While a code defect can be fixed by some other programmer, “fixing” a wrong decision at the management level by another manager is usually very expensive. The higher the level of management, the costlier the fix.
Me: Ok, I get it now… An organization should always hire two people for the same position?
Dr.Pear: No. I meant more resposibility-spread in all levels. Imagine your reporting manager, who knows and appreciates all your hard work, quits. And your new manager is like Clark Gable from Gone with the Wind. A pair reporting manager would still know your work and share that with the new manager. The same technique goes all the way to CEOs.
Me: I see…and how is this measurable?
Dr.Pear: Pairing effectiveness can be measured quantitatively. You have Resources on the X-axis, Domain on the Y-axis. More vertical dots means one resource knows more stuff. More horizontal dots means many people know one stuff. Ideally connecting dots of edges will form a rectangle – everyone knows everything. Realistically it is a histogram. This is called Pear’s Knowledge-Responsibility-Spread graph. The closer the ratio of area of rectangle to area of histogram to 1.0, the better the organization’s base.
Me: Ok, I feel a lot better now. I think…
Dr.Pear: Sure, glad to help. Also here are a few business cards, please distribute in your office to your pairs and managers.
Me: (hesitantly) Er.. .Thanks!

Dr.Pear seriously types some notes about me on his laptop. I glean over him gently and looking at the screen and say “Er… doctor, I think you have a typo, you spelled my name incorrectly…”

Dr.Pear: What? Are you trying to pair with me now?

Unassumable Abstractions

In the recently concluded annual technology conference of my organization, the keynote speaker Kirby Ferguson presented his talk “Everything is a Remix“. It also appears in TED. Watch it, its pretty good. Sometime last year, I had posted a blog on the similar theme, that we do not really invent anything. All we do is look into recycle bin and pull out existing ideas, but “innovate” just enough to present in a different form.

I remember one of my first implementation of a functionality in Visual Basic, back in 1995 – drag-n-drop a combobox from Toolbar panel, open the properties page and set the database connectivity properties, select a few columns, click on some validation checks and voila, the client retrieved the data from db! How simple can it be? Apparently not enough.

In writing web applications, there are two main styles – component based frameworks (CBF) and action based frameworks (ABF/MVC). Both have pros and cons based on the needs and perspectives and the war between the two seems to shift in balance of victory from time to time. Microsoft started with a CBF Asp.net solution and eventually dumped it to go towards MVC. Java started predominantly with ABF and later geared towards a few CBFs (JSF, Tapestry, Wicket etc).

I was recently working on a custom application that should create a component based solution based on a ABF. Before even designing the solution, I was giving a lot of thought about the philosophy of these styles. CBFs completely abstract the underlying mechanisms (models, views and data access) to provide a uniform API to create solutions, while the ABFs abstract each layer separately and lets you wire them up as required. Both, obviously have their utility value, but one problem I see with building component solutions is that if the underlying protocols change, almost the whole framework has to change (eg. Tapestry 4->5 and also Tapestry 5.4 having to replace the existing javascript layer).

The reality is that the underlying mechanisms have been evolving. And evolving fast. For eg., if a component solution had abstracted the rdbms access, javascript and web requests, the former two have changed significantly over the last two years. NoSqls have already disrupted the rdbms space and javascript frameworks like AngularJS are directly exploiting html, quite ingeniously. A CBF abstracting with an assumption of only HTML5 and CSS3 is already behind the curve. There is another way though – create a very high level of abstraction where all these layers could be switched at will. Even if there is such an abstraction, it would need intermediate-abstractions to bridge them to low-level code. Not too maintenable in the long run. This is one of the reasons that ASP.Net, which is tightly coupled to all layers, just could not scale. Other similar component solutions will meet a road block somewhere down the road, depending on the level of abstractions they provide. The reason is same – the assumption of stable assets, does not hold anymore.

In the ABFs, the individual layers are abstracted on its own – abstraction of views by GSP, jQuery, Angular, Express, Ember etc; abstraction of server-side by Grails, Rails, Spring MVC, Asp.Net MVC etc. and abstraction of data access by GORM, JPA, Hibernate, Spring Batch, Spring Data etc. In case any of these frameworks go out of fad, it is only necessary to refactor that abstraction and its relations, instead of rewriting the whole solution.

The challenge with the web technologies now is that newer simpler ways of achieving the same results are happening too fast. Granted, the protocol (http) itself is not changing, at least for now (WebSockets anyone?), but what changes is how data is being mined, how it is being groomed and how it is being presented. Let us take a short look at the three layers – mining, grooming and presenting.

Data-mining: The advantage with document dbs is you get to store your model as you want to be, not as what the datastore dictates you to be. If you evaporate all the data objects, it will basically residue down to two: Map and List. Neither is “natively” supported in an rdbms. You have to decompose them to some id and, more importantly a relation.

Data-grooming: Many data-grooming happen around xml and json. Unfortunately for xml, from being a mere human readable representation by strings, it took a very impractical turn – complicated schemas, explosion of verbosity, confusing binding libraries, generic representation of data types, configuration and code logic. It falls by its own weight of abstractions. In a traditional web application, think about the data conversions that you do – they take about 60% of the effort – converting from request input to command objects to domain or service layer objects to stored procedure models to actual data! And all the way back to presentation layer! (N+1)-Tier FWIW!

Data-presenting: Of the three, this layer has proven to change wildly of late. With javascript back as a first-class language, several frameworks have challenged the “traditional” notions. CBF solutions, which try to neatly wrap intricate mechanisms of mvc, would work only if what they wrap are stable. If what they wrap start to evolve frenetically or completely stop evolving, they just wont scale any more. Instead of providing real new features, CBF solutions end up reworking newer abstractions.

I was working on a project with Grails + MongoDB + AngularJS, the conversion layer is reduced to request from html and saving as json; Reading as json and passing it to Javascript. From thin-client to thin-server, moved we have, O Dooku! And while I was looking for something even simpler than this and I came across Breeze.js framework and found this code:

breeze.EntityQuery.from(“Orders”)
.where(“Customer.CompanyName”, “startsWith”, “Alfreds”)
.expand(“Customer, OrderDetails.Product”)
.using(manager)
.execute().then(querySucceeded).fail(queryFailed);

So querying the database domain models directly from presentation layer? Heard of it somewhere? Yeah, I heard that first in Visual Basic. Everything is a remix indeed!

The class of methodologies

Back in early 1990s, as a college student in Russia, one of the rituals I had to undergo was Collective Farming (колхоз), mostly comprised of picking up fall vegetables like potatoes, carrots, onions etc from the fields. While a typical student could be working in a restaurant or lab to earn the college funding, the government instead funded the education and got the students to pick potatoes. A good trade off, I guess, that also taught me an invaluable experience on team work. You pick potatoes one by one, the sum of which eventually feeds the whole country for the winter.

Modern economies can be broadly classified into three flavors: capitalist, socialist and socialist-disguised-as-capitalist. Having lived and closely observed the economy and culture of three major countries (India, Russia and USA), I see a glaring similarity between a project methodology and economy – both have people, leader, finance, budget, success, failure, debt, credit and blame in common. The economy, or at least in the perception of a society, over the course of several years, seem to shift between capitalism and socialism.

Waterfall methodology resembles a capitalist economy. The dominant voice (rich) leads, most others merely toeing the line. The powerful voice, could be the project manager, architect, backend developer, frontend developer or qa – typically whoever has the most experience and knowledge sets the pulse of the project. The client (people or ordinary stockholders) are told they are the most important, but are usually damned until a crisis happens. Everybody’s stock value is very different and there is always a hero who saves the early morning. It is not always that the most hard working get the credit, but the loudest. Also everyone is or appears pretty happy, until the last minute of the release, that is when the project starts to crumble, just like panic strikes in when the bubble bursts.

When the economy went burst a few years ago, it was surprising to see streaks of socialism in US economy, which touts free market capitalism in every possible opportunity. Repeated failures of large projects using waterfall methodology have given rise to Agile flavors, which is the equivalent of a socialist economy. The team decides what to do, the team shares the glory, the team shares blame, everybody works, no one is idle for long time, no single person is a hero, everybody is accountable, everyone is perceived equal in value and yet, true to socialism, some, like the product owners are more equal than others. But socialist economy also has problems. In the absence of a leader (= product owner) it will go nowhere. They also eventually yield to corruption, suppression of creativity, inertia and devaluation.

The third type, socialist-disguised-as-capitalist economy – I like to shorten it as “chaotic”. “Startup” companies adapt this methodology. Just like India, extremely agile yet filled with huge portions of inertia between. Things appear totally haphazard, but some how click at the end, just like the traffic. No documentation (traffic rules) are necessary, communication is more by intuition, you just look at each other and determine who makes the move first. Awesome sauce really, QA looks at the Developer and instantly knows what features are implemented and what aint. When you see a crisis, there is a very simple solution: Just Honk.

Given that Agile resembles socialist economy, in about 10 years from now, it may not be very popular and may become what “EJB” is now – people know what it is, but not many want to go there. Agile works now for today’s issues, but so did waterfall for yesterday’s issues.

Human psychology, human behavior may not fit into Agile eventually. Egoism is the basis of any work, and without the expression of that, work will be undervalued. Throughout history, personal glory has always been coveted more than anything else. While team work tends to be appreciated from time to time, thats not in the nature of human beings. Why do you think “team work” is so much emphasized in Dora, Diego, Wow wow Wubbzy, Mickey Mouse clubhouse and those several hundred Disney cartoons? Why lay so much stress on it, why because, humans are born as “individuals” first, and have to be inculcated the concept and benefits of team work into them from the early age. But again, inherent human tendencies may resurface much stronger than ever before, ushering a modified version of waterfall methodology.

The motorcycle diaries of webservices

Software is a big grandpa pendulum clock. It oscillates between simplicity and complexity perpetually. Ideas and concepts that start off simple, rapidly become complex and eventually complex ideas are broken down into simple components and again these simple components evolve into complex creatures of their own. After all we evolved from a single cell into complex cell living beings, why blame the software for that?

Webservices necessitated from complex interaction between disparate systems (remember CORBA, JNI, DCOM, RPC, mainframe apis?). All those were dissolved into ‘simple’ wsdl based services, as long as client and server maintain the contract, everything was promised to be a simple hookup.

But it didnt end there. Generating wsdl clients either in Java or .NET were complex in its own terms. Java discovered its way, as usual, with hundreds of wsdl clients and xml jars (anyone still using xerces, xalan, sax4j directly?) – jaxb, jax-rs, jws, cxf and several more proprietary ones). .Net, was not a A-shot either: first asmx, then wcf and then several flavors of .net versions on top of that. Most of the time, its not a one-step process to call a WCF wsdl from a Java based client, there are always some surprises around the corner in terms of configurations. Its heartening to see nobody got it right the first time.

If you think about it, what was the necessity to generate classes from wsdl in order to send plain xml text data? The early xml apis described the xml in terms of hardcore hierarchy: root, nodes, parent, children, siblings, single. It could have been easily extended to represent a oh-so loveable joint-family: cousins, uncles, aunts, nephews and nieces:

public getAuntyNode() {
return currentNode.getParent().getFather().getSister().toNode();
}

public getDistantUncleNode() {
return currentNode.getParentNode().getMother().getSisterInLaw().getHusband().getSibling().toNode()
}

public getImMyOwnGrandpaNode() {
//Outsourced
}

Such family friendly APIs would have thrilled developers who often work only after the family goes to sleep.

I am also starting to see that xml and xsd schemas are not an ideal way to describe data models. Describing a model in xsd using the semantics of xml itself is over complicated. On top of that extensions, simpleTypes, complexType – are way too tedious and garrulous. If you are exposed to C++/Ruby/Groovy world, you can visualize the models pretty quickly in terms of encapsulations, compositions and interfaces. In the xml world, I only see tags and attribute names. Lots and lots of them. The content is lost in the tag-tsunami.

While reviewing a project recently, where all our models are xsd based, data models extending from base xsds, and several simple, complex and xsd:any types, pretty soon I found them going above my head. Im not even thinking how xsds maintain versions of models and references.

Coming to actually using them in our Grails project, a colleague of mine tried to generate the client classes using jaxb (before trying a few other clients) and pretty soon found himself in a maze of conflicting classpaths from different generated wsdls. Jaxb generates classes for all the dependent xsds (obviously) and generating them for different wsdls (in the same project) ended up with several classpath issues.

Taking a break, we decided to ask a fundamental question: “Why do we have to worry about all the xml types in the Grails code, if all we need is to send a plain textual xml data?”.

Who cares what the model hierarchy of xsd is? Most of the time that model is useless to the Grails code, where typically the representation of model is different from the xsd models. All those xsd generated classes, objects, factories, abstract factories were required because the complex familial Java APIs were the only way to process xml (unless using StringBuffer and appending data manually).

Enter Groovy’s MarkupBuilder, it is much easier to build an xml than generating the client. So if we all care is to send data and generating that data itself is easier than generating clients, why bother at all with the later. And we switched to Grails wslite plugin, with which we are directly creating soap requests and sending it to server.

So thats how the pendulum switched back to its other position, back to creating hand made soaps. Eventually this may get complex too. But again, the important thing to remember is that the original complexity is not the same level of complexity of the broken down components. The “absolute complexity” remains unaltered and the “perceived complexity” depends on how lazy or smart we are.