Why is Software Development on the Web so messed up?

It's been said that those who do not learn history are doomed to repeat it. In the spirit of absolutely not wanting to go again through the excruciating pain of software development that was the norm in the '90s and the first 5 - 6 years of the 2000s, I am going to dissect the following historical document: XML, Java, and the Future of the Web, published on October 2, 1997, by Jon Bosak.

As I walk through some of the salient points exposed in that portentous documents, I will try to bring to your attention certain glaring errors in understanding and judgment regarding what is the goal of computing and of the information processing on the web, in particular.

Let us then begin with the opening sentence:

"The extraordinary growth of the World Wide Web has been fueled by the ability it gives authors to easily and cheaply distribute electronic documents to an international audience."

Right out of the gate, Mr. Bosak takes the left turn and commits himself to the implementational aspect of the technology he is discussing. The key concept, of course, is 'electronic documents'. In his view, the web is useful in its ability to offer an affordable distribution channels for prefabricated electronic documents.

The problem with the Ivory Tower views of the web is that, being high-tech bent, the high priesthood inhabiting the Ivory Tower tends to view the smallest unit of processable information as an object, such as electronic document.

Following at the heels of the opening salvo, the second sentence in this paper claims:

"As Web documents have become larger and more complex, however, Web content providers have begun to experience the limitations of a medium that does not provide the extensibility, structure, and data checking needed for large-scale commercial publishing."

This is now quite confusing. If the smallest unit of information processing on the web is an electronic document, what need for extensibility, structure and data checking is he talking about?

Let's look at the third sentence; maybe we'll find some clarification in there?

"The ability of Java applets to embed powerful data manipulation capabilities in Web clients makes even clearer the limitations of current methods for the transmittal of document data."

Huh? Java applets? OK, this is completely obsolete, but still belies the unexplained bias toward powerful data manipulation capabilities in Web clients.

In the fourth sentence (and opening a new paragraph), the author introduces the concept of Extensible Markup Language, and explains:

"... an Extensible Markup Language (XML) for applications that require functionality beyond the current Hypertext Markup Language (HTML)."

What is not clear yet is what kind of "functionality beyond the current Hypertext Markup Language (HTML)" is he referring to?

HTML -- imaginary limitations

After delivering the opening salvo, as exposed above, the author goes on to elaborate on the severe constraints that HTML introduces and how debilitating these constraints appear to him. Here is the author's list:

• Lack of Extensibility
• Absence of Structure
• Inability to perform Validations
  
  

Extensibility

This feature is the holly grail to developers who are keen on creating their own, proprietary protocols. The much celebrated power that Extensible Markup Language (XML) brings to the table -- namely extensibility, is at the same time the most troublesome aspect of the failed attempts to engage in collaborative computing. If anyone and everyone is free to use the technology that enables them to daydream any protocol they wish (in effect creating their own unique language), the Babilonian confusion created that way must quickly reach cosmic proportions. This is exactly what's happening right now on the web -- everyone feels entitled to dream up their own bloody protocol, and the potential online conversation tends to fall mostly on deaf ears.

The author states this clearly: "HTML does not allow users to specify their own tags or attributes in order to parameterize or otherwise semantically qualify their data." The key phrase is "their own" -- which amounts to "invent your own language".

Structure

Similar to the old adage 'form follows function', structure typically follows the meaning of the information. Mr. Bosak's lament in this respect is largely unfounded, since HTML does offer fairly comprehensive structure for expressing the semantics of the represented information.

Whether the author was aware of that ability to express semantics using HTML, remains a bit of a mystery. In any event, what the author seems to be complaining about with regards to the lack of structure in HTML is the erroneously perceived need for representing complex hierarchical dependencies.

There is really no evidence that an average consumer perusing the web resources has such a pressing need for consuming large and increasingly complex hierarchical constructs. In author's own words: "HTML does not support the specification of deep structures needed to represent database schemas or object-oriented hierarchies."

Yes, the above is true, but the question is: why would anyone have the need to consume 'deep structures, object-oriented hierarchies, or database schemas'? The question, of course, remains unanswered.

Validations

Let's examine Mr. Bosak's objection regarding HTML's inability to perform validations: "HTML does not support the kind of language specification that allows consuming applications to check data for structural validity on importation."

I must admit that I'm a bit confused -- why would anyone use a semantic markup language, such as HTML, to perform validations on 'importation'? Importation to where?

Responsibilities of the Web Client

The author then enlists some imaginary responsibilities that are, in his view, mandatory for the web client:

1. Web client may have to mediate between two or more heterogeneous databases
2. There is a need to distribute a significant proportion of the processing load from the Web server to the Web client
3. Web client often has to present different views of the same data to different users
4. Intelligent Web agents will attempt to tailor information discovery to the needs of individual users

The above list belies a strong desktop-centric approach to software development. All of the 'needs' listed in the four items above fall naturally into the server-side jurisdiction. The client is only responsible for rendering the representation that the server side business logic prepares.

Linking

Toward the end of his paper, Mr. Bosak addresses some perceived inefficiencies that he sees in the hypertext system as implemented using HTML. His list is, again, very revealing. Here is his list of perceived inefficiencies when implementing linking via HTML:

• HTML lacks location-independent naming
• It lacks bidirectional links
• It lacks the ability to define links that can be specified and managed outside of documents to which they apply
• No N-ary hyperlinks (e.g., rings, multiple windows)
• No aggregate links (multiple sources)
• No transclusion (the link target document appears to be part of the link source document)
• No attributes on links (link types)

Below is my response:

Location-independent naming: Huh? If I link to a resource representation, such as google.com, how's that location dependent? Where in that hypertext link do we see any information that would reveal location of the resource representation named google.com?

Bidirectional links: This is the engineer's wet dream; having links that will never get broken. Not gonna happen, time to grow up and deal with reality.

Links that can be specified and managed outside of documents to which they apply: It's the damned documents again! Snap out of the document dogma and start living in the world of resources and their representations. In other words -- content!

N-ary hyperlinks (e.g., rings, multiple windows): Not sure what this means, but I really cannot see any real life use for it.

Aggregate links (multiple sources): OK, 10 years ago no one apparently knew about real simple syndication and such, plus the aggregators etc.

Transclusion (the link target document appears to be part of the link source document): Also, no clue about the mashups that will become all the rage 10 years later...

Attributes on links (link types): What on earth does that mean? Link types? Like what?

Giving Java something to do!?!

Finally, the author cites the slogan "XML gives Java something to do" (no sources quoted in the original paper).

What does he mean by this battle cry? What would he envision Java will do with XML?

It's hard to say after more than 10 years had passed since that battle cry was uttered, but my hunch would be that he was referring to the 'nomadic' nature of Java. In the olden days, Sun Microsystems was selling the "network is the computer" slogan, and so at that time Sun's trailblazing language was Java. According to Sun, Java lives on the network, and is nomadic, courtesy of RMI (Remote Method Invocation). Snippets of Java code can travel the network, on demand, in order to supply just-in-time computations on the requesting client.

Supposing that the client had just received an XML document but doesn't know how to present it to the user, a snippet of Java code could be invoked and downloaded on a whim. Once installed on the client, Java routine will massage the XML etc.

This dated model of 'distributed computing' has long been pronounced dead and buried. The problem is, we still have armies of software developers who tend to think in a similar vein. Because of that, software development on the web has turned into a terrible chore, and the resulting web sites tend to be nightmarish.

The moral of this story is to learn the lesson from the history, become aware of the blind alleys that have been paved by the developers a decade ago, and learn how to avoid the minefields, pitfalls and traps of the obsolete software development practices.

You're welcome!

Resource Representation and Discoverability

We now seem to be entering the phase where we are in the process of reclaiming the web. Who are we reclaiming it from? Why, from no one else but the hordes of software developers.

The web has been hijacked by software developers. It is only natural that the people who dedicate their lives to becoming as intimate with the machines as is humanly possible get to be the people who take center stage in building the web. But, there is trouble in paradise. The web is, in its essence, the exact opposite of the machines. The web is all about humans.

Now, software developers are humans, however they are humans who value serving machines more than they value serving other human beings. Because of that, the web is a horrific place right now, fully dedicated to serving the machines.

This terrifying state of affairs must change. And the only way it will change is if humans reclaim the web, pry it out of the slaves-to-the-machine greedy hands, and install it at its proper place -- as a platform for serving the needs of the human mind and the needs of the human social dimension.

Resources and representation

As has been argued elsewhere, resources on the web are mere abstractions. They are not the tangibles that one could indulge in. At best, one can hope to consume the representation of the abstract resources scattered around the web.

It is often erroneously assumed that resources are directly exposed on the web, in their raw form. One example would be the unanimous conviction that URLs are web's native resources. Thus, a URL such as craigslist.org is viewed as a web resource.

Nothing could be further from the truth. A URL (such as craigslist.org) is not a web resource, it is merely a representation of a web resource. There seems to be an abstraction labeled as craigslist.org and hosted somewhere on the web. But what that abstraction (i.e. craigslist.org) really is, we have no way of knowing.

What we can learn about is one or more representations of that resource. For example, one representation of the craigslist.org resource is its URL. Another one could be the list of all the sites offered by craigslist.org. And so on.

What's the use of representation?

Some people fail to see the usefulness of the representation. They'd much rather get their hands on the resource itself, instead of beating around the bush of resource representation. The best way to explain this problem is to employ assistance of some heavy duty science:

Map is not the territory

This famous premise was issued by Alfred Korzybski, one of the seminal thinkers who helped shape the communication and information theory.

A map is the representation of the territory. Without the map, we'd be lost when traveling through the territory.

In a similar fashion, resource representation could be viewed as a map that eases our voyage through the territory (i.e. the resource we're exploring).

Very few people tend to question the usefulness of maps. It is our hope that, similarly, people will learn to embrace the usefulness of the resource representation.

Discoverability

The web is intended for human consumption. Humans are notorious for having fairly constrained short-term memory buffer, and are thus forced to consume information in a piecemeal fashion. The architecture of the web is therefore tailor-built to serve exactly that constraint -- consume the resource representation at a fairly leisurely pace. Expecting humans to consume the resource representation in a single giant gulp would be utterly unrealistic, and so the fundamental architecture of the web is based on the principle of discoverability.

What that means is that, on the web, we are serving byte-sized chunks of resource representations. These byte-sized chunks are intended to be consumed by human users pretty much at a glance. Any need for consuming more, for learning more, gets fulfilled by letting the users explore further, allowing them to discover more intricate details of the resource representation.

Away from the sociopath web

Today's web is mostly built by humans who crave serving the machines. As such, the way resource representations are typically architected on the web today belies the optimization to the way machines consume information. There is typically very little discoverability offered on most web sites, and the consumers of the representation are usually expected to digest insanely vast quantities of intricate information in a single gulp.

It is painfully obvious to even a very casual observer that the web today is not tailored for easy consumption by humans. It is as if some sociopaths, who lack any degree of empathy with fellow human beings, have built most of the web sites in operation today. Come to think of it, sociopaths is the fair characteristic of individuals who value interactions with the machines more than they value interacting with other human beings.

It is high time we start working on getting out of this sociopath hell.

Why is ROA Hard to Understand?

Resource Oriented Architecture is the term I've coined a year ago, only to discover almost immediately afterwards that others have been thinking independently along the same lines. During the past 12 months, I've witnessed numerous discussion threads and debates centered around the importance of ROA. Most of these discussion threads belie complete misunderstanding of the Resource Oriented Architecture, which is very alarming, given that ROA is the native architecture of the web.

And web is the native infrastructure for information dissemination, exchange, and sharing.

Old habits die hard

Most software developers in circulation today have cut their teeth by learning how to implement computing algorithms. Something akin to: "here is a jumbled sequence of numbers, write a computer program that will sort them in ascending order".

The emphasis in such an exercise is always on exhibiting reasonable amount of direct control over the execution logic. The would-be programmers have been groomed for generations to master the skills of gaining full control over the stream of instructions that the CPU executes. In other words, to be a computer programmer/software developer implies to them that a full control over all the processing aspects must be in place.

It's a single point of human control, same as it's a single point of machine control (i.e. the Central Processing Unit, or CPU).

But in the world of the web, such notions of centralized control are not only ridiculous, they are down right harmful. Still, armies of CPU-trained programmers have now been employed and deployed to develop software that will run on the web.

Is it a small wonder then that such 'central point of control' minded individuals are creating fully dysfunctional web sites? Indeed, old habits do die hard.

New habits are slow to emerge

What the above described situation then means is that the existing workforce that is dedicated to developing software programs is pretty much a write-off. Because old habits die hard, it is quite likely that it will be much harder to unlearn the old habits and relearn the new ones, than to start from a clean slate.

However, given the vast demand for web-based software that is out there, we must work on creating some sort of a compromise, in the attempt to bring the crusty old 'central point of control' software development workforce into the web fold. For that reason, we need to continue explaining the difference between the Central Processing Unit software architecture, and the Resource Oriented Architecture.

Transitioning from doing to managing

One of the hardest things to do in life is to make a successful transition from doing something to managing that same thing. The temptation to roll up one's sleeves while managing the process is simply too strong to resist. And that's why not too many people end up being good management material.

In the world of software development, right now there is a huge demand for people who are capable of abstaining from rolling up their sleeves and instead delegating the responsibilities to the 'workforce' out there.

What does that mean? In a more technical parlance, it's been proven that once software developers get trained in making procedure calls when attempting to process some information, they tend to get addicted to that gateway drug. Similar to how to a person with a hammer everything looks like a nail, to a programmer who had learned how to make procedure calls, every information processing challenge looks like a string of procedure calls. This algorithmic frame of mind is very hard to get rid of, once a person gets it contracted. And in order to become an efficient web developer, each and every procedure-besotted person must learn to let go of the procedure call driven world view.

Transitioning from local to distributed procedure calls

Most people learn how to program computers by implementing procedure calls that are local. What that means is that their world view is CPU-centric. In other words, the executable code in its entirety is sitting on a local box, usually right next to the developer's desk. The developer then intuitively grasps the world as consisting of a single Central Processing Unit which acts as a traffic cop, deciding on who gets to do what and when. The 'who' in the previous sentence refers to the procedure. Typically, a programmer will modularize his code into a fairly large number of procedures. These procedures will then be called at opportune moments, will perform the intended processing, and will then recede in the background.

This situation is the prescribed way for teaching newcomers how to develop software. It willfully ignores the real world issues, such as that there is very little use and need in having a 'desert island', 'single sink' scenario, where a computer user will be sitting alone engaged in the 'glass bead game' on the computer. In the world of business at least (and also in the world of entertainment), multiple users are a must. Meaning, CPUs and people will be networked (999 times out of 1,000).

There inevitably comes a time when a software developer wannabe realizes that a transition from a single CPU to multiple CPUs connected via the network, is unavoidable. At that point, the newbie usually gets struck by the immense complexities that such transition entails. And it is at that point that the newbies all over the world decide to drag in the procedure call world view into the networked world.

Now the problem of control in a multi-CPU environment arises. Who's in charge here? It used to be that a single human mind was in full charge and control over the execution of computing instructions, via the single CPU. Now, all of a sudden, multiple CPUs which are distributed across vast networks enter into the equation. The remote nodes start posing the problem when it comes to controlling the proceedings.

The only way for such developers to try and tame this complexity is to wiggle in their chairs and to fabricate some hairy-brained scheme for enabling remote procedure calls (or, RPC). Various mind-boggling protocols get implemented (CORBA, RMI, DCOM, SOAP, SOA, etc.) All of these protocols are incredibly arbitrary, non-compelling, frighteningly complex and brittle.

As we can see, this transition from making local, simple-minded procedure calls to making distributed, remote procedure calls never goes well. In the end, it inevitably results in fragile, brittle software that requires constant duct-taping and chewing-gumming in order to stay afloat.

Who's going down with the ship?

It is obvious to even a casual observer that the majority of software projects, today as in the past, fail miserably. As the complexities of the business scenarios continue to increase at a frightening rate, this rate of failure will only skyrocket.

At this pace, it won't be long before the software development industry turns into another Titanic, if the majority of developers continue to charge down the remote procedure call path. Developers who refuse to go down with the ship have only one option -- abandon the RPC ship and hop into the ROA boat.

But as we've already seen, the transition from navigating a huge, unwieldy ship to rowing a tiny nimble ROA canoe is proving more challenging than most developers are prepared to bear. That's why there must lie a major shakeout ahead of us, a shakeout that will separate the monolithic totalitarian software practices from the nimble, distributed ones. And ROA practices are certainly on the opposite side of the monolithic remote procedure calls practices.

How to make the transition?

It seems to me that, in the process of making the transition from RPC to ROA, the biggest stumbling block for the entrenched software developers lies in the fact that ROA proposes erecting an impenetrable barrier between the client and the resourses on the server. In the bad old RPC world, such barriers are unheard of. In other words, if the client is privy to the protocol that the service-oriented server had arbitrarily and unilaterally made up, the client can intimately manipulate the state of the server.

No such thing is ever possible in the ROA world. And that's the biggest put off for the honest-to-god procedural crowd. They simply don't seem physically capable of conceiving the world where it would not be possible to 'crack the code', learn the protocol, and then rule the server on the other end.

So in order to make a successful transition to ROA, these software dictators must learn how to abdicate from their dictatorial throne. What are the chances of that happening en masse any time soon?

Slim to none? What do you think?

Resource Representation

Software developers seem largely incapable of understanding the difference between resources and their representation. Understanding that difference is absolutely vital in understanding REST and the Resource Oriented Architecture (ROA).

Is it a small wonder then that we are beginning to see articles like this one ("Why is REST so hard to understand?") pop up in the publications devoted to the web architecture? It is absolutely clear that we won't be in the position to advance ROA practices unless we reach a critical mass of people who do possess a solid grasp of what is the difference between a resource and its representations.

Right now, it seems like only an infinitesimally small percentage of software developers are aware of the significance of that difference. For example, if we say that there are ten million software developers worldwide, 9,999,000 of these developers are blissfully ignorant that such a thing as resources and their representation even exists. Which leaves us with only about 1,000 developers who truly understand what it all means. Not a very encouraging statistics. As a matter of fact, the situation today is right down dismal.

We need to do something quickly in order to remedy this catastrophic situation. My small contribution here is to attempt to clarify the difference between resources and their representation. I will use mostly concepts from everyday life for illustration purposes. I will then attempt to draw the parallel between these concepts and the concepts one encounters in the arena of software development.

What is a Resource?

Let's start from the top: a resource is anything that humans find useful. For example, a house might be a resource. Or, money might be a resource. We find houses very useful, because they offer certain level of comfort in our daily living. We also find money very useful, because it's a resource that enables us to get other resources, such as houses.

Ten years ago, when I was buying my first house, I wanted to see what kind of a house can I afford. To that end I met with a mortgage broker, who interviewed me in the attempt to assess my buying power. Among other things, he needed to know how much money can I allocate towards the down payment on the house.

Once I told him how much money I have for the down payment, he was able to plug that information into his spreadsheet and then tell me what kind of mortgage do I qualify for. So I was then all set to go out and shop for the house, right?

Well, not so fast! Just because I've blurted out a dollar figure that I had in mind for the down payment, didn't automatically mean that I fully qualify for that mortgage. Because, you see, my word in these matters was viewed as being largely subjective, that is, prone to miscalculations and all kinds of other aberrations. What the bank needed is a more objective assessment of my buying power. Talk is cheap, and anyone can claim that they have one hundred thousand dollars tucked away in their bank account, ready to be plopped in toward the down payment, but the reality might be somewhat different.

Because of that, the banks have devised a more impartial practice, whereby the applicant must provide what is deemed to be a more objective proof that the money really is in the applicant's bank account.

In this example, the money that has been allocated for the down payment on the house is regarded as a resource.

What is a Representation?

Now, the down payment money we are talking about may indeed be sitting in my bank account. But how am I to truly convince the mortgage broker that it is really there? Well, I use the means of representation, that is to say, I use something else that acts on behalf of that money. For example, I use my words to convey the fact that I do have $100,000 in my bank account. My words then represent my money.

But, as we've seen, that representation (my words uttered in a casual conversation) is apparently not good enough for the mortgage broker. He needs something more 'solid' before he is fully convinced that the money is actually there. Meaning, he needs a different form of representation.

Typically, what the mortgage broker would deem a fairly impartial and objective representation of my money is a piece of paper issued by my bank that claims that it is true that I do have $100,000 in my account. In other words, once the mortgage broker receives that piece of paper, he can 'take it to the bank', so to speak.

That piece of paper is what we call a representation of a resource. The real money is nowhere to be found in that representation, yet in a somewhat superstitious manner, all the parties involved almost religiously obey and agree that a piece of paper is as good as the real thing.

Back on the Web

The exact same protocol abides on the web. There, as in everyday, non-virtual life, we have resources which are tucked away somewhere, and all we get to see, feel and touch are the representations. Resources on the web are merely concepts that are being represented using other concepts. On the web, as in real life, no one ever gets to see, hear, feel and touch a resource. All we get is mere representations.

We are free to attempt to manipulate those representations. And even in the cases when our manipulation yields perceptible differences, we have no grounds to claim the we have actually manipulated the resource itself. All we can conclude is that the resource representation has changed under the influence of our actions, nothing more, nothing less.

For example, if there is a resource such as a volleyball tournament published on the web, I can get its representation by sending a request for it. I may then decide to destroy that tournament by manipulating the tournament's representation. And my action may result in an affirmative representation (i.e. the resource may respond by sending me its representation which states that it has destroyed itself). I may then conclude that the volleyball tournament has, indeed, been destroyed. But I really have no way of ascertaining that. All I can claim is that I have acted upon the initial representation of that volleyball tournament, I have sent it the request to destroy itself, and I have received the confirmation that it is now destroyed. However, the resource itself may still be alive and well, existing somewhere beyond the reach of the resource representation. The only thing that is manifestly certain is that from that moment on, the resource will refuse to render its representation for the incoming requests.

This is similar to how, in the real world, no one would ever suggest that I take the mortgage broker into my bank's safety vault and point him to the pile of cash that's sitting there, and tell him: "Here, here's my $100,000, please count them and let's get on with it!" No one goes straight to the resource itself in real life, and the same holds true in the digital life as it gets distributed on the web (not to mention that even the physical cash is merely a representation of some other concept, etc.; one can never possibly get to the bottom of it, that is, the real resource is nouminal, unreachable by us mere mortals).

Understanding this distinction is vitally important if we are to move forward successfully in building the functional web of information. It is my experience that none of the developers I've ever worked with understand this situation, and that all of them erroneously believe that they are, at all times, working with the 'real thing'. Nothing could be farther from the truth, so I implore all developers to take a moment and try to digest the difference between resources and their representations.

Software Development Detox Part 4: Tribal Computing

One of the absolutely most heinous types of software intoxication is what I like to call Tribal Computing. This is the form of computing that is tied in with the primitive, proprietary setup and is a remnant of the old days of early adoption of computing by the businesses.

Ontogeny recapitulates Phylogeny

Ernst Haeckel (a German natural historian) wrote in 1868: “Ontogeny, or the development of the individual, is a short and quick recapitulation of phylogeny, or the development of the tribe to which it belongs.” (in this context, ontogeny is the development of the fetus, and phylogeny is the evolution of a species). Haeckel was referring to the way the fetal development of mammals seems to parallel the evolution of all life on earth. The fertilized mammalian egg first resembles a single-celled amoeba, then a multi-celled sponge, then a jellyfish, then an amphibian, then a reptile, then finally becomes recognizable as a mammal.

Applied to the field of computing, one could say that the development of computing seems to parallel the development of human society. At first, the society was segmented into small primitive tribes, which slowly evolved into larger units, such as cities, counties, provinces, regions, countries, nations, and so on.

We are today standing on a threshold of global computing (i.e. the web). But, in certain ways, we still seem deeply entrenched in the primitive, stone age world of tribal computing.

Think Locally

Most software in operation today is severely localized. Meaning, it was built to live on a single box, single CPU. Everything about it is very closed, very proprietary, and extremely local.

Some software allows for certain level of connectivity, whereby other software systems are given an opportunity to connect to the localized software and share/exchange some information.

Only the latest, most recent batch of software products (the so-called social software) has left the world of tribal computing and is reaching out to the global computing space.

Control Locally

Most tribal software was/is built with an engineering frame of mind. Whenever we approach building something with an engineering outlook, we are striving to introduce maximum level of control into the system.

One of the most detrimental side effects of building software with an engineering slant is the temptation to retain the state of the conversation. As we've seen in our first installment (State), the best way to create brittle and buggy software is to insist on retaining the state of the conversation that had transpired during the operation of the software product.

In addition to that, insisting on staying local (i.e. tribal, single box, single CPU etc.) means that the point of control also stays tribal. There is a single authoritative instance that claims to know everything and that controls what can and cannot happen on the system. That instance then becomes a single point of catastrophic failure.

Relinquish the Control Globally

In contrast, non-tribal software exhibits stunning capabilities for growth thanks to relinquishing the rigid engineering attitude. One of the fundamental reasons why web is such a spectacularly successive computing platform lies precisely in this abandoning of the tribal past and moving beyond the need to control and retain the state of the conversation.

By deciding to not care about the state of the systems engaged in the conversation, the non-tribal, globally oriented software is free to grow in any direction and to scale to any level of complexity.

We will see in the next installment what are the most optimal ways to achieve that level of robustness. Stay tuned.

Software Development Detox Part 3: Join the Conversation

So far we've established that the unsophisticated software solutions tend to be brittle and unreliable due to the following erroneous decisions:

1. Keeping track of the state of the conversation between the client and the server
2. Choosing to expose services through messaging interfaces (a.k.a. Remote Procedure Calls)

Today we're going to look into the minimal requirements for establishing the platform for enabling successful implementation of the business computing. We will see how important it is to separate the state of the conversation, that may have transpired between two or more business parties, from the transformations of the state of the participating parties. Also, we'll look into the advantages of abandoning the messaging interfaces when conducting business conversations.

Sustainable Business Strategies

Abandoning the world of software for a moment, let us examine the strategies that facilitate success in the world of business. Experience has shown that short term success is certainly achievable by exclusivity (i.e. by locking customers into some sort of a proprietary business model). However, for achieving any level of sustainability, businesses must, willy-nilly, open up and introduce free choices and embrace diversity.

In order to attain happy equilibrium of a sustainable business growth, one component of the business interaction must be optimized -- conversation. Businesses thrive on open-ended conversation. What that means is that while initially only two privileged parties may begin business conversation, other interested parties should be able to join and participate at any point.

For that to happen successfully, the barriers to entering the stream of conversation must be placed extremely low. In practical terms, anyone interested in joining the conversation should not be expected to learn any arbitrary set of rules, any new language, or any new protocol.

In addition to this fundamental requirement, parties joining the conversation midstream should also be able to easily examine the history. Anyone joining the conversation should be able to immediately gain access to the 'minutes' of all the previous conversations. That way, an interested party could, at their own convenience, examine the minutes and learn about some vital stats of the ongoing business transactions.

These two (the ability to join the conversation at any point in time and the ability to examine what had transpired up until that moment) are the most vital and essential requirements for conducting successful, sustainable business transactions.

Sustainable Business Computing Strategies

When it comes to achieving sustainable levels of business computing, the challenges tend to be even greater. Computing infrastructure and practices, that are being placed between various business parties interested in joining the conversation, serve to raise the barriers to entry.

Still, the promise of information technology is to make those barriers even lower than they tend to be for the non-digital business interactions. And in all truthfulness, that really is the true calling of this technology. So, the question then is: what went wrong?

The simplistic answer to the above question is that someone along the lines made a couple of wrong decisions (see the erroneous decisions listed above) and created a complicated situation resulting in businesses being unable to join interesting and potentially profitable conversations that are happening online.

Our job, then, is to remedy this situation, and to bring the technology back to the level where pretty much any interested business party is capable of easily joining the conversation. Not only that, but any party should qualify for easily examining the history of the conversation by replaying the 'minutes' of what had transpired while they were away.

Simplifying the Medium

In the world of information processing, bits and bytes could represent anything. But in the world of business computing, the medium that is of most interest is text. So in most situations, what is of particular business interest is content that is rendered as text.

Of a much lesser significance, but still figuring rather prominently in the world of business computing, are images. Less prominent media types would be audio and video.

So the viable computing platform that would fully facilitate successful business conversations is mostly focused on the content rendered as text. That content should be marked up in order to achieve certain level of semantic order. But the markup shouldn't be blown out of proportions.

Simplifying the Protocol

Once the medium for exchanging business content is sufficiently simplified, we must make sure that the channels for communicating that content are also fully simplified. Instead of expecting any interested party to learn the intricacies of a foreign language that is unilaterally enforced by the vendor, we must offer a severely restricted list of possible actions that are publicly vetted and extremely non-volatile.

For any resource that the businesses may be interested in, we can safely establish that only four actions would suffice when it comes to maintaining that resource:

1. Add resource
2. Remove resource
3. Modify resource
4. Fetch resource

The above four-pronged protocol is sufficient for conducting any business conversation. No further elaboration on the business computing protocol should ever be required (there is no need for any kind of messaging interfaces).

Joining the Conversation

When an interested party learns about the ongoing business conversation and wants to join in, all it has to do is fetch the state representation of the resource in question. From that point on, the interested party is free to propose any transition of the state of that resource. In addition to that, the interested party is also free to replay the conversation as it had unfolded prior to joining the conversation.

The above scenario is universal, and it applies across the board. Following this model, we can ensure that any business parties will have a successful transition to participating and contributing to the ongoing stream of business conversations.

Software Development Detox Part 2: Remote Procedure Calls

We've seen in our first detox installment that it is detrimental to maintain the state of the conversation between participating parties. The conversation that may have occurred between the interested parties and a targeted resource may have affected the state of the targeted resource. But that doesn't mean that the various stages of the conversation itself need to be retained.

Today we'll look into another form of software intoxication; this one has to do with how is the conversation implemented.

Typically, when two or more parties get engaged in a conversation, they tend to accomplish successful conversation by sending each other specific messages. For example, if an airplane is approaching the airport, the messaging between the pilot and the control tower gets peppered with code words such as "roger" and "over". That way, the conversation protocol gets established in the attempt to minimize the noise and maximize the signal.

In a loosely coupled world of business computing, free market forces dictate that many solutions providers can compete for the most successful solution. That situation creates a wild diversity of proposed conversation protocols. What's happening is that basically any vendor (i.e. solution provider) is free to come up with a completely arbitrary set of formal rules outlining how is the conversation going to take place. That creates a hell of a noise in the solution space, resulting in brittle and faulty implementations.

The culprit most of the time boils down to the methodology known as Remote Procedure Call (RPC). The remote procedure proposed by the vendor is a completely arbitrary, unilaterally constructed expression that the vendor expects the consumer to learn. In addition, the vendor reserves a unilateral right to change the expression encapsulated at the RPC level, and the consumer has no recourse but to re-learn the intricacies of how to talk to the vendor.

This unfortunate situation creates endless problems and headaches. All the consumers of the proposed vendor services are expected to keep an ever growing inventory of those custom Remote Procedures, and are on top of that left vulnerable to the future changes of that inventory.

Because of that, the effective detox therapy strongly advises against using the RPC model when architecting and designing your software solution. We will see in the next installment what is a much better way to go about accomplishing the robust software architecture.

Software Development Detox Part 1: State

In the first installment of the software development detox program, I am going to review misconceptions and misunderstandings of one of the most fundamental software concepts -- state.

The problem with understanding state in the context of information processing/software development arises when people fail to recognize and acknowledge that there are two distinct aspects of state in the arena of software development. By bundling up these two aspects, people end up projecting an incorrect picture and consequently paint themselves into a corner by choosing the unsuitable architecture.

I am now going to (temporarily) abandon metaphors (such as 'paint oneself into a corner' etc.) and switch to using simple, albeit somewhat exaggerated examples.

Software Types

In this example, I am going to review a common occurrence of a typical software construct, such as date. Date is an abstraction devised to encapsulate and express human concept of time. In the world of information processing, we use software constructs, such as types, to encapsulate and express abstractions such as calendar date.

Suppose someone offers a software abstraction (i.e. type) called CustomDate. This abstraction is supposedly capable of doing accurate date calculations, and is endowed with certain conveniences. One such convenience being the ability to express, in calendar terms, such human concepts as 'tomorrow', 'yesterday', 'next week', etc.

So we see that this type is capable of certain behavior (such as being able to answer the question 'what date is tomorrow?', etc.) But, in addition to discernible behavior, software types typically also possess state. For example, our CustomDate may possess a state of knowing what date is year-end.

This state may change (different corporations have different year-end dates). And the instance of the type is expected to remember the changed state.

What can you say and how it gets interpreted

Upon acquiring a new software type, such as CustomDate, we will be expected to learn about its capabilities. We are not expected to understand how it is working. We're not even expected to understand all of its capabilities. We are free to pick and choose.

For example, if the CustomDate possesses 50 different capabilities, and all we want from it is the ability to tell us what date is the year-end, we should be able to safely ignore the remaining 49 capabilities.

To violate this basic agreement would result in creating brittle, unreliable software. Here is one fictitious example that illustrates this problem:

If we instantiate CustomDate and assign that instance a handle such as customDate, we should then be able to talk to that instance. If we are only interested in learning about our company's year-end date, we can send a message to our customDate instance, as follows:

customDate.year-end

In response to receiving that message, the customDate instance will return the actual year-end date to us.

The above described scenario should always yield the same behavior. There shouldn't be any surprises in how an instance of customDate behaves upon receiving the year-end message. If there is even a slightest possibility that the established message may render different, unexpected behavior, our software is not only brittle, but extremely buggy.

By now you may be wondering how could there be a possibility that the above scenario ever yields any different behavior than expected? Let me explain with another example:

We've learned so far that, when dealing with an instance of customDate, we can say year-end and it will be interpreted as a question that reads: "could you please tell me what is the year-end date?" Consequently, the representation of the correct year-end will get rendered and served as a response to our query. We've thus realized that an instance of customDate has state. That state (i.e. the actual value of company's year-end date) is the only state we're interested in when dealing with this software construct.

However, as we've mentioned earlier, this software construct may have 49 other capabilities and states, of which we know nothing. Now, the fundamental principle of software engineering dictates that we are absolutely not required to know anything about any additional, extraneous states and behaviors that a software construct may bring to the table.

Regardless of that prime directive, people who are not well versed in designing software solutions tend to violate this dictum on a daily basis. The way to violate the prime directive would be to introduce certain state/behavior combo that will modify how the question gets interpreted. One can imagine how easy would it be to add a capability to CustomDate which will turn it into a currency conversion utility. This example is admittedly unrealistic and exaggerated, but I chose it to illustrate the foolhardiness of arbitrarily assigning various capabilities to a software construct.

In this example, an overzealous green developer may add a capability to CustomDate that will put it into a "currency conversion" mode. If someone else is using the same instance of CustomDate and puts it into this "currency conversion" mode, that change in its state may modify the behavior of an instance of CustomDate, rendering the response to the year-end question unintelligible.

Let's now run this hypothetical scenario:

1. CustomDate gets instantiated as a resource on the server


2. A message arrives from a client asking the resource to convert 100 USD to Canadian dollars


3. An instance of CustomDate (i.e. customDate) puts itself into the "currency conversion" mode and renders the proper currency conversion


4. The client then sends a message to customDate asking it for a year-end


5. The instance renders an answer that corresponds to the value of 100 US dollars at the year-end

The above answer at step 5 comes as a complete shock to the client who asked for a year-end; the client wasn't aware that the instance can be shape-shifting and consequently may not always be returning dates when asked about the year-end.

In other words, what you can say and how it gets interpreted changes based on the state that an instance of the type may be in. A very bad situation, guaranteed to render that particular software program dysfunctional.

Statelessness

We can see from the above example how disastrous it can be to attempt to manage the state of a resource. In our case, we've been managing the state of an instance of CustomDate, keeping track of when is it a date rendering machine, and when is it a currency conversion machine.

This tracking of the state resulted in the breakage of the working code. If we had abstained from keeping track of the state of the instance, the problems wouldn't have emerged in the first place.

From this we see that the only way to achieve robust and reliable software is to ensure that its constituent components are stateless. No memory of what had transpired during previous conversations should be retained.

However, keep in mind that we must distinguish here two types of states:

• Entity state
• Conversation state

It is this conversation state that is troublesome. Entity state is perfectly valid, and should be memorized. In this instance, entity state would be the fact that our company's year-end is October 31.

Keeping track of what transpired as clients have interrogated an instance of a software component, and then retaining that state, is always disastrous. And yet that is how most inexperienced software developers tend to architect and design their software.

Coming up

In the next installment, we'll look more closely into how to architect and design stateless software.

Contracts and Coupling in Resource Oriented Architecture

We've seen that most software developers in operation today are besotted by the vendor model. One of the symptoms of that malaise is the deeply ingrained idea of a contract. Basically, it would appear that, in the minds of typical software developers, there cannot possibly occur an interaction between software components without a predefined contract.

Of course, the contract they are talking about follows (albeit covertly) the implicit revenue model as ingrained in a typical software vendor's business plan. Software vendors make their living by locking into a contract with their customers.

But in cases where the utilization of the software product is not tied with any revenue model, the notion of a contract tends to lose its appeal. Like, for example, in cases where an open source community is bettering the services by utilizing the publicly vetted web standards and protocols.

There are no Unilateral Contracts

The web is a platform that publishes its capabilities in a unilateral fashion. Similar to any other publicly vetted service, the web is calibrated to serve the least common denominator.

For example, in all developed countries in the world there is a publicly vetted service called traffic. Various vehicles are free to enjoy the commodities that the public roads offer, mostly free of charge. However, the reason this service is made so freely available without strings attached lies in the fact that it comes with an extremely pronounced and unbendable bias. That bias states that all vehicles moving along the public roads must keep to the right side of the road.

If this system wasn't calibrated in such a way, it would be impossible to offer the amenities for conducting the public traffic.

So we see that the utmost flexibility of an infinitely complex system, such as public traffic, is only possible if that system is heavily calibrated in a non-negotiable fashion.

And because that calibration is non-negotiable, it does not qualify as being called a contract. In other words, if you're driving your car down the road, you're not holding the traffic contract in your hands that outlines all the details that bind you and the road owner. The road owner (i.e. the public) dictates the rules, and does so unilaterally. You have no say in whether you accept to keep to the right side of the road while driving. You simply must obey the unilateral rule, or suffer the consequences.

To put it more mildly, you must 'get used to it'.

Same principles apply when you're 'driving' around the web. The web specifies upfront the non-bendable rules and regulations, and you have no say in whether you like them, don't like them, whether you're planning to obey them or disobey them. The web doesn't expect you to make your stance known, and the web absolutely doesn't care.

Because of this unilateral set of rules, there cannot be a contract between the sever software on the web and the web client.

The Web is Uncoupled

Because of the lack of a bilateral contract between a web server and a web client, it is not possible to talk about any form of coupling on the web. Tight or loose coupling, which are terms tied with the world of software vendors' applications, lose any meaning on the web.

There are two reasons for this change:

1. On the web, the server need not know anything about the client
2. On the web, the server is blissfully unaware of the existence of any other server

All that is known on the web is that it is a sprawling collection of resources; these resources are easily accessible via the publicly vetted protocol (i.e. URI), and these resources know how to represent themselves and how to make their own state transition upon request.

Thanks to this particular bias, the web is the most scalable, the most interoperable, and the most robust, resilient and comprehensive computing platform ever.

Resource Oriented Architecture and Quality

In his recent post exposing imaginary limitations of the REST paradigm, Udi Dahan laments the lack of explicit dialog that would discuss the benefits of adopting REST. More specifically, he writes:

"What I haven’t heard is how I can use it (i.e. REST) to build large-scale distributed systems that are controlled by complex logic."

Here, I believe he is talking about his definition of quality. Building large-scale distributed systems that are controlled by complex logic sounds like a very costly proposition. Something one would not attempt just for the heck of it, just to kill a rainy weekend.

Because such a project entails a price tag of hundreds of thousands, if not millions of dollars, it is not to be taken lightly. Also, it is not something that happens very often. I would venture out to say that most of us have never been involved in such a project, nor will we ever find ourselves involved in such a thing. It is not very often that businesses embark upon building such complex distributed systems. Yes, eBay and Amazon etc. are such systems, but you could almost count these beasts on the fingers of one hand.

So the question that mystifies me a bit here is: why? Why worry about building such a large-scale distributed complex system? What's the business case for it? How many of those are you planning to build next year? What's your compelling reason for going after such a risky venture?

But assuming that Udi has his valid reasons, and that he has millions of budget dollars to back him up (which, frankly, I seriously doubt), let's engage him in his quest, for the sake of having fun.

What I'm assuming here is that, to Udi, large-scale complex distributed systems spell quality.

So You Want to Build a Multi-million Dollar System?

Why would a system we're building end up costing millions of dollars? There are several possible reasons:

1. We are using primitive technology, and thus it's taking us too long to build
2. We don't understand what is the system supposed to be doing, so we're making things up as we go
3. The scope of the system is on a runaway curve, and so we end up with explosion of features (i.e. featuritis)

In the case of primitive technology, we can imagine trying to dig the Panama canal by giving each worker a wooden spoon. That technology would be considered very primitive for the intended job, and would therefore result in the astronomical cost of the project.

Similarly, if we attempt to build an equivalent of amazon.com or ebay.com using assembler, we'll end up paying astronomical price for using such primitive technology.

In the case of not understanding the goals that the system under construction is supposed to deliver, 'fishing for results' would invariably turn out to be prohibitively expensive.

Finally, in the case of scope runaway, featuritis and bloat are well known for being insanely expensive.

Let's say Udi is unphased after reading this, and still wants to proceed with building his multi-million dollar large-scale complex distributed system. He is now curious how can REST and ROA be of any benefit to him. Let's explore.

Can REST and ROA Help?

My knee-jerk answer is: not really. But let's not give up so easily.

First, let's make two things clear:

1. If we don't understand what is the system supposed to be doing, then no amount of technology can salvage us from dying a miserable death
2. If we contract the featuritis disease and start bloating, there equally is no technological cure; we'll be doomed to collapse under our own weight

So let's assume here, for the sake of having fun, that Udi is immune from the two forms of malaise mentioned above. Let's say he has a very clear understanding of the goals of the large-scale distributed complex system he is trying to build, and let's also assume that he is immune from the featuritis and bloat (both assumptions being extremely unlikely, by the way, but let's pretend they've been licked).

Seeing a Problem where there isn't Any

OK, so Udi is concerned about the following:

"There are business rules that control which data should be returned and when. All this has been known for quite a while and has been modeled around transaction isolation levels in the OLTP sense, and needs to be handled at the application level when it comes to intelligent caching. I haven’t heard anything about REST that deals with this."

My question is: why on earth should REST be expected to deal with that? REST and ROA, as their names imply, specialize in dealing with Resources and their Representations. They don't specialize in dealing with the infrastructure that is the underpinning for the resources.

For all we know, that infrastructure might very well be implemented as OLTP, or perhaps as something else. Same as the ability of the RDBMS to index the data stored within it might be implemented using various algorithms. But in reality, who cares? The important thing is that it works and meets our expectations.

Next, Udi asks:

"If, as a result of updating one entity, other entities are affected how is that information propagated back to the caller?"

This is simple: the caller consumes resources by dealing with their representations. This is the essence of REST. The ground rule in ROA is that raw resource can never be touched by the client, or by the caller. All one is allowed to work with is the resource's representation.

Udi talks about updating an entity. I'll try and translate this into ROA speak. Instead of updating an entity, in ROA we speak of requesting a state transition for a resource. The way it works is as follows:

1. A resource gets identified (via the URI)
2. That resource then gets located (via the corresponding URL)
3. A request is then sent to that resource to change its state (i.e. to make a state transition)

At the end of that chain of events, other resources (or, 'entities' in Udi's parlance) may or may not get affected.

His question, then, is: how is that fact (i.e. that other resource got affected during the state transition) propagated back to the caller?

The answer is glaringly simple: it all depends on the use case. If the use case requires that the caller be notified of the state transition, then the appropriate representation of the new state will be presented to the caller in the form of a response.

One really feels like stopping at this point, and asking: what's the big mystery? Why is it that all these highly paid software architects are having such hellish time grasping this extremely simple concept of resources, their states, and their state transitions?

I am indeed forced to conclude that they seem to working really hard on trying to see the problem where there isn't any. Is that why they get paid those big bucks?

Here's the Skinny

Udi even gives us a tiny homework:

"Let’s say I have a simple rule: When the sum of all orders for a customer in the past quarter increases above X, the customer is to become a preferred customer. Do I PUT the new order resource to the customer resource, or should I have a generic orders resource that is “partitioned” by customer Id? Once the change to the customer occurs, should I PUT some change information somewhere so that the initiator can find out where it should go look for the change? And how should I be handling transactions around all these resources?"

Again, he is confusing apples with oranges. Let's break it down gently:

A customer is a resource. We don't know how is that resource implemented behind the firewall, and we really, really don't want to know! This point is crucial to understanding ROA. Please keep in mind that ROA is definitely not about the how.

OK, we now need to understand that a resource (i.e. customer) has state. For example, the resource could be in the state of being a preferred customer, or not being a preferred customer. However, we don't know, we have no idea how is that state implemented. Also, we don't want to know anything about those gory details.

On to how does that state change. Recall that REST stands for "Resource State Transition" (actually, this is incorrect, and this section talks about ROA, not REST; please see the correction notice at the end of this post). So 'state transition' is the crucial phrase here. Obviously, a resource is capable of making a transition from one state to another. Such as making the transition from the state of not preferred customer to the state of preferred customer.

How/when does that state occur, you ask? Well, that transition is driven by a particular event that must be described in the use case scenario. Like in Udi's use case above, in case of an event when the resource's state (such as total value of accumulated orders) reaches a predefined threshold, that resource knows how to make the transition from not being the preferred customer to becoming a preferred customer.

Again, we don't know how is this transition taking place. There are millions of conceivable ways that transition could take place, but the important thing is that it does.

All of Udi's questions surrounding this hypothetical scenario boil down to how, but that's largely irrelevant. Somehow the resource knows the proper way to make the transition from one state to the other state. When the client then makes a request for getting that resource, the resource in question will know how to represent its state, including the newly won 'preferred customer' feather-in-the-cap.

Correction: I've just realized that, in my eagerness to explain the encapsulation that Resource Oriented Architecture offers, I've made a mistake and erroneously ascribed state transition to REST. Actually, REST stands for Representational State Transfer, not Resource State Transition. The latter is a characteristic of ROA, not REST.

Sorry for the inadvertent error.

RESTafarian Jihad

I've just been accused of waging a RESTafarian Jihad. I feel that this accusation is unjust, and that it is uncalled for. So I need to explain and defend my case here.

Jihad is a strong word in this context, because it came to mean 'holy war' in the modern parlance. And I'm nowhere near waging a holy war on anything, least of on the non-REST platforms.

To be more specific, I am absolutely not interested in converting the Remote Procedure Call (RPC) people to REST. True, approximately 98% to 99% of all software developers I've ever met cannot imagine building a distributed software system without utilizing some form of RPC, so for these people, Resource Oriented Architecture (ROA) is meaningless.

But as I've already explained in one of my recent posts, I view these people as a write-off. Basically, I think there is nothing we can do about that, and that it's time we cut our losses and move on with our lives.

So you see, I'm leaving the RPC crowd (which means almost all software developers in existence today) alone. I'm not waging any Jihads, I'm not plotting to convert anyone.

Instead, I strongly believe that, as it has always been the case, the simpler system, with lower barriers to entry, will somehow win in the end. I don't really know how, but God's ways are mysterious, so we'll see. But same as CORBA had to disappear because it was just too unwieldy, web services and SOA will also be forced to vanish from the scene.

At that point, ROA will take the lead. Right now, we're preparing for that portentous event. It will come in its own sweet time. We're not rushing anything.

We know that, eventually, people will abandon the ways of the past century, and will embrace the twenty first century ways of employing high technology.

Resource Oriented Architecture and Vendors

Suppose we get to the point where Resource Oriented Architecture (ROA) becomes prominent and gains wide adoption. Given the ubiquity, pervasiveness and popularity of the web, it is not a far fetched proposition. After all, the native web is strictly resource-based, and a Resource Oriented Architecture that utilizes that orientation seems like a perfectly natural fit.

Now, if this re-orientation happens, what will the vendors do?

Vendor Business Model

Software vendors are in the business of making profit from selling software based solutions. Traditionally, the most efficient solution, from the vendors' profitability point of view, tends to come in the form of a tool. This is why we tend to call these vendors 'tool vendors'.

Because of this orientation, vendors tend to favor system architectures that are complex, convoluted, difficult to fathom and master. Of course, anything can be quite easily made more complicated than it already is. In other words, it is quite easy to muddle the issue, to add complexity, to increase brittleness. You don't have to be very smart of very competent in order to make things more complicated.

As a matter if fact, complicated situations are always a surefire sign of stupidity. If, on the other hand, you find systems that work but are nevertheless very simple and easy to navigate, that invariably means that someone quite intelligent has devised such a solution.

But, since an easy to understand, easy to navigate and use system is non-profitable from the vendors' point of view (i.e. such systems do not require 'tools' in order to be utilized), vendors do not see any value in playing along with them. Vendors prefer murky, complicated architectures. Getting you involved with such ungodly systems and architectures is a very lucrative proposition for the vendors, because then they have a very good justification to approach you and to start selling you their tools.

Do We Need Tools?

Take a practitioner of a well established profession, such as a lawyer, or a mathematician. These professionals deal with extremely elaborate, intricate systems and bodies of knowledge. They are expected to navigate very sophisticated waters of highly formalized systems.

But what kind of tools do they need in order to do their jobs? In reality, very few. For a lawyer may need nothing more than a typewriter, and a mathematician actually needs even less tools -- a paper and a pen will do just fine.

How come? It's simple -- their job is largely conceptual by nature.

Now, the question is: how is that different from software development? Why do we think that software development is not conceptual by nature, in ways similar to mathematical development?

The thing is, software development is not different in any significant ways from mathematical, or legal developments. They are all largely conceptual activities.

And realistically speaking, all these conceptual activities don't need many specialized tools. It is a great fallacy, perpetrated on us by the tool vendors with their business agendas, that software development needs an elaborate set of tools.

Choose the Right Foundation

And since we are dealing with a fairly conceptual activity when developing software, it is of crucial importance that we select the right foundation, the right philosophy, and the right paradigm from where we can operate successfully.

Choosing the right starting point will ensure that we don't get entangled in the morass of complexity, brittleness, analysis paralysis and the like. Otherwise, it would be devilishly easy to slip and end up in the blind alley.

If we now review the philosophy and the paradigms that the tool vendors are offering us, we see that they prefer to side with complexity, heavy lifting and hard work. The vendors are proposing we adopt systems that are architected for ever growing complexity. Theirs is the business model of introducing a big hairy problem, and then selling the solutions that would be pain killers.

So what's the alternative? We propose that instead of aiming at acquiring pain killers from the vendors, we focus our efforts on acquiring vitamins.

And in our vision, vitamins would be the publicly vetted standards upon which the web, as a universal computing platform, is built. The web offers us simple, extensible, scalable, interoperable and linkable architecture. As such, it is very beneficial to us in the sense that it does not introduce the pain of complexity which then needs to be handled with pain killers.

It's Not the How, it's the What

By now, we all know the intentions are older than implementation. There were times when foolish software developers used to think the reverse was true, but thankfully today most of them have been browbeaten into aligning with the program.

So everyone seems to agree nowadays that articulating the what (i.e. the intention) is much more important than articulating the how (i.e. the implementation). So where's the problem then?

Web Development Challenges

While every prudent software developer had by now learned the lesson in separating the what from the how, they still seem to be applying it at too low a level. Similar to how the budding software developers of the olden days used to build those monstrous huge bloated monolithic programs that ended up being totally unwieldy and unmanageable, today's web developers are falling into the same trap of building similarly huge, bloated, unwieldy web sites.

Or, let me put the question up to you like this: what would you rather build -- a huge monolithic tightly coupled software program, or an open-ended set of smallish, self-contained, loosely coupled software units, that are easy to implement, easy to test, easy to troubleshoot and debug, easy to deploy?

Of course, if you are a software developer worth even a tiny bit of your own salt, you'd go for the latter. Some of us still recoil in horror when the flashbacks of bad old days of monstrous monolithic software programs come back to haunt us. Ugh, not a happy memory!

But it's really funny how these time honored principles of separation of labor, inversion of control, etc., do not seem to translate at all when being ported to the web development. Here, when developing web sites, budding software developers are more than happy to fall into the same old trap of indulging in muddling the concerns and seeking a single point of control. Which ends up in incredibly messy, monolithic, coupled and unmanageable web sites.

This, again, is the outcome of blindly rushing in to grab the how! How do I do such-and-such in Rails? How do I do such-and-such in Ajax?How do I do such-and-such in Flex? And on and on...

People never seem to stop to think what. What do I need to do in order to make this work? Forget about how to do something, and focus on the what.

How you do something is really not a big deal. 99 percent of the time it's just a short google search away. So why sweat the small stuff?

The what is much tougher. Google will not be able to help you there. You'd need your own cool head, your own common sense to figure that one out.

The surefire sign that you're charging down the wrong path is if you find yourself building large monolithic web sites. You know the ones that rely on the function calls, with one central point of control? The ones where it gets progressively difficult to introduce any new capability?

You know the ones I'm talking about. I bet you that right now you're working on one such site.

Well stop! Same as you wouldn't work on a single humongous big ass class, but would rather chop it up into dozens and dozens of smaller, more manageable classes, you must do the same with that web site. Abandon it. Chop it up. Give up the single point of control, give up the irresistible need to shoehorn everything through the single funnel of omniscient point of control.

Think back to your other software development skills. Remember how tough it was until you've learn how to properly delegate?

Well, now's the time to do the same delegating, only not with objects, but with web sites. Treat each web site as a single independent component, that specializes in offering some useful resources. Then open us the traffic gates. Let those independent, standalone 'components' interact, and see what happens.

See, I gave you the what; the how should be a piece of cake for you now.

ROA and Microformats

The most recent feedback I've been getting on my ruminations regarding the Resource Oriented Architecture have been mostly concerned with the programmability of the web. In its vanilla state, the web is very easy to program. Basically, all the computer program needs to know how to do is identify the resource and then send it one of the three or four rigid, predefined messages that apply no matter what. These messages are:

1. Add this resource
2. Give me the representation of this resource
3. Make the resource state transition
4. Destroy the resource

Works like a charm every time. The beauty of this model is that it is unbreakable. Adhering to this model, one will never be forced to go through the pain of 'upgrading the web'. One's code will keep working no matter what.

Is Simplicity the Problem?

To talk to most web developers, you'd get an impression that this beautiful simplicity is more of a problem than a solution. Basically, it all boils down to the fact that programmers complain how this protocol (we're talking HTTP here) is too plain. It doesn't give them the 'power' they're used to when working with the Java API, or with .NET API and so on.

For example, in Java we have an open-ended world of unlimited custom made, home grown protocols. Anyone in the world is free to create their own monster mash, and invent their own capabilities and name them however they feel like. This is what programmers call 'power'.

But that's what I call 'weakness'. Why? Simply because it's so goddamn confusing. How can a thing that's so confusing be considered powerful?

The Problem of Discoverability

Some developers do recognize this problem (i.e. the problem with open-ended, unlimited world of home grown capabilities). Yes, it may be wonderful to have this vast world of incredibly sophisticated capabilities, but what's the point if no one knows about them? It would be absolutely unrealistic to expect that there be a central control instance that would maintain the world-wide inventory of all the ever growing capabilities that are being added to the web daily.

So instead of abandoning the wild geese chase, these architects suggest we use methods of piecemeal discovery. Various techniques have been proposed to that end: reflection, introspection, Web Services Description Language (WSDL), god knows what else. None of these really work, because even after you've discovered that there is a capability out there you had no idea existed, there isn't anything you can do to use it. This is because, while you may be able to discover the remote procedure call signature of that capability (i.e. how to call it, what types of parameters is it expecting, and what type of parameters is it returning), you still have absolutely no way of deciphering the meaning of that capability. What does it really mean, what does it really do?

You could always assume, but there is inevitably a big ass in every assumption.

It is very hard trying to interpret the intentions that some content conveys by relying on the formally measurable parameters. That would be akin to trying to figure out whether a person likes something or not by measuring that person's pulse, blood pressure, blood sugar level, brain wave activity, etc. Sure, all these things are measurable, but are they really conducive to attaining unambiguous conclusion?

Work from the Known, not from the Assumed

All the RPC methodologies prefer to work from the assumed standpoint. In other words, the RPC client prefers to engage the server in a preliminary conversation. The conversation goes something like this:

Client: "Hi, I am about to request that you render a service for me. Could you please tell me what you're capable of?"

Server: "Hi there, I offer wide variety of top-notch services for your exquisite enjoyment. What would be your pleasure today?"

Client: "Oh, I was hoping that you could help me convert inches to centimeters. Can you do that?"

Server: "Here is the list of things I can do (offers a long list of convoluted names)."

Client: "OK, let's see... (tries to find the name that would resemble the inches-to-centimeters conversion)"

Once the client makes a decision, the real conversation commences, meaning the real data may be exchanged.

In contrast, resource oriented client does not engage the resource in any sort of preliminary chit-chat. The client simply identifies the resource and asks it to send its representation to the client. The client examines the received representation and decides to either give it a miss, ask the resource to make a state transition, or ask it to destroy itself (or perhaps ask it to add a new resource). Simple as that. The conversation between the client and the resource commences right out of the gate. There's no pussyfootin'.

The Problem of Enriching the Protocol

So discoverability didn't really get us anywhere, nor could it ever do so. People are slowly but surely beginning to reach the conclusion that it is much safer and ultimately much better to ask the resource for its representation, than to interrogate it about its dubious capabilities. At least by sticking to the representation model, we know that our request will always get serviced in the predictable way.

But the problem now seems to be that the representation of the resource is not structured enough. What does that mean? Let's go back to my tennis court example for a minute -- if we identify certain tennis court in our town, and request to get its representation, the response will travel to our client and will be rendered for our consumption. We will then be able to read about it in more details. For instance, we may be able to see that this tennis court is not booked on Saturday morning, which is exactly the information we've been looking for (i.e. we've been searching for a tennis court in our town that would be free this coming Saturday morning).

So right there we see that this resource (i.e. tennis court) is endowed with the capability to be in a booked or free state. And that's all we need to know in order to fulfill our goal (and thus we'll find the web site that's hosting this resource to be very useful to us).

Now, most programmers see this situation as being very problematic. Basically, they are complaining that this representation of the resource is only human-friendly, and that machines have been left out of the equation. The highly unstructured content of the resource's representation may be fine for humans, but is all but useless for the machines.

Because of that, they propose that the rock solid HTTP protocol be enriched, ameliorated, and opened up for allowing us to enforce more structure upon the content of the resource's representation.

How do they propose to do that? Microformats is one way that seems to be getting many people's hope quite high. So let's look at how do Microformats propose to enrich the HTTP protocol.

The 80/20 Myth

Microformats offer a very non-intrusive approach to ameliorating the protocol. That approach is based on the more 'organic' view of things. In other words, it's bazaar rather than a cathedral, a garden rather than a crystal palace.

The so-called Zen of Microformats states that it only makes sense to cope with the 80% of the problem space, and leave the remaining 20% of the unsolved portion to take care of itself.

This, of course, is very reasonable. It is rather unacceptable from the engineering standpoint, but we all know by now that software development is as close to engineering as tap dancing is close to Dave Chapelle's Block Party.

In the nutshell, then, Microformats propose to open up the playing field for structuring the wild and woolly content as it is being served on the web as we speak.

Right now, it is possible to see some of the Microformats in action. Plenty of good ideas that definitely add value to the meaning of the structure of the resource representation.

So where's the problem? It's in the unsubstantiated belief that this additional structuring of the resource representation will catch on in approximately 80% of the cases. My hunch is that this expectation is hugely blown out of proportion.

The Selfish Web

One of the fascinating qualities of the web is that it offers one of the most altruistic experiences that emerge out of the most selfish motives. This is called 'harvesting the collective intelligence'. Each individual on the web pursues his/her proprietary, selfish goals, and yet the community tends to benefit vastly from such selfish pursuits.

But it would behoove us to keep in mind that, on the web, work avoidance is the norm. People mostly blurt things out on the web and then go on their merry ways. No one has the time nor any intention to stop and carefully structure their content.

Presently, the content offered on the web is at best structured to offer the following semantics:

• HTML head with a half-meaningful title (hopefully)
• body with (hopefully) only one H1 (heading one) markup tag
• ordered/unordered lists enumerating some collection
• divisions with semi-meaningful class names and ids

If one is extremely lucky, one may find an HTML representation of a resource that offers such well-formedness. But in most cases, the representations we do get are even below such extremely lax standards.

How are we then to expect that Microformats will pick up and reach the 80% of all representations? I think it's a pipe dream. I am doubtful that Microformats will ever reach even 20% of the representations out on the web. I hate to say this, but I'm afraid that we're more realistically looking at 2% to 5% rate of adoption.

Only time will tell, as always.

Is the Web Machine-Friendly?

People would like the web to be machine-friendly. What does that mean? Basically, people would like be able to teach the machines to go out on the web and do the legwork for them.

Now, machines are notoriously very brittle, and tend to easily break. You throw an unexpected piece of information at the machine, and it freaks out.

Humans are different. Humans can cope with irregularities. That's because we are blessed with common-sense.

Is the Web Broken?

We know for a fact that the web is not machine-friendly. But does that mean that it is broken? Some people tend to think yes, since the web cannot offer regular, uniform and predictable experience to the machines, it is broken.

Some other people, myself included, tend to think differently. I don't believe that the web should be made uniform, just so that the machines could traverse it without experiencing any hiccups.

So, in my view, the web is not broken. The web is just fine the way it is. It is the expectation that the web must be machine-friendly that is broken.

Smart Servant does not Imply Automation

Most people make a primordial mistake upon hearing about the Smart Servant and think that it means some really smart piece of automation. But that's very 19th century thinking. Today, in the 21st century, this is not what we're after anymore.

We're really after humanization of the technology. We want machines to learn to bend over backwards and kiss their own ass and serve our human needs. Nothing more.

And for that, we don't need massive automation. We don't need to turn the web into the wasteland of bland uniformity. Let the web be what it already is -- an enormous mass of messy, irregular, wacky and crazy stuff. That's life. That's what human beings thrive on.

We need to harness the technology that will help us participate and contribute to this mess. We don't need technology that will help us clean up and solve this mess. It is up to us, humans, to decide what's a problem and what isn't a problem.

Should the Web be Easy to Manipulate Programmatically?

My answer is: why? Who needs programmatic ways for accessing the content on the web? This is because I want to be in charge. I am the one who's in the driver's seat. Even when I hire a chauffeur to drive me, and am sitting in the back of the car, it is still me who is in the driver's seat.

Same is with the web. I am the consumer, the participator, the contributor on the web. I don't want machines to do that. I don't see any value or benefit in expecting the machines to do that.

Should the Purpose of Conversation be Pre-Established?

In response to my recent post on Resource Oriented Architecture (ROA), Abhijit Nadgouda made the following comment:

Not sure if this meant that the capability was a surprise to the client. If the client did not know of the capability, why did the client engage with the resource? Shouldn't the purpose of conversation be pre-established? Should the client expect any capability from the resource?

Just to clue you in, I was talking about the absence of a need for a client to know the particulars for the resource the client is interacting with. So long as the resource understands the request to represent itself, to make a state transition, and to eventually destroy itself, the client can accomplish its goal.

Now, when the client identifies and locates the resource, it is most natural for the client to expect to receive the representation of that resource. To use the example from my original post, if I, as a client, am looking for a tennis court where I could play with my friends, upon identifying and locating the potential court, I'd like to see its representation.

At that point, the identified tennis court will ship its representation to me. This representation will then be rendered in my browser. By examining the representation I've just received, I should be able to get a better picture about the resource's capability.

Now, Abhijit's question is: is this capability a surprise to me, the client? Well, hard to say, isn't it? I mean, it all depends on what I, as a client, was expecting when engaging in the conversation with the resource.

If, for instance, I was expecting the resource to offer shower facilities and it didn't, then yes, maybe I'd be surprised. But then again maybe not, because many of the public tennis courts in my city do not offer any amenities.

The Web Is About Exploratory Behavior

Another interesting question is this:

If the client did not know of the capability, why did the client engage with the resource?

It's called exploration. And web is all about exploring. Poking around. Does this tennis court have a wall to bounce the ball off of, or not? I don't know, let's explore and find out.

Shouldn't the Purpose of Conversation be Pre-Established?

Why enforce such a constraint? The web is also about freedom. Let me start by an informal chat, and see where it takes us. There's no need to impose an authoritarian, military type of a rigid conversation that mustn't deviate from the very narrow 'norm'.

Should the Client Expect any Capability from the Resource?

Yes, and we've covered that expectation in very many details already. To reiterate, a web client invariably expects any resource on the web to be capable of representing itself, of making the transition of its state, and of destroying itself.

Learn to Walk Before You Try to Run

It is wonderful that so many people are nowadays serendipitously discovering the power of Resource Oriented Architecture (ROA). However, one of the most problematic aspects of human nature is impatience. I feel qualified to speak about this foible because I am one of the most impatient people around.

And that's always been the downfall in any venture. So it is with rediscovering the world of resources. When I say 'rediscovering', I'm trying to remind the reader that resources have been built into the web from the day one. But somehow, we chose not to notice them. Instead, we chose to focus on remote procedure calls (i.e. the services).

The Follies of Impatience

Humans like to complicate things. Here we have a very robust and simple situation -- an ever growing collection of resources which could be accessed via three simple commands: represent yourself, modify yourself, destroy yourself. But in our blind impatience we've jumped at the conclusion that things are just too simple and that we need a much more complicated protocol in order to make things work. We rushed to invent the dreaded web services.

This impatient over-engineering is so foolish, that one is at a loss when trying to find an equivalent folly in other engineering fields. The closest one I could come up is with the system of traffic lights.

Today, we have this traffic regulation system that consist of the protocol based on the state of the physical semaphores out in the streets. At any point in time, each semaphore could assume one of the three possible states: it could turn turn green, which means 'go', it could turn yellow, which means 'get ready to stop' or it could turn red, which means 'stop' .

That's all there is to it. Very simple very elegant, and serves its purpose perfectly -- to regulate otherwise astronomically complex traffic patterns.

Now imagine if we've given in to the engineering follies of impatience, and let civic engineers take over the asylum and go nuts with their 'solutions'. We'd be then quite easily looking at 64 different colors that a traffic semaphore could take. Mauve would possibly mean 'go, but be forewarned that there's construction ahead', blue would mean 'speed limit around the corner', purple would mean 'food and lodging ahead', brown would mean 'traffic congestion after the next intersection', and so on. The possibilities are endless.

Would that be an improvement? Quite the contrary, it would be a veritable disaster. Not only would the chances of drivers getting totally confused be blown out of all proportions, but we'd probably witness the 'vendor wars' all over again. Different municipalities would strive to come up with their own version of comprehensive traffic regulating color scheme. Drivers would have to learn and memorize numerous color patterns, and to be additionally very aware when they have crossed the boundaries from one jurisdiction to another jurisdiction.

All in all an incredible mess which thankfully got avoided by patiently sticking to the dull but extremely reliable protocol of only three traffic lights.

Same as reducing the number of decisions when driving leads to the more reliable public traffic, reducing the number of decisions when traveling the web leads to the more reliable web software. On the web, all we need to know is that a resource is identifiable and locatable, and that it can represent itself to us, and can respond to our requests to change its state or to destroy itself. And if we keep these radically simple things in mind when developing web software, we'll realize that we can do absolutely anything on the web.

Not only that, but by observing these five simple rules (identify the resource, locate it, ask it to represent itself, ask it to make a state transition, ask it to destroy itself), we open up the world of endless possibilities for other interested parties to join in on the conversation and to ameliorate and augment our business. All that without expecting anyone to learn anything in particular.

This is the world of true democracy, where everyone is welcome and everyone is qualified and is free to feel adequate to contribute their own value.

Contrast that with the world of web services where each and every business is free to invent its own system of 'traffic lights' and then insist that anyone driving through their territory learns their convoluted system and obeys it. It is painfully obvious how web services had regressed us and damaged our credibility big time. It'll take many years before the damage can get undone. In the meantime, we'd do well to really learn how to walk properly on the web, before we try to run and again fall flat on our faces.