Inventing the Future

The best way to predict the future is to invent it. ~ Alan Kay

"If I'd asked people what they wanted they would have said a faster horse." ~ Henry Ford

I really like NoSQL data stores and I appreciate the new approaches because they tackle one of the most hidebound parts of information technology - enterprise storage. I still (dimly) recall the days before the predominance of the relational model where "database analyst" was a specific (rather than general) skill, and terms like "current of run-unit" could pretend to have meaning. The relational data model brought order and reason to enterprise data, but the seductiveness of the relational answer to the question of "How do we organize this?" over time came to preclude thoughts of any other form of organization.

It's not that there haven't been other successful data models and stores, but the other successful models have mostly been transparent -- you won't see the data model working because it is invisible to you. Microsoft Office applications have terrific data stores, but they don't place data in a relational model because there is simply no need to -- for Office apps, speed and flexibility trump the structure and order of the relational model. Wikipedia lists dozens of Windows file types, and across different operating systems and application types there are probably hundreds of data store types in general usage.

NoSQL approaches are so interesting and promising because they represent a break from the functional fixation that the relational model is the only conceivable model for enterprise data. In an earlier posting, I wrote:

Relational data models have ruled enterprise data management for more than 20 years -- to the point where it may be hard for generations of developers to imagine that there could be any kind of data model other than rows and columns.

This is the world we've lived in -- terrific for tabulations and Accounts Payable, but not necessarily well aligned for other problem domains. The challenge in moving beyond Relational is that our assimilation of "Relational" hinders us in looking at problems in anything other than Relational terms. This is a manifestation of a pattern that psychologist Karl Duncker called functional fixity -- the condition in which we grasp our principal tool as a hammer, thus our whole world comes to look like nails.

Eric Haseltine writes about functional fixity, creativity and invention in his book Long Fuse, Big Bang, in which he describes some of the opportunities that arise from seeing the world with new eyes. Haseltine writes:

In a classic functional fixity experiment, test subjects are asked how they would-- without grabbing the end of the rope-- get the ends of a rope suspended from a ceiling to touch opposite walls. Aside from the rope, the only object in the room is a pair of pliers sitting on the floor. Most test subjects don't realize that the only way to solve the problem is to tie the pliers to the end of the rope, then swing it so that the pliers-acting as a weight-carry the end of the rope to one wall, then, in pendulum style, to the other. The few subjects who perceive the pliers as a weight instead of a tool, solve the problem.

The progression of Moore's Law and advancement of NoSQL data approaches, cloud deployment, and modern languages and tools can open doors to advancements that would have been inconceivable without the synchronicity of these technological advances. We have new solution archetypes:

• Key-value storage (Memcached, Redis, Voldemort) for caches and transient data...
• Hadoop and MapReduce for massively parallel data processing
• Document databases (MongoDB, CouchDB) for log files and serial data
• Graph databases (Neo4j) for the social graph and applications

...but the kinds of solutions possible with these new approaches are far broader than that. We just need to keep building the skills to see such solutions as they become possible.

“The real voyage of discovery consists not in seeking new landscapes but in having new eyes.” ~ Proust